From owner-ips@ece.cmu.edu Thu Nov 1 02:06:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA22773 for ; Thu, 1 Nov 2001 02:06:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA16C5207899 for ips-outgoing; Thu, 1 Nov 2001 01:12:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA16C2l07893 for ; Thu, 1 Nov 2001 01:12:02 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA18848 for ; Thu, 1 Nov 2001 07:11:56 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA16Bu839728 for ; Thu, 1 Nov 2001 07:11:56 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI:SCSI responses for iSCSI errors X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 1 Nov 2001 08:11:52 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 08:11:55, Serialize complete at 01/11/2001 08:11:55 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ken, It is entirely up to the implementer. IMHO R2Ts that where not sent yet don't have to be sent. A response PDU will close nicely the cycle. The whole purpose of the delay is to avoid having PDUs carrying the ITT hit the target when the target has cleared its own control structures referring to the task. Julo "Ken Sandars" Sent by: owner-ips@ece.cmu.edu 31-10-01 19:08 Please respond to "Ken Sandars" To: cc: Subject: iSCSI:SCSI responses for iSCSI errors Gidday all, With reference to 3.4.2, when a target detects an error does: '...MUST wait until it receives a Data PDU with the F bit set, in the last expected sequence, before sending the Response PDU' imply the target waits for the entire Expected Data Transfer Length worth to be transferred? If so, the target may have to R2T for the solicited data! Should it R2T with 'correct' values, regardless of what the initiator sent? In this case it is possible that this clean-up action actually solicits all the required data, however, the command is still going to get a response with sense data indicating an abort. Is it possible to just reject the original command PDU when validity checking picks up one of these errors? Any Data-Out PDUs in the pipeline which belong to this rejected command can be silently discarded by the target. What am I missing? Thoughts? Ken Sandars ksandars@eurologic.com Eurologic Systems +44 117 930 9616 From owner-ips@ece.cmu.edu Thu Nov 1 03:44:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA04597 for ; Thu, 1 Nov 2001 03:44:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA17q4M13482 for ips-outgoing; Thu, 1 Nov 2001 02:52:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sansrv1.san-rad.co.il ([212.199.104.228]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA17pRl13436 for ; Thu, 1 Nov 2001 02:51:30 -0500 (EST) content-class: urn:content-classes:message Subject: RE: iSCSI: New iSCSI MIB draft MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Disposition-Notification-To: "Michele Hallak - Stamler" X-MimeOLE: Produced By Microsoft Exchange V6.0.4417.0 Date: Thu, 1 Nov 2001 09:52:53 +0200 Message-ID: <838D8D2617300146B7F47E4D9AE7FF10114A0E@SANSRV1.SAN-RAD.CO.IL> Thread-Topic: iSCSI: New iSCSI MIB draft Thread-Index: AcFiRAs1SBgPmPydQ/SAe6ZBP+HQ0AAZXx9w From: "Michele Hallak - Stamler" To: "Mark Bakke" , "IPS" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA17pYl13440 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hi, 1.I see that the target access list is still read-only. How do you think that it should be configured? 2. Who should configure the aliases of the iscsi targets and initiators? 3. Same question for the portals. Thanks, Michele -----Original Message----- From: Mark Bakke [mailto:mbakke@cisco.com] Sent: Wednesday, October 31, 2001 9:01 PM To: IPS Subject: Re: iSCSI: New iSCSI MIB draft For those who are more pictorially inclined when looking at MIBs, I have an updated MIB tree drawing available at: ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-mib-structure-03.pd f -- Mark Mark Bakke wrote: > > We have submitted a new iSCSI MIB draft to the repository. > Until it becomes available, it may be retrieved as either > a gzipped or text version from: > > ftp://ftpeng.cisco.com/mbakke/iscsi/draft-ietf-ips-iscsi-mib-03.txt.gz > > ftp://ftpeng.cisco.com/mbakke/iscsi/draft-ietf-ips-iscsi-mib-03.txt > > A matching UML drawing is available at: > > ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-uml-model-03.pdf > > The table hierarchy has been significantly flattened. This does > not change the object model, but does reduce the number of redundant > levels of OIDs when address individual objects. I will send a > new table structure drawing soon. > > We will send a list of open issues to the IPS list shortly. > > Thanks, > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Nov 1 05:15:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA09189 for ; Thu, 1 Nov 2001 05:15:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA19G1J18109 for ips-outgoing; Thu, 1 Nov 2001 04:16:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from best.eurologic.com ([193.120.246.5]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA19Fwl18102 for ; Thu, 1 Nov 2001 04:15:58 -0500 (EST) Received: from COORS (coors.eurologic.com [192.168.7.111] (may be forged)) by best.eurologic.com (8.9.3+Sun/8.9.3) with SMTP id JAA14293; Thu, 1 Nov 2001 09:12:16 GMT Message-ID: <004001c162b5$e87a8230$6f07a8c0@COORS> From: "Ron Cohen" To: "ips" , "Santosh Rao" References: <008f01c16232$4d808640$3d7db184@trebiaazner66z> <3BE0569C.A6CCCAF5@cup.hp.com> Subject: Re: iSCSI: current UNH Plugfest: Autosense Date: Thu, 1 Nov 2001 09:16:21 -0000 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh, I don't see how you can have autosense mandatory and not have sense data available. I would tend to agree with Anshul that the sense data MUST be provided. The Check Condition/Request Sense mechanisms of parallel SCSI are archaic. In the old days, you couldn't have autosense because you couldn't have sense data coming in when the command had data going out (Write). In the end, this mechanism was a pain in the ass -- its basically saying that an error occurred but I'm not telling you what it is. Returning a Check Condition with no sense data in the example you give with an error in the data segment does not seem correct to me. I would assume that the SCSI LLP would return a driver error in this case indicating that the data segment was lost rather than a SCSI error. Ron ----- Original Message ----- From: "Santosh Rao" To: "Anshul Chadda" Cc: Sent: Wednesday, October 31, 2001 7:53 PM Subject: Re: iSCSI: current UNH Plugfest > Anshul, > > I don't know why the initiator should care if the sense data arrived > along with the CHECK CONDITION scsi status or not. (?) > > Most, if not all, SCSI initiator stacks have some form of indicating > sense_status from SCSI LLP (hba driver, device driver, mini-port driver, > etc) to SCSI ULP (class driver, target driver, etc), which indicates > what the status of the sense operation was and whether sense data is > available or not. > > For instance, you also need to deal with a scenario where the target > sends a SCSI status of CHECK CONDITION and accompanies it with the sense > data in the data segment. However, the initiator encounters a data > digest error on the sense data data segment and drops the sense data > data segment. The initiator can always choose to complete the I/O and > return the SCSI status back to its SCSI ULP, indicating that no sense > data was available. > > I don't see why you would need the wording tightened to a MUST. > Initiators must not assume that the sense data will always be available > on a check condition. The sense operation may be unsuccessful and no > sense data may be available. > > Regards, > Santosh > > > > Anshul Chadda wrote: > > > > Hello: > > As this issue has come up with setting CHECK CONDITION in the SCSI > > Response. It is assumed that if CHECK CONDITION is set in the SCSI > > Response PDU, then there has to be sense data accompanied with it. So > > as far as I see it, it would help if the following sentence in the > > draft has the MUST/must in there. In the current wording, i can think > > that if there is no data segment in the SCSI Response PDU for a CHECK > > CONDITION, it is still OK. > > > > In draft 8, the sentence looks like the following: > > ------------------------------------------------------- > > 3.4.6 Data Segment - Sense and Response Data Segment > > > > iSCSI targets MUST support and enable autosense. If Status is CHECK > > CONDITION (0x02), then the Data Segment contains sense data for the > > failed command. > > > > ------------------------------------------------------- > > > > It can be changed to the following: > > > > ------------------------------------------------------- > > 3.4.6 Data Segment - Sense and Response Data Segment > > > > iSCSI targets MUST support and enable autosense. If Status is CHECK > > CONDITION (0x02), then the target MUST have sense data in the data > > segment for the failed command. > > > > ------------------------------------------------------- > > > > I don't know if there is a reason that the draft has the wording in > > the current way. Apologies if this subject has already been > > discussed. > > > > Regards, > > Anshul > > > > -------------------------------------------------------------------------- --------------------------------------- > > > > 5. Some common error situations: > > > > 1) - when a SCSI Response contains a CHECK CONDITION (Status=0x02), > > some targets are not including the SenseLength as the first 2 > > bytes in the data segment. Although the format of the data > > segment > > is clear from the diagram in section 3.4.6 on page 62 of draft 8 > > (page 63 of draft 8a), the last entry in the diagram for the > > SCSI > > Response PDU on page 58 of draft 8 (page 59 of draft 8a) is > > misleading because it mentions only the Sense Data and Response > > Data and omits the Sense Length. It would therefore be helpful > > if the last entry in the diagram on page 58 were changed to > > explicitly > > reference the diagram on page 62, as in: > > > > Data Segment -- see section 3.4.6 (optional) > > > > > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## > From owner-ips@ece.cmu.edu Thu Nov 1 05:30:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA09486 for ; Thu, 1 Nov 2001 05:30:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA19q2Y19983 for ips-outgoing; Thu, 1 Nov 2001 04:52:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA19q0l19976 for ; Thu, 1 Nov 2001 04:52:00 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA31824 for ; Thu, 1 Nov 2001 10:51:54 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA19pr827476 for ; Thu, 1 Nov 2001 10:51:53 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 1 Nov 2001 11:51:49 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 11:51:53, Serialize complete at 01/11/2001 11:51:53 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, Comments in text. Thanks, Julo "Robert D. Russell" Sent by: owner-ips@ece.cmu.edu 31-10-01 02:06 Please respond to "Robert D. Russell" To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: current UNH Plugfest Attached are some new issues that arose during the iSCSI plugfest at UNH on Tuesday 29-Oct-2001. (Note: these issues do not take into account any modifications or clarifications that occured in the standard due to the issues raised on Monday.) Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 ----------------------------------------------------------------------------- 1. Situation: as the first command on a new TCP connection, the initiator sends a login with T=1, CSG=1, NSG=3, and valid InitiatorName, TargetName, and SessionType keys. However, there is also a valid key having an invalid value, such as MaxConnections=abcd (i.e., not a number after the '=') or MaxConnections4 (i.e., missing the '='). What should the target do? Interpretation 1: According to section 3.10.4 page 82 of draft 8 (page 83 of draft 8a), "Any other key not understood by the target may be ignored by the target without affecting basic function. However the Text Response for a key that was not understood MUST be key=NotUnderstood." Two things have to be clarified: 1. Does this section also apply to keys received in a login? 2. Can "NotUnderstood" also apply to "values of keys" that are not understood, even if the key word itself is understood? +++ 1.NotUnderstood appears also in the negotiation section so it applies to login. I will remove it from the text section 2.It does not apply to values +++ If the answer to these 2 of these questions is "yes", then the appropriate response would seem to be for the target to just ignore the key and send back MaxConnections=NotUnderstood as part of its next login response. Interpratation 2: According to section 8.7 page 129 of draft 8 (page 130 of draft 8a), "A negotiation failure is considered one or both of the following: - None of the choices or the stated value is acceptable to one negotiating side. ..." Clearly this stated value ("abcd") is not acceptable to the target. Therefore, the following rule on page 129, draft 8 (page 130, draft 8a) applies: "- During Login, any failure in negotiation MUST be considered as the login process failure and the connection must be dropped." Therefore, the target should just drop the connection without sending any login response back to the initiator. Interpretation 3: This is a login command that contains an error caused by the initiator. Therefore, the target should send back a login response with a status code of 0x0200 and then close the TCP connection. +++ I think this is the correct interpretation and we will fix the wording in section 8 to say this. I suggest: - During Login, any failure in negotiation MUST be considered as the login process failure and the login phase must be terminated. If the failure is detected by the target, it must reject the login with the appropriate code. The connection must be terminated by the initiator. +++ 2. Situation: on the first login command in operational parameter negotiation stage, the initiator sends no operational keys, thereby telling the target that it accepts all the default values for these keys. However, the target wants to negotiate the value of MaxConnections, so in the login response it sends back "MaxConnections=3" (for example). Should the initiator send a response to this key or not? The statement in section 2.2.4 on page 30 of draft 8 and 8a: "For numerical (and single literal) negotiations, the responding party MUST respond with the required key.(...)" makes it clear that the responding party MUST respond. However, in this situation, it is not clear who the responding party is. +++ we are using the term offering and responding to distinguish them from initiator and target and explicitly say (in 2.2.4) that the target may offer keys +++ Interpretation 1: By not explicitly sending this key in the login command, the initiator is implicitly offering the default value and therefore is the offering party and the target is the responding party. The conclusion is that the initiator does not have to send a response to this key from the target. +++ there is no implicit ofering of defaults. A default is accepted only if the two parties are silent. +++ Interpretation 2: The target is the offering party because it is the party that explicitly stated the key for the first time during these negotiations. The conclusion is that the initiator MUST send a response to this key from the target. NOTE 1: If interpretation 1 is correct, it would seem to imply that the target MUST respond to every key whether or not it is present in the login from the initiator, even if it does not want to change the default value. The reason is that a missing key is an implicit offer of the default value, and the responding party MUST respond. Is this a correct interpretation? NOTE 2: The following statements in section 2.2.4 page 29 of draft 8a: Originator-> = Responder-> =|NOtUnderstood seem to imply that the originator is the party (initiator or target) that explicitly offers a key, and that omitting a key is not an implicit offer of that key with the default value. However, even in the revised draft 8a there is no definition of "Originator" and/or "Responder" that would make this clear. Adding to the standard these definitions, and an explicit statement that "a missing key does not constitute an implicit offer of the default" would help eliminate misunderstandings. In addition, including an example of this situation (where an initiator omits a key and the target offers the key) would be a big help. +++ will do +++ 3. Some of the login phase examples given in Appendix A of both draft 8 and 8a do not follow the rule in section 3.12.4 page 87 of draft 8 (page 88 of draft 8a): "The next stage value is valid only when the T bit is 1 and is reserved otherwise." and the rule in section 3 page 48 of draft 8 (page 49 of draft 8a): "Any reserved fields and values MUST be 0 unless specified otherwise." If these rules are applied, all examples having T=0 should also have NSG=0. Presently all of them with T=0 also have NSG=1 or NSG=3. +++ Will fix the examples +++ 4. Situation: The initiator and target have successfully completed the login phase for a discovery session and are now in full feature phase. The initiator sends a text command containing the single key: "SendTargets=". What response is expected from the target? Interpretation 1: According to the explanation on page 188 of draft 8 (page 189 of draft 8a): "If no target name is specified, the session should respond with addresses only for the target to which the session is logged in. This MUST be supported on operational sessions, and MAY NOT return targets other than the one to which the session is logged in." However, for a discovery session there is no target per se (the initiator does not specify a TargetName= during login), so the target therefore follows the rule on page 188 of draft 8 (page 189 of draft 8a): +++ That is not strictly correct. You may use discovery to determine the addresses and portal groups for a specific target - i.e. you may give a target-name. If you give a target name that is all you get. If don't give a name you get all the targets you are allowed to get to. ++++ "A SendTargets response MAY contain no target names, if there are no targets for the requesting initiator to access." and sends back a Text Response with no keys in it. Interpretation 2: In a discovery session, the key "SendTargets=" makes no sense and should be treated by the target in the same manner as the key "SendTargets=all". 5. Some common error situations: 1) - when a SCSI Response contains a CHECK CONDITION (Status=0x02), some targets are not including the SenseLength as the first 2 bytes in the data segment. Although the format of the data segment is clear from the diagram in section 3.4.6 on page 62 of draft 8 (page 63 of draft 8a), the last entry in the diagram for the SCSI Response PDU on page 58 of draft 8 (page 59 of draft 8a) is misleading because it mentions only the Sense Data and Response Data and omits the Sense Length. It would therefore be helpful if the last entry in the diagram on page 58 were changed to explicitly reference the diagram on page 62, as in: Data Segment -- see section 3.4.6 (optional) +++ will do +++ 2) - after sending a CmdSN on an initial login, some initiators are incrementing it before sending their first non-immediate command. (i.e., if the initial login contains CmdSN=123, they are sending CmdSN=124 on the first non-immediate command after the login phase). Section 3.12.10 on page 89 of draft 8 (page 90 of draft 8a) is clear that in this example the first non-immediate command should carry CmdSN=123, not 124. This was an issue at the July plugfest and apparently some implementations have not been updated to conform to the draft 8 standard in their handling of CmdSN. From owner-ips@ece.cmu.edu Thu Nov 1 06:06:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA10028 for ; Thu, 1 Nov 2001 06:06:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1ACGg21166 for ips-outgoing; Thu, 1 Nov 2001 05:12:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1ACDl21154 for ; Thu, 1 Nov 2001 05:12:13 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA33480 for ; Thu, 1 Nov 2001 11:12:02 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1AC2858072 for ; Thu, 1 Nov 2001 11:12:02 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 1 Nov 2001 12:11:57 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 12:12:02, Serialize complete at 01/11/2001 12:12:02 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Anshul, IMHO a single MUST in the paragraph is strong enough and covers all the cases. Julo "Anshul Chadda" Sent by: owner-ips@ece.cmu.edu 31-10-01 19:34 Please respond to "Anshul Chadda" To: cc: Subject: Re: iSCSI: current UNH Plugfest Hello: As this issue has come up with setting CHECK CONDITION in the SCSI Response. It is assumed that if CHECK CONDITION is set in the SCSI Response PDU, then there has to be sense data accompanied with it. So as far as I see it, it would help if the following sentence in the draft has the MUST/must in there. In the current wording, i can think that if there is no data segment in the SCSI Response PDU for a CHECK CONDITION, it is still OK. In draft 8, the sentence looks like the following: ------------------------------------------------------- 3.4.6 Data Segment - Sense and Response Data Segment iSCSI targets MUST support and enable autosense. If Status is CHECK CONDITION (0x02), then the Data Segment contains sense data for the failed command. ------------------------------------------------------- It can be changed to the following: ------------------------------------------------------- 3.4.6 Data Segment - Sense and Response Data Segment iSCSI targets MUST support and enable autosense. If Status is CHECK CONDITION (0x02), then the target MUST have sense data in the data segment for the failed command. ------------------------------------------------------- I don't know if there is a reason that the draft has the wording in the current way. Apologies if this subject has already been discussed. Regards, Anshul ----------------------------------------------------------------------------------------------------------------- 5. Some common error situations: 1) - when a SCSI Response contains a CHECK CONDITION (Status=0x02), some targets are not including the SenseLength as the first 2 bytes in the data segment. Although the format of the data segment is clear from the diagram in section 3.4.6 on page 62 of draft 8 (page 63 of draft 8a), the last entry in the diagram for the SCSI Response PDU on page 58 of draft 8 (page 59 of draft 8a) is misleading because it mentions only the Sense Data and Response Data and omits the Sense Length. It would therefore be helpful if the last entry in the diagram on page 58 were changed to explicitly reference the diagram on page 62, as in: Data Segment -- see section 3.4.6 (optional) From owner-ips@ece.cmu.edu Thu Nov 1 07:48:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA13243 for ; Thu, 1 Nov 2001 07:48:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1Baxw09111 for ips-outgoing; Thu, 1 Nov 2001 06:36:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Bawl09106 for ; Thu, 1 Nov 2001 06:36:58 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA21124 for ; Thu, 1 Nov 2001 12:36:49 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1Ban887888 for ; Thu, 1 Nov 2001 12:36:49 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 1 Nov 2001 13:36:45 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 13:36:49, Serialize complete at 01/11/2001 13:36:49 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, I assume that your statement about request sense refers to a data-in PDU not a data segment (e.g., not the data segment of a response PDU). Normally a request sense will result in a single input PDU - a Data-IN including also the good status for the request sense (if the implementer had enough goodwill to collapse the 2 phases). As for the MUST the paragraph has already a MUST (MUST support autosense). Julo Robert Snively Sent by: owner-ips@ece.cmu.edu 31-10-01 22:39 Please respond to Robert Snively To: "'Santosh Rao'" , Anshul Chadda cc: ips@ece.cmu.edu Subject: RE: iSCSI: current UNH Plugfest The "MUST have sense information if CHECK CONDITION" is a property of SCSI that requires the use of autosensing, as iSCSI does. It is only non autosensing drivers that will present CHECK CONDITION indications with sense data missing. Sense Data will never be generated in a data segment to any command except REQUEST SENSE. In that case, CHECK CONDITION will never be indicated unless the REQUEST SENSE command failed in some way, and (if autosensing is used), the CHECK CONDITION will be associated with its own set of sense information in the response PDU. Note that if the unit cannot assemble relevant sense information from the attached hardware, that is in itself a CHECK CONDITION, and the corresponding inability to access critical hardware must be posted in the sense information. That is why MUST is the proper requirement. Bob > -----Original Message----- > From: Santosh Rao [mailto:santoshr@cup.hp.com] > Sent: Wednesday, October 31, 2001 11:53 AM > To: Anshul Chadda > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI: current UNH Plugfest > > > Anshul, > > I don't know why the initiator should care if the sense data arrived > along with the CHECK CONDITION scsi status or not. (?) > > Most, if not all, SCSI initiator stacks have some form of indicating > sense_status from SCSI LLP (hba driver, device driver, > mini-port driver, > etc) to SCSI ULP (class driver, target driver, etc), which indicates > what the status of the sense operation was and whether sense data is > available or not. > > For instance, you also need to deal with a scenario where the target > sends a SCSI status of CHECK CONDITION and accompanies it > with the sense > data in the data segment. However, the initiator encounters a data > digest error on the sense data data segment and drops the sense data > data segment. The initiator can always choose to complete the I/O and > return the SCSI status back to its SCSI ULP, indicating that no sense > data was available. > > I don't see why you would need the wording tightened to a MUST. > Initiators must not assume that the sense data will always be > available > on a check condition. The sense operation may be unsuccessful and no > sense data may be available. > > Regards, > Santosh > > > > Anshul Chadda wrote: > > > > Hello: > > As this issue has come up with setting CHECK CONDITION in the SCSI > > Response. It is assumed that if CHECK CONDITION is set in the SCSI > > Response PDU, then there has to be sense data accompanied > with it. So > > as far as I see it, it would help if the following sentence in the > > draft has the MUST/must in there. In the current wording, i > can think > > that if there is no data segment in the SCSI Response PDU > for a CHECK > > CONDITION, it is still OK. > > > > In draft 8, the sentence looks like the following: > > ------------------------------------------------------- > > 3.4.6 Data Segment - Sense and Response Data Segment > > > > iSCSI targets MUST support and enable autosense. If Status is CHECK > > CONDITION (0x02), then the Data Segment contains sense data for the > > failed command. > > > > ------------------------------------------------------- > > > > It can be changed to the following: > > > > ------------------------------------------------------- > > 3.4.6 Data Segment - Sense and Response Data Segment > > > > iSCSI targets MUST support and enable autosense. If Status is CHECK > > CONDITION (0x02), then the target MUST have sense data in the data > > segment for the failed command. > > > > ------------------------------------------------------- > > > > I don't know if there is a reason that the draft has the wording in > > the current way. Apologies if this subject has already been > > discussed. > > > > Regards, > > Anshul > > > > > -------------------------------------------------------------- > --------------------------------------------------- > > > > 5. Some common error situations: > > > > 1) - when a SCSI Response contains a CHECK CONDITION > (Status=0x02), > > some targets are not including the SenseLength as the first 2 > > bytes in the data segment. Although the format of the data > > segment > > is clear from the diagram in section 3.4.6 on page 62 > of draft 8 > > (page 63 of draft 8a), the last entry in the diagram for the > > SCSI > > Response PDU on page 58 of draft 8 (page 59 of draft 8a) is > > misleading because it mentions only the Sense Data > and Response > > Data and omits the Sense Length. It would therefore > be helpful > > if the last entry in the diagram on page 58 were changed to > > explicitly > > reference the diagram on page 62, as in: > > > > Data Segment -- see section 3.4.6 (optional) > > > > > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## > From owner-ips@ece.cmu.edu Thu Nov 1 08:24:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA15560 for ; Thu, 1 Nov 2001 08:24:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1CS3311869 for ips-outgoing; Thu, 1 Nov 2001 07:28:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar2ab.compuserve.com (siaar2ab.compuserve.com [149.174.40.138]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1CS0l11863 for ; Thu, 1 Nov 2001 07:28:01 -0500 (EST) Received: (from mailgate@localhost) by siaar2ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id HAA16963 for ips@ece.cmu.edu; Thu, 1 Nov 2001 07:27:55 -0500 (EST) Received: from compuserve.com (dal-tgn-tkl-vty21.as.wcom.net [216.192.224.21]) by siaar2ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id HAA16946 for ; Thu, 1 Nov 2001 07:27:48 -0500 (EST) Message-ID: <3BE140FC.1E188857@compuserve.com> Date: Thu, 01 Nov 2001 06:33:00 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: FCencap: Missing SOF/EOF characters Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Off and on I hear complaints that it is not obvious why draft-ietf-ips-fcencapsulation-03.txt fails to list all the possible Fibre Channel SOF and EOF characters in 5.3. To address this concern, I propose adding the following text at the end of 5.3. Note: Some of the SOF and EOF characters defined by Fibre Channel are not listed in Table 1 and Table 2 because the design an operating characteristics of an IP Network make it inappropriate for transporting FC Frames containing those SOF and/or EOF characters. I proposed to make this addition in a new draft that will be available in time for consideration in Salt Lake. Comment appreciated. Ralph Weber From owner-ips@ece.cmu.edu Thu Nov 1 08:29:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA15773 for ; Thu, 1 Nov 2001 08:29:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1CULs11985 for ips-outgoing; Thu, 1 Nov 2001 07:30:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1CUJl11981 for ; Thu, 1 Nov 2001 07:30:19 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id NAA118966 for ; Thu, 1 Nov 2001 13:30:12 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1CUC808296 for ; Thu, 1 Nov 2001 13:30:12 +0100 Importance: Normal Subject: iSCSI: IPsec tunnel / transport mode decision To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Ofer Biran" Date: Thu, 1 Nov 2001 14:31:26 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 14:30:12 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Thu Nov 1 09:36:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA19150 for ; Thu, 1 Nov 2001 09:36:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1DFFQ14647 for ips-outgoing; Thu, 1 Nov 2001 08:15:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls16.mediaone.net (chmls16.mediaone.net [24.147.1.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1DFDl14642 for ; Thu, 1 Nov 2001 08:15:13 -0500 (EST) Received: from eddylaptop (equicksall.ne.mediaone.net [24.61.64.49]) by chmls16.mediaone.net (8.11.1/8.11.1) with SMTP id fA1DFwT04167 for ; Thu, 1 Nov 2001 08:15:58 -0500 (EST) From: "Eddy Quicksall" To: Subject: RE: iSCSI: current UNH Plugfest Date: Thu, 1 Nov 2001 08:14:48 -0500 Message-ID: <000101c162d7$330d39c0$0102a8c0@eddylaptop> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit I am reluctant to say this because I think most people think the initiator/target must check for correctness ... but, it is my feeling that that job should be up to a basher program. The target should not be in the business of diagnosing the initiator. The only time a target should check a field is when it could crash the system or data. Some format errors may have no consequences whatsoever. Eddy -----Original Message----- From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] Sent: Wednesday, October 31, 2001 05:39 PM To: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest Attached are the new issues that arose during the iSCSI plugfest at UNH on Wednesday 31-Oct-2001. (Note: these issues do not take into account any modifications or clarifications that occured in the standard due to the issues raised on Monday or Tuesday.) Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 ------------------------------------------------------------------------ ---- 1. Are receivers (initiator or target) REQUIRED to check that reserved bits and/or fields are set to 0? Section 3 on page 48 of draft 8 says: "Any bits not defined MUST be set to zero. Any reserved fields and values MUST be 0 unless specified otherwise." and Section 8.3 on page 127 of draft 8 says: "Explicit violations of the PDU layout rules stated in this document are format errors. This when detected, usually indicates a major implementation flaw in one of the parties. When a target or an initiator receives an iSCSI PDU with a format error, it MUST reset all transport connections in the session immediately and escalate the format error to session recovery (section 8.11.4)." According to these rules, a PDU with reserved bits and/or fields that are not set to 0 violates the PDU layout rules. Therefore, if an initiator or target receives such a PDU, it should immediately close all connections in the session and go to session recovery. Clearly a format error has extremely severe consequences! Although all vendors are setting reserved bits and fields to 0 on PDUs they are sending, many are NOT checking PDUs they are receiving to see if these bits and fields are set to 0. Basically, vendors are saying "who cares if reserved bits and/or fields in incoming PDUs are not zero? We do not want to take the time to do this checking, and there is no benefit to doing it. As long as the non-reserved bits and fields are set properly, we can and should proceed. Any time devoted to doing this checking is wasted in 99+% of the cases, and in the (unlikely) case that a non-zero bit or field is found, the consequences are too severe." There should be some statement in the standard to clarify what checking is required and what is optional. 2. A similar situation arises with respect to checking the consistency of fields such as Version-max, Version-min and Version-active in Login Requests and Login Responses. For example, consider the Version-max field. Section 3.12.5 says: "All Login requests within the Login phase MUST carry the same Version-max." All vendor initiators are setting Version-max correctly on all login requests they are sending, but many vendor targets are NOT checking received login requests to ensure that this rule is enforced. In particular, many targets simply use the Version-max and Version-min on the first login request they receive on a new connection, and then they ignore these fields on all subsequent login requests in the same login phase. Strictly speaking, a change in the Version-max field during the login phase constitutes a protocol error according to section 8.8 on page 130 of draft 8: "All violations of iSCSI PDU exchange sequences specified in this draft are also protocol errors. This category of errors can be addressed only by fixing the implementations; iSCSI defines Reject and response codes to enable this". Therefore the target should send back a login response with a status of 0x0200 and then close the connection. However, Section 3.12.5 also says: "The target MUST use the value presented with the first login request." This rule seems to imply that the value CAN change, because if it cannot change, then it doesn't matter which one of the login requests it is taken from, they are all the same anyway. The suggestion is to keep the requirement that the target MUST use the value presented on the first login request, but to allowed the target to ignore the value presented on all subsequent login requests in the same login phase. A similar rewording should be done for the other fields. 3. Can commands be sent out of order on the same connection? The behavior of targets is clearly specified in Section 2.2.2.3 on page 25 of draft 8, which says: "Except for the commands marked for immediate delivery the iSCSI target layer MUST eliver the commands for execution in the order specified by CmdSN." Section 2.2.2.3 on page 26 of draft 8 also says: "- CmdSN - the current command Sequence Number advanced by 1 on each command shipped except for commands marked for immediate delivery." but the meaning of the term "shipped" is vague, and does not necessarily require that the PDUs arrive on the other end of a TCP connection in the same order that the CmdSN values were assigned to these PDUs. Some initiators have been designed to send commands out of CmdSN order on one connection. Consider the situation where there is only one connection and a high-level dispatcher creates a PDU for a SCSI command that involves writing immediate data to the target. This PDU is enqueued to a lower-level layer which has to setup, start, and wait-for a DMA operation to move the immediate data into an onboard buffer before the PDU can be put onto the wire. While this is happening, the dispatcher creates another unrelated PDU for a SCSI read command (for example), and when this PDU is passed to the lower-level layer it can be sent immediately, ahead of the previous write PDU and therefore out of order on this connection. The standard clearly allows this to happen if the two PDUs were sent on different connections, and seems to imply that this can also happen when the two PDUs are sent on the same connection. The suggestion is to put in the standard an explicit statement that this is allowed or not allowed, as appropriate. If this is allowed, such a statement would avoid the erroneous assumption being made by some target implementers that within a single connection, commands will arrive in order. If this is not allowed, such a statement would avoid the erroneous assumption being made by some initiator implementers that within a single connection, commands can be put on the wire out of order. 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) now allow: "A value of 0 indicates no limit." Is this useful? Does it buy anything? The difficulties implementers are having with this are: 1) It is a special case. 2) It causes discontinuous ranges (for example, [0,64..2**24]) 3) It violates the min/max function normally used for the key. 4) There is always a limit anyway. Consider FirstBurstSize, which can have a value that is described as "<0|number-64-2**24>", and for which the minimum of the 2 numbers is selected. I one side offers 0 to mean unlimited, and the other side has a limit, it will reply with that limit, say 128 Kbytes. Therefore, the result is not min(0, 128K) but rather max(0, 128K). The statement in the standard that "the minimum of the 2 numbers is selected" is therefore wrong when one of the numbers is 0. Furthermore, when an initiator or target receives an offer for one of these keys, it cannot simply check that the offered value is legal by testing it against some minimum and maximum. It must first check for 0 and then only if that check shows the value is non-zero can it do the min/max range check for legality (i.e., 64-2**24). Finally, there is always a limit. If nothing else it is the limit imposed by the 24-bit DataSegmentLength field of the PDU requesting the transfer. It is useless to specify a FirstBurstSize (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because the largest possible DataSegmentLength in any PDU that can use this value is 2**24-1. The suggestion is to just eliminate this special case of 0 and require that the range 64-to-(2**24-1) be used instead -- it has exactly the same effect in all cases, it is easier to describe in the standard because it avoids all the extra words, and it is easier to code because it avoids all the special cases. NOTE: the standard should specify the limit in the ranges for MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead of 2**24. The number 2**24 cannot be represented in the 24-bit DataSegmentLength field and therefore can never be used. 5. This is a suggestion for a minor rewording in the standard to avoid misunderstandings. In Appendix E on page 188 of draft 8 it says: "The response to this command is a text response containing a list of targets and their addresses. Each target is returned as a target record. A target record begins with the TargetName text key, followed by a list of TargetAddress text keys, ..." In fact, there are situations where there are no targets and/or no addresses. These situations are clearly defined in the draft after the sentences quoted above, but it would help if those sentences at least hinted at the possibility that the lists could be empty or might not contain addresses. A possible rewording would be: "The response to this command is a text response containing a list of zero or more targets and, optionally, their addresses. Each target is returned as a target record. A target record begins with the TargetName text key, followed by a list of zero or more TargetAddress text keys, ..." 6. This is a suggestion for another very minor rewording in the standard. At the end of section 2.2.3 on page 29 of draft 8 it says: "Before full feature phase is established, only Login PDUs are allowed. ..." The suggested rewording is: "Before full feature phase is established, only Login Request and Login Response PDUs are allowed. ..." From owner-ips@ece.cmu.edu Thu Nov 1 11:00:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA22235 for ; Thu, 1 Nov 2001 11:00:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1EoQp21178 for ips-outgoing; Thu, 1 Nov 2001 09:50:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1EoOl21173 for ; Thu, 1 Nov 2001 09:50:24 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id JAA02944; Thu, 1 Nov 2001 09:50:04 -0500 (EST) Date: Thu, 1 Nov 2001 09:50:04 -0500 From: "Robert D. Russell" To: Julian Satran cc: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: IMHO if we mean MUST then we MUST say MUST. It is clear from the exchange between Anshul, Santosh and Robert that we already have different interpretations of what it means without MUST and hence we have interoperability problems. In this particular case I agree with Anshul and Robert that the standard should say MUST. Santosh's argument that in an error case the data gets lost does not seem to be relevant -- in error cases lots of information gets lost and recovery is necessary to get that information back. The spec provides for this. We should not be introducing interoperability problems because of a situation that may arise in the rare case of an error, especially when the spec already deals with recovery of that information when the error is detected. Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Thu, 1 Nov 2001, Julian Satran wrote: > Anshul, > > IMHO a single MUST in the paragraph is strong enough and covers all the > cases. > > Julo > > > > > "Anshul Chadda" > Sent by: owner-ips@ece.cmu.edu > 31-10-01 19:34 > Please respond to "Anshul Chadda" > > > To: > cc: > Subject: Re: iSCSI: current UNH Plugfest > > > > Hello: > As this issue has come up with setting CHECK CONDITION in the SCSI > Response. It is assumed that if CHECK CONDITION is set in the SCSI > Response PDU, then there has to be sense data accompanied with it. So as > far as I see it, it would help if the following sentence in the draft has > the MUST/must in there. In the current wording, i can think that if there > is no data segment in the SCSI Response PDU for a CHECK CONDITION, it is > still OK. > > In draft 8, the sentence looks like the following: > ------------------------------------------------------- > 3.4.6 Data Segment - Sense and Response Data Segment > iSCSI targets MUST support and enable autosense. If Status is CHECK > CONDITION (0x02), then the Data Segment contains sense data for the failed > command. > ------------------------------------------------------- > > It can be changed to the following: > > ------------------------------------------------------- > 3.4.6 Data Segment - Sense and Response Data Segment > iSCSI targets MUST support and enable autosense. If Status is CHECK > CONDITION (0x02), then the target MUST have sense data in the data segment > for the failed command. > ------------------------------------------------------- > I don't know if there is a reason that the draft has the wording in the > current way. Apologies if this subject has already been discussed. > > Regards, > Anshul > > ----------------------------------------------------------------------------------------------------------------- > > 5. Some common error situations: > > 1) - when a SCSI Response contains a CHECK CONDITION (Status=0x02), > some targets are not including the SenseLength as the first 2 > bytes in the data segment. Although the format of the data segment > is clear from the diagram in section 3.4.6 on page 62 of draft 8 > (page 63 of draft 8a), the last entry in the diagram for the SCSI > Response PDU on page 58 of draft 8 (page 59 of draft 8a) is > misleading because it mentions only the Sense Data and Response > Data and omits the Sense Length. It would therefore be helpful > if the last entry in the diagram on page 58 were changed to > explicitly > reference the diagram on page 62, as in: > > Data Segment -- see section 3.4.6 (optional) > > > > > From owner-ips@ece.cmu.edu Thu Nov 1 12:00:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA23709 for ; Thu, 1 Nov 2001 12:00:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1FltX25921 for ips-outgoing; Thu, 1 Nov 2001 10:47:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Flrl25913 for ; Thu, 1 Nov 2001 10:47:53 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA130154 for ; Thu, 1 Nov 2001 16:47:46 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1Flf888876 for ; Thu, 1 Nov 2001 16:47:41 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 1 Nov 2001 17:47:37 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 17:47:46, Serialize complete at 01/11/2001 17:47:46 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, The spec already says MUST. I don't agree that saying MUST twice is a good practice. Julo "Robert D. Russell" 01-11-01 16:50 Please respond to "Robert D. Russell" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest Julian: IMHO if we mean MUST then we MUST say MUST. It is clear from the exchange between Anshul, Santosh and Robert that we already have different interpretations of what it means without MUST and hence we have interoperability problems. In this particular case I agree with Anshul and Robert that the standard should say MUST. Santosh's argument that in an error case the data gets lost does not seem to be relevant -- in error cases lots of information gets lost and recovery is necessary to get that information back. The spec provides for this. We should not be introducing interoperability problems because of a situation that may arise in the rare case of an error, especially when the spec already deals with recovery of that information when the error is detected. Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Thu, 1 Nov 2001, Julian Satran wrote: > Anshul, > > IMHO a single MUST in the paragraph is strong enough and covers all the > cases. > > Julo > > > > > "Anshul Chadda" > Sent by: owner-ips@ece.cmu.edu > 31-10-01 19:34 > Please respond to "Anshul Chadda" > > > To: > cc: > Subject: Re: iSCSI: current UNH Plugfest > > > > Hello: > As this issue has come up with setting CHECK CONDITION in the SCSI > Response. It is assumed that if CHECK CONDITION is set in the SCSI > Response PDU, then there has to be sense data accompanied with it. So as > far as I see it, it would help if the following sentence in the draft has > the MUST/must in there. In the current wording, i can think that if there > is no data segment in the SCSI Response PDU for a CHECK CONDITION, it is > still OK. > > In draft 8, the sentence looks like the following: > ------------------------------------------------------- > 3.4.6 Data Segment - Sense and Response Data Segment > iSCSI targets MUST support and enable autosense. If Status is CHECK > CONDITION (0x02), then the Data Segment contains sense data for the failed > command. > ------------------------------------------------------- > > It can be changed to the following: > > ------------------------------------------------------- > 3.4.6 Data Segment - Sense and Response Data Segment > iSCSI targets MUST support and enable autosense. If Status is CHECK > CONDITION (0x02), then the target MUST have sense data in the data segment > for the failed command. > ------------------------------------------------------- > I don't know if there is a reason that the draft has the wording in the > current way. Apologies if this subject has already been discussed. > > Regards, > Anshul > > ----------------------------------------------------------------------------------------------------------------- > > 5. Some common error situations: > > 1) - when a SCSI Response contains a CHECK CONDITION (Status=0x02), > some targets are not including the SenseLength as the first 2 > bytes in the data segment. Although the format of the data segment > is clear from the diagram in section 3.4.6 on page 62 of draft 8 > (page 63 of draft 8a), the last entry in the diagram for the SCSI > Response PDU on page 58 of draft 8 (page 59 of draft 8a) is > misleading because it mentions only the Sense Data and Response > Data and omits the Sense Length. It would therefore be helpful > if the last entry in the diagram on page 58 were changed to > explicitly > reference the diagram on page 62, as in: > > Data Segment -- see section 3.4.6 (optional) > > > > > From owner-ips@ece.cmu.edu Thu Nov 1 12:28:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA24446 for ; Thu, 1 Nov 2001 12:28:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1Gski01734 for ips-outgoing; Thu, 1 Nov 2001 11:54:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.24.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Gsil01728 for ; Thu, 1 Nov 2001 11:54:45 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [64.101.128.87]) by sj-msg-core-2.cisco.com (8.11.3/8.9.1) with ESMTP id fA1Gsba02308 for ; Thu, 1 Nov 2001 08:54:37 -0800 (PST) Received: from DAPW2K (sjc-vpn2-59.cisco.com [10.21.112.59]) by sponge.cisco.com (Mirapoint) with SMTP id AAY10940; Thu, 1 Nov 2001 10:51:59 -0600 (CST) From: "Dave Peterson" To: "Ips@Ece. Cmu. Edu" Subject: iSCSI: iSCSI Rev 8 Date: Thu, 1 Nov 2001 10:56:03 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Good Morning, Suggested rewording for clause 3.4.1 in iSCSI Rev 8: bit 4 (o) set for Bidirectional Read Residual Overflow. In this case, the Bidirectional Read Residual Count indicates the number of bytes that were not transferred to the initiator because the initiator's Expected Bidirectional Read Data Transfer Length was not sufficient. bit 3 (u) set for Bidirectional Read Residual Underflow. In this case, the Bidirectional Read Residual Count indicates the number of bytes that were not transferred to the initiator out of the number of bytes that were expected to be transferred. bit 2 (O) set for Residual Overflow. In this case, the Residual Count indicates the number of bytes that were not transferred because the initiator's Expected Data Transfer length was not sufficient. For a bidirectional operation, the Residual Count contains the residual for the write operation. bit 1 (U) set for Residual Underflow. In this case, the Residual Count indicates the number of bytes that were not transferred out of the number of bytes that were expected to be transferred. For a bidirectional operation, the Residual Count contains the residual for the write operation. Suggested rewording for clause 3.4.4 3.4.4 Residual Count The Residual Count field is valid only in the case where either the U bit or the O bit is set. If neither bit is set, the Residual Count field SHOULD be zero. If the O bit is set, the Residual Count indicates the number of bytes that were not transferred because the initiator's Expected Data Transfer Length was not sufficient. If the U bit is set, the Residual Count indicates the number of bytes that were not transferred out of the number of bytes expected to be transferred. 3.4.5 Bidirectional Read Residual Count The Bidirectional Read Residual Count field is valid only in the case where either the u bit or the o bit is set. If neither bit is set, the Bidirectional Read Residual Count field SHOULD be zero. If the o bit is set, the Bidirectional Read Residual Count indicates the number of bytes that were not transferred to the initiator because the initiator's Expected Bidirectional Read Transfer Length was not sufficient. If the u bit is set, the Bidirectional Read Residual Count indicates the number of bytes that were not transferred to the initiator out of the number of bytes that were expected to be transferred. Dave From owner-ips@ece.cmu.edu Thu Nov 1 12:30:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA24545 for ; Thu, 1 Nov 2001 12:30:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1GOrO29194 for ips-outgoing; Thu, 1 Nov 2001 11:24:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars0m9.nortelnetworks.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1GOgl29169 for ; Thu, 1 Nov 2001 11:24:46 -0500 (EST) Received: from zcars04e.ca.nortel.com (zcars04e.ca.nortel.com [47.129.242.56]) by zcars0m9.nortelnetworks.com (8.11.0/8.11.0) with ESMTP id fA1GNpS22779 for ; Thu, 1 Nov 2001 11:23:51 -0500 (EST) Received: from zcard00m.ca.nortel.com by zcars04e.ca.nortel.com; Thu, 1 Nov 2001 11:21:37 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zcard00m.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id VPDBJ6ZS; Thu, 1 Nov 2001 11:21:00 -0500 Received: from cse-ns-laptop.nortelnetworks.com (cse-ns-laptop.us.nortel.com [47.16.69.109]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id VW7FBLQ0; Thu, 1 Nov 2001 11:21:00 -0500 Message-Id: <5.1.0.14.2.20011101111644.02e8e500@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Thu, 01 Nov 2001 11:23:15 -0500 To: ENDL_TX@computer.org, IPS Reflector From: "Franco Travostino" Subject: Re: FCencap: Missing SOF/EOF characters In-Reply-To: <3BE140FC.1E188857@compuserve.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_10463525==_.ALT" X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_10463525==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed Ralph, why should we be waiting until 5.3 (last page) to break this news (e.g., Class 1 isn't supported)? Can we move your text up to section 1, aptly titled "Scope". Alternately, we could narrow down the very title of the document. 0.02 -franco At 07:33 AM 11/1/2001, Ralph Weber wrote: >Off and on I hear complaints that it is not obvious >why draft-ietf-ips-fcencapsulation-03.txt fails to >list all the possible Fibre Channel SOF and EOF >characters in 5.3. > >To address this concern, I propose adding the following >text at the end of 5.3. > > Note: Some of the SOF and EOF characters defined by > Fibre Channel are not listed in Table 1 and Table 2 > because the design an operating characteristics > of an IP Network make it inappropriate for > transporting FC Frames containing those SOF and/or > EOF characters. > >I proposed to make this addition in a new draft that >will be available in time for consideration in Salt >Lake. > >Comment appreciated. > >Ralph Weber Franco Travostino, Director Content Internetworking Lab Advanced Technology Investments Nortel Networks, Inc. 600 Technology Park Billerica, MA 01821 USA Tel: 978 288 7708 Fax: 978 288 4690 email: travos@nortelnetworks.com --=====================_10463525==_.ALT Content-Type: text/html; charset="us-ascii"
Ralph,
why should we be waiting until 5.3 (last page) to break this news (e.g., Class 1 isn't supported)?
Can we move your text up to section 1, aptly titled "Scope". Alternately, we could narrow down the very title of the document.

0.02
-franco

At 07:33 AM 11/1/2001, Ralph Weber wrote:
Off and on I hear complaints that it is not obvious
why draft-ietf-ips-fcencapsulation-03.txt fails to
list all the possible Fibre Channel SOF and EOF
characters in 5.3.

To address this concern, I propose adding the following
text at the end of 5.3.

  Note: Some of the SOF and EOF characters defined by
  Fibre Channel are not listed in Table 1 and Table 2
  because the design an operating characteristics
  of an IP Network make it inappropriate for
  transporting FC Frames containing those SOF and/or
  EOF characters.

I proposed to make this addition in a new draft that
will be available in time for consideration in Salt
Lake.

Comment appreciated.

Ralph Weber


Franco Travostino, Director Content Internetworking Lab
Advanced Technology Investments
Nortel Networks, Inc.
600 Technology Park
Billerica, MA 01821 USA
Tel: 978 288 7708 Fax: 978 288 4690
email: travos@nortelnetworks.com
 --=====================_10463525==_.ALT-- From owner-ips@ece.cmu.edu Thu Nov 1 12:35:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA24742 for ; Thu, 1 Nov 2001 12:35:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1GuK001850 for ips-outgoing; Thu, 1 Nov 2001 11:56:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1GuIl01842 for ; Thu, 1 Nov 2001 11:56:18 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA1GuBu00638; Thu, 1 Nov 2001 08:56:11 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA1Gu6630922; Thu, 1 Nov 2001 08:56:06 -0800 From: "Bill Strahm" To: "Eddy Quicksall" , Subject: RE: iSCSI: current UNH Plugfest Date: Thu, 1 Nov 2001 08:55:57 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Importance: Normal In-Reply-To: <000101c162d7$330d39c0$0102a8c0@eddylaptop> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Usually the "Conservative in what you send, Liberal in what you accept" policy is used... In otherwords, The sender MUST set to 0 (or some other value) The receiver MUST ignore the value... This allows for some tweaking of the implementation, if I control both ends I might set a reserved value to 1, then I know something... If I receive a reserve value set to 1 and I don't do anything the other end knows it is not talking to itself (this can even be a versioning thing as well) Now, we need to be VERY careful in defining this, do we plan on having Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean... is it possible, is it desirable ? will there be extensions ??? If you can truly answer NO to all of those things, I would argue for REMOVING the reserved fields (if possible), if not, the MUST set, MUST ignore policy seems better Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Eddy Quicksall Sent: Thursday, November 01, 2001 5:15 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: current UNH Plugfest I am reluctant to say this because I think most people think the initiator/target must check for correctness ... but, it is my feeling that that job should be up to a basher program. The target should not be in the business of diagnosing the initiator. The only time a target should check a field is when it could crash the system or data. Some format errors may have no consequences whatsoever. Eddy -----Original Message----- From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] Sent: Wednesday, October 31, 2001 05:39 PM To: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest Attached are the new issues that arose during the iSCSI plugfest at UNH on Wednesday 31-Oct-2001. (Note: these issues do not take into account any modifications or clarifications that occured in the standard due to the issues raised on Monday or Tuesday.) Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 ------------------------------------------------------------------------ ---- 1. Are receivers (initiator or target) REQUIRED to check that reserved bits and/or fields are set to 0? Section 3 on page 48 of draft 8 says: "Any bits not defined MUST be set to zero. Any reserved fields and values MUST be 0 unless specified otherwise." and Section 8.3 on page 127 of draft 8 says: "Explicit violations of the PDU layout rules stated in this document are format errors. This when detected, usually indicates a major implementation flaw in one of the parties. When a target or an initiator receives an iSCSI PDU with a format error, it MUST reset all transport connections in the session immediately and escalate the format error to session recovery (section 8.11.4)." According to these rules, a PDU with reserved bits and/or fields that are not set to 0 violates the PDU layout rules. Therefore, if an initiator or target receives such a PDU, it should immediately close all connections in the session and go to session recovery. Clearly a format error has extremely severe consequences! Although all vendors are setting reserved bits and fields to 0 on PDUs they are sending, many are NOT checking PDUs they are receiving to see if these bits and fields are set to 0. Basically, vendors are saying "who cares if reserved bits and/or fields in incoming PDUs are not zero? We do not want to take the time to do this checking, and there is no benefit to doing it. As long as the non-reserved bits and fields are set properly, we can and should proceed. Any time devoted to doing this checking is wasted in 99+% of the cases, and in the (unlikely) case that a non-zero bit or field is found, the consequences are too severe." There should be some statement in the standard to clarify what checking is required and what is optional. 2. A similar situation arises with respect to checking the consistency of fields such as Version-max, Version-min and Version-active in Login Requests and Login Responses. For example, consider the Version-max field. Section 3.12.5 says: "All Login requests within the Login phase MUST carry the same Version-max." All vendor initiators are setting Version-max correctly on all login requests they are sending, but many vendor targets are NOT checking received login requests to ensure that this rule is enforced. In particular, many targets simply use the Version-max and Version-min on the first login request they receive on a new connection, and then they ignore these fields on all subsequent login requests in the same login phase. Strictly speaking, a change in the Version-max field during the login phase constitutes a protocol error according to section 8.8 on page 130 of draft 8: "All violations of iSCSI PDU exchange sequences specified in this draft are also protocol errors. This category of errors can be addressed only by fixing the implementations; iSCSI defines Reject and response codes to enable this". Therefore the target should send back a login response with a status of 0x0200 and then close the connection. However, Section 3.12.5 also says: "The target MUST use the value presented with the first login request." This rule seems to imply that the value CAN change, because if it cannot change, then it doesn't matter which one of the login requests it is taken from, they are all the same anyway. The suggestion is to keep the requirement that the target MUST use the value presented on the first login request, but to allowed the target to ignore the value presented on all subsequent login requests in the same login phase. A similar rewording should be done for the other fields. 3. Can commands be sent out of order on the same connection? The behavior of targets is clearly specified in Section 2.2.2.3 on page 25 of draft 8, which says: "Except for the commands marked for immediate delivery the iSCSI target layer MUST eliver the commands for execution in the order specified by CmdSN." Section 2.2.2.3 on page 26 of draft 8 also says: "- CmdSN - the current command Sequence Number advanced by 1 on each command shipped except for commands marked for immediate delivery." but the meaning of the term "shipped" is vague, and does not necessarily require that the PDUs arrive on the other end of a TCP connection in the same order that the CmdSN values were assigned to these PDUs. Some initiators have been designed to send commands out of CmdSN order on one connection. Consider the situation where there is only one connection and a high-level dispatcher creates a PDU for a SCSI command that involves writing immediate data to the target. This PDU is enqueued to a lower-level layer which has to setup, start, and wait-for a DMA operation to move the immediate data into an onboard buffer before the PDU can be put onto the wire. While this is happening, the dispatcher creates another unrelated PDU for a SCSI read command (for example), and when this PDU is passed to the lower-level layer it can be sent immediately, ahead of the previous write PDU and therefore out of order on this connection. The standard clearly allows this to happen if the two PDUs were sent on different connections, and seems to imply that this can also happen when the two PDUs are sent on the same connection. The suggestion is to put in the standard an explicit statement that this is allowed or not allowed, as appropriate. If this is allowed, such a statement would avoid the erroneous assumption being made by some target implementers that within a single connection, commands will arrive in order. If this is not allowed, such a statement would avoid the erroneous assumption being made by some initiator implementers that within a single connection, commands can be put on the wire out of order. 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) now allow: "A value of 0 indicates no limit." Is this useful? Does it buy anything? The difficulties implementers are having with this are: 1) It is a special case. 2) It causes discontinuous ranges (for example, [0,64..2**24]) 3) It violates the min/max function normally used for the key. 4) There is always a limit anyway. Consider FirstBurstSize, which can have a value that is described as "<0|number-64-2**24>", and for which the minimum of the 2 numbers is selected. I one side offers 0 to mean unlimited, and the other side has a limit, it will reply with that limit, say 128 Kbytes. Therefore, the result is not min(0, 128K) but rather max(0, 128K). The statement in the standard that "the minimum of the 2 numbers is selected" is therefore wrong when one of the numbers is 0. Furthermore, when an initiator or target receives an offer for one of these keys, it cannot simply check that the offered value is legal by testing it against some minimum and maximum. It must first check for 0 and then only if that check shows the value is non-zero can it do the min/max range check for legality (i.e., 64-2**24). Finally, there is always a limit. If nothing else it is the limit imposed by the 24-bit DataSegmentLength field of the PDU requesting the transfer. It is useless to specify a FirstBurstSize (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because the largest possible DataSegmentLength in any PDU that can use this value is 2**24-1. The suggestion is to just eliminate this special case of 0 and require that the range 64-to-(2**24-1) be used instead -- it has exactly the same effect in all cases, it is easier to describe in the standard because it avoids all the extra words, and it is easier to code because it avoids all the special cases. NOTE: the standard should specify the limit in the ranges for MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead of 2**24. The number 2**24 cannot be represented in the 24-bit DataSegmentLength field and therefore can never be used. 5. This is a suggestion for a minor rewording in the standard to avoid misunderstandings. In Appendix E on page 188 of draft 8 it says: "The response to this command is a text response containing a list of targets and their addresses. Each target is returned as a target record. A target record begins with the TargetName text key, followed by a list of TargetAddress text keys, ..." In fact, there are situations where there are no targets and/or no addresses. These situations are clearly defined in the draft after the sentences quoted above, but it would help if those sentences at least hinted at the possibility that the lists could be empty or might not contain addresses. A possible rewording would be: "The response to this command is a text response containing a list of zero or more targets and, optionally, their addresses. Each target is returned as a target record. A target record begins with the TargetName text key, followed by a list of zero or more TargetAddress text keys, ..." 6. This is a suggestion for another very minor rewording in the standard. At the end of section 2.2.3 on page 29 of draft 8 it says: "Before full feature phase is established, only Login PDUs are allowed. ..." The suggested rewording is: "Before full feature phase is established, only Login Request and Login Response PDUs are allowed. ..." From owner-ips@ece.cmu.edu Thu Nov 1 12:42:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25040 for ; Thu, 1 Nov 2001 12:41:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1H17o02243 for ips-outgoing; Thu, 1 Nov 2001 12:01:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1H15l02238 for ; Thu, 1 Nov 2001 12:01:06 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA1H0xn15708 for ; Thu, 1 Nov 2001 12:00:59 -0500 (EST) Content-Class: urn:content-classes:message Subject: RE: FCencap: Missing SOF/EOF characters MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C162F6.C73518A4" Date: Thu, 1 Nov 2001 12:00:58 -0500 Disposition-Notification-To: "Elizabeth Rodriguez" X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B29B@nc8220exchange.ral.lucent.com> Thread-Topic: FCencap: Missing SOF/EOF characters Thread-Index: AcFi9au2o3NyDHH+Q723eTU6AwEItAAAHTcw From: "Elizabeth Rodriguez" To: "Franco Travostino" , , "IPS Reflector" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C162F6.C73518A4 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Chair hat off -- =20 Inclusion of this information sounds good to me. Locale of information does not matter to me as much. As mentioned in previous email, we may also want to include the 'why not appropriate' in annex, for reader's information. =20 Elizabeth -----Original Message----- From: Franco Travostino [mailto:travos@nortelnetworks.com] Sent: Thursday, November 01, 2001 10:23 AM To: ENDL_TX@computer.org; IPS Reflector Subject: Re: FCencap: Missing SOF/EOF characters Ralph, why should we be waiting until 5.3 (last page) to break this news (e.g., Class 1 isn't supported)? Can we move your text up to section 1, aptly titled "Scope". Alternately, we could narrow down the very title of the document. 0.02 -franco At 07:33 AM 11/1/2001, Ralph Weber wrote: Off and on I hear complaints that it is not obvious why draft-ietf-ips-fcencapsulation-03.txt fails to list all the possible Fibre Channel SOF and EOF characters in 5.3. To address this concern, I propose adding the following text at the end of 5.3. Note: Some of the SOF and EOF characters defined by Fibre Channel are not listed in Table 1 and Table 2 because the design an operating characteristics of an IP Network make it inappropriate for transporting FC Frames containing those SOF and/or EOF characters. I proposed to make this addition in a new draft that will be available in time for consideration in Salt Lake. Comment appreciated. Ralph Weber Franco Travostino, Director Content Internetworking Lab Advanced Technology Investments Nortel Networks, Inc. 600 Technology Park Billerica, MA 01821 USA Tel: 978 288 7708 Fax: 978 288 4690 email: travos@nortelnetworks.com ------_=_NextPart_001_01C162F6.C73518A4 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Chair=20 hat off --
 
Inclusion of this information sounds good to=20 me.
Locale=20 of information does not matter to me as much.
As=20 mentioned in previous email, we may also want to include the 'why not=20 appropriate' in annex, for reader's information.
 
Elizabeth
 -----Original = Message-----
From:=20 Franco Travostino [mailto:travos@nortelnetworks.com]
Sent: = Thursday,=20 November 01, 2001 10:23 AM
To: ENDL_TX@computer.org; IPS=20 Reflector
Subject: Re: FCencap: Missing SOF/EOF=20 characters


Ralph,
why=20 should we be waiting until 5.3 (last page) to break this news (e.g., = Class 1=20 isn't supported)?
Can we move your text up to section 1, aptly = titled=20 "Scope". Alternately, we could narrow down the very title of the=20 document.

0.02
-franco

At 07:33 AM 11/1/2001, Ralph = Weber=20 wrote:
Off and on I hear = complaints that it=20 is not obvious
why draft-ietf-ips-fcencapsulation-03.txt fails = to
list=20 all the possible Fibre Channel SOF and EOF
characters in = 5.3.

To=20 address this concern, I propose adding the following
text at the = end of=20 5.3.

  Note: Some of the SOF and EOF characters defined=20 by
  Fibre Channel are not listed in Table 1 and Table = 2
 =20 because the design an operating characteristics
  of an IP = Network=20 make it inappropriate for
  transporting FC Frames = containing those=20 SOF and/or
  EOF characters.

I proposed to make this = addition=20 in a new draft that
will be available in time for consideration = in=20 Salt
Lake.

Comment appreciated.

Ralph=20 Weber


Franco Travostino, Director Content Internetworking=20 Lab
Advanced Technology Investments
Nortel Networks, Inc.
600 = Technology Park
Billerica, MA 01821 USA
Tel: 978 288 7708 Fax: = 978 288=20 4690
email:=20 travos@nortelnetworks.com

------_=_NextPart_001_01C162F6.C73518A4-- From owner-ips@ece.cmu.edu Thu Nov 1 12:43:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25083 for ; Thu, 1 Nov 2001 12:43:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1GVZe29788 for ips-outgoing; Thu, 1 Nov 2001 11:31:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1GVXl29778 for ; Thu, 1 Nov 2001 11:31:33 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id LAA05738; Thu, 1 Nov 2001 11:31:27 -0500 (EST) Date: Thu, 1 Nov 2001 11:31:27 -0500 From: "Robert D. Russell" To: Julian Satran cc: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: Comments below: > 4. Situation: The initiator and target have successfully completed the > login phase for a discovery session and are now in full feature phase. > The initiator sends a text command containing the single key: > "SendTargets=". What response is expected from the target? > > > Interpretation 1: > > According to the explanation on page 188 of draft 8 (page 189 of > draft 8a): > "If no target name is specified, the session should respond with > addresses only for the target to which the session is logged in. > This MUST be supported on operational sessions, and MAY NOT > return targets other than the one to which the session is logged in." > > However, for a discovery session there is no target per se (the > initiator does not specify a TargetName= during login), so the > target therefore follows the rule on page 188 of draft 8 (page 189 > of draft 8a): > > +++ > That is not strictly correct. You may use discovery to determine the > addresses and portal groups for a specific target - i.e. you may give a > target-name. If you give a target name that is all you get. If don't give > a name you get all the targets you are allowed to get to. > ++++ If I understand this reply correctly, you are saying that in a discovery session, when the initiator sends: SendTargets= it should expect to receive all the targets it is allowed to get to. Correct? This leads to 2 further questions/points: 1. Is this response any different from the response the initiator will get when it sends: SendTargets=all 2. If these responses are the same, then the explanation of on page 188 of draft 8 should be changed to indicate that this is what happens for a discovery session. Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Thu Nov 1 12:52:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25510 for ; Thu, 1 Nov 2001 12:52:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1HGQO03478 for ips-outgoing; Thu, 1 Nov 2001 12:16:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1HGOl03474 for ; Thu, 1 Nov 2001 12:16:24 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id JAA25752 for ips@ece.cmu.edu; Thu, 1 Nov 2001 09:16:18 -0800 (PST) From: Keith McCloghrie Message-Id: <200111011716.JAA25752@cisco.com> Subject: iSCSI MIB - comments on iscsiAccessList To: ips@ece.cmu.edu Date: Thu, 1 Nov 2001 09:16:18 -0800 (PST) X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I have some questions/suggestions regarding iscsiAccessList in draft-ietf-ips-iscsi-mib-03.txt. > 6.9. iscsiAccessList > > The iscsiAccessListAttributesTable contains an entry for each > initiator that is allowed to access the target under which it > appears. If a target allows access to any initiator, an > AccessListAttributesEntry with the initiator's iSCSI name should be > used. I think the last sentence is a) confusing (do you mean "any initiator" ?) and b) may not always be true - with a wildcard mechanism ("iscsi" in the next paragaph), an initiator's name does not have to be in the table, right ? > This table does not cover all possible access control schemes that a > vendor could implement. If access to an initiator cannot be > determined just by its iSCSI name, an implementation may either > include a single entry per target with the initiator name "iscsi", or > may choose to place no entries in this table. Does no entries in the table allow any initiator access, or does it deny access to all initiators ? > iscsiAccessListAttributesTable OBJECT-TYPE > SYNTAX SEQUENCE OF IscsiAccessListAttributesEntry > MAX-ACCESS not-accessible > STATUS current > DESCRIPTION > "A list of iSCSI initiators which will be granted access > to iSCSI resources through targets within the iSCSI > instance." Can you say explicitly: - what does no entries in the table mean, and - how does the wildcard entry (an entry with name="iscsi") work. > ... > > iscsiALInitiatorName OBJECT-TYPE > SYNTAX SnmpAdminString > MAX-ACCESS read-only > STATUS current > DESCRIPTION > "An octet string that defines an initiator identified > by the key of the Login Command which will > be granted access. If this string has the value of 'iscsi', > then any initiator may access this target." > ::= { iscsiAccessListAttributesEntry 2 } If you intend that an entry of "iscsi" means that "any initiator name is allowed", then I think it's a little strange that a special meaning applies to a name that an administrator might just happen to use without realising it. Here are three alternatives which I think are better: 1. have a column in the iscsiTargetAttributesTable which enables/disables the use of the access-list. (Then, disabling has the same function as "icsci" entry.) 2. have the zero-length name allow access to any name; (this is a special case of #3 below.) 3. have an additional column, iscsiALInitiatorMatchType, which is an INTEGER { exact(1), prefix(2) } where 'exact' is the type you currently have, and 'prefix' says that longer initiator names will match if they can be truncated to the value of iscsiALInitiatorName. Keith. From owner-ips@ece.cmu.edu Thu Nov 1 12:54:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25584 for ; Thu, 1 Nov 2001 12:54:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1HUWe04661 for ips-outgoing; Thu, 1 Nov 2001 12:30:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel7.hp.com (atlrel7.hp.com [192.151.27.9]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1HUUl04655 for ; Thu, 1 Nov 2001 12:30:30 -0500 (EST) Received: from core.rose.hp.com (unknown [15.43.208.100]) by atlrel7.hp.com (Postfix) with ESMTP id 7AD2D1F90A; Thu, 1 Nov 2001 12:27:48 -0500 (EST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id JAA14784; Thu, 1 Nov 2001 09:31:50 -0800 (PST) Message-Id: <200111011731.JAA14784@core.rose.hp.com> Date: Thu, 01 Nov 2001 09:42:35 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.73 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: Julian Satran Cc: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, > +++ I think this is the correct interpretation and we will fix the wording > in section 8 to say this. I suggest: > > - During Login, any failure in negotiation MUST be considered as the login > process failure and the login phase must be terminated. If the failure is > detected by the target, it must reject the login with the appropriate > code. The connection must be terminated by the initiator. I agree. However, I suggest the following with a minor change (and leaving out the last sentence). - During Login, any failure in negotiation MUST be considered as the login process failure and the login phase must be terminated. If the failure is detected by the target, it must terminate the login with the appropriate login response status code. Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Julian Satran wrote: > > Bob, > > Comments in text. > > Thanks, > Julo > > "Robert D. Russell" > Sent by: owner-ips@ece.cmu.edu > 31-10-01 02:06 > Please respond to "Robert D. Russell" > > > To: ips@ece.cmu.edu > cc: > Subject: Re: iSCSI: current UNH Plugfest > > > > Attached are some new issues that arose during the iSCSI plugfest at > UNH on Tuesday 29-Oct-2001. > (Note: these issues do not take into account any modifications or > clarifications that occured in the standard due to the issues raised > on Monday.) > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > ----------------------------------------------------------------------------- > > 1. Situation: as the first command on a new TCP connection, the initiator > sends a login with T=1, CSG=1, NSG=3, and valid InitiatorName, > TargetName, > and SessionType keys. However, there is also a valid key having an > invalid value, such as MaxConnections=abcd (i.e., not a number after > the > '=') or MaxConnections4 (i.e., missing the '='). > What should the target do? > > Interpretation 1: > > According to section 3.10.4 page 82 of draft 8 (page 83 of draft 8a), > "Any other key not understood by the target may be ignored by the > target without affecting basic function. However the Text Response > for a key that was not understood MUST be key=NotUnderstood." > > Two things have to be clarified: > 1. Does this section also apply to keys received in a login? > 2. Can "NotUnderstood" also apply to "values of keys" that are not > understood, even if the key word itself is understood? > +++ > 1.NotUnderstood appears also in the negotiation section so it applies to > login. > I will remove it from the text section > 2.It does not apply to values > +++ > > If the answer to these 2 of these questions is "yes", then the > appropriate > response would seem to be for the target to just ignore the key and > send > back MaxConnections=NotUnderstood as part of its next login response. > > > Interpratation 2: > > According to section 8.7 page 129 of draft 8 (page 130 of draft 8a), > "A negotiation failure is considered one or both of the following: > - None of the choices or the stated value is acceptable to one > negotiating side. ..." > Clearly this stated value ("abcd") is not acceptable to the target. > Therefore, the following rule on page 129, draft 8 (page 130, draft 8a) > applies: > "- During Login, any failure in negotiation MUST be considered > as the login process failure and the connection must be dropped." > > Therefore, the target should just drop the connection without sending > any login response back to the initiator. > > Interpretation 3: > > This is a login command that contains an error caused by the > initiator. Therefore, the target should send back a login response > with a status code of 0x0200 and then close the TCP connection. > > +++ I think this is the correct interpretation and we will fix the wording > in section 8 to say this. I suggest: > > - During Login, any failure in negotiation MUST be considered as the login > process failure and the login phase must be terminated. If the failure is > detected by the target, it must reject the login with the appropriate > code. The connection must be terminated by the initiator. > > +++ > > 2. Situation: on the first login command in operational parameter > negotiation > stage, the initiator sends no operational keys, thereby telling the > target > that it accepts all the default values for these keys. However, the > target > wants to negotiate the value of MaxConnections, so in the login > response > it sends back "MaxConnections=3" (for example). Should the initiator > send a response to this key or not? > > The statement in section 2.2.4 on page 30 of draft 8 and 8a: > "For numerical (and single literal) negotiations, the responding party > MUST > respond with the required key.(...)" > makes it clear that the responding party MUST respond. However, in > this > situation, it is not clear who the responding party is. > > +++ we are using the term offering and responding to distinguish them from > initiator and target and explicitly say (in 2.2.4) that the target may > offer keys > +++ > > Interpretation 1: > > By not explicitly sending this key in the login command, the initiator > is implicitly offering the default value and therefore is the offering > party and the target is the responding party. The conclusion is that > the initiator does not have to send a response to this key from the > target. > > +++ there is no implicit ofering of defaults. A default is accepted only > if the two parties are silent. > > +++ > Interpretation 2: > > The target is the offering party because it is the party that > explicitly > stated the key for the first time during these negotiations. The > conclusion is that the initiator MUST send a response to this key from > the > target. > > NOTE 1: If interpretation 1 is correct, it would seem to imply that the > target MUST respond to every key whether or not it is present in the > login from the initiator, even if it does not want to change the > default > value. The reason is that a missing key is an implicit offer of the > default value, and the responding party MUST respond. Is this a > correct > interpretation? > > NOTE 2: The following statements in section 2.2.4 page 29 of draft 8a: > > Originator-> = > Responder-> =|NOtUnderstood > > seem to imply that the originator is the party (initiator or target) > that > explicitly offers a key, and that omitting a key is not an implicit > offer > of that key with the default value. However, even in the revised draft > 8a > there is no definition of "Originator" and/or "Responder" that would > make this clear. Adding to the standard these definitions, and an > explicit statement that "a missing key does not constitute an implicit > offer of the default" would help eliminate misunderstandings. In > addition, including an example of this situation (where an initiator > omits a key and the target offers the key) would be a big help. > > +++ will do +++ > > 3. Some of the login phase examples given in Appendix A of both draft 8 > and > 8a do not follow the rule in section 3.12.4 page 87 of draft 8 (page 88 > of draft 8a): > "The next stage value is valid only when the T bit is 1 and is > reserved otherwise." > and the rule in section 3 page 48 of draft 8 (page 49 of draft 8a): > "Any reserved fields and values MUST be 0 unless specified > otherwise." > > If these rules are applied, all examples having T=0 should also > have NSG=0. Presently all of them with T=0 also have NSG=1 or NSG=3. > > +++ Will fix the examples +++ > > 4. Situation: The initiator and target have successfully completed the > login phase for a discovery session and are now in full feature phase. > The initiator sends a text command containing the single key: > "SendTargets=". What response is expected from the target? > > Interpretation 1: > > According to the explanation on page 188 of draft 8 (page 189 of > draft 8a): > "If no target name is specified, the session should respond with > addresses only for the target to which the session is logged in. > This MUST be supported on operational sessions, and MAY NOT > return targets other than the one to which the session is logged in." > > However, for a discovery session there is no target per se (the > initiator does not specify a TargetName= during login), so the > target therefore follows the rule on page 188 of draft 8 (page 189 > of draft 8a): > > +++ > That is not strictly correct. You may use discovery to determine the > addresses and portal groups for a specific target - i.e. you may give a > target-name. If you give a target name that is all you get. If don't give > a name you get all the targets you are allowed to get to. > ++++ > > "A SendTargets response MAY contain no target names, if there are no > targets for the requesting initiator to access." > and sends back a Text Response with no keys in it. > > Interpretation 2: > > In a discovery session, the key "SendTargets=" makes no sense and > should > be treated by the target in the same manner as the key > "SendTargets=all". > > 5. Some common error situations: > > 1) - when a SCSI Response contains a CHECK CONDITION (Status=0x02), > some targets are not including the SenseLength as the first 2 > bytes in the data segment. Although the format of the data segment > is clear from the diagram in section 3.4.6 on page 62 of draft 8 > (page 63 of draft 8a), the last entry in the diagram for the SCSI > Response PDU on page 58 of draft 8 (page 59 of draft 8a) is > misleading because it mentions only the Sense Data and Response > Data and omits the Sense Length. It would therefore be helpful > if the last entry in the diagram on page 58 were changed to > explicitly > reference the diagram on page 62, as in: > > Data Segment -- see section 3.4.6 (optional) > +++ will do +++ > > 2) - after sending a CmdSN on an initial login, some initiators are > incrementing it before sending their first non-immediate command. > (i.e., if the initial login contains CmdSN=123, they are sending > CmdSN=124 on the first non-immediate command after the login phase). > Section 3.12.10 on page 89 of draft 8 (page 90 of draft 8a) is > clear that in this example the first non-immediate command should > carry CmdSN=123, not 124. This was an issue at the July plugfest > and apparently some implementations have not been updated to conform > to the draft 8 standard in their handling of CmdSN. From owner-ips@ece.cmu.edu Thu Nov 1 12:54:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25596 for ; Thu, 1 Nov 2001 12:54:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1HUGp04640 for ips-outgoing; Thu, 1 Nov 2001 12:30:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1HUDl04629 for ; Thu, 1 Nov 2001 12:30:13 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA32672 for ; Thu, 1 Nov 2001 18:30:07 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1HU6842482 for ; Thu, 1 Nov 2001 18:30:06 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 1 Nov 2001 19:30:02 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 19:30:06, Serialize complete at 01/11/2001 19:30:06 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, Rereading the text I understand what is confusing. The target name mentioned is the one from the previous item! I will reformulate it to read: The value must be one of: all The initiator is requesting that information on all relevant targets known to the implementation be returned. This value MUST be supported on a discovery session, and MAY NOT be supported on an operational session. If an iSCSI target name is specified, the session should respond with addresses for only the named target, if possible. This value MUST be supported on discovery sessions. A discovery session MUST be capable of returning addresses for those targets that would have been returned had value=all been designated. The session should respond with addresses only for the target to which the session is logged in. This MUST be supported on operational sessions, and MAY NOT return targets other than the one to which the session is logged in. Thanks, Julo "Robert D. Russell" 01-11-01 18:31 Please respond to "Robert D. Russell" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest Julian: Comments below: > 4. Situation: The initiator and target have successfully completed the > login phase for a discovery session and are now in full feature phase. > The initiator sends a text command containing the single key: > "SendTargets=". What response is expected from the target? > > > Interpretation 1: > > According to the explanation on page 188 of draft 8 (page 189 of > draft 8a): > "If no target name is specified, the session should respond with > addresses only for the target to which the session is logged in. > This MUST be supported on operational sessions, and MAY NOT > return targets other than the one to which the session is logged in." > > However, for a discovery session there is no target per se (the > initiator does not specify a TargetName= during login), so the > target therefore follows the rule on page 188 of draft 8 (page 189 > of draft 8a): > > +++ > That is not strictly correct. You may use discovery to determine the > addresses and portal groups for a specific target - i.e. you may give a > target-name. If you give a target name that is all you get. If don't give > a name you get all the targets you are allowed to get to. > ++++ If I understand this reply correctly, you are saying that in a discovery session, when the initiator sends: SendTargets= it should expect to receive all the targets it is allowed to get to. Correct? This leads to 2 further questions/points: 1. Is this response any different from the response the initiator will get when it sends: SendTargets=all 2. If these responses are the same, then the explanation of on page 188 of draft 8 should be changed to indicate that this is what happens for a discovery session. Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Thu Nov 1 13:47:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA27344 for ; Thu, 1 Nov 2001 13:47:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1IF7508569 for ips-outgoing; Thu, 1 Nov 2001 13:15:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1IF4l08557 for ; Thu, 1 Nov 2001 13:15:04 -0500 (EST) Received: from cisco.com (58509@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id NAA19673; Thu, 1 Nov 2001 13:14:57 -0500 (EST) Message-ID: <3BE18ECE.89BB64E0@cisco.com> Date: Thu, 01 Nov 2001 12:05:02 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Keith McCloghrie CC: ips@ece.cmu.edu Subject: Re: iSCSI MIB - comments on iscsiAccessList References: <200111011716.JAA25752@cisco.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Keith McCloghrie wrote: > > I have some questions/suggestions regarding iscsiAccessList > in draft-ietf-ips-iscsi-mib-03.txt. > > > 6.9. iscsiAccessList > > > > The iscsiAccessListAttributesTable contains an entry for each > > initiator that is allowed to access the target under which it > > appears. If a target allows access to any initiator, an > > AccessListAttributesEntry with the initiator's iSCSI name should be > > used. > > I think the last sentence is > > a) confusing (do you mean "any initiator" ?) and No. I meant "any particular initiator"; this was badly worded. > b) may not always be true - with a wildcard mechanism ("iscsi" in the next > paragaph), an initiator's name does not have to be in the table, right ? Yes. That is correct. > > This table does not cover all possible access control schemes that a > > vendor could implement. If access to an initiator cannot be > > determined just by its iSCSI name, an implementation may either > > include a single entry per target with the initiator name "iscsi", or > > may choose to place no entries in this table. > > Does no entries in the table allow any initiator access, or does it > deny access to all initiators ? I think that we need to decide that here. It was originally intended to mean that all initiators are allowed, but since we have a way to say that by putting in "iscsi", I think that it should mean either that all initiators are denied, or that the access list mechanism in the MIB is not enough to determine whether or not an initiator will be granted access. I would tend toward the latter; if there are no access list entries for a target, it means that there is not enough information to tell who gets to access it from this MIB. Implementations that provide value-add access control can augment this MIB with their own enterprise MIBs for access control. I would like to re-word this to: ... The value "iscsi" in an access list entry means that any initiator may access this target. If a target has no entries in the access list attributes table, it means that there is not enough information here to determine whether or not an initiator will be granted access. > > iscsiAccessListAttributesTable OBJECT-TYPE > > SYNTAX SEQUENCE OF IscsiAccessListAttributesEntry > > MAX-ACCESS not-accessible > > STATUS current > > DESCRIPTION > > "A list of iSCSI initiators which will be granted access > > to iSCSI resources through targets within the iSCSI > > instance." > > Can you say explicitly: > > - what does no entries in the table mean, and > - how does the wildcard entry (an entry with name="iscsi") work. OK. > > ... > > > > iscsiALInitiatorName OBJECT-TYPE > > SYNTAX SnmpAdminString > > MAX-ACCESS read-only > > STATUS current > > DESCRIPTION > > "An octet string that defines an initiator identified > > by the key of the Login Command which will > > be granted access. If this string has the value of 'iscsi', > > then any initiator may access this target." > > ::= { iscsiAccessListAttributesEntry 2 } > > If you intend that an entry of "iscsi" means that "any initiator > name is allowed", then I think it's a little strange that a special > meaning applies to a name that an administrator might just happen to > use without realising it. "iscsi" is a reserved initiator name value, and must not be assigned to an initiator or target by an adminstrator or manufacturer. There should be no problems with using it here. > > Here are three alternatives which I think are better: > > 1. have a column in the iscsiTargetAttributesTable which enables/disables > the use of the access-list. (Then, disabling has the same function as > "icsci" entry.) This is probably the cleanest; an implementation that does not do access lists would not have to implement the access list part of the MIB at all. > 2. have the zero-length name allow access to any name; (this is a special > case of #3 below.) > > 3. have an additional column, iscsiALInitiatorMatchType, which is an > INTEGER { exact(1), prefix(2) } where 'exact' is the type you currently > have, and 'prefix' says that longer initiator names will match > if they can be truncated to the value of iscsiALInitiatorName. > > Keith. -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Thu Nov 1 13:50:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA27446 for ; Thu, 1 Nov 2001 13:50:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1INAf09315 for ips-outgoing; Thu, 1 Nov 2001 13:23:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1IN8l09311 for ; Thu, 1 Nov 2001 13:23:08 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id KAA13407; Thu, 1 Nov 2001 10:22:59 -0800 (PST) From: Keith McCloghrie Message-Id: <200111011822.KAA13407@cisco.com> Subject: Re: iSCSI MIB - comments on iscsiAccessList To: mbakke@cisco.com (Mark Bakke) Date: Thu, 1 Nov 2001 10:22:58 -0800 (PST) Cc: kzm@cisco.com (Keith McCloghrie), ips@ece.cmu.edu In-Reply-To: <3BE18ECE.89BB64E0@cisco.com> from "Mark Bakke" at Nov 01, 2001 12:05:02 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > > > 6.9. iscsiAccessList > > > > > > The iscsiAccessListAttributesTable contains an entry for each > > > initiator that is allowed to access the target under which it > > > appears. If a target allows access to any initiator, an > > > AccessListAttributesEntry with the initiator's iSCSI name should be > > > used. > > > > I think the last sentence is > > > > a) confusing (do you mean "any initiator" ?) and > > No. I meant "any particular initiator"; this was badly worded. How about also changing "with the initiator's iSCSI name" to "with that initiator's iSCSI name". > > > This table does not cover all possible access control schemes that a > > > vendor could implement. If access to an initiator cannot be > > > determined just by its iSCSI name, an implementation may either > > > include a single entry per target with the initiator name "iscsi", or > > > may choose to place no entries in this table. > > > > Does no entries in the table allow any initiator access, or does it > > deny access to all initiators ? > > I think that we need to decide that here. It was originally intended > to mean that all initiators are allowed, but since we have a way to > say that by putting in "iscsi", I think that it should mean either that > all initiators are denied, or that the access list mechanism in the > MIB is not enough to determine whether or not an initiator will > be granted access. I would tend toward the latter; if there are no > access list entries for a target, it means that there is not enough > information to tell who gets to access it from this MIB. Implementations > that provide value-add access control can augment this MIB with their > own enterprise MIBs for access control. > > I would like to re-word this to: > > ... The value "iscsi" in an access list entry means that any > initiator may access this target. If a target has no entries > in the access list attributes table, it means that there is > not enough information here to determine whether or not an > initiator will be granted access. OK. > > > iscsiALInitiatorName OBJECT-TYPE > > > SYNTAX SnmpAdminString > > > MAX-ACCESS read-only > > > STATUS current > > > DESCRIPTION > > > "An octet string that defines an initiator identified > > > by the key of the Login Command which will > > > be granted access. If this string has the value of 'iscsi', > > > then any initiator may access this target." > > > ::= { iscsiAccessListAttributesEntry 2 } > > > > If you intend that an entry of "iscsi" means that "any initiator > > name is allowed", then I think it's a little strange that a special > > meaning applies to a name that an administrator might just happen to > > use without realising it. > > "iscsi" is a reserved initiator name value, and must not be assigned > to an initiator or target by an adminstrator or manufacturer. There > should be no problems with using it here. > > > > > Here are three alternatives which I think are better: > > > > 1. have a column in the iscsiTargetAttributesTable which enables/disables > > the use of the access-list. (Then, disabling has the same function as > > "icsci" entry.) > > This is probably the cleanest; an implementation that does not do > access lists would not have to implement the access list part of > the MIB at all. OK. Thanks, Keith. From owner-ips@ece.cmu.edu Thu Nov 1 13:56:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA27609 for ; Thu, 1 Nov 2001 13:56:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1Hn0x06366 for ips-outgoing; Thu, 1 Nov 2001 12:49:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Hmxl06362 for ; Thu, 1 Nov 2001 12:48:59 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA35618 for ; Thu, 1 Nov 2001 18:48:53 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1Hmqr120470 for ; Thu, 1 Nov 2001 18:48:52 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: iSCSI Rev 8 X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 1 Nov 2001 19:48:48 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 19:48:52, Serialize complete at 01/11/2001 19:48:52 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks - Julo "Dave Peterson" Sent by: owner-ips@ece.cmu.edu 01-11-01 18:56 Please respond to "Dave Peterson" To: "Ips@Ece. Cmu. Edu" cc: Subject: iSCSI: iSCSI Rev 8 Good Morning, Suggested rewording for clause 3.4.1 in iSCSI Rev 8: bit 4 (o) set for Bidirectional Read Residual Overflow. In this case, the Bidirectional Read Residual Count indicates the number of bytes that were not transferred to the initiator because the initiator's Expected Bidirectional Read Data Transfer Length was not sufficient. bit 3 (u) set for Bidirectional Read Residual Underflow. In this case, the Bidirectional Read Residual Count indicates the number of bytes that were not transferred to the initiator out of the number of bytes that were expected to be transferred. bit 2 (O) set for Residual Overflow. In this case, the Residual Count indicates the number of bytes that were not transferred because the initiator's Expected Data Transfer length was not sufficient. For a bidirectional operation, the Residual Count contains the residual for the write operation. bit 1 (U) set for Residual Underflow. In this case, the Residual Count indicates the number of bytes that were not transferred out of the number of bytes that were expected to be transferred. For a bidirectional operation, the Residual Count contains the residual for the write operation. Suggested rewording for clause 3.4.4 3.4.4 Residual Count The Residual Count field is valid only in the case where either the U bit or the O bit is set. If neither bit is set, the Residual Count field SHOULD be zero. If the O bit is set, the Residual Count indicates the number of bytes that were not transferred because the initiator's Expected Data Transfer Length was not sufficient. If the U bit is set, the Residual Count indicates the number of bytes that were not transferred out of the number of bytes expected to be transferred. 3.4.5 Bidirectional Read Residual Count The Bidirectional Read Residual Count field is valid only in the case where either the u bit or the o bit is set. If neither bit is set, the Bidirectional Read Residual Count field SHOULD be zero. If the o bit is set, the Bidirectional Read Residual Count indicates the number of bytes that were not transferred to the initiator because the initiator's Expected Bidirectional Read Transfer Length was not sufficient. If the u bit is set, the Bidirectional Read Residual Count indicates the number of bytes that were not transferred to the initiator out of the number of bytes that were expected to be transferred. Dave From owner-ips@ece.cmu.edu Thu Nov 1 14:29:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA28335 for ; Thu, 1 Nov 2001 14:29:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1Id3R10706 for ips-outgoing; Thu, 1 Nov 2001 13:39:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA1Icwl10699 for ; Thu, 1 Nov 2001 13:38:58 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fA1IcbM03325; Thu, 1 Nov 2001 13:38:37 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fA1Icab30981; Thu, 1 Nov 2001 13:38:37 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15329.38576.650000.176588@gargle.gargle.HOWL> Date: Thu, 1 Nov 2001 13:38:40 -0500 From: Paul Koning To: bill@sanera.net Cc: Eddy_Quicksall@ivivity.com, ips@ece.cmu.edu Subject: RE: iSCSI: current UNH Plugfest References: <000101c162d7$330d39c0$0102a8c0@eddylaptop> X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 1 November 2001) by Bill Strahm: > Usually the "Conservative in what you send, Liberal in what you accept" > policy is used... That's a very important principle... > In otherwords, The sender MUST set to 0 (or some other value) The receiver > MUST ignore the value... That's the way to express the principle. But the text quoted in the earlier notes does not express it the same way. In particular "... are format errors. This when detected..." implies that receivers may or may not detect format errors. In other words, it implies -- but does NOT state explicitly -- that receivers MAY check reserved fields for zeroness. > This allows for some tweaking of the implementation, if I control both ends > I might set a reserved value to 1, then I know something... If I receive a > reserve value set to 1 and I don't do anything the other end knows it is not > talking to itself (this can even be a versioning thing as well) > > Now, we need to be VERY careful in defining this, do we plan on having > Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean... > is it possible, is it desirable ? will there be extensions ??? > > If you can truly answer NO to all of those things, I would argue for > REMOVING the reserved fields (if possible), if not, the MUST set, MUST > ignore policy seems better I agree strongly. There is a lot of experience in the community on what helps and what hurts convenient version upgrade. In particular, there is a LOT of evidence that ANY checking of reserved fields is a nasty thing. Unless you plan never to go beyond V1, you really need to require the rule Bill stated, i.e., receivers MUST ignore the contents of reserved fields -- they are NOT allowed to "verify" them. So the spec needs to be clarified to say that random values in reserved fields are NOT format errors and receivers MUST NOT treat them as such. paul > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Eddy Quicksall > Sent: Thursday, November 01, 2001 5:15 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: current UNH Plugfest > > > I am reluctant to say this because I think most people think the > initiator/target must check for correctness ... but, it is my feeling that > that job should be up to a basher program. The target should not be in the > business of diagnosing the initiator. The only time a target should check a > field is when it could crash the system or data. Some format errors may have > no consequences whatsoever. > > Eddy > > > -----Original Message----- > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > Sent: Wednesday, October 31, 2001 05:39 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: current UNH Plugfest > > > Attached are the new issues that arose during the iSCSI plugfest > at UNH on Wednesday 31-Oct-2001. > > (Note: these issues do not take into account any modifications or > clarifications that occured in the standard due to the issues raised > on Monday or Tuesday.) > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > ------------------------------------------------------------------------ > ---- > > 1. Are receivers (initiator or target) REQUIRED to check that reserved > bits and/or fields are set to 0? > > Section 3 on page 48 of draft 8 says: > "Any bits not defined MUST be set to zero. Any reserved fields and > values MUST be 0 unless specified otherwise." > > and Section 8.3 on page 127 of draft 8 says: > "Explicit violations of the PDU layout rules stated in this > document > are format errors. This when detected, usually indicates a major > implementation flaw in one of the parties. > > When a target or an initiator receives an iSCSI PDU with a format > error, it MUST reset all transport connections in the session > immediately and escalate the format error to session recovery > (section 8.11.4)." > > According to these rules, a PDU with reserved bits and/or fields that > are not set to 0 violates the PDU layout rules. Therefore, if an > initiator or target receives such a PDU, it should immediately close > all connections in the session and go to session recovery. > > Clearly a format error has extremely severe consequences! > > Although all vendors are setting reserved bits and fields to 0 on > PDUs they are sending, many are NOT checking PDUs they are receiving > to see if these bits and fields are set to 0. Basically, vendors are > saying "who cares if reserved bits and/or fields in incoming PDUs are > not zero? We do not want to take the time to do this checking, and > there is no benefit to doing it. As long as the non-reserved bits > and > fields are set properly, we can and should proceed. Any time devoted > to doing this checking is wasted in 99+% of the cases, and in the > (unlikely) case that a non-zero bit or field is found, the > consequences are too severe." > > There should be some statement in the standard to clarify what > checking > is required and what is optional. > > 2. A similar situation arises with respect to checking the consistency > of fields such as Version-max, Version-min and Version-active in > Login > Requests and Login Responses. > > For example, consider the Version-max field. > > Section 3.12.5 says: > "All Login requests within the Login phase MUST carry the same > Version-max." > > All vendor initiators are setting Version-max correctly on all > login requests they are sending, but many vendor targets are NOT > checking received login requests to ensure that this rule is > enforced. > In particular, many targets simply use the Version-max and > Version-min > on the first login request they receive on a new connection, and then > they ignore these fields on all subsequent login requests in the same > login phase. > > Strictly speaking, a change in the Version-max field during the login > phase constitutes a protocol error according to section 8.8 on page > 130 > of draft 8: > > "All violations of iSCSI PDU exchange sequences specified in this > draft are also protocol errors. This category of errors can be > addressed only by fixing the implementations; iSCSI defines Reject > and response codes to enable this". > > Therefore the target should send back a login response with a status > of 0x0200 and then close the connection. > > However, Section 3.12.5 also says: > "The target MUST use the value presented with the first login > request." > > This rule seems to imply that the value CAN change, because if it > cannot > change, then it doesn't matter which one of the login requests it is > taken from, they are all the same anyway. > > The suggestion is to keep the requirement that the target MUST use > the > value presented on the first login request, but to allowed the target > to ignore the value presented on all subsequent login requests in the > same login phase. A similar rewording should be done for the other > fields. > > 3. Can commands be sent out of order on the same connection? > > The behavior of targets is clearly specified in Section 2.2.2.3 on > page 25 of draft 8, which says: > "Except for the commands marked for immediate delivery the iSCSI > target layer MUST eliver the commands for execution in the order > specified by CmdSN." > > Section 2.2.2.3 on page 26 of draft 8 also says: > "- CmdSN - the current command Sequence Number advanced by 1 on > each command shipped except for commands marked for immediate > delivery." > but the meaning of the term "shipped" is vague, and does not > necessarily > require that the PDUs arrive on the other end of a TCP connection > in the same order that the CmdSN values were assigned to these PDUs. > > Some initiators have been designed to send commands out of CmdSN > order on one connection. Consider the situation where there is only > one connection and a high-level dispatcher creates a PDU for a SCSI > command that involves writing immediate data to the target. This PDU > is enqueued to a lower-level layer which has to setup, start, and > wait-for a DMA operation to move the immediate data into an onboard > buffer before the PDU can be put onto the wire. While this is > happening, the dispatcher creates another unrelated PDU for a SCSI > read command (for example), and when this PDU is passed to the > lower-level layer it can be sent immediately, ahead of the previous > write PDU and therefore out of order on this connection. > > The standard clearly allows this to happen if the two PDUs were sent > on different connections, and seems to imply that this can also > happen > when the two PDUs are sent on the same connection. > > The suggestion is to put in the standard an explicit statement that > this is allowed or not allowed, as appropriate. > > If this is allowed, such a statement would avoid the erroneous > assumption being made by some target implementers that within a > single > connection, commands will arrive in order. > > If this is not allowed, such a statement would avoid the erroneous > assumption being made by some initiator implementers that within a > single connection, commands can be put on the wire out of order. > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > now allow: "A value of 0 indicates no limit." > > Is this useful? Does it buy anything? > > The difficulties implementers are having with this are: > > 1) It is a special case. > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > 3) It violates the min/max function normally used for the key. > 4) There is always a limit anyway. > > Consider FirstBurstSize, which can have a value that is described > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > is selected. > > I one side offers 0 to mean unlimited, and the other side > has a limit, it will reply with that limit, say 128 Kbytes. > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > The statement in the standard that "the minimum of the 2 numbers is > selected" is therefore wrong when one of the numbers is 0. > > Furthermore, when an initiator or target receives an offer for one > of these keys, it cannot simply check that the offered value is > legal by testing it against some minimum and maximum. It must first > check for 0 and then only if that check shows the value is non-zero > can it do the min/max range check for legality (i.e., 64-2**24). > > Finally, there is always a limit. If nothing else it is the > limit imposed by the 24-bit DataSegmentLength field of the PDU > requesting the transfer. It is useless to specify a FirstBurstSize > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > the largest possible DataSegmentLength in any PDU that can use > this value is 2**24-1. > > The suggestion is to just eliminate this special case of 0 and > require > that the range 64-to-(2**24-1) be used instead -- it has exactly the > same effect in all cases, it is easier to describe in the standard > because it avoids all the extra words, and it is easier to code > because it avoids all the special cases. > > NOTE: the standard should specify the limit in the ranges for > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > of 2**24. The number 2**24 cannot be represented in the 24-bit > DataSegmentLength field and therefore can never be used. > > 5. This is a suggestion for a minor rewording in the standard to avoid > misunderstandings. > > In Appendix E on page 188 of draft 8 it says: > > "The response to this command is a text response containing a list > of > targets and their addresses. Each target is returned as a target > record. A target record begins with the TargetName text key, > followed by a list of TargetAddress text keys, ..." > > In fact, there are situations where there are no targets and/or no > addresses. These situations are clearly defined in the draft after > the sentences quoted above, but it would help if those sentences > at least hinted at the possibility that the lists could be empty > or might not contain addresses. A possible rewording would be: > > "The response to this command is a text response containing a list > of > zero or more targets and, optionally, their addresses. Each target > is returned as a target record. A target record begins with the > TargetName text key, followed by a list of zero or more > TargetAddress text keys, ..." > > > 6. This is a suggestion for another very minor rewording in the > standard. > > At the end of section 2.2.3 on page 29 of draft 8 it says: > > "Before full feature phase is established, only Login PDUs are > allowed. ..." > > The suggested rewording is: > > "Before full feature phase is established, only Login Request and > Login Response PDUs are allowed. ..." From owner-ips@ece.cmu.edu Thu Nov 1 14:32:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA28408 for ; Thu, 1 Nov 2001 14:32:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1IspC12106 for ips-outgoing; Thu, 1 Nov 2001 13:54:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Isnl12100 for ; Thu, 1 Nov 2001 13:54:49 -0500 (EST) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA06444; Thu, 1 Nov 2001 11:54:31 -0700 (MST) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [172.20.248.1]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v2.1p1) with ESMTP id LAA13606; Thu, 1 Nov 2001 11:54:44 -0700 (MST) Received: from sun.com (nws-dhcp-199-212.Central.Sun.COM [172.20.193.212]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id LAA04388; Thu, 1 Nov 2001 11:54:44 -0700 (MST) Message-ID: <3BE19A82.1E3895A3@sun.com> Date: Thu, 01 Nov 2001 11:54:58 -0700 From: "Mark A. Carlson" Reply-To: mark.carlson@sun.com Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.75 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Mark Bakke CC: ips@ece.cmu.edu Subject: Re: iSCSI MIB - comments on iscsiAccessList References: <200111011716.JAA25752@cisco.com> <3BE18ECE.89BB64E0@cisco.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark Bakke wrote: > > Keith McCloghrie wrote: > > ... > > > This table does not cover all possible access control schemes that a > > > vendor could implement. If access to an initiator cannot be > > > determined just by its iSCSI name, an implementation may either > > > include a single entry per target with the initiator name "iscsi", or > > > may choose to place no entries in this table. > > > > Does no entries in the table allow any initiator access, or does it > > deny access to all initiators ? > > I think that we need to decide that here. It was originally intended > to mean that all initiators are allowed, but since we have a way to > say that by putting in "iscsi", I think that it should mean either that > all initiators are denied, or that the access list mechanism in the > MIB is not enough to determine whether or not an initiator will > be granted access. I would tend toward the latter; if there are no > access list entries for a target, it means that there is not enough > information to tell who gets to access it from this MIB. Implementations > that provide value-add access control can augment this MIB with their > own enterprise MIBs for access control. > > I would like to re-word this to: > > ... The value "iscsi" in an access list entry means that any > initiator may access this target. If a target has no entries > in the access list attributes table, it means that there is > not enough information here to determine whether or not an > initiator will be granted access. But isn't the fact that the iSCSI name is used, as part of what access control is based on, useful to applications? For example, if I use a combination of iSCSI name and host userid to determine access, I would still like to know that *some* access is available from this initiator so that I can relate this logical relationship (maybe draw a graph). How about a field that indicates "additional information required" rather than leaving the whole entry blank? (empty really should mean no access, rather than no standard access, and we need to differentiate these two). Even if a vendor extends iscsiAccessList for their own access control, at least the standard properties could be supported so that applications can see that access is allowed from somewhere. Having iscsiALInitiatorName = iscsi, and "additional information" = true, would indicate that the proprietary access control mechanism needs to be consulted to understand who has access, but at least there is some access allowed. -- mark From owner-ips@ece.cmu.edu Thu Nov 1 14:51:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA28717 for ; Thu, 1 Nov 2001 14:51:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1JQAT14679 for ips-outgoing; Thu, 1 Nov 2001 14:26:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1JQ8l14674 for ; Thu, 1 Nov 2001 14:26:08 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id UAA23286 for ; Thu, 1 Nov 2001 20:26:02 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1JQ1W48556 for ; Thu, 1 Nov 2001 20:26:01 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 1 Nov 2001 21:25:59 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 21:26:01, Serialize complete at 01/11/2001 21:26:01 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA1JQ9l14676 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Sandeep - your note made me look - and I found that I forgot the list - Sorry - Julo ----- Forwarded by Julian Satran/Haifa/IBM on 01-11-01 21:25 ----- Julian Satran 01-11-01 14:02 To: "Robert D. Russell" cc: From: Julian Satran/Haifa/IBM@IBMIL Subject: Re: iSCSI: current UNH Plugfest Bob, Comments in text - thanks - Julo "Robert D. Russell" Sent by: owner-ips@ece.cmu.edu 01-11-01 00:39 Please respond to "Robert D. Russell" To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: current UNH Plugfest Attached are the new issues that arose during the iSCSI plugfest at UNH on Wednesday 31-Oct-2001. (Note: these issues do not take into account any modifications or clarifications that occured in the standard due to the issues raised on Monday or Tuesday.) Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 ---------------------------------------------------------------------------- 1. Are receivers (initiator or target) REQUIRED to check that reserved bits and/or fields are set to 0? Section 3 on page 48 of draft 8 says: "Any bits not defined MUST be set to zero. Any reserved fields and values MUST be 0 unless specified otherwise." and Section 8.3 on page 127 of draft 8 says: "Explicit violations of the PDU layout rules stated in this document are format errors. This when detected, usually indicates a major implementation flaw in one of the parties. When a target or an initiator receives an iSCSI PDU with a format error, it MUST reset all transport connections in the session immediately and escalate the format error to session recovery (section 8.11.4)." According to these rules, a PDU with reserved bits and/or fields that are not set to 0 violates the PDU layout rules. Therefore, if an initiator or target receives such a PDU, it should immediately close all connections in the session and go to session recovery. Clearly a format error has extremely severe consequences! Although all vendors are setting reserved bits and fields to 0 on PDUs they are sending, many are NOT checking PDUs they are receiving to see if these bits and fields are set to 0. Basically, vendors are saying "who cares if reserved bits and/or fields in incoming PDUs are not zero? We do not want to take the time to do this checking, and there is no benefit to doing it. As long as the non-reserved bits and fields are set properly, we can and should proceed. Any time devoted to doing this checking is wasted in 99+% of the cases, and in the (unlikely) case that a non-zero bit or field is found, the consequences are too severe." There should be some statement in the standard to clarify what checking is required and what is optional. +++ As it was said already on this list there are things that a standard better leaves unsaid :-) In accordance with the good tradition of other iSCSI transports I suggest we stay silent on this. +++ 2. A similar situation arises with respect to checking the consistency of fields such as Version-max, Version-min and Version-active in Login Requests and Login Responses. For example, consider the Version-max field. Section 3.12.5 says: "All Login requests within the Login phase MUST carry the same Version-max." All vendor initiators are setting Version-max correctly on all login requests they are sending, but many vendor targets are NOT checking received login requests to ensure that this rule is enforced. In particular, many targets simply use the Version-max and Version-min on the first login request they receive on a new connection, and then they ignore these fields on all subsequent login requests in the same login phase. Strictly speaking, a change in the Version-max field during the login phase constitutes a protocol error according to section 8.8 on page 130 of draft 8: "All violations of iSCSI PDU exchange sequences specified in this draft are also protocol errors. This category of errors can be addressed only by fixing the implementations; iSCSI defines Reject and response codes to enable this". Therefore the target should send back a login response with a status of 0x0200 and then close the connection. However, Section 3.12.5 also says: "The target MUST use the value presented with the first login request." This rule seems to imply that the value CAN change, because if it cannot change, then it doesn't matter which one of the login requests it is taken from, they are all the same anyway. The suggestion is to keep the requirement that the target MUST use the value presented on the first login request, but to allowed the target to ignore the value presented on all subsequent login requests in the same login phase. A similar rewording should be done for the other fields. +++ As above. +++ 3. Can commands be sent out of order on the same connection? The behavior of targets is clearly specified in Section 2.2.2.3 on page 25 of draft 8, which says: "Except for the commands marked for immediate delivery the iSCSI target layer MUST eliver the commands for execution in the order specified by CmdSN." Section 2.2.2.3 on page 26 of draft 8 also says: "- CmdSN - the current command Sequence Number advanced by 1 on each command shipped except for commands marked for immediate delivery." but the meaning of the term "shipped" is vague, and does not necessarily require that the PDUs arrive on the other end of a TCP connection in the same order that the CmdSN values were assigned to these PDUs. Some initiators have been designed to send commands out of CmdSN order on one connection. Consider the situation where there is only one connection and a high-level dispatcher creates a PDU for a SCSI command that involves writing immediate data to the target. This PDU is enqueued to a lower-level layer which has to setup, start, and wait-for a DMA operation to move the immediate data into an onboard buffer before the PDU can be put onto the wire. While this is happening, the dispatcher creates another unrelated PDU for a SCSI read command (for example), and when this PDU is passed to the lower-level layer it can be sent immediately, ahead of the previous write PDU and therefore out of order on this connection. The standard clearly allows this to happen if the two PDUs were sent on different connections, and seems to imply that this can also happen when the two PDUs are sent on the same connection. The suggestion is to put in the standard an explicit statement that this is allowed or not allowed, as appropriate. If this is allowed, such a statement would avoid the erroneous assumption being made by some target implementers that within a single connection, commands will arrive in order. If this is not allowed, such a statement would avoid the erroneous assumption being made by some initiator implementers that within a single connection, commands can be put on the wire out of order. +++ will add an explicit statement saying that this behaviour is forbidden. 2.2.2.1 will contain: On any given connection, the iSCSI initiator MUST send the commands in the order specified by CmdSN. +++ 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) now allow: "A value of 0 indicates no limit." Is this useful? Does it buy anything? +++ I have already removed this. However the numerical negotiation will have 0 (wherever appropriate) as a means to indicate "I don't care" The text suggested for this is: For numerical negotiations, the value 0 MAY be specified by the offering party as a "don't care"/"unlimited" value for parameters that explicitly allow it; in this case, the responder may choose any legal value for the parameter. And in the MAXBurstSize etc: A value of 0 MAY be used as a "don't care" offer in negotiations. +++ The difficulties implementers are having with this are: 1) It is a special case. 2) It causes discontinuous ranges (for example, [0,64..2**24]) 3) It violates the min/max function normally used for the key. 4) There is always a limit anyway. Consider FirstBurstSize, which can have a value that is described as "<0|number-64-2**24>", and for which the minimum of the 2 numbers is selected. I one side offers 0 to mean unlimited, and the other side has a limit, it will reply with that limit, say 128 Kbytes. Therefore, the result is not min(0, 128K) but rather max(0, 128K). The statement in the standard that "the minimum of the 2 numbers is selected" is therefore wrong when one of the numbers is 0. Furthermore, when an initiator or target receives an offer for one of these keys, it cannot simply check that the offered value is legal by testing it against some minimum and maximum. It must first check for 0 and then only if that check shows the value is non-zero can it do the min/max range check for legality (i.e., 64-2**24). Finally, there is always a limit. If nothing else it is the limit imposed by the 24-bit DataSegmentLength field of the PDU requesting the transfer. It is useless to specify a FirstBurstSize (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because the largest possible DataSegmentLength in any PDU that can use this value is 2**24-1. The suggestion is to just eliminate this special case of 0 and require that the range 64-to-(2**24-1) be used instead -- it has exactly the same effect in all cases, it is easier to describe in the standard because it avoids all the extra words, and it is easier to code because it avoids all the special cases. NOTE: the standard should specify the limit in the ranges for MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead of 2**24. The number 2**24 cannot be represented in the 24-bit DataSegmentLength field and therefore can never be used. 5. This is a suggestion for a minor rewording in the standard to avoid misunderstandings. In Appendix E on page 188 of draft 8 it says: "The response to this command is a text response containing a list of targets and their addresses. Each target is returned as a target record. A target record begins with the TargetName text key, followed by a list of TargetAddress text keys, ..." In fact, there are situations where there are no targets and/or no addresses. These situations are clearly defined in the draft after the sentences quoted above, but it would help if those sentences at least hinted at the possibility that the lists could be empty or might not contain addresses. A possible rewording would be: "The response to this command is a text response containing a list of zero or more targets and, optionally, their addresses. Each target is returned as a target record. A target record begins with the TargetName text key, followed by a list of zero or more TargetAddress text keys, ..." +++ OK +++ 6. This is a suggestion for another very minor rewording in the standard. At the end of section 2.2.3 on page 29 of draft 8 it says: "Before full feature phase is established, only Login PDUs are allowed. ..." The suggested rewording is: "Before full feature phase is established, only Login Request and Login Response PDUs are allowed. ..." +++ OK +++ From owner-ips@ece.cmu.edu Thu Nov 1 15:24:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA28409 for ; Thu, 1 Nov 2001 14:32:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1J8rQ13182 for ips-outgoing; Thu, 1 Nov 2001 14:08:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pop.mainstreet.net (smtp.mainstreet.net [207.5.0.46]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Hwel07182 for ; Thu, 1 Nov 2001 12:58:40 -0500 (EST) Received: from vip (saqibj.mainstreet.net [207.5.1.168]) by pop.mainstreet.net (8.11.3/8.11.3) with SMTP id fA1HsrP22880; Thu, 1 Nov 2001 09:54:54 -0800 (PST) Reply-To: From: "Saqib Jang" To: "Ofer Biran" , Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Thu, 1 Nov 2001 10:03:29 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I thought the latest security draft already closed on this issue. >From Section 2.3 of -04 draft. iSCSI security implementations MUST support ESP in transport mode. Saqib -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Thursday, November 01, 2001 4:31 AM To: ips@ece.cmu.edu Subject: iSCSI: IPsec tunnel / transport mode decision I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Thu Nov 1 15:40:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29764 for ; Thu, 1 Nov 2001 15:40:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1Ju9317265 for ips-outgoing; Thu, 1 Nov 2001 14:56:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Ju7l17258 for ; Thu, 1 Nov 2001 14:56:07 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id UAA118736; Thu, 1 Nov 2001 20:56:00 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1Jtxw27002; Thu, 1 Nov 2001 20:55:59 +0100 Importance: Normal Subject: RE: iSCSI: IPsec tunnel / transport mode decision To: Cc: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "Ofer Biran" Date: Thu, 1 Nov 2001 21:57:08 +0100 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 01/11/2001 21:55:59 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk No, this issue never got rough consensus. The statement was put there just to provoke the discussion. And before the thread of the security draft official positioning is awaken - an effort will be made s.t. it will not include any normative text that doesn't match the protocols standards normative text. Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 01/11/2001 19:03:29 Please respond to To: Ofer Biran/Haifa/IBM@IBMIL, cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision I thought the latest security draft already closed on this issue. From Section 2.3 of -04 draft. iSCSI security implementations MUST support ESP in transport mode. Saqib -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Thursday, November 01, 2001 4:31 AM To: ips@ece.cmu.edu Subject: iSCSI: IPsec tunnel / transport mode decision I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Thu Nov 1 15:42:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29885 for ; Thu, 1 Nov 2001 15:42:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1KF8518952 for ips-outgoing; Thu, 1 Nov 2001 15:15:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1KEIl18879 for ; Thu, 1 Nov 2001 15:14:20 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA128846 for ; Thu, 1 Nov 2001 14:11:38 -0600 Received: from d04nms41.raleigh.ibm.com (d04nms41.raleigh.ibm.com [9.67.228.19]) by southrelay02.raleigh.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA1KEC9210008 for ; Thu, 1 Nov 2001 15:14:12 -0500 Subject: Re: is TargetName always mandatory or not? To: "Jim Hafner" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Andre Asselin" Date: Thu, 1 Nov 2001 15:15:13 -0500 X-MIMETrack: Serialize by Router on D04NMS41/04/M/IBM(Release 5.0.8 |June 18, 2001) at 11/01/2001 03:14:11 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, I agree with what you say, except for the part that mapping TSID=target portal group tag will work. Let's assume the following architecture: I have a network entity with one network portal (and thus one portal group). Inside this entity lives two iSCSI targets. Scenerio: An iSCSI initiator opens a connection to the network portal in order to add a connection to an already existing session (the network entity's networking code knows this because TSID in the initial login request is 0). The networking code needs to determine what session the initiator wants to add to, so it compares some items from the initial login request to each of the established sessions until it finds a match. The question is what items should it compare to identify a match? Section 3.12.9 of the spec reads: The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. As you say, this is clearly target centric (because, for example, an initiator could have 2 different sessions open to two different targets that have the same TSID). But from a target point of view, what that text means to me that the network entity's networking code should compare ISID + InitiatorName + TSID to determine a match. That implies that TSID must be unique per TargetName and per portal group ID. If TSID is just the target portal group tag, then the comparison wouldn't work. For example, using the architecture above where there is just one portal group, the target could have two sessions open: session A (InitiatorName=foo, ISID=1, TargetName=bar, TSID=0) and session B (InitiatorName=foo, ISID=1, TargetName=foobar, TSID=0). If it receives a login request with (InitiatorName=foo, ISID=1, TSID=0), which session is it for? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 06:33 PM From: Jim Hafner/Almaden/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, First, your comment "SSID + InitiatorName must be enough to uniquely identify a session" is target-centric. It would be different from the initiator's viewpoint. However, from the target's viewpoint, the target name is implicit and from the initiator viewpoint, the initiator name is implicit, so globally, the triple of the two names + SSID (made up of the ISID and TSID) form the identifier of the session. Locally (between two specific guys), the names are implicit so only the SSID is required. It's all a matter of point of view! As for the difference in identifiers, as I mentioned in the private note, the session is an iSCSI construct and is identified by iSCSI things. The nexus is a SCSI thing and is identified by SCSI constructs (based on how we've mapped the iSCSI things to SCSI things). However, you've brought to the fore again a related question: "what value does the TSID provide to the protocol?" I'm not going to answer that, but I will note that one target implementation that (I think) works is that the TSID = target portal group tag. The other thing to ask is "what value is there in the nexus identifier?" This is never really used in SCSI at all, so it's not a critical issue what composes it. However, it is important (IMO) that the SCSI ports have names and the logical derivative of that statement is that the nexus is identified by the concatenation of the SCSI port names (hence the definition we have). Jim Hafner Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 03:00:53 pm Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu Subject: Re: is TargetName always mandatory or not? John, WHOOPS! I was wrong; you are absolutely right that the spec says "TargetName" and not "TSID". When I was reading through it, I saw "TargetName", but read to myself "TSID". Apologies... In my defense (confusion?), however, 3.12.9 says TSID rather than TargetName is used to uniquely identify a session. Going by that, SSID + InitiatorName must be enough to uniquely identify a session. Jim Hafner pointed out to me that the I_T nexus is identified by one thing and the session is identified by another. If the two must have a 1-1 mapping in iSCSI, why are there two different identifiers? Why not just use the current definition of the I_T nexus to uniquely identify both the nexus and session (i.e. get rid of TSID)? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 04:42 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, I looked again through the document and No where could I find a statement that you claimed as "a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag". It is the InitiatorName, ISID, TSID, with the TargetName and PortalGroup. Please point out to me in the Spec (8 or above), where I can find your statement on I_T Nexus. I did find the following (please ignore for this conversation the "i" and 't" stuff): "- Session: The group of TCP connections that link an initiator with a target, form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one "target image". " " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + 'i'+ ISID, iSCSI Target Name + 't'+ Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). " I have not found a place where Session ID is tied to the TSID. Hence my comment that we also need to have the TargetName in the Initial Login on all Connections. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 10:40:55 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? John & Julian, How about this for the section 5 text: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The initial login request of the leading Login MAY also include the SessionType key=value pair, in which case if the SessionType is not "discovery", then the initial login request MUST also include the key=value pair TargetName. John, I disagree with your second statement: I don't see any way you could have 2 different sessions established within the same portal group with the same InitiatorName, ISID, and TSID. The spec. says that a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag. There is no mention of TargetName contributing to the identificaiton of a session. In my opinion, a non-leading connection should NOT have the TargetName, since it doesn't contribute anything. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: ips@ece.cmu.edu Sent by: Subject: Re: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 10/31/2001 04:20 AM Julian, I think the TargetName should be included on the Initial Login Request on the Leading Login. It seem strange to permit the Authentication functions to proceed if perhaps the intended Target is different then the one doing the Authentication. The way it currently is written, you could pass all the Security test and then find out just before going into Full Function Phase, that it was intended for some other Target. Seems like a waste. My I think that TargetName should also be on all connections on the Initial Login Request. Here is my thinking: The SSID (ISID+TSID) is unique only with regards to a Specific Initiator and Target Node Pair. It is therefore not clear that just knowing the SSID and InitiatorName is enough to understand what session the subsequent connections are being attached to. And it is possible that the CID could be also overlapped with another session. Therefore, I think it make since to determine all this on the initial Login on every Connection, so you know at the beginning what values can be negotiated, or that are being set to the right Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/30/2001 11:33:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: is TargetName always mandatory or not? It is the leading login: The section 5 paragraph will read: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. Julo Andre Asselin/Raleigh/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 31-10-01 02:08 Please respond to Andre Asselin To: "IPS Reflector (E-mail)" cc: Subject: is TargetName always mandatory or not? In section 5 of the spec, it says "If the SessionType is not "discovery" then the initial Login Request MUST also include the key=value pair TargetName.". However, in Appendix D, the description for TargetName says it is Leading Only. Should TargetName not be Leading Only, or should the text in section 5 say that TargetName must be in the initial Login Request of a leading connection? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Thu Nov 1 15:43:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29904 for ; Thu, 1 Nov 2001 15:43:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1Jvsu17410 for ips-outgoing; Thu, 1 Nov 2001 14:57:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Jvol17398 for ; Thu, 1 Nov 2001 14:57:50 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA1Jviu02362; Thu, 1 Nov 2001 11:57:44 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA1Jvc604150; Thu, 1 Nov 2001 11:57:38 -0800 From: "Bill Strahm" To: , "Ofer Biran" , Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Thu, 1 Nov 2001 11:57:21 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Funny because RFC 2401 says (Section 4.1) " In summary, a) A host MUST support both transport and tunnel mode. b) A security gateway is required to support only tunnel mode. If it supports transport mode, that should be used only when the security gateway is acting as a host, e.g., for network management. " I am assuming that at least one end of the iSCSI implementation is a Host (if not both ends) and therefore will have a conformant IPsec implementation... Now the question is where do we want to allow security endpoints to be. If we want to allow only host-host security (and the requisite policy nightmares) then Transport Mode will work. However if we want to allow Tunneling between hosts and security gateways, then Tunnel mode will need to be used. In reality I think we should stick with the 2401 requirements, that way I don't have to write my own implementation... I have not seen a call of consensus on this issue, have you issued it David ? Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Saqib Jang Sent: Thursday, November 01, 2001 10:03 AM To: Ofer Biran; ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision I thought the latest security draft already closed on this issue. >From Section 2.3 of -04 draft. iSCSI security implementations MUST support ESP in transport mode. Saqib -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Thursday, November 01, 2001 4:31 AM To: ips@ece.cmu.edu Subject: iSCSI: IPsec tunnel / transport mode decision I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Thu Nov 1 16:17:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00708 for ; Thu, 1 Nov 2001 16:17:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1KgDs21327 for ips-outgoing; Thu, 1 Nov 2001 15:42:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imo-m10.mx.aol.com (imo-m10.mx.aol.com [64.12.136.165]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1KgBl21322 for ; Thu, 1 Nov 2001 15:42:11 -0500 (EST) Received: from VAHUJA@aol.com by imo-m10.mx.aol.com (mail_out_v31_r1.8.) id 3.fc.e76e9bf (4240) for ; Thu, 1 Nov 2001 15:42:04 -0500 (EST) From: VAHUJA@aol.com Message-ID: Date: Thu, 1 Nov 2001 15:42:03 EST Subject: Fwd: iSCSI: IPsec tunnel / transport mode decision To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="part1_fc.e76e9bf.29130d9b_boundary" X-Mailer: AOL 5.0 for Windows sub 138 Sender: owner-ips@ece.cmu.edu Precedence: bulk --part1_fc.e76e9bf.29130d9b_boundary Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit My concern is with the requirement that a host MUST support both Tunnel and Transport Mode. A large enterprise most likely has its own VPN today. For it to not trust its Intranet is a question they already addressed. Instead of imposing another layer of IPSec for the enterprise, we ought to make it easy for them to use their existing VPNs for securing transport over the Internet. Bottom line, I think we should go with Tunnel Mode, that can be deployed anywhere in the customer environment, and leave it up to the customer and vendor to determine where exactly to deploy the Tunnel Mode. Transport Mode should be Optional and let the market demand vs cost drive its implementations. Trust the customer! --part1_fc.e76e9bf.29130d9b_boundary Content-Type: message/rfc822 Content-Disposition: inline Return-Path: Received: from rly-za03.mx.aol.com (rly-za03.mail.aol.com [172.31.36.99]) by air-za04.mail.aol.com (v82.22) with ESMTP id MAILINZA45-1101152150; Thu, 01 Nov 2001 15:21:50 -0500 Received: from ece.cmu.edu (ece.cmu.edu [128.2.136.200]) by rly-za03.mx.aol.com (v80.21) with ESMTP id MAILRELAYINZA310-1101152123; Thu, 01 Nov 2001 15:21:23 -0500 Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1Jvsu17410 for ips-outgoing; Thu, 1 Nov 2001 14:57:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1Jvol17398 for ; Thu, 1 Nov 2001 14:57:50 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA1Jviu02362; Thu, 1 Nov 2001 11:57:44 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA1Jvc604150; Thu, 1 Nov 2001 11:57:38 -0800 From: "Bill Strahm" To: , "Ofer Biran" , Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Thu, 1 Nov 2001 11:57:21 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Funny because RFC 2401 says (Section 4.1) " In summary, a) A host MUST support both transport and tunnel mode. b) A security gateway is required to support only tunnel mode. If it supports transport mode, that should be used only when the security gateway is acting as a host, e.g., for network management. " I am assuming that at least one end of the iSCSI implementation is a Host (if not both ends) and therefore will have a conformant IPsec implementation... Now the question is where do we want to allow security endpoints to be. If we want to allow only host-host security (and the requisite policy nightmares) then Transport Mode will work. However if we want to allow Tunneling between hosts and security gateways, then Tunnel mode will need to be used. In reality I think we should stick with the 2401 requirements, that way I don't have to write my own implementation... I have not seen a call of consensus on this issue, have you issued it David ? Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Saqib Jang Sent: Thursday, November 01, 2001 10:03 AM To: Ofer Biran; ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision I thought the latest security draft already closed on this issue. >From Section 2.3 of -04 draft. iSCSI security implementations MUST support ESP in transport mode. Saqib -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Thursday, November 01, 2001 4:31 AM To: ips@ece.cmu.edu Subject: iSCSI: IPsec tunnel / transport mode decision I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 --part1_fc.e76e9bf.29130d9b_boundary-- From owner-ips@ece.cmu.edu Thu Nov 1 16:17:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00721 for ; Thu, 1 Nov 2001 16:17:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1KhXu21454 for ips-outgoing; Thu, 1 Nov 2001 15:43:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1KhVl21448 for ; Thu, 1 Nov 2001 15:43:31 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id PAA12313; Thu, 1 Nov 2001 15:43:18 -0500 (EST) Date: Thu, 1 Nov 2001 15:43:18 -0500 From: "Robert D. Russell" To: Julian Satran cc: ips@ece.cmu.edu Subject: Re: is TargetName always mandatory or not? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: Please see comments below: Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 > From Julian_Satran@il.ibm.com Wed Oct 31 08:15:10 2001 > Date: Wed, 31 Oct 2001 09:33:50 +0200 > From: Julian Satran > To: ips@ece.cmu.edu > Subject: Re: is TargetName always mandatory or not? > > It is the leading login: > > The section 5 paragraph will read: > > The initial Login request of the first connection of a session (leading > login) MUST include the InitiatorName key=value pair. The leading Login > request MAY also include the SessionType key=value pair in which case if > the SessionType is not "discovery" then the leading Login Request MUST > also include the key=value pair TargetName. > > Julo I believe the phrase "in which case" should be taken out, because in the case when SessionType key is omited, the default session type is normal and the initial Login request MUST include the TargetName key. Furthermore, if the session is to be a discovery session, then the SessionType=discover pair MUST also be present in the initial login request. The new wording would then be: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. If the session is to be a discovery session, then the leading Login request MUST also include the SessionType=discovery pair. If the session is to be a normal session, then the leading Login request MUST also include the key=value pair TargetName. I also have some related changes that need to be made in order to maintain consistency with the above wording in section 5. 1. The following sentence must be removed from the end of section 5.1 (because the LO designation means that the InititatorName, TargetName, and SessionType keys cannot be sent at all during the login phase of new connections within existing sessions). remove If sent on new connections within an existing session the iSCSI Target Name and the iSCSI Initiator Name MUST be the same as the one used for the leading connection. 2. The wording in Appendix D for the InitiatorName and TargetName entries should be changed as follows: for InitiatorName: change "This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase." to "This key MUST be provided by the initiator on the initial Login request of the first connection of any type of session." for TargetName: change "This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase." to "This key MUST be provided by the initiator on the initial Login request of the first connection of a normal session." 3. The following phrase should be added to the wording in Appendix D for the SessionType entry: "This key can only be used on the initial Login request of the first connection of a session." 4. Finally, the wording in section 2.2.7 should be changed: change The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. to The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be omitted. The key "SessionType=discovery" MUST be sent by the initiator on the initial Login request of the first connection to indicate a discovery session. From owner-ips@ece.cmu.edu Thu Nov 1 16:17:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00746 for ; Thu, 1 Nov 2001 16:17:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1KWoM20506 for ips-outgoing; Thu, 1 Nov 2001 15:32:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1KWYl20473 for ; Thu, 1 Nov 2001 15:32:34 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA77522 for ; Thu, 1 Nov 2001 14:29:54 -0600 Received: from d04nms41.raleigh.ibm.com (d04nms41.raleigh.ibm.com [9.67.228.19]) by southrelay02.raleigh.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA1KWR9128212 for ; Thu, 1 Nov 2001 15:32:27 -0500 Subject: Re: is TargetName always mandatory or not? To: "John Hufferd" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Andre Asselin" Date: Thu, 1 Nov 2001 15:33:28 -0500 X-MIMETrack: Serialize by Router on D04NMS41/04/M/IBM(Release 5.0.8 |June 18, 2001) at 11/01/2001 03:32:26 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk John, Section 3.12.9 of the spec reads: The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. So according to that, TSID + ISID + InitiatorName must be enough information for a target to uniquely idenfity a session, so mapping TSID to target portal group ID won't always work (see my other note addressed to Jim for an example of where it won't). After thinking about this for a while, I've come to the same conclusion as you (that a target should compare ISID + InitiatorName + TargetName + target portal group to uniquely identify a session), but from a different angle. My reasoning: Since there is a 1-1 correspondence between an I_T nexus and a session, if you can identify the nexus, you've identified the session, and vice versa. Since 3.12.9 says that TSID + ISID + InitiatorName must uniquely identify a session, then TSID must be an alias for TargetName + target portal group ID, since (ISID + InitiatorName + TargetName + portal group ID) uniquely identifies a nexus. In my opinion, that aliasing only complicates (and confuses!) the protocol and should be done away with. If TargetName were required on every initial login request, then the target could compare (ISID + InitiatorName + TargetName + portal group) to determine a session match, and forget about TSID entirely. In that case, there is no need for TSID, except to flag whether the login request is to create a new session or to add a new connection to an existing session (i.e. indicate leading login), in which case it could be dropped and replaced with a single bit. To eliminate TSID, I believe the following changes would be needed: - Eliminate TSID from login request and login response - Use one of the bits in byte 1 of login request to indicate a leading login, and update any text that refers to TSID=0 to refer to this bit - Define session to be equivalent to I_T nexus (not "loosely equivalent to") - Define SSID/session ID to be the same as the I_T nexus identifier - Remove the "LO" from TargetName - If it hasn't already been done, remove the "LO" from InitiatorName Comments? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM 10/31/2001 cc: ips@ece.cmu.edu 06:32 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, TSID, and therefore SSID is a handle that can be use to queue stuff within the scope of the Initiator Node and Target Node. The target can use what ever it wants in this field. Some Folks might want to use the TSID to be the Portal Group Tag and I think then it might be unique. But even then only within the InitiatorName+ISID Space. Anyway, you need to think of it only as a handle. Also since most folks will not have 64k Sessions into their box, some (most?) implementations might find it unique. But it is not unique in the Architecture. Now the important thing is that, as things stand now, the Login needs the InitiatorName, and the TargetName on the Initial Login of ALL Connections in order to uniquely identify secondary Connections to the Original Connection of the Session. Does that seem correct to you and others on the Reflector? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin 10/31/2001 03:00 PM To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: John Hufferd) John, WHOOPS! I was wrong; you are absolutely right that the spec says "TargetName" and not "TSID". When I was reading through it, I saw "TargetName", but read to myself "TSID". Apologies... In my defense (confusion?), however, 3.12.9 says TSID rather than TargetName is used to uniquely identify a session. Going by that, SSID + InitiatorName must be enough to uniquely identify a session. Jim Hafner pointed out to me that the I_T nexus is identified by one thing and the session is identified by another. If the two must have a 1-1 mapping in iSCSI, why are there two different identifiers? Why not just use the current definition of the I_T nexus to uniquely identify both the nexus and session (i.e. get rid of TSID)? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 04:42 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, I looked again through the document and No where could I find a statement that you claimed as "a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag". It is the InitiatorName, ISID, TSID, with the TargetName and PortalGroup. Please point out to me in the Spec (8 or above), where I can find your statement on I_T Nexus. I did find the following (please ignore for this conversation the "i" and 't" stuff): "- Session: The group of TCP connections that link an initiator with a target, form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one "target image". " " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + 'i'+ ISID, iSCSI Target Name + 't'+ Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). " I have not found a place where Session ID is tied to the TSID. Hence my comment that we also need to have the TargetName in the Initial Login on all Connections. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 10:40:55 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? John & Julian, How about this for the section 5 text: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The initial login request of the leading Login MAY also include the SessionType key=value pair, in which case if the SessionType is not "discovery", then the initial login request MUST also include the key=value pair TargetName. John, I disagree with your second statement: I don't see any way you could have 2 different sessions established within the same portal group with the same InitiatorName, ISID, and TSID. The spec. says that a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag. There is no mention of TargetName contributing to the identificaiton of a session. In my opinion, a non-leading connection should NOT have the TargetName, since it doesn't contribute anything. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: ips@ece.cmu.edu Sent by: Subject: Re: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 10/31/2001 04:20 AM Julian, I think the TargetName should be included on the Initial Login Request on the Leading Login. It seem strange to permit the Authentication functions to proceed if perhaps the intended Target is different then the one doing the Authentication. The way it currently is written, you could pass all the Security test and then find out just before going into Full Function Phase, that it was intended for some other Target. Seems like a waste. My I think that TargetName should also be on all connections on the Initial Login Request. Here is my thinking: The SSID (ISID+TSID) is unique only with regards to a Specific Initiator and Target Node Pair. It is therefore not clear that just knowing the SSID and InitiatorName is enough to understand what session the subsequent connections are being attached to. And it is possible that the CID could be also overlapped with another session. Therefore, I think it make since to determine all this on the initial Login on every Connection, so you know at the beginning what values can be negotiated, or that are being set to the right Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/30/2001 11:33:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: is TargetName always mandatory or not? It is the leading login: The section 5 paragraph will read: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. Julo Andre Asselin/Raleigh/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 31-10-01 02:08 Please respond to Andre Asselin To: "IPS Reflector (E-mail)" cc: Subject: is TargetName always mandatory or not? In section 5 of the spec, it says "If the SessionType is not "discovery" then the initial Login Request MUST also include the key=value pair TargetName.". However, in Appendix D, the description for TargetName says it is Leading Only. Should TargetName not be Leading Only, or should the text in section 5 say that TargetName must be in the initial Login Request of a leading connection? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Thu Nov 1 16:26:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA00952 for ; Thu, 1 Nov 2001 16:26:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1KeVX21188 for ips-outgoing; Thu, 1 Nov 2001 15:40:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1KeOl21168 for ; Thu, 1 Nov 2001 15:40:24 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA26448 for ; Thu, 1 Nov 2001 15:37:29 -0500 Received: from d03nm041.boulder.ibm.com (d03nm041.boulder.ibm.com [9.99.140.41]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1KdqC95554 for ; Thu, 1 Nov 2001 13:39:52 -0700 Importance: Normal Subject: Re: is TargetName always mandatory or not? To: "Andre Asselin" Cc: ips@ece.cmu.edu From: "Jim Hafner" Date: Thu, 1 Nov 2001 12:39:51 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM041/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/01/2001 12:39:52 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Andre, Your picture isn't quite right. The portal group tag is relative to the iSCSI target and that target is *uniquely* identified by it's name. So, in your picture, there are two targets (not one). Each can label its target portal group any way it wants (independent of the other). Each has independent control over the TSIDs it uses. Each may *share* use of a network portal, but that's a different issue. The model implies two independent targets (even if they live in the same network entity and share resources) in your scenario. In other words, the portal groups are wrt targets not the network entity. In your scenario, whatever layer (you put it in the network code) has to decide what session to add the connection to has the initiator name, the ISID, the TSID *and* the target name (you left this out) in the login request. That fully qualifies both the target and the session as well. Jim Hafner Andre Asselin 11/01/2001 12:15 pm To: Jim Hafner/Almaden/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: Database 'Jim Hafner', View '($Inbox)') Jim, I agree with what you say, except for the part that mapping TSID=target portal group tag will work. Let's assume the following architecture: I have a network entity with one network portal (and thus one portal group). Inside this entity lives two iSCSI targets. Scenerio: An iSCSI initiator opens a connection to the network portal in order to add a connection to an already existing session (the network entity's networking code knows this because TSID in the initial login request is 0). The networking code needs to determine what session the initiator wants to add to, so it compares some items from the initial login request to each of the established sessions until it finds a match. The question is what items should it compare to identify a match? Section 3.12.9 of the spec reads: The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. As you say, this is clearly target centric (because, for example, an initiator could have 2 different sessions open to two different targets that have the same TSID). But from a target point of view, what that text means to me that the network entity's networking code should compare ISID + InitiatorName + TSID to determine a match. That implies that TSID must be unique per TargetName and per portal group ID. If TSID is just the target portal group tag, then the comparison wouldn't work. For example, using the architecture above where there is just one portal group, the target could have two sessions open: session A (InitiatorName=foo, ISID=1, TargetName=bar, TSID=0) and session B (InitiatorName=foo, ISID=1, TargetName=foobar, TSID=0). If it receives a login request with (InitiatorName=foo, ISID=1, TSID=0), which session is it for? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: 10/31/2001 cc: 06:33 PM From: Subject: (Document link: Andre Asselin) Andre, First, your comment "SSID + InitiatorName must be enough to uniquely identify a session" is target-centric. It would be different from the initiator's viewpoint. However, from the target's viewpoint, the target name is implicit and from the initiator viewpoint, the initiator name is implicit, so globally, the triple of the two names + SSID (made up of the ISID and TSID) form the identifier of the session. Locally (between two specific guys), the names are implicit so only the SSID is required. It's all a matter of point of view! As for the difference in identifiers, as I mentioned in the private note, the session is an iSCSI construct and is identified by iSCSI things. The nexus is a SCSI thing and is identified by SCSI constructs (based on how we've mapped the iSCSI things to SCSI things). However, you've brought to the fore again a related question: "what value does the TSID provide to the protocol?" I'm not going to answer that, but I will note that one target implementation that (I think) works is that the TSID = target portal group tag. The other thing to ask is "what value is there in the nexus identifier?" This is never really used in SCSI at all, so it's not a critical issue what composes it. However, it is important (IMO) that the SCSI ports have names and the logical derivative of that statement is that the nexus is identified by the concatenation of the SCSI port names (hence the definition we have). Jim Hafner Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 03:00:53 pm Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu Subject: Re: is TargetName always mandatory or not? John, WHOOPS! I was wrong; you are absolutely right that the spec says "TargetName" and not "TSID". When I was reading through it, I saw "TargetName", but read to myself "TSID". Apologies... In my defense (confusion?), however, 3.12.9 says TSID rather than TargetName is used to uniquely identify a session. Going by that, SSID + InitiatorName must be enough to uniquely identify a session. Jim Hafner pointed out to me that the I_T nexus is identified by one thing and the session is identified by another. If the two must have a 1-1 mapping in iSCSI, why are there two different identifiers? Why not just use the current definition of the I_T nexus to uniquely identify both the nexus and session (i.e. get rid of TSID)? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 04:42 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, I looked again through the document and No where could I find a statement that you claimed as "a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag". It is the InitiatorName, ISID, TSID, with the TargetName and PortalGroup. Please point out to me in the Spec (8 or above), where I can find your statement on I_T Nexus. I did find the following (please ignore for this conversation the "i" and 't" stuff): "- Session: The group of TCP connections that link an initiator with a target, form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one "target image". " " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + 'i'+ ISID, iSCSI Target Name + 't'+ Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). " I have not found a place where Session ID is tied to the TSID. Hence my comment that we also need to have the TargetName in the Initial Login on all Connections. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 10:40:55 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? John & Julian, How about this for the section 5 text: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The initial login request of the leading Login MAY also include the SessionType key=value pair, in which case if the SessionType is not "discovery", then the initial login request MUST also include the key=value pair TargetName. John, I disagree with your second statement: I don't see any way you could have 2 different sessions established within the same portal group with the same InitiatorName, ISID, and TSID. The spec. says that a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag. There is no mention of TargetName contributing to the identificaiton of a session. In my opinion, a non-leading connection should NOT have the TargetName, since it doesn't contribute anything. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: ips@ece.cmu.edu Sent by: Subject: Re: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 10/31/2001 04:20 AM Julian, I think the TargetName should be included on the Initial Login Request on the Leading Login. It seem strange to permit the Authentication functions to proceed if perhaps the intended Target is different then the one doing the Authentication. The way it currently is written, you could pass all the Security test and then find out just before going into Full Function Phase, that it was intended for some other Target. Seems like a waste. My I think that TargetName should also be on all connections on the Initial Login Request. Here is my thinking: The SSID (ISID+TSID) is unique only with regards to a Specific Initiator and Target Node Pair. It is therefore not clear that just knowing the SSID and InitiatorName is enough to understand what session the subsequent connections are being attached to. And it is possible that the CID could be also overlapped with another session. Therefore, I think it make since to determine all this on the initial Login on every Connection, so you know at the beginning what values can be negotiated, or that are being set to the right Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/30/2001 11:33:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: is TargetName always mandatory or not? It is the leading login: The section 5 paragraph will read: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. Julo Andre Asselin/Raleigh/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 31-10-01 02:08 Please respond to Andre Asselin To: "IPS Reflector (E-mail)" cc: Subject: is TargetName always mandatory or not? In section 5 of the spec, it says "If the SessionType is not "discovery" then the initial Login Request MUST also include the key=value pair TargetName.". However, in Appendix D, the description for TargetName says it is Leading Only. Should TargetName not be Leading Only, or should the text in section 5 say that TargetName must be in the initial Login Request of a leading connection? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Thu Nov 1 17:38:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA02222 for ; Thu, 1 Nov 2001 17:38:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1LdLw25757 for ips-outgoing; Thu, 1 Nov 2001 16:39:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1LdIl25750 for ; Thu, 1 Nov 2001 16:39:18 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA29668 for ; Thu, 1 Nov 2001 16:36:43 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA1LdBC155350 for ; Thu, 1 Nov 2001 14:39:11 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: is TargetName always mandatory or not? To: "Andre Asselin" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 1 Nov 2001 13:38:16 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/01/2001 02:39:11 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Andre, I think Jim has just answered your question on the use of portal groups. In any event the wordage you point out in 3.12.9 is a bit loose. When it says: "The TSID is the target assigned tag ...." it is only implied that the TSID is chosen in the context of the Target (iSCSI Target Node) and not a iSCSI Network Entity. Therefore, the implication that the code would only need the TSID + ISID + InitiatorName to bind a connection to a Session, is only valid in the context of a Specific iSCSI Target Node. The Fact that more then one iSCSI Target Node could be located in a iSCSI Network Entity, means that the implementation needs more complete information to direct the connection binding with the approprate Session (and hence the approprate Target Node). Depending on the implementation, it is probably possible to use the 64K different possibilities to make the TSID unique across the entire iSCSI Network Entity, but that is not a hard requirement of the protocol. I do not mind having the TSID being left in, it can be most useful in many (most) installations is sufficient as a unique qualifier with the ISID + InitiatorName, but that is SIZE and implementation determined. I think we need to update the wordage in 3.12.9 and other places to make sure that everyone understands that TSID is only specified as being unique in the setting of a specific iSCSI Node. And we need to update the Draft to ensure that Both the InitiatorName and TargetName are always used on all non Discovery Connection Logins, and always on the Initial Login request of each connection. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin 11/01/2001 12:33 PM To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: John Hufferd) John, Section 3.12.9 of the spec reads: The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. So according to that, TSID + ISID + InitiatorName must be enough information for a target to uniquely idenfity a session, so mapping TSID to target portal group ID won't always work (see my other note addressed to Jim for an example of where it won't). After thinking about this for a while, I've come to the same conclusion as you (that a target should compare ISID + InitiatorName + TargetName + target portal group to uniquely identify a session), but from a different angle. My reasoning: Since there is a 1-1 correspondence between an I_T nexus and a session, if you can identify the nexus, you've identified the session, and vice versa. Since 3.12.9 says that TSID + ISID + InitiatorName must uniquely identify a session, then TSID must be an alias for TargetName + target portal group ID, since (ISID + InitiatorName + TargetName + portal group ID) uniquely identifies a nexus. In my opinion, that aliasing only complicates (and confuses!) the protocol and should be done away with. If TargetName were required on every initial login request, then the target could compare (ISID + InitiatorName + TargetName + portal group) to determine a session match, and forget about TSID entirely. In that case, there is no need for TSID, except to flag whether the login request is to create a new session or to add a new connection to an existing session (i.e. indicate leading login), in which case it could be dropped and replaced with a single bit. To eliminate TSID, I believe the following changes would be needed: - Eliminate TSID from login request and login response - Use one of the bits in byte 1 of login request to indicate a leading login, and update any text that refers to TSID=0 to refer to this bit - Define session to be equivalent to I_T nexus (not "loosely equivalent to") - Define SSID/session ID to be the same as the I_T nexus identifier - Remove the "LO" from TargetName - If it hasn't already been done, remove the "LO" from InitiatorName Comments? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM 10/31/2001 cc: ips@ece.cmu.edu 06:32 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: Andre Asselin) Andre, TSID, and therefore SSID is a handle that can be use to queue stuff within the scope of the Initiator Node and Target Node. The target can use what ever it wants in this field. Some Folks might want to use the TSID to be the Portal Group Tag and I think then it might be unique. But even then only within the InitiatorName+ISID Space. Anyway, you need to think of it only as a handle. Also since most folks will not have 64k Sessions into their box, some (most?) implementations might find it unique. But it is not unique in the Architecture. Now the important thing is that, as things stand now, the Login needs the InitiatorName, and the TargetName on the Initial Login of ALL Connections in order to uniquely identify secondary Connections to the Original Connection of the Session. Does that seem correct to you and others on the Reflector? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin 10/31/2001 03:00 PM To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: John Hufferd) John, WHOOPS! I was wrong; you are absolutely right that the spec says "TargetName" and not "TSID". When I was reading through it, I saw "TargetName", but read to myself "TSID". Apologies... In my defense (confusion?), however, 3.12.9 says TSID rather than TargetName is used to uniquely identify a session. Going by that, SSID + InitiatorName must be enough to uniquely identify a session. Jim Hafner pointed out to me that the I_T nexus is identified by one thing and the session is identified by another. If the two must have a 1-1 mapping in iSCSI, why are there two different identifiers? Why not just use the current definition of the I_T nexus to uniquely identify both the nexus and session (i.e. get rid of TSID)? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 04:42 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: Andre Asselin) Andre, I looked again through the document and No where could I find a statement that you claimed as "a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag". It is the InitiatorName, ISID, TSID, with the TargetName and PortalGroup. Please point out to me in the Spec (8 or above), where I can find your statement on I_T Nexus. I did find the following (please ignore for this conversation the "i" and 't" stuff): "- Session: The group of TCP connections that link an initiator with a target, form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one "target image". " " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + 'i'+ ISID, iSCSI Target Name + 't'+ Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). " I have not found a place where Session ID is tied to the TSID. Hence my comment that we also need to have the TargetName in the Initial Login on all Connections. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 10:40:55 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? John & Julian, How about this for the section 5 text: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The initial login request of the leading Login MAY also include the SessionType key=value pair, in which case if the SessionType is not "discovery", then the initial login request MUST also include the key=value pair TargetName. John, I disagree with your second statement: I don't see any way you could have 2 different sessions established within the same portal group with the same InitiatorName, ISID, and TSID. The spec. says that a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag. There is no mention of TargetName contributing to the identificaiton of a session. In my opinion, a non-leading connection should NOT have the TargetName, since it doesn't contribute anything. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: ips@ece.cmu.edu Sent by: Subject: Re: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 10/31/2001 04:20 AM Julian, I think the TargetName should be included on the Initial Login Request on the Leading Login. It seem strange to permit the Authentication functions to proceed if perhaps the intended Target is different then the one doing the Authentication. The way it currently is written, you could pass all the Security test and then find out just before going into Full Function Phase, that it was intended for some other Target. Seems like a waste. My I think that TargetName should also be on all connections on the Initial Login Request. Here is my thinking: The SSID (ISID+TSID) is unique only with regards to a Specific Initiator and Target Node Pair. It is therefore not clear that just knowing the SSID and InitiatorName is enough to understand what session the subsequent connections are being attached to. And it is possible that the CID could be also overlapped with another session. Therefore, I think it make since to determine all this on the initial Login on every Connection, so you know at the beginning what values can be negotiated, or that are being set to the right Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/30/2001 11:33:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: is TargetName always mandatory or not? It is the leading login: The section 5 paragraph will read: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. Julo Andre Asselin/Raleigh/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 31-10-01 02:08 Please respond to Andre Asselin To: "IPS Reflector (E-mail)" cc: Subject: is TargetName always mandatory or not? In section 5 of the spec, it says "If the SessionType is not "discovery" then the initial Login Request MUST also include the key=value pair TargetName.". However, in Appendix D, the description for TargetName says it is Leading Only. Should TargetName not be Leading Only, or should the text in section 5 say that TargetName must be in the initial Login Request of a leading connection? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Thu Nov 1 18:19:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA02725 for ; Thu, 1 Nov 2001 18:19:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1MX0329928 for ips-outgoing; Thu, 1 Nov 2001 17:33:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from wellington.xo.com (wellington.xo.com [207.155.252.73]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1MWwl29923 for ; Thu, 1 Nov 2001 17:32:58 -0500 (EST) Received: from blekinge ([66.89.96.162]) by wellington.xo.com id RAA12331; Thu, 1 Nov 2001 17:32:21 -0500 (EST) [ConcentricHost SMTP Relay 1.14] Message-ID: <015f01c16324$f8725720$1001010a@blekinge> From: "Peter Mellquist" To: "Michele Hallak - Stamler" Cc: "IPS" References: <838D8D2617300146B7F47E4D9AE7FF10114A0E@SANSRV1.SAN-RAD.CO.IL> Subject: Re: iSCSI: New iSCSI MIB draft Date: Thu, 1 Nov 2001 14:31:35 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I agree. I would assume that I can use SNMPv3 to manage the ACL in a secure manner. This would require read/write access. Basically, it should be possible to manage and control an iSCSI target or initiator via SNMP. -peter Peter Mellquist Seven Systems Technology peterm@seven-systems.com ----- Original Message ----- From: "Michele Hallak - Stamler" To: "Mark Bakke" ; "IPS" Sent: Wednesday, October 31, 2001 11:52 PM Subject: RE: iSCSI: New iSCSI MIB draft > Hi, > 1.I see that the target access list is still read-only. How do you think > that it should be configured? > 2. Who should configure the aliases of the iscsi targets and initiators? > 3. Same question for the portals. > Thanks, > Michele > > -----Original Message----- > From: Mark Bakke [mailto:mbakke@cisco.com] > Sent: Wednesday, October 31, 2001 9:01 PM > To: IPS > Subject: Re: iSCSI: New iSCSI MIB draft > > > For those who are more pictorially inclined when looking at > MIBs, I have an updated MIB tree drawing available at: > > ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-mib-structure-03.pd > f > > -- > Mark > > Mark Bakke wrote: > > > > We have submitted a new iSCSI MIB draft to the repository. > > Until it becomes available, it may be retrieved as either > > a gzipped or text version from: > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/draft-ietf-ips-iscsi-mib-03.txt.gz > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/draft-ietf-ips-iscsi-mib-03.txt > > > > A matching UML drawing is available at: > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-uml-model-03.pdf > > > > The table hierarchy has been significantly flattened. This does > > not change the object model, but does reduce the number of redundant > > levels of OIDs when address individual objects. I will send a > > new table structure drawing soon. > > > > We will send a list of open issues to the IPS list shortly. > > > > Thanks, > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 > From owner-ips@ece.cmu.edu Thu Nov 1 19:22:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA03647 for ; Thu, 1 Nov 2001 19:22:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA1NVpc04193 for ips-outgoing; Thu, 1 Nov 2001 18:31:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA1NVnl04186 for ; Thu, 1 Nov 2001 18:31:49 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id SAA15624 for ; Thu, 1 Nov 2001 18:31:48 -0500 (EST) Date: Thu, 1 Nov 2001 18:31:48 -0500 From: "Robert D. Russell" To: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Attached are the new issues that arose during the iSCSI plugfest at UNH on Thursday 1-Nov-2001. Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 ---------------------------------------------------------------------- There were no new issues today! Lots of people were busy sending data and interoperating with each other! There were, however, a few suggestions/requests for clarifications in the standard: 1. Since all Login Requests MUST be issued as immediate requests, the diagram in section 3.12 should show bit 6 of byte 0 as "1", not "I", and section 3.12.2 should simply be eliminated. 2. The first paragraph of Appendix D should describe the special role of the first Login request on the first connection of a new session. In particular, there are 3 keys that MUST be sent in that Login PDU (InitiatorName, SessionType if it is discovery, TargetName if the session is normal). This is, in fact, yet another classification that is currently included within LO, but LO is too general, since LO allows keys to appear at any time during the leading login phase, and these keys must appear on the very first login PDU in that leading login phase. 3. A target cannot release its resources for a command until it receives an ExpStatSN back from the initiator that acknowledges receipt of the StatSN carried in the SCSI Response to that command. However, if the initiator does not have any more I/O to perform, that ExpStatSN will not come back to the target, and the target may end up holding on to those resources indefinitely. This is a situation where initiator non-action can impact target performance, and could even lead to denial of service attacks if several initiators were to do this simultaneously to the same target. The standard provides the NOP-In/Out mechanism to deal with this: The last paragraph of section 2.2.2 says: During periods when traffic on a connection is unidirectional, iSCSI NOP-Out/In PDUs may be utilized to synchronize the command and status ordering counters of the target and initiator. and section 3.18 says: A NOP-Out may also be used to confirm a changed ExpStatSN if there is no other PDU to carry it for a long time. The question is, is there a recommended mechanism to trigger these NOP-pings? Clearly the target could set a timer, and if ExpStatSN is not received back at the end of the time interval, the target could send a NOP-in as a ping in order to get the initiator to send back a NOP-out with the updated ExpStatSN. Also clearly the initiator could set a timer, and if it has no traffic to send during the time interval, the initiator could send a NOP-out to deliver the updated ExpStatSN to the target. This approach is probably not as reliable as the first, since a) if the initiator does not do it, only the target gets hurt, and b) if the connection drops, the target will never receive the NOP-out. On the other hand, it may be more efficient for the initiator to do this, since a typical target may be dealing with hundreds of connections, and the initiator only a few. Which is the best procedure? Are there other ways to do this? What is the recommended way, or is there a reason for the standard not to recommend a triggering mechanism, in which case, could the standard suggest methods or provide guidance, especially if there are better, less obvious ways to do it? From owner-ips@ece.cmu.edu Thu Nov 1 21:24:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA06668 for ; Thu, 1 Nov 2001 21:24:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA21JS112038 for ips-outgoing; Thu, 1 Nov 2001 20:19:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA21JRl12033 for ; Thu, 1 Nov 2001 20:19:27 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 214AF2F53; Thu, 1 Nov 2001 18:19:26 -0700 (MST) Received: from axcsbh2.cos.agilent.com (axcsbh2.cos.agilent.com [130.29.152.144]) by msgrel1.cos.agilent.com (Postfix) with SMTP id ED00225B; Thu, 1 Nov 2001 18:19:25 -0700 (MST) Received: from 130.29.152.144 by axcsbh2.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 01 Nov 2001 18:19:25 -0700 Received: by axcsbh2.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Thu, 1 Nov 2001 18:19:25 -0700 Message-ID: <01A7DAF31F93D511AEE300D0B706ED920CF68D@axcs13.cos.agilent.com> From: "CAVANNA,VICENTE V (A-Roseville,ex1)" To: "'Ofer Biran'" , ips@ece.cmu.edu Cc: "CAVANNA,VICENTE V (A-Roseville,ex1)" , "SHEEHY,DAVE (A-Americas,unix1)" Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Thu, 1 Nov 2001 18:19:24 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk It seems to me (who has not had the experience of implementing IpSec) that tunnel mode, even when implemented in the end host (rather than in a router), is a superset of transport mode whose only significant disadvantage is that tunnel mode requires more overhead in the form of the extra IP header. On the other hand, tunnel mode offers more flexibility in implementation as it is easier to implement in BITS and BITW implementations whereas transport mode can only be easily implemented when IPSec is implemented as part of the network layer i.e. integrated into the OS. The reason is that the IPSec headers are inserted AFTER the IP payload is constructed. This means that IPSec has to duplicate IP functionality because it has to recalculate the IP checksum and fragment the packet when necessary. I would support making tunnel mode the favored mode in iSCSI. IPSec compliance requires that transport mode be implemented but if iSCSI discourages it the implementation need not be as efficient as tunnel mode. Vince |-----Original Message----- |From: Ofer Biran [mailto:BIRAN@il.ibm.com] |Sent: Thursday, November 01, 2001 4:31 AM |To: ips@ece.cmu.edu |Subject: iSCSI: IPsec tunnel / transport mode decision | | |I'd like to drive this open issue into group consensus. It seems to |me that the tendency was more toward making tunnel mode a MUST as iFCP |and FCIP did, mainly due the option of integrating an existing IPsec |chip/box with the iSCSI implementation offering. If we reach |this decision, |we may choose even not to mention transport mode (as MAY or some other |recommending text). | |There is an excellent analysis made by Bernard Aboba in Section |"5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 |( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) |that can help us with this decision (also Section "5.2. NAT |traversal" is |relevant). | | Regards, | Ofer | |Ofer Biran |Storage and Systems Technology |IBM Research Lab in Haifa |biran@il.ibm.com 972-4-8296253 | From owner-ips@ece.cmu.edu Fri Nov 2 02:06:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA22569 for ; Fri, 2 Nov 2001 02:06:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA26Hse01170 for ips-outgoing; Fri, 2 Nov 2001 01:17:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA26Hql01166 for ; Fri, 2 Nov 2001 01:17:52 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA26Hju05796; Thu, 1 Nov 2001 22:17:45 -0800 Received: from strahmw2k (ras-pc-7-4.sanera.net [172.16.7.4]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA26Hc618038; Thu, 1 Nov 2001 22:17:39 -0800 From: "Bill Strahm" To: "CAVANNA,VICENTE V \(A-Roseville,ex1\)" , "'Ofer Biran'" , Cc: "SHEEHY,DAVE \(A-Americas,unix1\)" Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Thu, 1 Nov 2001 22:17:32 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <01A7DAF31F93D511AEE300D0B706ED920CF68D@axcs13.cos.agilent.com> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ok, First off with complexity comes a configuration nightmare... Second I can implement both a BITS and a BITW IPsec Transport and Tunnel mode... it really isn't all that hard. BITS implies that you are making OS changes, or at least changes under the TCP/UDP layer that is traditionally exposed to user applications, so I think most of the argument you are trying to make in your second paragraph doesn't hold much water for me. I would rather use Tunnel mode myself, however my reasons are that I can completely offload the implementation off of the host and target and let intermediate devices deal with security policies and such things... However most of the tone of the security draft is to require for at least one end if not both to be intimately aware of what is going on with security... Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of CAVANNA,VICENTE V (A-Roseville,ex1) Sent: Thursday, November 01, 2001 5:19 PM To: 'Ofer Biran'; ips@ece.cmu.edu Cc: CAVANNA,VICENTE V (A-Roseville,ex1); SHEEHY,DAVE (A-Americas,unix1) Subject: RE: iSCSI: IPsec tunnel / transport mode decision It seems to me (who has not had the experience of implementing IpSec) that tunnel mode, even when implemented in the end host (rather than in a router), is a superset of transport mode whose only significant disadvantage is that tunnel mode requires more overhead in the form of the extra IP header. On the other hand, tunnel mode offers more flexibility in implementation as it is easier to implement in BITS and BITW implementations whereas transport mode can only be easily implemented when IPSec is implemented as part of the network layer i.e. integrated into the OS. The reason is that the IPSec headers are inserted AFTER the IP payload is constructed. This means that IPSec has to duplicate IP functionality because it has to recalculate the IP checksum and fragment the packet when necessary. I would support making tunnel mode the favored mode in iSCSI. IPSec compliance requires that transport mode be implemented but if iSCSI discourages it the implementation need not be as efficient as tunnel mode. Vince |-----Original Message----- |From: Ofer Biran [mailto:BIRAN@il.ibm.com] |Sent: Thursday, November 01, 2001 4:31 AM |To: ips@ece.cmu.edu |Subject: iSCSI: IPsec tunnel / transport mode decision | | |I'd like to drive this open issue into group consensus. It seems to |me that the tendency was more toward making tunnel mode a MUST as iFCP |and FCIP did, mainly due the option of integrating an existing IPsec |chip/box with the iSCSI implementation offering. If we reach |this decision, |we may choose even not to mention transport mode (as MAY or some other |recommending text). | |There is an excellent analysis made by Bernard Aboba in Section |"5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 |( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) |that can help us with this decision (also Section "5.2. NAT |traversal" is |relevant). | | Regards, | Ofer | |Ofer Biran |Storage and Systems Technology |IBM Research Lab in Haifa |biran@il.ibm.com 972-4-8296253 | From owner-ips@ece.cmu.edu Fri Nov 2 04:44:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA28525 for ; Fri, 2 Nov 2001 04:44:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA28jsC09041 for ips-outgoing; Fri, 2 Nov 2001 03:45:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA28jrl09036 for ; Fri, 2 Nov 2001 03:45:53 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA37786 for ; Fri, 2 Nov 2001 09:45:46 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA28jk567180 for ; Fri, 2 Nov 2001 09:45:46 +0100 To: ips@ece.cmu.edu Importance: High MIME-Version: 1.0 Subject: iSCSI-countdown X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 2 Nov 2001 10:45:44 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 02/11/2001 10:45:46, Serialize complete at 02/11/2001 10:45:46 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, Since I will probably not be able to finish the editorial changes to the draft untill the cutoff date for IETF I intend to send 09 in the current format and merge with the editorial changes in the next version. I started the "countdown" - you can see the 08-90 in both txt and pdf-with-changebar format on my site: http://www.haifa.il.ibm.com/satran/ips Julo From owner-ips@ece.cmu.edu Fri Nov 2 09:15:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA05061 for ; Fri, 2 Nov 2001 09:15:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2CjrE04253 for ips-outgoing; Fri, 2 Nov 2001 07:45:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from exch-connector.netcomsystems.com (mushroom.netcomsystems.com [12.9.24.195]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2Cjol04249 for ; Fri, 2 Nov 2001 07:45:50 -0500 (EST) Received: by exch-connector.netcomsystems.com with Internet Mail Service (5.5.2653.19) id ; Fri, 2 Nov 2001 04:45:44 -0800 Message-ID: <9384475DFC05D2118F9C00805F6F263106DDB912@exchange1.netcomsystems.com> From: "Cunningham, Howard" To: "'Jim Hafner'" , Andre Asselin Cc: ips@ece.cmu.edu Subject: RE: is TargetName always mandatory or not? Date: Fri, 2 Nov 2001 04:45:43 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C1639C.49086720" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C1639C.49086720 Content-Type: text/plain; charset="iso-8859-1" Andre: I apologize if I have missed something in this thread, but... A non-zero TSID means either a connection re-start (X) bit or add a new connection to an existing session. It seems to me ISID-TSID uniquely determines the session, while a new CID (unique within the session only) indicates a new connection. In draft 8 section 3.12.9, it says: 3.12.9 TSID The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. On a Login request a TSID value of 0 indicates a request to open a new session. A non zero TSID indicates a request to add a connection to an existing session. Regards... -----Original Message----- From: Jim Hafner [mailto:hafner@almaden.ibm.com] Sent: Thursday, November 01, 2001 3:40 PM To: Andre Asselin Cc: ips@ece.cmu.edu Subject: Re: is TargetName always mandatory or not? Andre, Your picture isn't quite right. The portal group tag is relative to the iSCSI target and that target is *uniquely* identified by it's name. So, in your picture, there are two targets (not one). Each can label its target portal group any way it wants (independent of the other). Each has independent control over the TSIDs it uses. Each may *share* use of a network portal, but that's a different issue. The model implies two independent targets (even if they live in the same network entity and share resources) in your scenario. In other words, the portal groups are wrt targets not the network entity. In your scenario, whatever layer (you put it in the network code) has to decide what session to add the connection to has the initiator name, the ISID, the TSID *and* the target name (you left this out) in the login request. That fully qualifies both the target and the session as well. Jim Hafner Andre Asselin 11/01/2001 12:15 pm To: Jim Hafner/Almaden/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: Database 'Jim Hafner', View '($Inbox)') Jim, I agree with what you say, except for the part that mapping TSID=target portal group tag will work. Let's assume the following architecture: I have a network entity with one network portal (and thus one portal group). Inside this entity lives two iSCSI targets. Scenerio: An iSCSI initiator opens a connection to the network portal in order to add a connection to an already existing session (the network entity's networking code knows this because TSID in the initial login request is 0). The networking code needs to determine what session the initiator wants to add to, so it compares some items from the initial login request to each of the established sessions until it finds a match. The question is what items should it compare to identify a match? Section 3.12.9 of the spec reads: The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. As you say, this is clearly target centric (because, for example, an initiator could have 2 different sessions open to two different targets that have the same TSID). But from a target point of view, what that text means to me that the network entity's networking code should compare ISID + InitiatorName + TSID to determine a match. That implies that TSID must be unique per TargetName and per portal group ID. If TSID is just the target portal group tag, then the comparison wouldn't work. For example, using the architecture above where there is just one portal group, the target could have two sessions open: session A (InitiatorName=foo, ISID=1, TargetName=bar, TSID=0) and session B (InitiatorName=foo, ISID=1, TargetName=foobar, TSID=0). If it receives a login request with (InitiatorName=foo, ISID=1, TSID=0), which session is it for? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: 10/31/2001 cc: 06:33 PM From: Subject: (Document link: Andre Asselin) Andre, First, your comment "SSID + InitiatorName must be enough to uniquely identify a session" is target-centric. It would be different from the initiator's viewpoint. However, from the target's viewpoint, the target name is implicit and from the initiator viewpoint, the initiator name is implicit, so globally, the triple of the two names + SSID (made up of the ISID and TSID) form the identifier of the session. Locally (between two specific guys), the names are implicit so only the SSID is required. It's all a matter of point of view! As for the difference in identifiers, as I mentioned in the private note, the session is an iSCSI construct and is identified by iSCSI things. The nexus is a SCSI thing and is identified by SCSI constructs (based on how we've mapped the iSCSI things to SCSI things). However, you've brought to the fore again a related question: "what value does the TSID provide to the protocol?" I'm not going to answer that, but I will note that one target implementation that (I think) works is that the TSID = target portal group tag. The other thing to ask is "what value is there in the nexus identifier?" This is never really used in SCSI at all, so it's not a critical issue what composes it. However, it is important (IMO) that the SCSI ports have names and the logical derivative of that statement is that the nexus is identified by the concatenation of the SCSI port names (hence the definition we have). Jim Hafner Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 03:00:53 pm Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu Subject: Re: is TargetName always mandatory or not? John, WHOOPS! I was wrong; you are absolutely right that the spec says "TargetName" and not "TSID". When I was reading through it, I saw "TargetName", but read to myself "TSID". Apologies... In my defense (confusion?), however, 3.12.9 says TSID rather than TargetName is used to uniquely identify a session. Going by that, SSID + InitiatorName must be enough to uniquely identify a session. Jim Hafner pointed out to me that the I_T nexus is identified by one thing and the session is identified by another. If the two must have a 1-1 mapping in iSCSI, why are there two different identifiers? Why not just use the current definition of the I_T nexus to uniquely identify both the nexus and session (i.e. get rid of TSID)? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 04:42 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, I looked again through the document and No where could I find a statement that you claimed as "a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag". It is the InitiatorName, ISID, TSID, with the TargetName and PortalGroup. Please point out to me in the Spec (8 or above), where I can find your statement on I_T Nexus. I did find the following (please ignore for this conversation the "i" and 't" stuff): "- Session: The group of TCP connections that link an initiator with a target, form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one "target image". " " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + 'i'+ ISID, iSCSI Target Name + 't'+ Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). " I have not found a place where Session ID is tied to the TSID. Hence my comment that we also need to have the TargetName in the Initial Login on all Connections. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 10:40:55 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? John & Julian, How about this for the section 5 text: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The initial login request of the leading Login MAY also include the SessionType key=value pair, in which case if the SessionType is not "discovery", then the initial login request MUST also include the key=value pair TargetName. John, I disagree with your second statement: I don't see any way you could have 2 different sessions established within the same portal group with the same InitiatorName, ISID, and TSID. The spec. says that a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag. There is no mention of TargetName contributing to the identificaiton of a session. In my opinion, a non-leading connection should NOT have the TargetName, since it doesn't contribute anything. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: ips@ece.cmu.edu Sent by: Subject: Re: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 10/31/2001 04:20 AM Julian, I think the TargetName should be included on the Initial Login Request on the Leading Login. It seem strange to permit the Authentication functions to proceed if perhaps the intended Target is different then the one doing the Authentication. The way it currently is written, you could pass all the Security test and then find out just before going into Full Function Phase, that it was intended for some other Target. Seems like a waste. My I think that TargetName should also be on all connections on the Initial Login Request. Here is my thinking: The SSID (ISID+TSID) is unique only with regards to a Specific Initiator and Target Node Pair. It is therefore not clear that just knowing the SSID and InitiatorName is enough to understand what session the subsequent connections are being attached to. And it is possible that the CID could be also overlapped with another session. Therefore, I think it make since to determine all this on the initial Login on every Connection, so you know at the beginning what values can be negotiated, or that are being set to the right Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/30/2001 11:33:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: is TargetName always mandatory or not? It is the leading login: The section 5 paragraph will read: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. Julo Andre Asselin/Raleigh/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 31-10-01 02:08 Please respond to Andre Asselin To: "IPS Reflector (E-mail)" cc: Subject: is TargetName always mandatory or not? In section 5 of the spec, it says "If the SessionType is not "discovery" then the initial Login Request MUST also include the key=value pair TargetName.". However, in Appendix D, the description for TargetName says it is Leading Only. Should TargetName not be Leading Only, or should the text in section 5 say that TargetName must be in the initial Login Request of a leading connection? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC ------_=_NextPart_001_01C1639C.49086720 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: is TargetName always mandatory or not?

Andre:

I apologize if I have missed something in this = thread, but...

A non-zero TSID means either a connection re-start = (X) bit or add a new connection to an existing session. It seems to me = ISID-TSID uniquely determines the session, while a new CID (unique = within the session only) indicates a new connection.

In draft 8 section 3.12.9, it says:

        3.12.9 = TSID
         =
        The TSID = is the target assigned tag for a session with a specific
        named = initiator that, together with the ISID uniquely identifies a
        session = with that initiator.
         =
        On a = Login request a TSID value of 0 indicates a request to open a
        new = session.
         =
        A non = zero TSID indicates a request to add a connection to an
        existing = session.

Regards...

-----Original Message-----
From: Jim Hafner [mailto:hafner@almaden.ibm.com= ]
Sent: Thursday, November 01, 2001 3:40 PM
To: Andre Asselin
Cc: ips@ece.cmu.edu
Subject: Re: is TargetName always mandatory or = not?



Andre,

Your picture isn't quite right.  The portal = group tag is relative to the
iSCSI target and that target is *uniquely* = identified by it's name.  So, in
your picture, there are two targets (not one).  = Each can label its target
portal group any way it wants (independent of the = other).  Each has
independent control over the TSIDs it uses.  = Each may *share* use of a
network portal, but that's a different issue.  = The model implies two
independent targets (even if they live in the same = network entity and share
resources) in your scenario.
In other words, the portal groups are wrt targets = not the network entity.

In your scenario, whatever layer (you put it in the = network code) has to
decide what session to add the connection to has the = initiator name, the
ISID, the TSID *and* the target name (you left this = out) in the login
request.  That fully qualifies both the target = and the session as well.

Jim Hafner

          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;       
          &nb= sp;     Andre = Asselin           = ;            = ;  
          &nb= sp;     11/01/2001 12:15 = pm           &nbs= p;       
          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;       
          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;       



To:   Jim Hafner/Almaden/IBM@IBMUS
cc:   ips@ece.cmu.edu
From: Andre Asselin/Raleigh/IBM@IBMUS
Subject:  Re: is TargetName always mandatory or = not?  (Document link:
      Database 'Jim = Hafner', View '($Inbox)')

Jim,
     I agree with what you say, = except for the part that mapping
TSID=3Dtarget portal group tag will work.

Let's assume the following architecture:  I have = a network entity with one
network portal (and thus one portal group).  = Inside this entity lives two
iSCSI targets.

Scenerio: An iSCSI initiator opens a connection to = the network portal in
order to add a connection to an already existing = session (the network
entity's networking code knows this because TSID in = the initial login
request is 0).  The networking code needs to = determine what session the
initiator wants to add to, so it compares some items = from the initial login
request to each of the established sessions until it = finds a match.  The
question is what items should it compare to identify = a match?

Section 3.12.9 of the spec reads:

        The TSID = is the target assigned tag for a session with a specific
        named = initiator that, together with the ISID uniquely identifies a
        session = with that initiator.

As you say, this is clearly target centric (because, = for example, an
initiator could have 2 different sessions open to = two different targets
that have the same TSID).  But from a target = point of view, what that text
means to me that the network entity's networking = code should compare ISID +
InitiatorName + TSID to determine a match.  = That implies that TSID must be
unique per TargetName and per portal group ID.  = If TSID is just the target
portal group tag, then the comparison wouldn't = work.  For example, using
the architecture above where there is just one = portal group, the target
could have two sessions open: session A = (InitiatorName=3Dfoo, ISID=3D1,
TargetName=3Dbar, TSID=3D0) and session B = (InitiatorName=3Dfoo, ISID=3D1,
TargetName=3Dfoobar, TSID=3D0).  If it receives = a login request with
(InitiatorName=3Dfoo, ISID=3D1, TSID=3D0), which = session is it for?

Andre Asselin
IBM ServeRAID Software Development
Research Triangle Park, NC



          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;  

          &nb= sp;         Jim = Hafner           =             =             =             =             =             =             =             =         

          &nb= sp;           &nb= sp;           &nb= sp;      = To:           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;     

          &nb= sp;         = 10/31/2001           = cc:           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;     

          &nb= sp;         06:33 = PM           &nbs= p; = From:           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;           &= nbsp;   

          &nb= sp;           &nb= sp;           &nb= sp;      Subject:     = (Document link: Andre = Asselin)          &nbs= p;           &nbs= p;           &nbs= p;           &nbs= p;  

          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;  

          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;  

          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;  

          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;  

          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;  

          &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;           &nb= sp;  



Andre,

First, your comment "SSID + InitiatorName must = be enough to uniquely
identify a session" is target-centric.  It = would be different from the
initiator's viewpoint.  However, from the = target's viewpoint, the target
name is implicit and from the initiator viewpoint, = the initiator name is
implicit, so globally, the triple of the two names + = SSID (made up of the
ISID and TSID) form the identifier of the = session.  Locally (between two
specific guys), the names are implicit so only the = SSID is required.  It's
all a matter of point of view!

As for the difference in identifiers, as I mentioned = in the private note,
the session is an iSCSI construct and is identified = by iSCSI things.  The
nexus is a SCSI thing and is identified by SCSI = constructs (based on how
we've mapped the iSCSI things to SCSI = things).

However, you've brought to the fore again a related = question:  "what value
does the TSID provide to the protocol?"

I'm not going to answer that, but I will note that = one target
implementation that (I think) works is that the TSID = =3D target portal group
tag.

The other thing to ask is "what value is there = in  the nexus identifier?"
This is never really used in SCSI at all, so it's = not a critical issue what
composes it.  However, it is important (IMO) = that the SCSI ports have names
and the logical derivative of that statement is that = the nexus is
identified by the concatenation of the SCSI port = names (hence the
definition we have).

Jim Hafner


Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on = 10/31/2001 03:00:53 pm

Sent by:  owner-ips@ece.cmu.edu


To:   John Hufferd/San = Jose/IBM@IBMUS
cc:   ips@ece.cmu.edu
Subject:  Re: is TargetName always mandatory or = not?



John,
     WHOOPS!  I was wrong; = you are absolutely right that the spec says
"TargetName" and not = "TSID".  When I was reading through it, I saw
"TargetName", but read to myself = "TSID".  Apologies...
     In my defense (confusion?), = however, 3.12.9 says TSID rather than
TargetName is used to uniquely identify a = session.  Going by that, SSID +
InitiatorName must be enough to uniquely identify a = session.

     Jim Hafner pointed out to me = that the I_T nexus is identified by one
thing and the session is identified by = another.  If the two must have a 1-1
mapping in iSCSI, why are there two different = identifiers?  Why not just
use the current definition of the I_T nexus to = uniquely identify both the
nexus and session (i.e. get rid of TSID)?

Andre Asselin
IBM ServeRAID Software Development
Research Triangle Park, NC




          &nb= sp;         John Hufferd
          &nb= sp;           &nb= sp;           &nb= sp;      To:     = Andre
Asselin/Raleigh/IBM@IBMUS
          &nb= sp;         = 10/31/2001           = cc:     ips@ece.cmu.edu
          &nb= sp;         04:42 = PM           &nbs= p; From:   John Hufferd/San
Jose/IBM@IBMUS
          &nb= sp;           &nb= sp;           &nb= sp;      Subject:     Re: = is TargetName
always mandatory or not?(Document link: Andre = Asselin)









Andre,
I looked again through the document and No where = could I find a statement
that you claimed as "a nexus, and therefore an = iSCSI session, is uniquely
identified by the InitiatorName, ISID, TSID, and = portal group tag".  It is
the InitiatorName, ISID, TSID, with the TargetName = and PortalGroup.

Please point out to me in the Spec (8 or above), = where  I can find your
statement on I_T Nexus.

I did find the following (please ignore for this = conversation the "i" and
't" stuff):

"- Session: The group of TCP connections that = link an initiator with a
target, form a session (loosely equivalent to a SCSI = I-T nexus). TCP
connections can be added and removed from a session. = Across all connections
within a session, an initiator sees one "target = image".  "

" - I_T nexus: According to [SAM2], the I_T = nexus is a relationship between
a SCSI Initiator Port and a SCSI Target Port. For = iSCSI, this relationship
is a session, defined as a relationship between an = iSCSI Initiator's end of
the session (SCSI Initiator Port) and the iSCSI = Target's Portal Group. The
I_T nexus can be identified by the conjunction of = the SCSI port names; that
is, the I_T nexus identifier is the tuple (iSCSI = Initiator Name + 'i'+
ISID, iSCSI Target Name + 't'+ Portal Group Tag). = NOTE: The I_T nexus
identifier is not equal to the session identifier = (SSID).  "

I have not found a place where Session ID is tied to = the TSID.

Hence my comment that we also need to have the = TargetName in the Initial
Login on all Connections.



.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  = eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: hufferd@us.ibm.com


Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on = 10/31/2001 10:40:55 AM

Sent by:  owner-ips@ece.cmu.edu


To:   ips@ece.cmu.edu
cc:   John Hufferd/San = Jose/IBM@IBMUS
Subject:  Re: is TargetName always mandatory or = not?



John & Julian,
     How about this for the = section 5 text:

The initial Login request of the first connection of = a session (leading
login) MUST include the InitiatorName key=3Dvalue = pair. The initial login
request
of the leading Login MAY also include the = SessionType key=3Dvalue pair, in
which case if the SessionType is not = "discovery", then the initial login
request
MUST also include the key=3Dvalue pair = TargetName.

John,
     I disagree with your second = statement: I don't see any way you could
have 2 different sessions established within the = same portal group with the
same InitiatorName, ISID, and TSID.  The spec. = says that a nexus, and
therefore an iSCSI session, is uniquely identified = by the InitiatorName,
ISID, TSID, and portal group tag.  There is no = mention of TargetName
contributing to the identificaiton of a = session.  In my opinion, a
non-leading connection should NOT have the = TargetName, since it doesn't
contribute anything.

Andre Asselin
IBM ServeRAID Software Development
Research Triangle Park, NC




          &nb= sp;         John
          &nb= sp;         = Hufferd/San          = To:     Julian
Satran/Haifa/IBM@IBMIL
          &nb= sp;         = Jose/IBM@IBMUS       = cc:     ips@ece.cmu.edu
          &nb= sp;         Sent = by:           &nb= sp; Subject:     Re: is TargetName
always mandatory or not?
          &nb= sp;         = owner-ips@ece.
          &nb= sp;         cmu.edu


          &nb= sp;         10/31/2001
          &nb= sp;         04:20 AM






Julian,
I think the TargetName should be included on the = Initial Login Request on
the Leading Login.  It seem strange to permit = the Authentication functions
to proceed if perhaps the intended Target is = different then the one doing
the Authentication.  The way it currently is = written, you could pass all
the Security test and then find out just before = going into Full Function
Phase, that it was intended for some other = Target.  Seems like a waste.

My I think that TargetName should also be on all = connections on the Initial
Login Request.  Here is my thinking:

The SSID (ISID+TSID) is unique only with regards to a = Specific Initiator
and Target Node Pair.  It is therefore not = clear that just knowing the SSID
and InitiatorName is enough to understand what = session the subsequent
connections are being attached to.  And it is = possible that the CID could
be also overlapped with another session.

Therefore, I think it make since to determine all = this on the initial Login
on every Connection, so you know at the beginning = what values can be
negotiated, or that are being set to the right = Session.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  = eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: hufferd@us.ibm.com


Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on = 10/30/2001 11:33:50 PM

Sent by:  owner-ips@ece.cmu.edu


To:   ips@ece.cmu.edu
cc:
Subject:  Re: is TargetName always mandatory or = not?



It is the leading login:

The section 5 paragraph will read:

The initial Login request of the first connection of = a session (leading
login) MUST include the InitiatorName key=3Dvalue = pair. The leading Login
request MAY also include the SessionType key=3Dvalue = pair in which case if
the SessionType is not "discovery" then = the leading Login Request MUST
also include the key=3Dvalue pair TargetName.

Julo




Andre Asselin/Raleigh/IBM@IBMUS
Sent by: owner-ips@ece.cmu.edu
31-10-01 02:08
Please respond to Andre Asselin


        = To:     "IPS Reflector (E-mail)" = <ips@ece.cmu.edu>
        = cc:
        = Subject:        is TargetName always = mandatory or not?



     In section 5 of the spec, it = says "If the SessionType is not
"discovery" then the initial Login Request = MUST also include the key=3Dvalue
pair TargetName.".  However, in Appendix = D, the description for TargetName
says it is Leading Only.
     Should TargetName not be = Leading Only, or should the text in section
5
say that TargetName must be in the initial Login = Request of a leading
connection?

Andre Asselin
IBM ServeRAID Software Development
Research Triangle Park, NC




















------_=_NextPart_001_01C1639C.49086720-- From owner-ips@ece.cmu.edu Fri Nov 2 10:27:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA07773 for ; Fri, 2 Nov 2001 10:27:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2E82Y09326 for ips-outgoing; Fri, 2 Nov 2001 09:08:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mayfair.vxindia.veritas.com (bay-bridge.veritas.com [143.127.3.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2E7wl09318 for ; Fri, 2 Nov 2001 09:07:59 -0500 (EST) Received: from RAHULB (rahulb.vxindia.veritas.com [10.212.80.59]) by mayfair.vxindia.veritas.com (8.9.3/8.9.3) with SMTP id TAA17208 for ; Fri, 2 Nov 2001 19:37:48 +0530 (IST) Message-ID: <018501c163a7$d70371e0$3b50d40a@vxindia.veritas.com> From: "Rahul Bhagwat" To: Subject: Data re-read from Device Server Date: Fri, 2 Nov 2001 19:38:22 +0530 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0182_01C163D5.EE5CFD60" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0182_01C163D5.EE5CFD60 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, Section 8 mentions "It is also assumed that at target, incoming data = (read data) MAY be kept for recovery or it can be re-read from a device server." Is it assumed here that the data from the device server is idempotent = i.e will not change when tried to re-read. If that is not the case and we have more = than one Data In PDU being sent in the response (for one of which a = retransmission is=20 requested), it will result in in-correct view of the data being given to = the initiator. As an example, consider that the following response needs to be given ------------------ | 1 | 2 | 3 | 4 | ----------------- which gets transmitted in two Data In PDUs like this -------- --------- | 1 | 2 | | 3 | 4 | -------- --------- Now if a re-read generates ------------------ | 1 | 3 | 4 | 5 | ------------------ and retransmission is requested for 2nd PDU. It will get transmitted = like=20 -------- | 4 | 5 | -------- which means the initiator neither got 1,2,3,4 nor got 1,3,4,5 (both of = which are valid snapshots at some point in time), instead it got 1,2,4,5 which = never existed at the target. If we try to re-read only data pertaining to retransmitting PDU, the = problem persists. Regards, Rahul ------=_NextPart_000_0182_01C163D5.EE5CFD60 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
 
Hi,
 
Section 8 mentions "It is also assumed = that at=20 target, incoming  data (read data)
MAY be kept for recovery or it can be = re-read from=20 a device server."
 
Is it assumed here that the data from = the device=20 server is idempotent i.e will not
change when tried to re-read. If that = is not the=20 case and we have more than one
Data In PDU being sent in the response = (for one of=20 which a retransmission is
requested), it will result in = in-correct view of=20 the data being given to the initiator.
 
As an example, consider that the = following response=20 needs to be given
 
  ------------------
 | 1 | 2 | 3 | 4 |
  -----------------
 
which gets transmitted in two Data In = PDUs like=20 this
 
 =20 --------           = ;      =20 ---------
 | 1 | 2=20 |            =     | =20 3 | 4 |
 =20 --------           = ;      =20 ---------
 
Now if a re-read generates
 
  ------------------
 | 1 |  3 | 4 | 5 = |
  ------------------
 
and retransmission is requested for 2nd PDU. It will get = transmitted like=20
 
  --------
 | 4 | 5 |
  --------
 
which means the initiator neither got 1,2,3,4 nor got 1,3,4,5 (both = of=20 which are
valid snapshots at some point in time), instead it got 1,2,4,5 = which never=20 existed
at the target.
 
If we try to re-read only data pertaining to retransmitting PDU, = the=20 problem persists.
 
Regards,
Rahul
------=_NextPart_000_0182_01C163D5.EE5CFD60-- From owner-ips@ece.cmu.edu Fri Nov 2 11:01:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08759 for ; Fri, 2 Nov 2001 11:01:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2FNNA14808 for ips-outgoing; Fri, 2 Nov 2001 10:23:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA2FNLl14800 for ; Fri, 2 Nov 2001 10:23:21 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by dirty; Fri Nov 2 10:22:38 EST 2001 Received: from aura.research.bell-labs.com (aura.research.bell-labs.com [135.104.46.10]) by grubby.research.bell-labs.com (8.11.6/8.11.6) with ESMTP id fA2FLuR30835 for ; Fri, 2 Nov 2001 10:21:56 -0500 (EST) Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id KAA07972 for ; Fri, 2 Nov 2001 10:21:56 -0500 (EST) Message-ID: <3BE2BA13.63F2B387@research.bell-labs.com> Date: Fri, 02 Nov 2001 10:21:55 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Data re-read from Device Server References: <018501c163a7$d70371e0$3b50d40a@vxindia.veritas.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I believe the implicit assumption is that the ULP is holding the right locks. Since retransmission has been requested, the particular command has not yet completed at the initiator. That, in turn, implies that the ULP is still waiting for the data and has not yet released its relevant locks. -Sandeep > Rahul Bhagwat wrote: > > > Hi, > > Section 8 mentions "It is also assumed that at target, incoming data > (read data) > MAY be kept for recovery or it can be re-read from a device server." > > Is it assumed here that the data from the device server is idempotent > i.e will not > change when tried to re-read. If that is not the case and we have more > than one > Data In PDU being sent in the response (for one of which a > retransmission is > requested), it will result in in-correct view of the data being given > to the initiator. > > As an example, consider that the following response needs to be given > > ------------------ > | 1 | 2 | 3 | 4 | > ----------------- > > which gets transmitted in two Data In PDUs like this > > -------- --------- > | 1 | 2 | | 3 | 4 | > -------- --------- > > Now if a re-read generates > > ------------------ > | 1 | 3 | 4 | 5 | > ------------------ > > and retransmission is requested for 2nd PDU. It will get transmitted > like > > -------- > | 4 | 5 | > -------- > > which means the initiator neither got 1,2,3,4 nor got 1,3,4,5 (both of > which are > valid snapshots at some point in time), instead it got 1,2,4,5 which > never existed > at the target. > > If we try to re-read only data pertaining to retransmitting PDU, the > problem persists. > > Regards, > Rahul From owner-ips@ece.cmu.edu Fri Nov 2 11:04:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08824 for ; Fri, 2 Nov 2001 11:04:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2Ewlf12889 for ips-outgoing; Fri, 2 Nov 2001 09:58:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from best.eurologic.com ([193.120.246.5]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2Ewel12876 for ; Fri, 2 Nov 2001 09:58:41 -0500 (EST) Received: from COORS (coors.eurologic.com [192.168.7.111]) by best.eurologic.com (8.9.3+Sun/8.9.3) with SMTP id OAA17677; Fri, 2 Nov 2001 14:55:05 GMT Message-ID: <001901c163ae$f30a6a90$6f07a8c0@COORS> From: "Ron Cohen" To: "ips" , "Paul Koning" References: <000101c162d7$330d39c0$0102a8c0@eddylaptop> <15329.38576.650000.176588@gargle.gargle.HOWL> Subject: Re: iSCSI: current UNH Plugfest: Reserved bits Date: Fri, 2 Nov 2001 14:59:18 -0000 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The problem with accepting values in reserved bits is that when the reserved bit is no longer reserved (evolution in the standard) it gives the impression to the initiator that a target implements the new feature when it may actually only be ignoring it. Ron ----- Original Message ----- From: "Paul Koning" To: Cc: ; Sent: Thursday, November 01, 2001 6:38 PM Subject: RE: iSCSI: current UNH Plugfest > Excerpt of message (sent 1 November 2001) by Bill Strahm: > > Usually the "Conservative in what you send, Liberal in what you accept" > > policy is used... > > That's a very important principle... > > > In otherwords, The sender MUST set to 0 (or some other value) The receiver > > MUST ignore the value... > > That's the way to express the principle. But the text quoted in the > earlier notes does not express it the same way. > > In particular "... are format errors. This when detected..." implies > that receivers may or may not detect format errors. In other words, > it implies -- but does NOT state explicitly -- that receivers MAY > check reserved fields for zeroness. > > > This allows for some tweaking of the implementation, if I control both ends > > I might set a reserved value to 1, then I know something... If I receive a > > reserve value set to 1 and I don't do anything the other end knows it is not > > talking to itself (this can even be a versioning thing as well) > > > > Now, we need to be VERY careful in defining this, do we plan on having > > Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean... > > is it possible, is it desirable ? will there be extensions ??? > > > > If you can truly answer NO to all of those things, I would argue for > > REMOVING the reserved fields (if possible), if not, the MUST set, MUST > > ignore policy seems better > > I agree strongly. There is a lot of experience in the community on > what helps and what hurts convenient version upgrade. In particular, > there is a LOT of evidence that ANY checking of reserved fields is a > nasty thing. Unless you plan never to go beyond V1, you really need > to require the rule Bill stated, i.e., receivers MUST ignore the > contents of reserved fields -- they are NOT allowed to "verify" them. > > So the spec needs to be clarified to say that random values in > reserved fields are NOT format errors and receivers MUST NOT treat > them as such. > > paul > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Eddy Quicksall > > Sent: Thursday, November 01, 2001 5:15 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > I am reluctant to say this because I think most people think the > > initiator/target must check for correctness ... but, it is my feeling that > > that job should be up to a basher program. The target should not be in the > > business of diagnosing the initiator. The only time a target should check a > > field is when it could crash the system or data. Some format errors may have > > no consequences whatsoever. > > > > Eddy > > > > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Wednesday, October 31, 2001 05:39 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: current UNH Plugfest > > > > > > Attached are the new issues that arose during the iSCSI plugfest > > at UNH on Wednesday 31-Oct-2001. > > > > (Note: these issues do not take into account any modifications or > > clarifications that occured in the standard due to the issues raised > > on Monday or Tuesday.) > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > ------------------------------------------------------------------------ > > ---- > > > > 1. Are receivers (initiator or target) REQUIRED to check that reserved > > bits and/or fields are set to 0? > > > > Section 3 on page 48 of draft 8 says: > > "Any bits not defined MUST be set to zero. Any reserved fields and > > values MUST be 0 unless specified otherwise." > > > > and Section 8.3 on page 127 of draft 8 says: > > "Explicit violations of the PDU layout rules stated in this > > document > > are format errors. This when detected, usually indicates a major > > implementation flaw in one of the parties. > > > > When a target or an initiator receives an iSCSI PDU with a format > > error, it MUST reset all transport connections in the session > > immediately and escalate the format error to session recovery > > (section 8.11.4)." > > > > According to these rules, a PDU with reserved bits and/or fields that > > are not set to 0 violates the PDU layout rules. Therefore, if an > > initiator or target receives such a PDU, it should immediately close > > all connections in the session and go to session recovery. > > > > Clearly a format error has extremely severe consequences! > > > > Although all vendors are setting reserved bits and fields to 0 on > > PDUs they are sending, many are NOT checking PDUs they are receiving > > to see if these bits and fields are set to 0. Basically, vendors are > > saying "who cares if reserved bits and/or fields in incoming PDUs are > > not zero? We do not want to take the time to do this checking, and > > there is no benefit to doing it. As long as the non-reserved bits > > and > > fields are set properly, we can and should proceed. Any time devoted > > to doing this checking is wasted in 99+% of the cases, and in the > > (unlikely) case that a non-zero bit or field is found, the > > consequences are too severe." > > > > There should be some statement in the standard to clarify what > > checking > > is required and what is optional. > > > > 2. A similar situation arises with respect to checking the consistency > > of fields such as Version-max, Version-min and Version-active in > > Login > > Requests and Login Responses. > > > > For example, consider the Version-max field. > > > > Section 3.12.5 says: > > "All Login requests within the Login phase MUST carry the same > > Version-max." > > > > All vendor initiators are setting Version-max correctly on all > > login requests they are sending, but many vendor targets are NOT > > checking received login requests to ensure that this rule is > > enforced. > > In particular, many targets simply use the Version-max and > > Version-min > > on the first login request they receive on a new connection, and then > > they ignore these fields on all subsequent login requests in the same > > login phase. > > > > Strictly speaking, a change in the Version-max field during the login > > phase constitutes a protocol error according to section 8.8 on page > > 130 > > of draft 8: > > > > "All violations of iSCSI PDU exchange sequences specified in this > > draft are also protocol errors. This category of errors can be > > addressed only by fixing the implementations; iSCSI defines Reject > > and response codes to enable this". > > > > Therefore the target should send back a login response with a status > > of 0x0200 and then close the connection. > > > > However, Section 3.12.5 also says: > > "The target MUST use the value presented with the first login > > request." > > > > This rule seems to imply that the value CAN change, because if it > > cannot > > change, then it doesn't matter which one of the login requests it is > > taken from, they are all the same anyway. > > > > The suggestion is to keep the requirement that the target MUST use > > the > > value presented on the first login request, but to allowed the target > > to ignore the value presented on all subsequent login requests in the > > same login phase. A similar rewording should be done for the other > > fields. > > > > 3. Can commands be sent out of order on the same connection? > > > > The behavior of targets is clearly specified in Section 2.2.2.3 on > > page 25 of draft 8, which says: > > "Except for the commands marked for immediate delivery the iSCSI > > target layer MUST eliver the commands for execution in the order > > specified by CmdSN." > > > > Section 2.2.2.3 on page 26 of draft 8 also says: > > "- CmdSN - the current command Sequence Number advanced by 1 on > > each command shipped except for commands marked for immediate > > delivery." > > but the meaning of the term "shipped" is vague, and does not > > necessarily > > require that the PDUs arrive on the other end of a TCP connection > > in the same order that the CmdSN values were assigned to these PDUs. > > > > Some initiators have been designed to send commands out of CmdSN > > order on one connection. Consider the situation where there is only > > one connection and a high-level dispatcher creates a PDU for a SCSI > > command that involves writing immediate data to the target. This PDU > > is enqueued to a lower-level layer which has to setup, start, and > > wait-for a DMA operation to move the immediate data into an onboard > > buffer before the PDU can be put onto the wire. While this is > > happening, the dispatcher creates another unrelated PDU for a SCSI > > read command (for example), and when this PDU is passed to the > > lower-level layer it can be sent immediately, ahead of the previous > > write PDU and therefore out of order on this connection. > > > > The standard clearly allows this to happen if the two PDUs were sent > > on different connections, and seems to imply that this can also > > happen > > when the two PDUs are sent on the same connection. > > > > The suggestion is to put in the standard an explicit statement that > > this is allowed or not allowed, as appropriate. > > > > If this is allowed, such a statement would avoid the erroneous > > assumption being made by some target implementers that within a > > single > > connection, commands will arrive in order. > > > > If this is not allowed, such a statement would avoid the erroneous > > assumption being made by some initiator implementers that within a > > single connection, commands can be put on the wire out of order. > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > now allow: "A value of 0 indicates no limit." > > > > Is this useful? Does it buy anything? > > > > The difficulties implementers are having with this are: > > > > 1) It is a special case. > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > 3) It violates the min/max function normally used for the key. > > 4) There is always a limit anyway. > > > > Consider FirstBurstSize, which can have a value that is described > > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > > is selected. > > > > I one side offers 0 to mean unlimited, and the other side > > has a limit, it will reply with that limit, say 128 Kbytes. > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > The statement in the standard that "the minimum of the 2 numbers is > > selected" is therefore wrong when one of the numbers is 0. > > > > Furthermore, when an initiator or target receives an offer for one > > of these keys, it cannot simply check that the offered value is > > legal by testing it against some minimum and maximum. It must first > > check for 0 and then only if that check shows the value is non-zero > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > Finally, there is always a limit. If nothing else it is the > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > requesting the transfer. It is useless to specify a FirstBurstSize > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > the largest possible DataSegmentLength in any PDU that can use > > this value is 2**24-1. > > > > The suggestion is to just eliminate this special case of 0 and > > require > > that the range 64-to-(2**24-1) be used instead -- it has exactly the > > same effect in all cases, it is easier to describe in the standard > > because it avoids all the extra words, and it is easier to code > > because it avoids all the special cases. > > > > NOTE: the standard should specify the limit in the ranges for > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > DataSegmentLength field and therefore can never be used. > > > > 5. This is a suggestion for a minor rewording in the standard to avoid > > misunderstandings. > > > > In Appendix E on page 188 of draft 8 it says: > > > > "The response to this command is a text response containing a list > > of > > targets and their addresses. Each target is returned as a target > > record. A target record begins with the TargetName text key, > > followed by a list of TargetAddress text keys, ..." > > > > In fact, there are situations where there are no targets and/or no > > addresses. These situations are clearly defined in the draft after > > the sentences quoted above, but it would help if those sentences > > at least hinted at the possibility that the lists could be empty > > or might not contain addresses. A possible rewording would be: > > > > "The response to this command is a text response containing a list > > of > > zero or more targets and, optionally, their addresses. Each target > > is returned as a target record. A target record begins with the > > TargetName text key, followed by a list of zero or more > > TargetAddress text keys, ..." > > > > > > 6. This is a suggestion for another very minor rewording in the > > standard. > > > > At the end of section 2.2.3 on page 29 of draft 8 it says: > > > > "Before full feature phase is established, only Login PDUs are > > allowed. ..." > > > > The suggested rewording is: > > > > "Before full feature phase is established, only Login Request and > > Login Response PDUs are allowed. ..." > > From owner-ips@ece.cmu.edu Fri Nov 2 11:58:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA11473 for ; Fri, 2 Nov 2001 11:58:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2G4Dn18160 for ips-outgoing; Fri, 2 Nov 2001 11:04:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2G4Cl18154 for ; Fri, 2 Nov 2001 11:04:12 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA46222 for ; Fri, 2 Nov 2001 11:01:35 -0500 Received: from d03nm041.boulder.ibm.com (d03nm041.boulder.ibm.com [9.99.140.41]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA2G44j124592 for ; Fri, 2 Nov 2001 09:04:04 -0700 Importance: Normal Subject: RE: is TargetName always mandatory or not? To: ips@ece.cmu.edu From: "Jim Hafner" Date: Fri, 2 Nov 2001 08:04:03 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM041/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/02/2001 08:04:04 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, I think this thread may be exposing either an ambiguity in the draft or perhaps a gap. I think it's possible to interpret the requirement that TargetName be supplied (for normal sessions) only on the *first login message on the first connection of a new session* and is not required on the *first login message of subsequent connections intended for the same session". My personal interpretation is that the TargetName is required on the *first login message on every connection (in all cases, new session or adding connections to existing sessions)*. I also think Andre's example of a network entity containing many iSCSI targets with shared network portals (and the scoping of portal group tags and TSIDs by iSCSI target) points out the need for the more restrictive requirement (and the rest of my response to Andre assumed this). But I've heard other comments that can be interpreted as the less restrictive requirement is what is intended. So, I suggest (and I don't think this is asking much) that the requirement be that TargetName be supplied on the first Login message of every connection (not just the leading one). Jim Hafner From owner-ips@ece.cmu.edu Fri Nov 2 13:09:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13756 for ; Fri, 2 Nov 2001 13:09:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2HSlR24652 for ips-outgoing; Fri, 2 Nov 2001 12:28:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2HSkl24647 for ; Fri, 2 Nov 2001 12:28:46 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 7B3E31969; Fri, 2 Nov 2001 10:28:45 -0700 (MST) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 2C44465C; Fri, 2 Nov 2001 10:28:45 -0700 (MST) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 02 Nov 2001 10:28:44 -0700 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 2 Nov 2001 10:28:44 -0700 Message-ID: <01A7DAF31F93D511AEE300D0B706ED920CF692@axcs13.cos.agilent.com> From: "CAVANNA,VICENTE V (A-Roseville,ex1)" To: "'Bill Strahm'" Cc: "SHEEHY,DAVE (A-Americas,unix1)" , "CAVANNA,VICENTE V (A-Roseville,ex1)" , "'Ofer Biran'" , ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Fri, 2 Nov 2001 10:28:42 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Good Morning Bill, To me, a BITS implementation of IPSec does not require changes to the OS (with its built-in network stack) and is in fact useful when such changes are not feasible. In BITS, I assume IPSec is implemented as a shim between the network and data link layer. I thought, from reading the literature, that this was a conventional definition of BITS. If you are allowed to make changes to the network stack when adding IPSec functionality then, as I noted in my original memo, I consider transport mode to be feasible . If I understand you correctly then, what you really favor is a BITW implementation which I understand to be essentially an external crypto device. I would like to be able to use that too, but only provisionally, as a BITW implementation will eventually (as iSCSI matures) be considered too expensive and IPSec will have to be incorporated more tightly into the network stack. Vince -----Original Message----- From: Bill Strahm [mailto:bill@Sanera.net] Sent: Thursday, November 01, 2001 10:18 PM To: CAVANNA,VICENTE V (A-Roseville,ex1); 'Ofer Biran'; ips@ece.cmu.edu Cc: SHEEHY,DAVE (A-Americas,unix1) Subject: RE: iSCSI: IPsec tunnel / transport mode decision Ok, First off with complexity comes a configuration nightmare... Second I can implement both a BITS and a BITW IPsec Transport and Tunnel mode... it really isn't all that hard. BITS implies that you are making OS changes, or at least changes under the TCP/UDP layer that is traditionally exposed to user applications, so I think most of the argument you are trying to make in your second paragraph doesn't hold much water for me. I would rather use Tunnel mode myself, however my reasons are that I can completely offload the implementation off of the host and target and let intermediate devices deal with security policies and such things... However most of the tone of the security draft is to require for at least one end if not both to be intimately aware of what is going on with security... Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of CAVANNA,VICENTE V (A-Roseville,ex1) Sent: Thursday, November 01, 2001 5:19 PM To: 'Ofer Biran'; ips@ece.cmu.edu Cc: CAVANNA,VICENTE V (A-Roseville,ex1); SHEEHY,DAVE (A-Americas,unix1) Subject: RE: iSCSI: IPsec tunnel / transport mode decision It seems to me (who has not had the experience of implementing IpSec) that tunnel mode, even when implemented in the end host (rather than in a router), is a superset of transport mode whose only significant disadvantage is that tunnel mode requires more overhead in the form of the extra IP header. On the other hand, tunnel mode offers more flexibility in implementation as it is easier to implement in BITS and BITW implementations whereas transport mode can only be easily implemented when IPSec is implemented as part of the network layer i.e. integrated into the OS. The reason is that the IPSec headers are inserted AFTER the IP payload is constructed. This means that IPSec has to duplicate IP functionality because it has to recalculate the IP checksum and fragment the packet when necessary. I would support making tunnel mode the favored mode in iSCSI. IPSec compliance requires that transport mode be implemented but if iSCSI discourages it the implementation need not be as efficient as tunnel mode. Vince |-----Original Message----- |From: Ofer Biran [mailto:BIRAN@il.ibm.com] |Sent: Thursday, November 01, 2001 4:31 AM |To: ips@ece.cmu.edu |Subject: iSCSI: IPsec tunnel / transport mode decision | | |I'd like to drive this open issue into group consensus. It seems to |me that the tendency was more toward making tunnel mode a MUST as iFCP |and FCIP did, mainly due the option of integrating an existing IPsec |chip/box with the iSCSI implementation offering. If we reach |this decision, |we may choose even not to mention transport mode (as MAY or some other |recommending text). | |There is an excellent analysis made by Bernard Aboba in Section |"5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 |( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) |that can help us with this decision (also Section "5.2. NAT |traversal" is |relevant). | | Regards, | Ofer | |Ofer Biran |Storage and Systems Technology |IBM Research Lab in Haifa |biran@il.ibm.com 972-4-8296253 | From owner-ips@ece.cmu.edu Fri Nov 2 13:13:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13907 for ; Fri, 2 Nov 2001 13:13:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2HlJR26004 for ips-outgoing; Fri, 2 Nov 2001 12:47:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2HlGl25998 for ; Fri, 2 Nov 2001 12:47:16 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA2HlAu08377; Fri, 2 Nov 2001 09:47:10 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA2Hl5625494; Fri, 2 Nov 2001 09:47:05 -0800 From: "Bill Strahm" To: "CAVANNA,VICENTE V \(A-Roseville,ex1\)" Cc: "SHEEHY,DAVE \(A-Americas,unix1\)" , "'Ofer Biran'" , Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Fri, 2 Nov 2001 09:46:55 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: <01A7DAF31F93D511AEE300D0B706ED920CF692@axcs13.cos.agilent.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ok, I have implemented IPsec (both Transport & Tunnel) as a Win32 Intermediate Driver for Win 9x/NT 4 (We didn't need it in W2K because it is implemented there all ready). I have also implemented it as a set of Linux Netfilter rules for the 2.4 kernel... I would consider both of these BITS implementations and I consider these as OS modifications (all though depending on if you consider the drivers as a part of the OS they may not be) There are a few optimizations that you can make if you actually own the TCP stack. 1) Setting the MTU for the connection to take out the size of the IPsec overhead, however you can use ICMP error messages from the IPsec layer to do the same thing. 2) Handling TCP timeouts while a SA is negotiated. My implementation would drop the first TCP/SYN packet while the negotiation was going on, then send the second packet that comes down, usually after the negotiation went through... The other option was to send the TCP/SYN packet in the clear... BITW implementations have many of the same things to do, the only difference is that it happens after it goes through Ethernet processing... I have heard of an implementation that went through a Linux TUN(???) driver to pass the full Ethernet packet up to user space to have the IPsec processing done on it before passing back down as a raw Ethernet packet to go out eth0... Then there are external gateways that we are all familiar with... I am interested in external gateways mainly because they are 1) Purchasable off of the shelf 2) Don't have integration requirements - just tell the customer to buy two and get them to work, we run over it Bill -----Original Message----- From: CAVANNA,VICENTE V (A-Roseville,ex1) [mailto:vince_cavanna@agilent.com] Sent: Friday, November 02, 2001 9:29 AM To: 'Bill Strahm' Cc: SHEEHY,DAVE (A-Americas,unix1); CAVANNA,VICENTE V (A-Roseville,ex1); 'Ofer Biran'; ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Good Morning Bill, To me, a BITS implementation of IPSec does not require changes to the OS (with its built-in network stack) and is in fact useful when such changes are not feasible. In BITS, I assume IPSec is implemented as a shim between the network and data link layer. I thought, from reading the literature, that this was a conventional definition of BITS. If you are allowed to make changes to the network stack when adding IPSec functionality then, as I noted in my original memo, I consider transport mode to be feasible . If I understand you correctly then, what you really favor is a BITW implementation which I understand to be essentially an external crypto device. I would like to be able to use that too, but only provisionally, as a BITW implementation will eventually (as iSCSI matures) be considered too expensive and IPSec will have to be incorporated more tightly into the network stack. Vince -----Original Message----- From: Bill Strahm [mailto:bill@Sanera.net] Sent: Thursday, November 01, 2001 10:18 PM To: CAVANNA,VICENTE V (A-Roseville,ex1); 'Ofer Biran'; ips@ece.cmu.edu Cc: SHEEHY,DAVE (A-Americas,unix1) Subject: RE: iSCSI: IPsec tunnel / transport mode decision Ok, First off with complexity comes a configuration nightmare... Second I can implement both a BITS and a BITW IPsec Transport and Tunnel mode... it really isn't all that hard. BITS implies that you are making OS changes, or at least changes under the TCP/UDP layer that is traditionally exposed to user applications, so I think most of the argument you are trying to make in your second paragraph doesn't hold much water for me. I would rather use Tunnel mode myself, however my reasons are that I can completely offload the implementation off of the host and target and let intermediate devices deal with security policies and such things... However most of the tone of the security draft is to require for at least one end if not both to be intimately aware of what is going on with security... Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of CAVANNA,VICENTE V (A-Roseville,ex1) Sent: Thursday, November 01, 2001 5:19 PM To: 'Ofer Biran'; ips@ece.cmu.edu Cc: CAVANNA,VICENTE V (A-Roseville,ex1); SHEEHY,DAVE (A-Americas,unix1) Subject: RE: iSCSI: IPsec tunnel / transport mode decision It seems to me (who has not had the experience of implementing IpSec) that tunnel mode, even when implemented in the end host (rather than in a router), is a superset of transport mode whose only significant disadvantage is that tunnel mode requires more overhead in the form of the extra IP header. On the other hand, tunnel mode offers more flexibility in implementation as it is easier to implement in BITS and BITW implementations whereas transport mode can only be easily implemented when IPSec is implemented as part of the network layer i.e. integrated into the OS. The reason is that the IPSec headers are inserted AFTER the IP payload is constructed. This means that IPSec has to duplicate IP functionality because it has to recalculate the IP checksum and fragment the packet when necessary. I would support making tunnel mode the favored mode in iSCSI. IPSec compliance requires that transport mode be implemented but if iSCSI discourages it the implementation need not be as efficient as tunnel mode. Vince |-----Original Message----- |From: Ofer Biran [mailto:BIRAN@il.ibm.com] |Sent: Thursday, November 01, 2001 4:31 AM |To: ips@ece.cmu.edu |Subject: iSCSI: IPsec tunnel / transport mode decision | | |I'd like to drive this open issue into group consensus. It seems to |me that the tendency was more toward making tunnel mode a MUST as iFCP |and FCIP did, mainly due the option of integrating an existing IPsec |chip/box with the iSCSI implementation offering. If we reach |this decision, |we may choose even not to mention transport mode (as MAY or some other |recommending text). | |There is an excellent analysis made by Bernard Aboba in Section |"5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 |( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) |that can help us with this decision (also Section "5.2. NAT |traversal" is |relevant). | | Regards, | Ofer | |Ofer Biran |Storage and Systems Technology |IBM Research Lab in Haifa |biran@il.ibm.com 972-4-8296253 | From owner-ips@ece.cmu.edu Fri Nov 2 13:13:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13929 for ; Fri, 2 Nov 2001 13:13:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2HOKw24291 for ips-outgoing; Fri, 2 Nov 2001 12:24:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2HOFl24280 for ; Fri, 2 Nov 2001 12:24:15 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA2HO5u08192; Fri, 2 Nov 2001 09:24:05 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA2HNx624806; Fri, 2 Nov 2001 09:23:59 -0800 From: "Bill Strahm" To: "Ron Cohen" , "ips" , "Paul Koning" Subject: RE: iSCSI: current UNH Plugfest: Reserved bits Date: Fri, 2 Nov 2001 09:23:49 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: <001901c163ae$f30a6a90$6f07a8c0@COORS> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit In many cases that is acceptable... Imagine what would happen if Router vendors threw out packets with headers that set reserved bits to anything but 1. Anyone that wanted to play with DiffServ would have to buy new routers that implemented the functionality of the TOS bits... This when it is perfectly acceptable to just ignore them and send packets on without applying any extra functionality - What a pain. Some of the upgrade problem can be handled with a new version number (if you don't speak the new version, I will drop the packet) but there are MANY extensions that I can think of where using a bit or two of a reserved field would allow potentially better performance if the endpoint knows what to do, and no real degradation of service if it doesn't know what to do... Why should I have to rev the version number to handle this case ? Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ron Cohen Sent: Friday, November 02, 2001 6:59 AM To: ips; Paul Koning Subject: Re: iSCSI: current UNH Plugfest: Reserved bits The problem with accepting values in reserved bits is that when the reserved bit is no longer reserved (evolution in the standard) it gives the impression to the initiator that a target implements the new feature when it may actually only be ignoring it. Ron ----- Original Message ----- From: "Paul Koning" To: Cc: ; Sent: Thursday, November 01, 2001 6:38 PM Subject: RE: iSCSI: current UNH Plugfest > Excerpt of message (sent 1 November 2001) by Bill Strahm: > > Usually the "Conservative in what you send, Liberal in what you accept" > > policy is used... > > That's a very important principle... > > > In otherwords, The sender MUST set to 0 (or some other value) The receiver > > MUST ignore the value... > > That's the way to express the principle. But the text quoted in the > earlier notes does not express it the same way. > > In particular "... are format errors. This when detected..." implies > that receivers may or may not detect format errors. In other words, > it implies -- but does NOT state explicitly -- that receivers MAY > check reserved fields for zeroness. > > > This allows for some tweaking of the implementation, if I control both ends > > I might set a reserved value to 1, then I know something... If I receive a > > reserve value set to 1 and I don't do anything the other end knows it is not > > talking to itself (this can even be a versioning thing as well) > > > > Now, we need to be VERY careful in defining this, do we plan on having > > Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean... > > is it possible, is it desirable ? will there be extensions ??? > > > > If you can truly answer NO to all of those things, I would argue for > > REMOVING the reserved fields (if possible), if not, the MUST set, MUST > > ignore policy seems better > > I agree strongly. There is a lot of experience in the community on > what helps and what hurts convenient version upgrade. In particular, > there is a LOT of evidence that ANY checking of reserved fields is a > nasty thing. Unless you plan never to go beyond V1, you really need > to require the rule Bill stated, i.e., receivers MUST ignore the > contents of reserved fields -- they are NOT allowed to "verify" them. > > So the spec needs to be clarified to say that random values in > reserved fields are NOT format errors and receivers MUST NOT treat > them as such. > > paul > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Eddy Quicksall > > Sent: Thursday, November 01, 2001 5:15 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > I am reluctant to say this because I think most people think the > > initiator/target must check for correctness ... but, it is my feeling that > > that job should be up to a basher program. The target should not be in the > > business of diagnosing the initiator. The only time a target should check a > > field is when it could crash the system or data. Some format errors may have > > no consequences whatsoever. > > > > Eddy > > > > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Wednesday, October 31, 2001 05:39 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: current UNH Plugfest > > > > > > Attached are the new issues that arose during the iSCSI plugfest > > at UNH on Wednesday 31-Oct-2001. > > > > (Note: these issues do not take into account any modifications or > > clarifications that occured in the standard due to the issues raised > > on Monday or Tuesday.) > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > ------------------------------------------------------------------------ > > ---- > > > > 1. Are receivers (initiator or target) REQUIRED to check that reserved > > bits and/or fields are set to 0? > > > > Section 3 on page 48 of draft 8 says: > > "Any bits not defined MUST be set to zero. Any reserved fields and > > values MUST be 0 unless specified otherwise." > > > > and Section 8.3 on page 127 of draft 8 says: > > "Explicit violations of the PDU layout rules stated in this > > document > > are format errors. This when detected, usually indicates a major > > implementation flaw in one of the parties. > > > > When a target or an initiator receives an iSCSI PDU with a format > > error, it MUST reset all transport connections in the session > > immediately and escalate the format error to session recovery > > (section 8.11.4)." > > > > According to these rules, a PDU with reserved bits and/or fields that > > are not set to 0 violates the PDU layout rules. Therefore, if an > > initiator or target receives such a PDU, it should immediately close > > all connections in the session and go to session recovery. > > > > Clearly a format error has extremely severe consequences! > > > > Although all vendors are setting reserved bits and fields to 0 on > > PDUs they are sending, many are NOT checking PDUs they are receiving > > to see if these bits and fields are set to 0. Basically, vendors are > > saying "who cares if reserved bits and/or fields in incoming PDUs are > > not zero? We do not want to take the time to do this checking, and > > there is no benefit to doing it. As long as the non-reserved bits > > and > > fields are set properly, we can and should proceed. Any time devoted > > to doing this checking is wasted in 99+% of the cases, and in the > > (unlikely) case that a non-zero bit or field is found, the > > consequences are too severe." > > > > There should be some statement in the standard to clarify what > > checking > > is required and what is optional. > > > > 2. A similar situation arises with respect to checking the consistency > > of fields such as Version-max, Version-min and Version-active in > > Login > > Requests and Login Responses. > > > > For example, consider the Version-max field. > > > > Section 3.12.5 says: > > "All Login requests within the Login phase MUST carry the same > > Version-max." > > > > All vendor initiators are setting Version-max correctly on all > > login requests they are sending, but many vendor targets are NOT > > checking received login requests to ensure that this rule is > > enforced. > > In particular, many targets simply use the Version-max and > > Version-min > > on the first login request they receive on a new connection, and then > > they ignore these fields on all subsequent login requests in the same > > login phase. > > > > Strictly speaking, a change in the Version-max field during the login > > phase constitutes a protocol error according to section 8.8 on page > > 130 > > of draft 8: > > > > "All violations of iSCSI PDU exchange sequences specified in this > > draft are also protocol errors. This category of errors can be > > addressed only by fixing the implementations; iSCSI defines Reject > > and response codes to enable this". > > > > Therefore the target should send back a login response with a status > > of 0x0200 and then close the connection. > > > > However, Section 3.12.5 also says: > > "The target MUST use the value presented with the first login > > request." > > > > This rule seems to imply that the value CAN change, because if it > > cannot > > change, then it doesn't matter which one of the login requests it is > > taken from, they are all the same anyway. > > > > The suggestion is to keep the requirement that the target MUST use > > the > > value presented on the first login request, but to allowed the target > > to ignore the value presented on all subsequent login requests in the > > same login phase. A similar rewording should be done for the other > > fields. > > > > 3. Can commands be sent out of order on the same connection? > > > > The behavior of targets is clearly specified in Section 2.2.2.3 on > > page 25 of draft 8, which says: > > "Except for the commands marked for immediate delivery the iSCSI > > target layer MUST eliver the commands for execution in the order > > specified by CmdSN." > > > > Section 2.2.2.3 on page 26 of draft 8 also says: > > "- CmdSN - the current command Sequence Number advanced by 1 on > > each command shipped except for commands marked for immediate > > delivery." > > but the meaning of the term "shipped" is vague, and does not > > necessarily > > require that the PDUs arrive on the other end of a TCP connection > > in the same order that the CmdSN values were assigned to these PDUs. > > > > Some initiators have been designed to send commands out of CmdSN > > order on one connection. Consider the situation where there is only > > one connection and a high-level dispatcher creates a PDU for a SCSI > > command that involves writing immediate data to the target. This PDU > > is enqueued to a lower-level layer which has to setup, start, and > > wait-for a DMA operation to move the immediate data into an onboard > > buffer before the PDU can be put onto the wire. While this is > > happening, the dispatcher creates another unrelated PDU for a SCSI > > read command (for example), and when this PDU is passed to the > > lower-level layer it can be sent immediately, ahead of the previous > > write PDU and therefore out of order on this connection. > > > > The standard clearly allows this to happen if the two PDUs were sent > > on different connections, and seems to imply that this can also > > happen > > when the two PDUs are sent on the same connection. > > > > The suggestion is to put in the standard an explicit statement that > > this is allowed or not allowed, as appropriate. > > > > If this is allowed, such a statement would avoid the erroneous > > assumption being made by some target implementers that within a > > single > > connection, commands will arrive in order. > > > > If this is not allowed, such a statement would avoid the erroneous > > assumption being made by some initiator implementers that within a > > single connection, commands can be put on the wire out of order. > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > now allow: "A value of 0 indicates no limit." > > > > Is this useful? Does it buy anything? > > > > The difficulties implementers are having with this are: > > > > 1) It is a special case. > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > 3) It violates the min/max function normally used for the key. > > 4) There is always a limit anyway. > > > > Consider FirstBurstSize, which can have a value that is described > > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > > is selected. > > > > I one side offers 0 to mean unlimited, and the other side > > has a limit, it will reply with that limit, say 128 Kbytes. > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > The statement in the standard that "the minimum of the 2 numbers is > > selected" is therefore wrong when one of the numbers is 0. > > > > Furthermore, when an initiator or target receives an offer for one > > of these keys, it cannot simply check that the offered value is > > legal by testing it against some minimum and maximum. It must first > > check for 0 and then only if that check shows the value is non-zero > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > Finally, there is always a limit. If nothing else it is the > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > requesting the transfer. It is useless to specify a FirstBurstSize > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > the largest possible DataSegmentLength in any PDU that can use > > this value is 2**24-1. > > > > The suggestion is to just eliminate this special case of 0 and > > require > > that the range 64-to-(2**24-1) be used instead -- it has exactly the > > same effect in all cases, it is easier to describe in the standard > > because it avoids all the extra words, and it is easier to code > > because it avoids all the special cases. > > > > NOTE: the standard should specify the limit in the ranges for > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > DataSegmentLength field and therefore can never be used. > > > > 5. This is a suggestion for a minor rewording in the standard to avoid > > misunderstandings. > > > > In Appendix E on page 188 of draft 8 it says: > > > > "The response to this command is a text response containing a list > > of > > targets and their addresses. Each target is returned as a target > > record. A target record begins with the TargetName text key, > > followed by a list of TargetAddress text keys, ..." > > > > In fact, there are situations where there are no targets and/or no > > addresses. These situations are clearly defined in the draft after > > the sentences quoted above, but it would help if those sentences > > at least hinted at the possibility that the lists could be empty > > or might not contain addresses. A possible rewording would be: > > > > "The response to this command is a text response containing a list > > of > > zero or more targets and, optionally, their addresses. Each target > > is returned as a target record. A target record begins with the > > TargetName text key, followed by a list of zero or more > > TargetAddress text keys, ..." > > > > > > 6. This is a suggestion for another very minor rewording in the > > standard. > > > > At the end of section 2.2.3 on page 29 of draft 8 it says: > > > > "Before full feature phase is established, only Login PDUs are > > allowed. ..." > > > > The suggested rewording is: > > > > "Before full feature phase is established, only Login Request and > > Login Response PDUs are allowed. ..." > > From owner-ips@ece.cmu.edu Fri Nov 2 14:19:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA15764 for ; Fri, 2 Nov 2001 14:19:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2IKJ328612 for ips-outgoing; Fri, 2 Nov 2001 13:20:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2IKHl28607 for ; Fri, 2 Nov 2001 13:20:17 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id DF16F307 for ; Fri, 2 Nov 2001 13:20:16 -0500 (EST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id KAA04104 for ; Fri, 2 Nov 2001 10:21:43 -0800 (PST) Message-Id: <200111021821.KAA04104@core.rose.hp.com> Date: Fri, 02 Nov 2001 10:32:30 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: iSCSI: new state diagrams for rev09 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit All: Consequent to the feedback I received both online and offline, iSCSI state diagrams are considerably revamped. Thanks to all those who provided the feedback. The new state diagrams are posted at: http://storage-arch.hp.com/iscsi_states_rev09.pdf The ASCII forms of the above are currently part of rev 08-90 draft that Julian posted on his web. The major changes are - - The initiator and target state machines are split for both standard connection diagram and session state diagram. It is hoped that this separation makes it easier to follow. - Several (informative, but) non-essential states were eliminated, since they were waiting for internal events (like acquiring resources) that are likely non-existent in good implementations. That leaves us with a 7-state machine for both initiator and target. - The reduction of states has a salutary effect on the ASCII representation. I was able to translate all the diagrams into ASCII! - The state descriptions are more descriptive now, specifically calling out the event(s) that the state machine is waiting for in that state. - The transition descriptions are also more descriptive, listing the set of events that causes the given transition for each of the roles. - The "connection recovery state diagram" has been renamed as "connection cleanup state diagram" since the word "recovery" was inadvertently overloaded. Connection recovery (the act of reassigning the task allegiance to a different connection) happens outside the scope of this state machine depending on the operational ErrorRecoveryLevel - all this diagram specifies is the dynamics of gracefully closing an active iSCSI connection, perhaps replacing with a new connection. - Consequent to the above and for general clarity, some states have been renamed to reflect the "wait reason" better. - Certain missing/incorrect transitions are now fixed - like the "close the session" Logout handling etc. - I added some text on the first slide that describes the design philosophy behind the current model, perhaps also can be called design assumptions. I have only one (likely) open issue on my list. I received feedback from one reviewer that the _description_ of the diagrams is not "traditional" - for example, the current tabular format captures the transitions (each caused by a set of events) in a state-state matrix, but does not describe them in terms of an event-state matrix for each state (which would run into several pages). The current description scheme seemed quite acceptable to myself (and also to Julian among others) - but I am open on this. Please comment if you strongly feel about this issue. Thanks! -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Fri Nov 2 15:59:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18016 for ; Fri, 2 Nov 2001 15:59:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2JfoP05643 for ips-outgoing; Fri, 2 Nov 2001 14:41:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2Jfml05637 for ; Fri, 2 Nov 2001 14:41:48 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA39548 for ; Fri, 2 Nov 2001 14:39:12 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA2Jfjj208464 for ; Fri, 2 Nov 2001 12:41:45 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: is TargetName always mandatory or not? To: "Julian Satran" Cc: "Jim Hafner" , ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Fri, 2 Nov 2001 11:40:50 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/02/2001 12:41:45 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I agree with Jim Hafner on this issue. The Draft should say: "The TargetName is required on the first login message on every connection (in all cases, new session or adding connections to existing sessions)." Also I think we need to change section 3.12.9, which currently says: "The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator". Should be changed to: "The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific Target (Not necessarily Unique within the iSCSI Target Network Entity)." . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Jim Hafner/Almaden/IBM@IBMUS@ece.cmu.edu on 11/02/2001 08:04:03 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: is TargetName always mandatory or not? Folks, I think this thread may be exposing either an ambiguity in the draft or perhaps a gap. I think it's possible to interpret the requirement that TargetName be supplied (for normal sessions) only on the *first login message on the first connection of a new session* and is not required on the *first login message of subsequent connections intended for the same session". My personal interpretation is that the TargetName is required on the *first login message on every connection (in all cases, new session or adding connections to existing sessions)*. I also think Andre's example of a network entity containing many iSCSI targets with shared network portals (and the scoping of portal group tags and TSIDs by iSCSI target) points out the need for the more restrictive requirement (and the rest of my response to Andre assumed this). But I've heard other comments that can be interpreted as the less restrictive requirement is what is intended. So, I suggest (and I don't think this is asking much) that the requirement be that TargetName be supplied on the first Login message of every connection (not just the leading one). Jim Hafner From owner-ips@ece.cmu.edu Fri Nov 2 17:17:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA20176 for ; Fri, 2 Nov 2001 17:17:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2LJig14028 for ips-outgoing; Fri, 2 Nov 2001 16:19:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2LJgl14023 for ; Fri, 2 Nov 2001 16:19:42 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 5B71231CD for ; Fri, 2 Nov 2001 14:19:41 -0700 (MST) Received: from acropora.rose.agilent.com (acropora.rose.agilent.com [130.30.179.5]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 6E6FF66D for ; Fri, 2 Nov 2001 14:19:40 -0700 (MST) Received: (from dbs@localhost) by acropora.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) id NAA17335 for ips@ece.cmu.edu; Fri, 2 Nov 2001 13:18:35 -0800 (PST) From: Dave Sheehy Message-Id: <200111022118.NAA17335@acropora.rose.agilent.com> Subject: Re: iSCSI: current UNH Plugfest To: ips@ece.cmu.edu (IETF IP SAN Reflector) Date: Fri, 02 Nov 2001 13:18:34 PST X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk > 3. Can commands be sent out of order on the same connection? > > The behavior of targets is clearly specified in Section 2.2.2.3 on > page 25 of draft 8, which says: > "Except for the commands marked for immediate delivery the iSCSI > target layer MUST eliver the commands for execution in the order > specified by CmdSN." > > Section 2.2.2.3 on page 26 of draft 8 also says: > "- CmdSN - the current command Sequence Number advanced by 1 on > each command shipped except for commands marked for immediate > delivery." > but the meaning of the term "shipped" is vague, and does not > necessarily > require that the PDUs arrive on the other end of a TCP connection > in the same order that the CmdSN values were assigned to these PDUs. > > Some initiators have been designed to send commands out of CmdSN > order on one connection. Consider the situation where there is only > one connection and a high-level dispatcher creates a PDU for a SCSI > command that involves writing immediate data to the target. This PDU > is enqueued to a lower-level layer which has to setup, start, and > wait-for a DMA operation to move the immediate data into an onboard > buffer before the PDU can be put onto the wire. While this is > happening, the dispatcher creates another unrelated PDU for a SCSI > read command (for example), and when this PDU is passed to the > lower-level layer it can be sent immediately, ahead of the previous > write PDU and therefore out of order on this connection. > > The standard clearly allows this to happen if the two PDUs were sent > on different connections, and seems to imply that this can also happen > when the two PDUs are sent on the same connection. > > The suggestion is to put in the standard an explicit statement that > this is allowed or not allowed, as appropriate. > > If this is allowed, such a statement would avoid the erroneous > assumption being made by some target implementers that within a single > connection, commands will arrive in order. > > If this is not allowed, such a statement would avoid the erroneous > assumption being made by some initiator implementers that within a > single connection, commands can be put on the wire out of order. > > +++ > > will add an explicit statement saying that this behaviour is forbidden. > 2.2.2.1 will contain: > > On any given connection, the iSCSI initiator MUST send the commands in the > order specified by CmdSN. > > +++ Why do you feel this behavior should be forbidden? Targets already have to order commands across the session. I don't see why it's a problem to extend that to the connection as well. I, for one, believe we should take a liberal stance on this. Dave Sheehy From owner-ips@ece.cmu.edu Fri Nov 2 17:20:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA20230 for ; Fri, 2 Nov 2001 17:20:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2La8815357 for ips-outgoing; Fri, 2 Nov 2001 16:36:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2La4l15345 for ; Fri, 2 Nov 2001 16:36:04 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA332154 for ; Fri, 2 Nov 2001 15:33:05 -0600 Received: from d04nms41.raleigh.ibm.com (d04nms41.raleigh.ibm.com [9.67.228.19]) by southrelay02.raleigh.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA2LZdb199830 for ; Fri, 2 Nov 2001 16:35:39 -0500 Subject: portal group is a component of ___? To: "Jim Hafner" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Andre Asselin" Date: Fri, 2 Nov 2001 16:36:41 -0500 X-MIMETrack: Serialize by Router on D04NMS41/04/M/IBM(Release 5.0.8 |June 18, 2001) at 11/02/2001 04:35:38 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, First, let me say thanks-- this thread has educated me immensly on some of the subtle aspects of iSCSI. >Your picture isn't quite right. The portal group tag is relative to the iSCSI target and that target is *uniquely* identified by it's name. So, in your picture, there are two targets (not one). I think you mean portal groups, not targets. >Each can label its target portal group any way it wants (independent of the other). Each has independent control over the TSIDs it uses. Each may *share* use of a network portal, but that's a different issue. The model implies two independent targets (even if they live in the same network entity and share resources) in your scenario. In other words, the portal groups are wrt targets not the network entity. Thanks for the clarification-- I was missing this understanding of the architecture. > In your scenario, whatever layer (you put it in the network code) has to decide what session to add the connection to has the initiator name, the ISID, the TSID *and* the target name (you left this out) in the login request. That fully qualifies both the target and the session as well. I concur that that is how a target * should * identify a session, however, that's not what the spec currently says, and needs to be updated. Now, if you can put up with one more question, is a network portal a component of a network entity or an iSCSI node? 2.5.1 (c) says its a component of a network entity, but 2.5.1(d) says A Portal Group defines a set of Network Portals within an iSCSI Node that collectively supports the capability of coordinating a session with connections spanning these portals. which is kind of ambiguous as to whether it means portal group is within an iSCSI node, or the network portals are within the iSCSI node. I'd suggest this to clarify it: A Portal Group is a component of an iSCSI node that defines a set of Network Portals that collectively supports the capability of coordinating a session with connections spanning these portals. Also if a network portal really is a component of a network entity, shouldn't the diagram on the following page be redrawn? (original) ----------------------------IP Network--------------------- | | | +--------|---------------|--------------------|---------------------+ | +----|---------------|-----+ +----|---------+ | | | +---------+ +---------+ | | +---------+ | | | | | Network | | Network | | | | Network | | | | | | Portal | | Portal | | | | Portal | | | | | +--|------+ +---------+ | | +---------+ | | | | | | | | | | | | | | Portal | | | | Portal | | | | | Group 1 | | | | Group 2 | | | +--------------------------+ +--------------+ | | | | | | | +----------------------------+ +-----------------------------+ | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | +----------------------------+ +-----------------------------+ | | | | iSCSI Target Node | | (within Network Entity, not shown) | +-------------------------------------------------------------------+ (corrected?) ----------------------------IP Network--------------------- | | | +----------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +--------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +------|-------+ | | | | | Portal | | Portal | | | | | | Group 1 | | Group 2 | | | | | +--------------------------+ +--------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-------------------------------------------------------------------+ | | | | Network Entity | +-----------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: Andre Asselin/Raleigh/IBM@IBMUS 11/01/2001 cc: ips@ece.cmu.edu 03:39 PM From: Jim Hafner/Almaden/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, Your picture isn't quite right. The portal group tag is relative to the iSCSI target and that target is *uniquely* identified by it's name. So, in your picture, there are two targets (not one). Each can label its target portal group any way it wants (independent of the other). Each has independent control over the TSIDs it uses. Each may *share* use of a network portal, but that's a different issue. The model implies two independent targets (even if they live in the same network entity and share resources) in your scenario. In other words, the portal groups are wrt targets not the network entity. In your scenario, whatever layer (you put it in the network code) has to decide what session to add the connection to has the initiator name, the ISID, the TSID *and* the target name (you left this out) in the login request. That fully qualifies both the target and the session as well. Jim Hafner Andre Asselin 11/01/2001 12:15 pm To: Jim Hafner/Almaden/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: Database 'Jim Hafner', View '($Inbox)') Jim, I agree with what you say, except for the part that mapping TSID=target portal group tag will work. Let's assume the following architecture: I have a network entity with one network portal (and thus one portal group). Inside this entity lives two iSCSI targets. Scenerio: An iSCSI initiator opens a connection to the network portal in order to add a connection to an already existing session (the network entity's networking code knows this because TSID in the initial login request is 0). The networking code needs to determine what session the initiator wants to add to, so it compares some items from the initial login request to each of the established sessions until it finds a match. The question is what items should it compare to identify a match? Section 3.12.9 of the spec reads: The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. As you say, this is clearly target centric (because, for example, an initiator could have 2 different sessions open to two different targets that have the same TSID). But from a target point of view, what that text means to me that the network entity's networking code should compare ISID + InitiatorName + TSID to determine a match. That implies that TSID must be unique per TargetName and per portal group ID. If TSID is just the target portal group tag, then the comparison wouldn't work. For example, using the architecture above where there is just one portal group, the target could have two sessions open: session A (InitiatorName=foo, ISID=1, TargetName=bar, TSID=0) and session B (InitiatorName=foo, ISID=1, TargetName=foobar, TSID=0). If it receives a login request with (InitiatorName=foo, ISID=1, TSID=0), which session is it for? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 06:33 PM cc: ips@ece.cmu.edu From: Jim Hafner/Almaden/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, First, your comment "SSID + InitiatorName must be enough to uniquely identify a session" is target-centric. It would be different from the initiator's viewpoint. However, from the target's viewpoint, the target name is implicit and from the initiator viewpoint, the initiator name is implicit, so globally, the triple of the two names + SSID (made up of the ISID and TSID) form the identifier of the session. Locally (between two specific guys), the names are implicit so only the SSID is required. It's all a matter of point of view! As for the difference in identifiers, as I mentioned in the private note, the session is an iSCSI construct and is identified by iSCSI things. The nexus is a SCSI thing and is identified by SCSI constructs (based on how we've mapped the iSCSI things to SCSI things). However, you've brought to the fore again a related question: "what value does the TSID provide to the protocol?" I'm not going to answer that, but I will note that one target implementation that (I think) works is that the TSID = target portal group tag. The other thing to ask is "what value is there in the nexus identifier?" This is never really used in SCSI at all, so it's not a critical issue what composes it. However, it is important (IMO) that the SCSI ports have names and the logical derivative of that statement is that the nexus is identified by the concatenation of the SCSI port names (hence the definition we have). Jim Hafner Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 03:00:53 pm Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu Subject: Re: is TargetName always mandatory or not? John, WHOOPS! I was wrong; you are absolutely right that the spec says "TargetName" and not "TSID". When I was reading through it, I saw "TargetName", but read to myself "TSID". Apologies... In my defense (confusion?), however, 3.12.9 says TSID rather than TargetName is used to uniquely identify a session. Going by that, SSID + InitiatorName must be enough to uniquely identify a session. Jim Hafner pointed out to me that the I_T nexus is identified by one thing and the session is identified by another. If the two must have a 1-1 mapping in iSCSI, why are there two different identifiers? Why not just use the current definition of the I_T nexus to uniquely identify both the nexus and session (i.e. get rid of TSID)? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 04:42 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, I looked again through the document and No where could I find a statement that you claimed as "a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag". It is the InitiatorName, ISID, TSID, with the TargetName and PortalGroup. Please point out to me in the Spec (8 or above), where I can find your statement on I_T Nexus. I did find the following (please ignore for this conversation the "i" and 't" stuff): "- Session: The group of TCP connections that link an initiator with a target, form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one "target image". " " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + 'i'+ ISID, iSCSI Target Name + 't'+ Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). " I have not found a place where Session ID is tied to the TSID. Hence my comment that we also need to have the TargetName in the Initial Login on all Connections. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 10:40:55 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? John & Julian, How about this for the section 5 text: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The initial login request of the leading Login MAY also include the SessionType key=value pair, in which case if the SessionType is not "discovery", then the initial login request MUST also include the key=value pair TargetName. John, I disagree with your second statement: I don't see any way you could have 2 different sessions established within the same portal group with the same InitiatorName, ISID, and TSID. The spec. says that a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag. There is no mention of TargetName contributing to the identificaiton of a session. In my opinion, a non-leading connection should NOT have the TargetName, since it doesn't contribute anything. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: ips@ece.cmu.edu Sent by: Subject: Re: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 10/31/2001 04:20 AM Julian, I think the TargetName should be included on the Initial Login Request on the Leading Login. It seem strange to permit the Authentication functions to proceed if perhaps the intended Target is different then the one doing the Authentication. The way it currently is written, you could pass all the Security test and then find out just before going into Full Function Phase, that it was intended for some other Target. Seems like a waste. My I think that TargetName should also be on all connections on the Initial Login Request. Here is my thinking: The SSID (ISID+TSID) is unique only with regards to a Specific Initiator and Target Node Pair. It is therefore not clear that just knowing the SSID and InitiatorName is enough to understand what session the subsequent connections are being attached to. And it is possible that the CID could be also overlapped with another session. Therefore, I think it make since to determine all this on the initial Login on every Connection, so you know at the beginning what values can be negotiated, or that are being set to the right Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/30/2001 11:33:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: is TargetName always mandatory or not? It is the leading login: The section 5 paragraph will read: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. Julo Andre Asselin/Raleigh/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 31-10-01 02:08 Please respond to Andre Asselin To: "IPS Reflector (E-mail)" cc: Subject: is TargetName always mandatory or not? In section 5 of the spec, it says "If the SessionType is not "discovery" then the initial Login Request MUST also include the key=value pair TargetName.". However, in Appendix D, the description for TargetName says it is Leading Only. Should TargetName not be Leading Only, or should the text in section 5 say that TargetName must be in the initial Login Request of a leading connection? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Fri Nov 2 17:20:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA20243 for ; Fri, 2 Nov 2001 17:20:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2LwEn17086 for ips-outgoing; Fri, 2 Nov 2001 16:58:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2LwBl17074 for ; Fri, 2 Nov 2001 16:58:11 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel12.hp.com (Postfix) with ESMTP id B66AB1FAFA; Fri, 2 Nov 2001 13:57:56 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id NAA04949; Fri, 2 Nov 2001 13:57:52 -0800 (PST) Message-ID: <3BE318DB.DC5C65C7@cup.hp.com> Date: Fri, 02 Nov 2001 14:06:19 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Dave Sheehy Cc: IETF IP SAN Reflector Subject: Re: iSCSI: current UNH Plugfest References: <200111022118.NAA17335@acropora.rose.agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I agree with Dave. I think a wire protocol is stepping out of bounds in imposing this kind of restrictions on implementations. As long as an end to end ordering scheme exists and is mandatory to implement/use, I don't see why ordering within the initiator stack must be mandated ? - Santosh Dave Sheehy wrote: > > > 3. Can commands be sent out of order on the same connection? > > > > The behavior of targets is clearly specified in Section 2.2.2.3 on > > page 25 of draft 8, which says: > > "Except for the commands marked for immediate delivery the iSCSI > > target layer MUST eliver the commands for execution in the order > > specified by CmdSN." > > > > Section 2.2.2.3 on page 26 of draft 8 also says: > > "- CmdSN - the current command Sequence Number advanced by 1 on > > each command shipped except for commands marked for immediate > > delivery." > > but the meaning of the term "shipped" is vague, and does not > > necessarily > > require that the PDUs arrive on the other end of a TCP connection > > in the same order that the CmdSN values were assigned to these PDUs. > > > > Some initiators have been designed to send commands out of CmdSN > > order on one connection. Consider the situation where there is only > > one connection and a high-level dispatcher creates a PDU for a SCSI > > command that involves writing immediate data to the target. This PDU > > is enqueued to a lower-level layer which has to setup, start, and > > wait-for a DMA operation to move the immediate data into an onboard > > buffer before the PDU can be put onto the wire. While this is > > happening, the dispatcher creates another unrelated PDU for a SCSI > > read command (for example), and when this PDU is passed to the > > lower-level layer it can be sent immediately, ahead of the previous > > write PDU and therefore out of order on this connection. > > > > The standard clearly allows this to happen if the two PDUs were sent > > on different connections, and seems to imply that this can also happen > > when the two PDUs are sent on the same connection. > > > > The suggestion is to put in the standard an explicit statement that > > this is allowed or not allowed, as appropriate. > > > > If this is allowed, such a statement would avoid the erroneous > > assumption being made by some target implementers that within a single > > connection, commands will arrive in order. > > > > If this is not allowed, such a statement would avoid the erroneous > > assumption being made by some initiator implementers that within a > > single connection, commands can be put on the wire out of order. > > > > +++ > > > > will add an explicit statement saying that this behaviour is forbidden. > > 2.2.2.1 will contain: > > > > On any given connection, the iSCSI initiator MUST send the commands in the > > order specified by CmdSN. > > > > +++ > > Why do you feel this behavior should be forbidden? Targets already have to > order commands across the session. I don't see why it's a problem to extend > that to the connection as well. I, for one, believe we should take a liberal > stance on this. > > Dave Sheehy -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Fri Nov 2 17:25:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA20364 for ; Fri, 2 Nov 2001 17:25:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2Lrvw16742 for ips-outgoing; Fri, 2 Nov 2001 16:53:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2Lrnl16733 for ; Fri, 2 Nov 2001 16:53:50 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA343110 for ; Fri, 2 Nov 2001 15:50:27 -0600 Received: from d04nms41.raleigh.ibm.com (d04nms41.raleigh.ibm.com [9.67.228.19]) by southrelay02.raleigh.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA2Lr1b104962 for ; Fri, 2 Nov 2001 16:53:01 -0500 Subject: RE: is TargetName always mandatory or not? To: "John Hufferd" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Andre Asselin" Date: Fri, 2 Nov 2001 16:54:04 -0500 X-MIMETrack: Serialize by Router on D04NMS41/04/M/IBM(Release 5.0.8 |June 18, 2001) at 11/02/2001 04:53:00 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk John, The wording you have below, I assume that is for the description of TargetName in Appendix D, right? If so, since TargetName isn't required for a discovery session, could I suggest this wording instead? This key must be provided by the initiator of the TCP connection to the remote endpoint in the first login request if the initiator is not establishing a discovery session. Also, there are a few other places in the spec that need to be updated to be consistent with TargetName being required in the leading login of all normal sessions. The current version of the 6th paragraph in chapter 5 reads: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. A suggested rewrite would be (building on the text suggested by Bob Russell): All initial Login requests MUST include the InitiatorName key=value pair. If the initial Login request is also a leading Login (TSID=0) and the new session is to be a discovery session, then the initial Login request MUST also include the SessionType=discovery key=value pair. If the initial Login request is a leading Login and the new session is to be a normal session, then the initial Login request MUST also include the TargetName key=value pair and MAY also include the SessionType=normal key=value pair. All initial Login requests that are not also a leading Login (TSID != 0) MUST include the TargetName key=value pair. Also, this text appears in 2.2.7: The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session. The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. A suggested rewrite would be: The initiator must present its iSCSI Initiator Name in the first login request. If the initiator is not establishing a discovery session (see 2.4), it also must present the iSCSI Target Name to which it wishes to connect in the first login request. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. See chapter 5 for a more detailed description of the login process. Rationale for making the MUST lowercase: I think specs should state the MUSTs, MAYs, and SHOULDs in only one place so that a requirement isn't stated multiple times in slightly different language. If the powers that be disagree, then feel free to ignore my opinion... Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu Sent by: Subject: RE: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 11/02/2001 02:40 PM Julian, I agree with Jim Hafner on this issue. The Draft should say: "The TargetName is required on the first login message on every connection (in all cases, new session or adding connections to existing sessions)." Also I think we need to change section 3.12.9, which currently says: "The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator". Should be changed to: "The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific Target (Not necessarily Unique within the iSCSI Target Network Entity)." . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Jim Hafner/Almaden/IBM@IBMUS@ece.cmu.edu on 11/02/2001 08:04:03 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: is TargetName always mandatory or not? Folks, I think this thread may be exposing either an ambiguity in the draft or perhaps a gap. I think it's possible to interpret the requirement that TargetName be supplied (for normal sessions) only on the *first login message on the first connection of a new session* and is not required on the *first login message of subsequent connections intended for the same session". My personal interpretation is that the TargetName is required on the *first login message on every connection (in all cases, new session or adding connections to existing sessions)*. I also think Andre's example of a network entity containing many iSCSI targets with shared network portals (and the scoping of portal group tags and TSIDs by iSCSI target) points out the need for the more restrictive requirement (and the rest of my response to Andre assumed this). But I've heard other comments that can be interpreted as the less restrictive requirement is what is intended. So, I suggest (and I don't think this is asking much) that the requirement be that TargetName be supplied on the first Login message of every connection (not just the leading one). Jim Hafner From owner-ips@ece.cmu.edu Fri Nov 2 17:32:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA20503 for ; Fri, 2 Nov 2001 17:32:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA2M03U17246 for ips-outgoing; Fri, 2 Nov 2001 17:00:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA2M01l17241 for ; Fri, 2 Nov 2001 17:00:01 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA23034 for ; Fri, 2 Nov 2001 16:57:25 -0500 Received: from d03nm041.boulder.ibm.com (d03nm041.boulder.ibm.com [9.99.140.41]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA2Lxxj19094 for ; Fri, 2 Nov 2001 14:59:59 -0700 Importance: Normal Subject: Re: portal group is a component of ___? To: "Andre Asselin" Cc: ips@ece.cmu.edu From: "Jim Hafner" Date: Fri, 2 Nov 2001 13:59:57 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM041/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/02/2001 01:59:58 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Andre, The network portals are components of the network entity (these are possibly shared resources). The portal groups are components of the iSCSI Node. One portal group for one iSCSI Node may span an overlapping but non-equal set of network portals as another portal group for a different iSCSI Node within the same network entity. In effect, the portal groups are possibly logical constructs within a node. This is implied by the current language of the model because session coordination is a function of an iSCSI node and portal groups are collections of network portals over which the session coordinator can handle multi-connection sessions. BUT, I agree that your words: A Portal Group is a component of an iSCSI node that defines a set of Network Portals that collectively supports the capability of coordinating a session with connections spanning these portals. make this more clear. Julo? Can you make this change (in the definition of Portal Groups in the model clause)? In your note attached, you respond to my first comment with "you mean portal groups, not targets". No, I meant targets! One named "bar" and one named "foobar". They share the same network portal. Each may have a different view of how they lay portal groups across the shared (or not shared) network portals. But I think we have this straight now. In the previous thread, and concurred by John Hufferd, I've already made the claim that the draft is ambiguous about the TargetName use -- and have advocated clarification of the language. As for redrawing the picture... It is very difficult in ASCII to draw a picture that shows the complexity of possible ways of putting all these pieces together. It's even difficult (as John will attest) to put them in a fancy graphics picture. Also, the intent of the drawing is to encapsulate what ONE target sees. Perhaps your change that the network portals are "attached to" but not "contained in" portal groups is better and more consistent with the first picture in the model clause. I have no objection to this change either. (Julo?). Thanks for the contributions! Jim Hafner Andre Asselin 11/02/2001 01:36 pm To: Jim Hafner/Almaden/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: portal group is a component of ___? (Document link: Database 'Jim Hafner', View '($Inbox)') Jim, First, let me say thanks-- this thread has educated me immensly on some of the subtle aspects of iSCSI. >Your picture isn't quite right. The portal group tag is relative to the iSCSI target and that target is *uniquely* identified by it's name. So, in your picture, there are two targets (not one). I think you mean portal groups, not targets. >Each can label its target portal group any way it wants (independent of the other). Each has independent control over the TSIDs it uses. Each may *share* use of a network portal, but that's a different issue. The model implies two independent targets (even if they live in the same network entity and share resources) in your scenario. In other words, the portal groups are wrt targets not the network entity. Thanks for the clarification-- I was missing this understanding of the architecture. > In your scenario, whatever layer (you put it in the network code) has to decide what session to add the connection to has the initiator name, the ISID, the TSID *and* the target name (you left this out) in the login request. That fully qualifies both the target and the session as well. I concur that that is how a target * should * identify a session, however, that's not what the spec currently says, and needs to be updated. Now, if you can put up with one more question, is a network portal a component of a network entity or an iSCSI node? 2.5.1 (c) says its a component of a network entity, but 2.5.1(d) says A Portal Group defines a set of Network Portals within an iSCSI Node that collectively supports the capability of coordinating a session with connections spanning these portals. which is kind of ambiguous as to whether it means portal group is within an iSCSI node, or the network portals are within the iSCSI node. I'd suggest this to clarify it: A Portal Group is a component of an iSCSI node that defines a set of Network Portals that collectively supports the capability of coordinating a session with connections spanning these portals. Also if a network portal really is a component of a network entity, shouldn't the diagram on the following page be redrawn? (original) ----------------------------IP Network--------------------- | | | +--------|---------------|--------------------|---------------------+ | +----|---------------|-----+ +----|---------+ | | | +---------+ +---------+ | | +---------+ | | | | | Network | | Network | | | | Network | | | | | | Portal | | Portal | | | | Portal | | | | | +--|------+ +---------+ | | +---------+ | | | | | | | | | | | | | | Portal | | | | Portal | | | | | Group 1 | | | | Group 2 | | | +--------------------------+ +--------------+ | | | | | | | +----------------------------+ +-----------------------------+ | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | +----------------------------+ +-----------------------------+ | | | | iSCSI Target Node | | (within Network Entity, not shown) | +-------------------------------------------------------------------+ (corrected?) ----------------------------IP Network--------------------- | | | +----------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +--------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +------|-------+ | | | | | Portal | | Portal | | | | | | Group 1 | | Group 2 | | | | | +--------------------------+ +--------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-------------------------------------------------------------------+ | | | | Network Entity | +-----------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: 11/01/2001 cc: 03:39 PM From: Subject: (Document link: Andre Asselin) Andre, Your picture isn't quite right. The portal group tag is relative to the iSCSI target and that target is *uniquely* identified by it's name. So, in your picture, there are two targets (not one). Each can label its target portal group any way it wants (independent of the other). Each has independent control over the TSIDs it uses. Each may *share* use of a network portal, but that's a different issue. The model implies two independent targets (even if they live in the same network entity and share resources) in your scenario. In other words, the portal groups are wrt targets not the network entity. In your scenario, whatever layer (you put it in the network code) has to decide what session to add the connection to has the initiator name, the ISID, the TSID *and* the target name (you left this out) in the login request. That fully qualifies both the target and the session as well. Jim Hafner Andre Asselin 11/01/2001 12:15 pm To: Jim Hafner/Almaden/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: Database 'Jim Hafner', View '($Inbox)') Jim, I agree with what you say, except for the part that mapping TSID=target portal group tag will work. Let's assume the following architecture: I have a network entity with one network portal (and thus one portal group). Inside this entity lives two iSCSI targets. Scenerio: An iSCSI initiator opens a connection to the network portal in order to add a connection to an already existing session (the network entity's networking code knows this because TSID in the initial login request is 0). The networking code needs to determine what session the initiator wants to add to, so it compares some items from the initial login request to each of the established sessions until it finds a match. The question is what items should it compare to identify a match? Section 3.12.9 of the spec reads: The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. As you say, this is clearly target centric (because, for example, an initiator could have 2 different sessions open to two different targets that have the same TSID). But from a target point of view, what that text means to me that the network entity's networking code should compare ISID + InitiatorName + TSID to determine a match. That implies that TSID must be unique per TargetName and per portal group ID. If TSID is just the target portal group tag, then the comparison wouldn't work. For example, using the architecture above where there is just one portal group, the target could have two sessions open: session A (InitiatorName=foo, ISID=1, TargetName=bar, TSID=0) and session B (InitiatorName=foo, ISID=1, TargetName=foobar, TSID=0). If it receives a login request with (InitiatorName=foo, ISID=1, TSID=0), which session is it for? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 06:33 PM cc: ips@ece.cmu.edu From: Jim Hafner/Almaden/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, First, your comment "SSID + InitiatorName must be enough to uniquely identify a session" is target-centric. It would be different from the initiator's viewpoint. However, from the target's viewpoint, the target name is implicit and from the initiator viewpoint, the initiator name is implicit, so globally, the triple of the two names + SSID (made up of the ISID and TSID) form the identifier of the session. Locally (between two specific guys), the names are implicit so only the SSID is required. It's all a matter of point of view! As for the difference in identifiers, as I mentioned in the private note, the session is an iSCSI construct and is identified by iSCSI things. The nexus is a SCSI thing and is identified by SCSI constructs (based on how we've mapped the iSCSI things to SCSI things). However, you've brought to the fore again a related question: "what value does the TSID provide to the protocol?" I'm not going to answer that, but I will note that one target implementation that (I think) works is that the TSID = target portal group tag. The other thing to ask is "what value is there in the nexus identifier?" This is never really used in SCSI at all, so it's not a critical issue what composes it. However, it is important (IMO) that the SCSI ports have names and the logical derivative of that statement is that the nexus is identified by the concatenation of the SCSI port names (hence the definition we have). Jim Hafner Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 03:00:53 pm Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu Subject: Re: is TargetName always mandatory or not? John, WHOOPS! I was wrong; you are absolutely right that the spec says "TargetName" and not "TSID". When I was reading through it, I saw "TargetName", but read to myself "TSID". Apologies... In my defense (confusion?), however, 3.12.9 says TSID rather than TargetName is used to uniquely identify a session. Going by that, SSID + InitiatorName must be enough to uniquely identify a session. Jim Hafner pointed out to me that the I_T nexus is identified by one thing and the session is identified by another. If the two must have a 1-1 mapping in iSCSI, why are there two different identifiers? Why not just use the current definition of the I_T nexus to uniquely identify both the nexus and session (i.e. get rid of TSID)? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 04:42 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, I looked again through the document and No where could I find a statement that you claimed as "a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag". It is the InitiatorName, ISID, TSID, with the TargetName and PortalGroup. Please point out to me in the Spec (8 or above), where I can find your statement on I_T Nexus. I did find the following (please ignore for this conversation the "i" and 't" stuff): "- Session: The group of TCP connections that link an initiator with a target, form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one "target image". " " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + 'i'+ ISID, iSCSI Target Name + 't'+ Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). " I have not found a place where Session ID is tied to the TSID. Hence my comment that we also need to have the TargetName in the Initial Login on all Connections. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 10:40:55 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? John & Julian, How about this for the section 5 text: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The initial login request of the leading Login MAY also include the SessionType key=value pair, in which case if the SessionType is not "discovery", then the initial login request MUST also include the key=value pair TargetName. John, I disagree with your second statement: I don't see any way you could have 2 different sessions established within the same portal group with the same InitiatorName, ISID, and TSID. The spec. says that a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag. There is no mention of TargetName contributing to the identificaiton of a session. In my opinion, a non-leading connection should NOT have the TargetName, since it doesn't contribute anything. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: ips@ece.cmu.edu Sent by: Subject: Re: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 10/31/2001 04:20 AM Julian, I think the TargetName should be included on the Initial Login Request on the Leading Login. It seem strange to permit the Authentication functions to proceed if perhaps the intended Target is different then the one doing the Authentication. The way it currently is written, you could pass all the Security test and then find out just before going into Full Function Phase, that it was intended for some other Target. Seems like a waste. My I think that TargetName should also be on all connections on the Initial Login Request. Here is my thinking: The SSID (ISID+TSID) is unique only with regards to a Specific Initiator and Target Node Pair. It is therefore not clear that just knowing the SSID and InitiatorName is enough to understand what session the subsequent connections are being attached to. And it is possible that the CID could be also overlapped with another session. Therefore, I think it make since to determine all this on the initial Login on every Connection, so you know at the beginning what values can be negotiated, or that are being set to the right Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/30/2001 11:33:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: is TargetName always mandatory or not? It is the leading login: The section 5 paragraph will read: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. Julo Andre Asselin/Raleigh/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 31-10-01 02:08 Please respond to Andre Asselin To: "IPS Reflector (E-mail)" cc: Subject: is TargetName always mandatory or not? In section 5 of the spec, it says "If the SessionType is not "discovery" then the initial Login Request MUST also include the key=value pair TargetName.". However, in Appendix D, the description for TargetName says it is Leading Only. Should TargetName not be Leading Only, or should the text in section 5 say that TargetName must be in the initial Login Request of a leading connection? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Fri Nov 2 21:09:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA23058 for ; Fri, 2 Nov 2001 21:09:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA31OZV00884 for ips-outgoing; Fri, 2 Nov 2001 20:24:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls05.mediaone.net (chmls05.mediaone.net [24.147.1.143]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA31OYl00880 for ; Fri, 2 Nov 2001 20:24:34 -0500 (EST) Received: from breinhold (h00e09871f366.ne.mediaone.net [24.128.217.119]) by chmls05.mediaone.net (8.11.1/8.11.1) with SMTP id fA31OTN26527; Fri, 2 Nov 2001 20:24:29 -0500 (EST) From: "Barry Reinhold" To: "Dave Sheehy" , "IETF IP SAN Reflector" Subject: RE: iSCSI: current UNH Plugfest Date: Fri, 2 Nov 2001 20:23:59 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <200111022118.NAA17335@acropora.rose.agilent.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Using features such as out of order command delivery on a connection tend to be the sort of things that lead to interoperability problems. It is unexpected and probably going to hit poorly tested code paths even if the standard is written to allow it. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Dave Sheehy >Sent: Friday, November 02, 2001 4:19 PM >To: IETF IP SAN Reflector >Subject: Re: iSCSI: current UNH Plugfest > > > >> 3. Can commands be sent out of order on the same connection? >> >> The behavior of targets is clearly specified in Section 2.2.2.3 on >> page 25 of draft 8, which says: >> "Except for the commands marked for immediate delivery the iSCSI >> target layer MUST eliver the commands for execution in the order >> specified by CmdSN." >> >> Section 2.2.2.3 on page 26 of draft 8 also says: >> "- CmdSN - the current command Sequence Number advanced by 1 on >> each command shipped except for commands marked for immediate >> delivery." >> but the meaning of the term "shipped" is vague, and does not >> necessarily >> require that the PDUs arrive on the other end of a TCP connection >> in the same order that the CmdSN values were assigned to these PDUs. >> >> Some initiators have been designed to send commands out of CmdSN >> order on one connection. Consider the situation where there is only >> one connection and a high-level dispatcher creates a PDU for a SCSI >> command that involves writing immediate data to the target. This PDU >> is enqueued to a lower-level layer which has to setup, start, and >> wait-for a DMA operation to move the immediate data into an onboard >> buffer before the PDU can be put onto the wire. While this is >> happening, the dispatcher creates another unrelated PDU for a SCSI >> read command (for example), and when this PDU is passed to the >> lower-level layer it can be sent immediately, ahead of the previous >> write PDU and therefore out of order on this connection. >> >> The standard clearly allows this to happen if the two PDUs were sent >> on different connections, and seems to imply that this can also happen >> when the two PDUs are sent on the same connection. >> >> The suggestion is to put in the standard an explicit statement that >> this is allowed or not allowed, as appropriate. >> >> If this is allowed, such a statement would avoid the erroneous >> assumption being made by some target implementers that within a single >> connection, commands will arrive in order. >> >> If this is not allowed, such a statement would avoid the erroneous >> assumption being made by some initiator implementers that within a >> single connection, commands can be put on the wire out of order. >> >> +++ >> >> will add an explicit statement saying that this behaviour is forbidden. >> 2.2.2.1 will contain: >> >> On any given connection, the iSCSI initiator MUST send the >commands in the >> order specified by CmdSN. >> >> +++ > >Why do you feel this behavior should be forbidden? Targets already have to >order commands across the session. I don't see why it's a problem to extend >that to the connection as well. I, for one, believe we should take >a liberal >stance on this. > >Dave Sheehy > From owner-ips@ece.cmu.edu Fri Nov 2 23:02:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA25771 for ; Fri, 2 Nov 2001 23:02:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA33C7O07354 for ips-outgoing; Fri, 2 Nov 2001 22:12:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA2N1kl21809 for ; Fri, 2 Nov 2001 18:01:46 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fA2N1XM12295; Fri, 2 Nov 2001 18:01:33 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fA2N1Wj27102; Fri, 2 Nov 2001 18:01:33 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15331.9681.232000.600964@gargle.gargle.HOWL> Date: Fri, 2 Nov 2001 18:01:37 -0500 From: Paul Koning To: rcohen@eurologic.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest: Reserved bits References: <000101c162d7$330d39c0$0102a8c0@eddylaptop> <15329.38576.650000.176588@gargle.gargle.HOWL> <001901c163ae$f30a6a90$6f07a8c0@COORS> X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 2 November 2001) by Ron Cohen: > The problem with accepting values in reserved bits is that when the reserved > bit is no longer reserved (evolution in the standard) it gives the > impression to the initiator that a target implements the new feature when it > may actually only be ignoring it. Yes, if you don't do some other things that are needed. You MUST have protocol version numbers in the initial messages. Otherwise you are utterly lost. With that, you know what version the other end has. The key point is that requiring reserved fields to be ignored is that you can rely on the fact that new things can be asked for in reserved fields with the knowledge that old implementations will ignore those things. Quite often this is what you want. Occasionally it is not; for those cases you make use of the version number. In any case, what I was describing is long established practice in many protocols that have successfully navigated the version number migration path. paul From owner-ips@ece.cmu.edu Sat Nov 3 22:33:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA22650 for ; Sat, 3 Nov 2001 22:33:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA42eLf04983 for ips-outgoing; Sat, 3 Nov 2001 21:40:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.windriver.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA42eEl04974 for ; Sat, 3 Nov 2001 21:40:19 -0500 (EST) Received: from london ([147.11.45.211]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id SAA14847; Sat, 3 Nov 2001 18:39:08 -0800 (PST) From: "Rod Harrison" To: "Barry Reinhold" , "Dave Sheehy" , "IETF IP SAN Reflector" Subject: RE: iSCSI: current UNH Plugfest Date: Sat, 3 Nov 2001 18:42:13 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Barry, In general I agree but I don't think this is as much of a corner case as it at first appears. Targets will have code very similar to that needed to handle out of order commands to deal with digest errors. Targets also need to queue commands whilst waiting for both solicited and unsolicited data to arrive. Queuing out of order commands seems little extra work. From an initiators point of view there are efficiency, and probably performance gains to be had from sending commands out of order. Bob Russell gave the example of a read being sent whilst write data DMA is happening, and a similar situation can arise with DMA for writes overtaking that of earlier writes if the initiator has multiple DMA engines. In this case the initiator might be forced to let the wire go idle if it can't send the data from completed DMAs as soon as possible. We already have a command queue at the target to enforce correct serialisation of commands, doing the same thing at the initiator is redundant. Finally, I don't believe we should be writing a standard to work around poor coding and test coverage, especially at the cost of potential efficiency gains. I agree with Dave and Santosh that commands being sent out of order on a single session should be allowed by the standard. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Barry Reinhold Sent: Friday, November 02, 2001 5:24 PM To: Dave Sheehy; IETF IP SAN Reflector Subject: RE: iSCSI: current UNH Plugfest Using features such as out of order command delivery on a connection tend to be the sort of things that lead to interoperability problems. It is unexpected and probably going to hit poorly tested code paths even if the standard is written to allow it. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Dave Sheehy >Sent: Friday, November 02, 2001 4:19 PM >To: IETF IP SAN Reflector >Subject: Re: iSCSI: current UNH Plugfest > > > >> 3. Can commands be sent out of order on the same connection? >> >> The behavior of targets is clearly specified in Section 2.2.2.3 on >> page 25 of draft 8, which says: >> "Except for the commands marked for immediate delivery the iSCSI >> target layer MUST eliver the commands for execution in the order >> specified by CmdSN." >> >> Section 2.2.2.3 on page 26 of draft 8 also says: >> "- CmdSN - the current command Sequence Number advanced by 1 on >> each command shipped except for commands marked for immediate >> delivery." >> but the meaning of the term "shipped" is vague, and does not >> necessarily >> require that the PDUs arrive on the other end of a TCP connection >> in the same order that the CmdSN values were assigned to these PDUs. >> >> Some initiators have been designed to send commands out of CmdSN >> order on one connection. Consider the situation where there is only >> one connection and a high-level dispatcher creates a PDU for a SCSI >> command that involves writing immediate data to the target. This PDU >> is enqueued to a lower-level layer which has to setup, start, and >> wait-for a DMA operation to move the immediate data into an onboard >> buffer before the PDU can be put onto the wire. While this is >> happening, the dispatcher creates another unrelated PDU for a SCSI >> read command (for example), and when this PDU is passed to the >> lower-level layer it can be sent immediately, ahead of the previous >> write PDU and therefore out of order on this connection. >> >> The standard clearly allows this to happen if the two PDUs were sent >> on different connections, and seems to imply that this can also happen >> when the two PDUs are sent on the same connection. >> >> The suggestion is to put in the standard an explicit statement that >> this is allowed or not allowed, as appropriate. >> >> If this is allowed, such a statement would avoid the erroneous >> assumption being made by some target implementers that within a single >> connection, commands will arrive in order. >> >> If this is not allowed, such a statement would avoid the erroneous >> assumption being made by some initiator implementers that within a >> single connection, commands can be put on the wire out of order. >> >> +++ >> >> will add an explicit statement saying that this behaviour is forbidden. >> 2.2.2.1 will contain: >> >> On any given connection, the iSCSI initiator MUST send the >commands in the >> order specified by CmdSN. >> >> +++ > >Why do you feel this behavior should be forbidden? Targets already have to >order commands across the session. I don't see why it's a problem to extend >that to the connection as well. I, for one, believe we should take >a liberal >stance on this. > >Dave Sheehy > From owner-ips@ece.cmu.edu Sat Nov 3 23:39:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA23349 for ; Sat, 3 Nov 2001 23:39:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA43kDN08740 for ips-outgoing; Sat, 3 Nov 2001 22:46:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.24.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA43kAl08731 for ; Sat, 3 Nov 2001 22:46:10 -0500 (EST) Received: from sponge.cisco.com (sponge.cisco.com [64.101.128.87]) by sj-msg-core-2.cisco.com (8.11.3/8.9.1) with ESMTP id fA43k4a20747 for ; Sat, 3 Nov 2001 19:46:04 -0800 (PST) Received: from DAPW2K (sjc-vpn2-971.cisco.com [10.21.115.203]) by sponge.cisco.com (Mirapoint) with SMTP id AAC02921; Sat, 3 Nov 2001 21:42:53 -0600 (CST) From: "Dave Peterson" To: "IETF IP SAN Reflector" Subject: RE: iSCSI: current UNH Plugfest Date: Sat, 3 Nov 2001 21:47:31 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit There is no reason why the Initiator needs to send commands in CmdSN order. The target should be able to queue up any out of order commands to the queuing capacity. As stated below the Target is responsible for delivering the commands for execution in the order specified by CmdSN. Dave > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Barry Reinhold > Sent: Friday, November 02, 2001 7:24 PM > To: Dave Sheehy; IETF IP SAN Reflector > Subject: RE: iSCSI: current UNH Plugfest > > > Using features such as out of order command delivery on a > connection tend to > be the sort of things that lead to interoperability problems. It is > unexpected and probably going to hit poorly tested code paths even if the > standard is written to allow it. > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > >Dave Sheehy > >Sent: Friday, November 02, 2001 4:19 PM > >To: IETF IP SAN Reflector > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > >> 3. Can commands be sent out of order on the same connection? > >> > >> The behavior of targets is clearly specified in Section 2.2.2.3 on > >> page 25 of draft 8, which says: > >> "Except for the commands marked for immediate delivery the iSCSI > >> target layer MUST eliver the commands for execution in the order > >> specified by CmdSN." > >> > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >> "- CmdSN - the current command Sequence Number advanced by 1 on > >> each command shipped except for commands marked for immediate > >> delivery." > >> but the meaning of the term "shipped" is vague, and does not > >> necessarily > >> require that the PDUs arrive on the other end of a TCP connection > >> in the same order that the CmdSN values were assigned to these PDUs. > >> > >> Some initiators have been designed to send commands out of CmdSN > >> order on one connection. Consider the situation where there is only > >> one connection and a high-level dispatcher creates a PDU for a SCSI > >> command that involves writing immediate data to the target. > This PDU > >> is enqueued to a lower-level layer which has to setup, start, and > >> wait-for a DMA operation to move the immediate data into an onboard > >> buffer before the PDU can be put onto the wire. While this is > >> happening, the dispatcher creates another unrelated PDU for a SCSI > >> read command (for example), and when this PDU is passed to the > >> lower-level layer it can be sent immediately, ahead of the previous > >> write PDU and therefore out of order on this connection. > >> > >> The standard clearly allows this to happen if the two PDUs were sent > >> on different connections, and seems to imply that this can > also happen > >> when the two PDUs are sent on the same connection. > >> > >> The suggestion is to put in the standard an explicit statement that > >> this is allowed or not allowed, as appropriate. > >> > >> If this is allowed, such a statement would avoid the erroneous > >> assumption being made by some target implementers that > within a single > >> connection, commands will arrive in order. > >> > >> If this is not allowed, such a statement would avoid the erroneous > >> assumption being made by some initiator implementers that within a > >> single connection, commands can be put on the wire out of order. > >> > >> +++ > >> > >> will add an explicit statement saying that this behaviour is forbidden. > >> 2.2.2.1 will contain: > >> > >> On any given connection, the iSCSI initiator MUST send the > >commands in the > >> order specified by CmdSN. > >> > >> +++ > > > >Why do you feel this behavior should be forbidden? Targets > already have to > >order commands across the session. I don't see why it's a > problem to extend > >that to the connection as well. I, for one, believe we should take > >a liberal > >stance on this. > > > >Dave Sheehy > > > From owner-ips@ece.cmu.edu Sun Nov 4 07:45:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA11782 for ; Sun, 4 Nov 2001 07:45:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA4CQeI20206 for ips-outgoing; Sun, 4 Nov 2001 07:26:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA4CQbl20198 for ; Sun, 4 Nov 2001 07:26:37 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA47694 for ; Sun, 4 Nov 2001 13:26:30 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA4CQUa30906 for ; Sun, 4 Nov 2001 13:26:30 +0100 To: ips@ece.cmu.edu Importance: High MIME-Version: 1.0 Subject: iSCSI - Configuration Examples by John Hufferd X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sun, 4 Nov 2001 14:26:29 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 04/11/2001 14:26:30, Serialize complete at 04/11/2001 14:26:30 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, John Hufferd has been kind enough to share with us all his configuration examples. You can see them at: http://www.haifa.il.ibm.com/satran/ips/iSCSIConfigurationExamples.pdf Thanks John, Julo From owner-ips@ece.cmu.edu Sun Nov 4 07:45:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA11793 for ; Sun, 4 Nov 2001 07:45:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA4Bw5N18781 for ips-outgoing; Sun, 4 Nov 2001 06:58:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA4Bw2l18773 for ; Sun, 4 Nov 2001 06:58:02 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id MAA22548; Sun, 4 Nov 2001 12:57:55 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA4BvrP110620; Sun, 4 Nov 2001 12:57:54 +0100 Importance: Normal Subject: RE: iSCSI: IPsec tunnel / transport mode decision To: Cc: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "Ofer Biran" Date: Sun, 4 Nov 2001 13:26:44 +0100 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 04/11/2001 13:57:54 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Saqib, Mandatory transport mode would make bundling of external IPSec impossible, while tunnel mode is not more difficult to implement within the iSCSI endpoint than transport mode. "Cost of ownership and complexity of deploying a stand-alone IPsec gateway" might be among the considerations of vendors and customers, but I don't think the standard should block such solutions (and it blocks more than just stand-alone IPsec gateway). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 02/11/2001 20:59:03 Please respond to To: "Bill Strahm" , "CAVANNA,VICENTE V (A-Roseville,ex1)" cc: "SHEEHY,DAVE (A-Americas,unix1)" , Ofer Biran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: IPsec tunnel / transport mode decision What about the cost of ownership and complexity of deploying a stand-alone IPsec gateway for use with IPsec end-points? If transport-mode IPsec is a must-to-implement capability in iSCSI end-points there is an opportunity to have much more coherent security for iSCSI. Saqib From owner-ips@ece.cmu.edu Sun Nov 4 15:30:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15882 for ; Sun, 4 Nov 2001 15:30:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA4JdkH15275 for ips-outgoing; Sun, 4 Nov 2001 14:39:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA4Jdil15270 for ; Sun, 4 Nov 2001 14:39:44 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA4Jddu22767; Sun, 4 Nov 2001 11:39:39 -0800 Received: from strahmw2k (ras-pc-7-8.sanera.net [172.16.7.8]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA4JdX601468; Sun, 4 Nov 2001 11:39:33 -0800 From: "Bill Strahm" To: "Ofer Biran" , Cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Sun, 4 Nov 2001 11:39:22 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ok, How does mandatory Transport mode remove the possibility of external IPsec... I have said before I can make IPsec transport & tunnel mode work in external devices, just like you can do SSL/TLS accelerators both internally and externally Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Sunday, November 04, 2001 4:27 AM To: saqibj@margallacomm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Saqib, Mandatory transport mode would make bundling of external IPSec impossible, while tunnel mode is not more difficult to implement within the iSCSI endpoint than transport mode. "Cost of ownership and complexity of deploying a stand-alone IPsec gateway" might be among the considerations of vendors and customers, but I don't think the standard should block such solutions (and it blocks more than just stand-alone IPsec gateway). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 02/11/2001 20:59:03 Please respond to To: "Bill Strahm" , "CAVANNA,VICENTE V (A-Roseville,ex1)" cc: "SHEEHY,DAVE (A-Americas,unix1)" , Ofer Biran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: IPsec tunnel / transport mode decision What about the cost of ownership and complexity of deploying a stand-alone IPsec gateway for use with IPsec end-points? If transport-mode IPsec is a must-to-implement capability in iSCSI end-points there is an opportunity to have much more coherent security for iSCSI. Saqib From owner-ips@ece.cmu.edu Mon Nov 5 04:02:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA09395 for ; Mon, 5 Nov 2001 04:02:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA58BUx29095 for ips-outgoing; Mon, 5 Nov 2001 03:11:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA58BSl29087 for ; Mon, 5 Nov 2001 03:11:28 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id JAA42796 for ; Mon, 5 Nov 2001 09:11:21 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA58BFv86634 for ; Mon, 5 Nov 2001 09:11:21 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 5 Nov 2001 10:11:13 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 05/11/2001 10:11:21, Serialize complete at 05/11/2001 10:11:21 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, Comments in text. Thanks, Julo "Robert D. Russell" Sent by: owner-ips@ece.cmu.edu 02-11-01 01:31 Please respond to "Robert D. Russell" To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: current UNH Plugfest Attached are the new issues that arose during the iSCSI plugfest at UNH on Thursday 1-Nov-2001. Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 ---------------------------------------------------------------------- There were no new issues today! Lots of people were busy sending data and interoperating with each other! There were, however, a few suggestions/requests for clarifications in the standard: 1. Since all Login Requests MUST be issued as immediate requests, the diagram in section 3.12 should show bit 6 of byte 0 as "1", not "I", and section 3.12.2 should simply be eliminated. ++++ OK - Julo ++++ 2. The first paragraph of Appendix D should describe the special role of the first Login request on the first connection of a new session. In particular, there are 3 keys that MUST be sent in that Login PDU (InitiatorName, SessionType if it is discovery, TargetName if the session is normal). This is, in fact, yet another classification that is currently included within LO, but LO is too general, since LO allows keys to appear at any time during the leading login phase, and these keys must appear on the very first login PDU in that leading login phase. +++ There is now a text in Section 5 that reads: The initial Login request of the first connection of a session (primary login) MUST include the InitiatorName key=value pair. The primary Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. +++ 3. A target cannot release its resources for a command until it receives an ExpStatSN back from the initiator that acknowledges receipt of the StatSN carried in the SCSI Response to that command. However, if the initiator does not have any more I/O to perform, that ExpStatSN will not come back to the target, and the target may end up holding on to those resources indefinitely. This is a situation where initiator non-action can impact target performance, and could even lead to denial of service attacks if several initiators were to do this simultaneously to the same target. The standard provides the NOP-In/Out mechanism to deal with this: The last paragraph of section 2.2.2 says: During periods when traffic on a connection is unidirectional, iSCSI NOP-Out/In PDUs may be utilized to synchronize the command and status ordering counters of the target and initiator. and section 3.18 says: A NOP-Out may also be used to confirm a changed ExpStatSN if there is no other PDU to carry it for a long time. The question is, is there a recommended mechanism to trigger these NOP-pings? Clearly the target could set a timer, and if ExpStatSN is not received back at the end of the time interval, the target could send a NOP-in as a ping in order to get the initiator to send back a NOP-out with the updated ExpStatSN. Also clearly the initiator could set a timer, and if it has no traffic to send during the time interval, the initiator could send a NOP-out to deliver the updated ExpStatSN to the target. This approach is probably not as reliable as the first, since a) if the initiator does not do it, only the target gets hurt, and b) if the connection drops, the target will never receive the NOP-out. On the other hand, it may be more efficient for the initiator to do this, since a typical target may be dealing with hundreds of connections, and the initiator only a few. Which is the best procedure? Are there other ways to do this? What is the recommended way, or is there a reason for the standard not to recommend a triggering mechanism, in which case, could the standard suggest methods or provide guidance, especially if there are better, less obvious ways to do it? +++ There are no better ways to do it. However we see this a purely an implementation matter. I assume that a target low on resources will try to recover some and ping the initiators. +++ From owner-ips@ece.cmu.edu Mon Nov 5 04:03:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA09414 for ; Mon, 5 Nov 2001 04:03:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA58g2900789 for ips-outgoing; Mon, 5 Nov 2001 03:42:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA58g0l00785 for ; Mon, 5 Nov 2001 03:42:00 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA17558 for ; Mon, 5 Nov 2001 09:41:53 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA58fq323958 for ; Mon, 5 Nov 2001 09:41:52 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: current UNH Plugfest: Reserved bits X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 5 Nov 2001 10:41:51 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 05/11/2001 10:41:52, Serialize complete at 05/11/2001 10:41:52 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk The usual way other standard bodies handle reserved fields are that they are mandated to be 0 but a compliant receiver does not have to check. However most of them don't say this explicitly. However to avoid having "compliance tests" test receivers we might want to say in Section 3: Any compliant sender MUST set to zero all bits not defined and all reserved fields unless specified otherwise. Any compliant receiver MUST ignore any bit not defined and all reserved fields unless specified otherwise. Julo ----- Forwarded by Julian Satran/Haifa/IBM on 05-11-01 10:37 ----- "Ron Cohen" Sent by: owner-ips@ece.cmu.edu 02-11-01 16:59 Please respond to "Ron Cohen" To: "ips" , "Paul Koning" cc: Subject: Re: iSCSI: current UNH Plugfest: Reserved bits The problem with accepting values in reserved bits is that when the reserved bit is no longer reserved (evolution in the standard) it gives the impression to the initiator that a target implements the new feature when it may actually only be ignoring it. Ron ----- Original Message ----- From: "Paul Koning" To: Cc: ; Sent: Thursday, November 01, 2001 6:38 PM Subject: RE: iSCSI: current UNH Plugfest > Excerpt of message (sent 1 November 2001) by Bill Strahm: > > Usually the "Conservative in what you send, Liberal in what you accept" > > policy is used... > > That's a very important principle... > > > In otherwords, The sender MUST set to 0 (or some other value) The receiver > > MUST ignore the value... > > That's the way to express the principle. But the text quoted in the > earlier notes does not express it the same way. > > In particular "... are format errors. This when detected..." implies > that receivers may or may not detect format errors. In other words, > it implies -- but does NOT state explicitly -- that receivers MAY > check reserved fields for zeroness. > > > This allows for some tweaking of the implementation, if I control both ends > > I might set a reserved value to 1, then I know something... If I receive a > > reserve value set to 1 and I don't do anything the other end knows it is not > > talking to itself (this can even be a versioning thing as well) > > > > Now, we need to be VERY careful in defining this, do we plan on having > > Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean... > > is it possible, is it desirable ? will there be extensions ??? > > > > If you can truly answer NO to all of those things, I would argue for > > REMOVING the reserved fields (if possible), if not, the MUST set, MUST > > ignore policy seems better > > I agree strongly. There is a lot of experience in the community on > what helps and what hurts convenient version upgrade. In particular, > there is a LOT of evidence that ANY checking of reserved fields is a > nasty thing. Unless you plan never to go beyond V1, you really need > to require the rule Bill stated, i.e., receivers MUST ignore the > contents of reserved fields -- they are NOT allowed to "verify" them. > > So the spec needs to be clarified to say that random values in > reserved fields are NOT format errors and receivers MUST NOT treat > them as such. > > paul > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Eddy Quicksall > > Sent: Thursday, November 01, 2001 5:15 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > I am reluctant to say this because I think most people think the > > initiator/target must check for correctness ... but, it is my feeling that > > that job should be up to a basher program. The target should not be in the > > business of diagnosing the initiator. The only time a target should check a > > field is when it could crash the system or data. Some format errors may have > > no consequences whatsoever. > > > > Eddy > > > > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Wednesday, October 31, 2001 05:39 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: current UNH Plugfest > > > > > > Attached are the new issues that arose during the iSCSI plugfest > > at UNH on Wednesday 31-Oct-2001. > > > > (Note: these issues do not take into account any modifications or > > clarifications that occured in the standard due to the issues raised > > on Monday or Tuesday.) > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > ------------------------------------------------------------------------ > > ---- > > > > 1. Are receivers (initiator or target) REQUIRED to check that reserved > > bits and/or fields are set to 0? > > > > Section 3 on page 48 of draft 8 says: > > "Any bits not defined MUST be set to zero. Any reserved fields and > > values MUST be 0 unless specified otherwise." > > > > and Section 8.3 on page 127 of draft 8 says: > > "Explicit violations of the PDU layout rules stated in this > > document > > are format errors. This when detected, usually indicates a major > > implementation flaw in one of the parties. > > > > When a target or an initiator receives an iSCSI PDU with a format > > error, it MUST reset all transport connections in the session > > immediately and escalate the format error to session recovery > > (section 8.11.4)." > > > > According to these rules, a PDU with reserved bits and/or fields that > > are not set to 0 violates the PDU layout rules. Therefore, if an > > initiator or target receives such a PDU, it should immediately close > > all connections in the session and go to session recovery. > > > > Clearly a format error has extremely severe consequences! > > > > Although all vendors are setting reserved bits and fields to 0 on > > PDUs they are sending, many are NOT checking PDUs they are receiving > > to see if these bits and fields are set to 0. Basically, vendors are > > saying "who cares if reserved bits and/or fields in incoming PDUs are > > not zero? We do not want to take the time to do this checking, and > > there is no benefit to doing it. As long as the non-reserved bits > > and > > fields are set properly, we can and should proceed. Any time devoted > > to doing this checking is wasted in 99+% of the cases, and in the > > (unlikely) case that a non-zero bit or field is found, the > > consequences are too severe." > > > > There should be some statement in the standard to clarify what > > checking > > is required and what is optional. > > > > 2. A similar situation arises with respect to checking the consistency > > of fields such as Version-max, Version-min and Version-active in > > Login > > Requests and Login Responses. > > > > For example, consider the Version-max field. > > > > Section 3.12.5 says: > > "All Login requests within the Login phase MUST carry the same > > Version-max." > > > > All vendor initiators are setting Version-max correctly on all > > login requests they are sending, but many vendor targets are NOT > > checking received login requests to ensure that this rule is > > enforced. > > In particular, many targets simply use the Version-max and > > Version-min > > on the first login request they receive on a new connection, and then > > they ignore these fields on all subsequent login requests in the same > > login phase. > > > > Strictly speaking, a change in the Version-max field during the login > > phase constitutes a protocol error according to section 8.8 on page > > 130 > > of draft 8: > > > > "All violations of iSCSI PDU exchange sequences specified in this > > draft are also protocol errors. This category of errors can be > > addressed only by fixing the implementations; iSCSI defines Reject > > and response codes to enable this". > > > > Therefore the target should send back a login response with a status > > of 0x0200 and then close the connection. > > > > However, Section 3.12.5 also says: > > "The target MUST use the value presented with the first login > > request." > > > > This rule seems to imply that the value CAN change, because if it > > cannot > > change, then it doesn't matter which one of the login requests it is > > taken from, they are all the same anyway. > > > > The suggestion is to keep the requirement that the target MUST use > > the > > value presented on the first login request, but to allowed the target > > to ignore the value presented on all subsequent login requests in the > > same login phase. A similar rewording should be done for the other > > fields. > > > > 3. Can commands be sent out of order on the same connection? > > > > The behavior of targets is clearly specified in Section 2.2.2.3 on > > page 25 of draft 8, which says: > > "Except for the commands marked for immediate delivery the iSCSI > > target layer MUST eliver the commands for execution in the order > > specified by CmdSN." > > > > Section 2.2.2.3 on page 26 of draft 8 also says: > > "- CmdSN - the current command Sequence Number advanced by 1 on > > each command shipped except for commands marked for immediate > > delivery." > > but the meaning of the term "shipped" is vague, and does not > > necessarily > > require that the PDUs arrive on the other end of a TCP connection > > in the same order that the CmdSN values were assigned to these PDUs. > > > > Some initiators have been designed to send commands out of CmdSN > > order on one connection. Consider the situation where there is only > > one connection and a high-level dispatcher creates a PDU for a SCSI > > command that involves writing immediate data to the target. This PDU > > is enqueued to a lower-level layer which has to setup, start, and > > wait-for a DMA operation to move the immediate data into an onboard > > buffer before the PDU can be put onto the wire. While this is > > happening, the dispatcher creates another unrelated PDU for a SCSI > > read command (for example), and when this PDU is passed to the > > lower-level layer it can be sent immediately, ahead of the previous > > write PDU and therefore out of order on this connection. > > > > The standard clearly allows this to happen if the two PDUs were sent > > on different connections, and seems to imply that this can also > > happen > > when the two PDUs are sent on the same connection. > > > > The suggestion is to put in the standard an explicit statement that > > this is allowed or not allowed, as appropriate. > > > > If this is allowed, such a statement would avoid the erroneous > > assumption being made by some target implementers that within a > > single > > connection, commands will arrive in order. > > > > If this is not allowed, such a statement would avoid the erroneous > > assumption being made by some initiator implementers that within a > > single connection, commands can be put on the wire out of order. > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > now allow: "A value of 0 indicates no limit." > > > > Is this useful? Does it buy anything? > > > > The difficulties implementers are having with this are: > > > > 1) It is a special case. > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > 3) It violates the min/max function normally used for the key. > > 4) There is always a limit anyway. > > > > Consider FirstBurstSize, which can have a value that is described > > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > > is selected. > > > > I one side offers 0 to mean unlimited, and the other side > > has a limit, it will reply with that limit, say 128 Kbytes. > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > The statement in the standard that "the minimum of the 2 numbers is > > selected" is therefore wrong when one of the numbers is 0. > > > > Furthermore, when an initiator or target receives an offer for one > > of these keys, it cannot simply check that the offered value is > > legal by testing it against some minimum and maximum. It must first > > check for 0 and then only if that check shows the value is non-zero > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > Finally, there is always a limit. If nothing else it is the > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > requesting the transfer. It is useless to specify a FirstBurstSize > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > the largest possible DataSegmentLength in any PDU that can use > > this value is 2**24-1. > > > > The suggestion is to just eliminate this special case of 0 and > > require > > that the range 64-to-(2**24-1) be used instead -- it has exactly the > > same effect in all cases, it is easier to describe in the standard > > because it avoids all the extra words, and it is easier to code > > because it avoids all the special cases. > > > > NOTE: the standard should specify the limit in the ranges for > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > DataSegmentLength field and therefore can never be used. > > > > 5. This is a suggestion for a minor rewording in the standard to avoid > > misunderstandings. > > > > In Appendix E on page 188 of draft 8 it says: > > > > "The response to this command is a text response containing a list > > of > > targets and their addresses. Each target is returned as a target > > record. A target record begins with the TargetName text key, > > followed by a list of TargetAddress text keys, ..." > > > > In fact, there are situations where there are no targets and/or no > > addresses. These situations are clearly defined in the draft after > > the sentences quoted above, but it would help if those sentences > > at least hinted at the possibility that the lists could be empty > > or might not contain addresses. A possible rewording would be: > > > > "The response to this command is a text response containing a list > > of > > zero or more targets and, optionally, their addresses. Each target > > is returned as a target record. A target record begins with the > > TargetName text key, followed by a list of zero or more > > TargetAddress text keys, ..." > > > > > > 6. This is a suggestion for another very minor rewording in the > > standard. > > > > At the end of section 2.2.3 on page 29 of draft 8 it says: > > > > "Before full feature phase is established, only Login PDUs are > > allowed. ..." > > > > The suggested rewording is: > > > > "Before full feature phase is established, only Login Request and > > Login Response PDUs are allowed. ..." > > ----- Forwarded by Julian Satran/Haifa/IBM on 05-11-01 10:37 ----- "Bill Strahm" Sent by: owner-ips@ece.cmu.edu 02-11-01 19:23 Please respond to "Bill Strahm" To: "Ron Cohen" , "ips" , "Paul Koning" cc: Subject: RE: iSCSI: current UNH Plugfest: Reserved bits In many cases that is acceptable... Imagine what would happen if Router vendors threw out packets with headers that set reserved bits to anything but 1. Anyone that wanted to play with DiffServ would have to buy new routers that implemented the functionality of the TOS bits... This when it is perfectly acceptable to just ignore them and send packets on without applying any extra functionality - What a pain. Some of the upgrade problem can be handled with a new version number (if you don't speak the new version, I will drop the packet) but there are MANY extensions that I can think of where using a bit or two of a reserved field would allow potentially better performance if the endpoint knows what to do, and no real degradation of service if it doesn't know what to do... Why should I have to rev the version number to handle this case ? Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ron Cohen Sent: Friday, November 02, 2001 6:59 AM To: ips; Paul Koning Subject: Re: iSCSI: current UNH Plugfest: Reserved bits The problem with accepting values in reserved bits is that when the reserved bit is no longer reserved (evolution in the standard) it gives the impression to the initiator that a target implements the new feature when it may actually only be ignoring it. Ron ----- Original Message ----- From: "Paul Koning" To: Cc: ; Sent: Thursday, November 01, 2001 6:38 PM Subject: RE: iSCSI: current UNH Plugfest > Excerpt of message (sent 1 November 2001) by Bill Strahm: > > Usually the "Conservative in what you send, Liberal in what you accept" > > policy is used... > > That's a very important principle... > > > In otherwords, The sender MUST set to 0 (or some other value) The receiver > > MUST ignore the value... > > That's the way to express the principle. But the text quoted in the > earlier notes does not express it the same way. > > In particular "... are format errors. This when detected..." implies > that receivers may or may not detect format errors. In other words, > it implies -- but does NOT state explicitly -- that receivers MAY > check reserved fields for zeroness. > > > This allows for some tweaking of the implementation, if I control both ends > > I might set a reserved value to 1, then I know something... If I receive a > > reserve value set to 1 and I don't do anything the other end knows it is not > > talking to itself (this can even be a versioning thing as well) > > > > Now, we need to be VERY careful in defining this, do we plan on having > > Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean... > > is it possible, is it desirable ? will there be extensions ??? > > > > If you can truly answer NO to all of those things, I would argue for > > REMOVING the reserved fields (if possible), if not, the MUST set, MUST > > ignore policy seems better > > I agree strongly. There is a lot of experience in the community on > what helps and what hurts convenient version upgrade. In particular, > there is a LOT of evidence that ANY checking of reserved fields is a > nasty thing. Unless you plan never to go beyond V1, you really need > to require the rule Bill stated, i.e., receivers MUST ignore the > contents of reserved fields -- they are NOT allowed to "verify" them. > > So the spec needs to be clarified to say that random values in > reserved fields are NOT format errors and receivers MUST NOT treat > them as such. > > paul > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Eddy Quicksall > > Sent: Thursday, November 01, 2001 5:15 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > I am reluctant to say this because I think most people think the > > initiator/target must check for correctness ... but, it is my feeling that > > that job should be up to a basher program. The target should not be in the > > business of diagnosing the initiator. The only time a target should check a > > field is when it could crash the system or data. Some format errors may have > > no consequences whatsoever. > > > > Eddy > > > > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Wednesday, October 31, 2001 05:39 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: current UNH Plugfest > > > > > > Attached are the new issues that arose during the iSCSI plugfest > > at UNH on Wednesday 31-Oct-2001. > > > > (Note: these issues do not take into account any modifications or > > clarifications that occured in the standard due to the issues raised > > on Monday or Tuesday.) > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > ------------------------------------------------------------------------ > > ---- > > > > 1. Are receivers (initiator or target) REQUIRED to check that reserved > > bits and/or fields are set to 0? > > > > Section 3 on page 48 of draft 8 says: > > "Any bits not defined MUST be set to zero. Any reserved fields and > > values MUST be 0 unless specified otherwise." > > > > and Section 8.3 on page 127 of draft 8 says: > > "Explicit violations of the PDU layout rules stated in this > > document > > are format errors. This when detected, usually indicates a major > > implementation flaw in one of the parties. > > > > When a target or an initiator receives an iSCSI PDU with a format > > error, it MUST reset all transport connections in the session > > immediately and escalate the format error to session recovery > > (section 8.11.4)." > > > > According to these rules, a PDU with reserved bits and/or fields that > > are not set to 0 violates the PDU layout rules. Therefore, if an > > initiator or target receives such a PDU, it should immediately close > > all connections in the session and go to session recovery. > > > > Clearly a format error has extremely severe consequences! > > > > Although all vendors are setting reserved bits and fields to 0 on > > PDUs they are sending, many are NOT checking PDUs they are receiving > > to see if these bits and fields are set to 0. Basically, vendors are > > saying "who cares if reserved bits and/or fields in incoming PDUs are > > not zero? We do not want to take the time to do this checking, and > > there is no benefit to doing it. As long as the non-reserved bits > > and > > fields are set properly, we can and should proceed. Any time devoted > > to doing this checking is wasted in 99+% of the cases, and in the > > (unlikely) case that a non-zero bit or field is found, the > > consequences are too severe." > > > > There should be some statement in the standard to clarify what > > checking > > is required and what is optional. > > > > 2. A similar situation arises with respect to checking the consistency > > of fields such as Version-max, Version-min and Version-active in > > Login > > Requests and Login Responses. > > > > For example, consider the Version-max field. > > > > Section 3.12.5 says: > > "All Login requests within the Login phase MUST carry the same > > Version-max." > > > > All vendor initiators are setting Version-max correctly on all > > login requests they are sending, but many vendor targets are NOT > > checking received login requests to ensure that this rule is > > enforced. > > In particular, many targets simply use the Version-max and > > Version-min > > on the first login request they receive on a new connection, and then > > they ignore these fields on all subsequent login requests in the same > > login phase. > > > > Strictly speaking, a change in the Version-max field during the login > > phase constitutes a protocol error according to section 8.8 on page > > 130 > > of draft 8: > > > > "All violations of iSCSI PDU exchange sequences specified in this > > draft are also protocol errors. This category of errors can be > > addressed only by fixing the implementations; iSCSI defines Reject > > and response codes to enable this". > > > > Therefore the target should send back a login response with a status > > of 0x0200 and then close the connection. > > > > However, Section 3.12.5 also says: > > "The target MUST use the value presented with the first login > > request." > > > > This rule seems to imply that the value CAN change, because if it > > cannot > > change, then it doesn't matter which one of the login requests it is > > taken from, they are all the same anyway. > > > > The suggestion is to keep the requirement that the target MUST use > > the > > value presented on the first login request, but to allowed the target > > to ignore the value presented on all subsequent login requests in the > > same login phase. A similar rewording should be done for the other > > fields. > > > > 3. Can commands be sent out of order on the same connection? > > > > The behavior of targets is clearly specified in Section 2.2.2.3 on > > page 25 of draft 8, which says: > > "Except for the commands marked for immediate delivery the iSCSI > > target layer MUST eliver the commands for execution in the order > > specified by CmdSN." > > > > Section 2.2.2.3 on page 26 of draft 8 also says: > > "- CmdSN - the current command Sequence Number advanced by 1 on > > each command shipped except for commands marked for immediate > > delivery." > > but the meaning of the term "shipped" is vague, and does not > > necessarily > > require that the PDUs arrive on the other end of a TCP connection > > in the same order that the CmdSN values were assigned to these PDUs. > > > > Some initiators have been designed to send commands out of CmdSN > > order on one connection. Consider the situation where there is only > > one connection and a high-level dispatcher creates a PDU for a SCSI > > command that involves writing immediate data to the target. This PDU > > is enqueued to a lower-level layer which has to setup, start, and > > wait-for a DMA operation to move the immediate data into an onboard > > buffer before the PDU can be put onto the wire. While this is > > happening, the dispatcher creates another unrelated PDU for a SCSI > > read command (for example), and when this PDU is passed to the > > lower-level layer it can be sent immediately, ahead of the previous > > write PDU and therefore out of order on this connection. > > > > The standard clearly allows this to happen if the two PDUs were sent > > on different connections, and seems to imply that this can also > > happen > > when the two PDUs are sent on the same connection. > > > > The suggestion is to put in the standard an explicit statement that > > this is allowed or not allowed, as appropriate. > > > > If this is allowed, such a statement would avoid the erroneous > > assumption being made by some target implementers that within a > > single > > connection, commands will arrive in order. > > > > If this is not allowed, such a statement would avoid the erroneous > > assumption being made by some initiator implementers that within a > > single connection, commands can be put on the wire out of order. > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > now allow: "A value of 0 indicates no limit." > > > > Is this useful? Does it buy anything? > > > > The difficulties implementers are having with this are: > > > > 1) It is a special case. > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > 3) It violates the min/max function normally used for the key. > > 4) There is always a limit anyway. > > > > Consider FirstBurstSize, which can have a value that is described > > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > > is selected. > > > > I one side offers 0 to mean unlimited, and the other side > > has a limit, it will reply with that limit, say 128 Kbytes. > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > The statement in the standard that "the minimum of the 2 numbers is > > selected" is therefore wrong when one of the numbers is 0. > > > > Furthermore, when an initiator or target receives an offer for one > > of these keys, it cannot simply check that the offered value is > > legal by testing it against some minimum and maximum. It must first > > check for 0 and then only if that check shows the value is non-zero > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > Finally, there is always a limit. If nothing else it is the > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > requesting the transfer. It is useless to specify a FirstBurstSize > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > the largest possible DataSegmentLength in any PDU that can use > > this value is 2**24-1. > > > > The suggestion is to just eliminate this special case of 0 and > > require > > that the range 64-to-(2**24-1) be used instead -- it has exactly the > > same effect in all cases, it is easier to describe in the standard > > because it avoids all the extra words, and it is easier to code > > because it avoids all the special cases. > > > > NOTE: the standard should specify the limit in the ranges for > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > DataSegmentLength field and therefore can never be used. > > > > 5. This is a suggestion for a minor rewording in the standard to avoid > > misunderstandings. > > > > In Appendix E on page 188 of draft 8 it says: > > > > "The response to this command is a text response containing a list > > of > > targets and their addresses. Each target is returned as a target > > record. A target record begins with the TargetName text key, > > followed by a list of TargetAddress text keys, ..." > > > > In fact, there are situations where there are no targets and/or no > > addresses. These situations are clearly defined in the draft after > > the sentences quoted above, but it would help if those sentences > > at least hinted at the possibility that the lists could be empty > > or might not contain addresses. A possible rewording would be: > > > > "The response to this command is a text response containing a list > > of > > zero or more targets and, optionally, their addresses. Each target > > is returned as a target record. A target record begins with the > > TargetName text key, followed by a list of zero or more > > TargetAddress text keys, ..." > > > > > > 6. This is a suggestion for another very minor rewording in the > > standard. > > > > At the end of section 2.2.3 on page 29 of draft 8 it says: > > > > "Before full feature phase is established, only Login PDUs are > > allowed. ..." > > > > The suggested rewording is: > > > > "Before full feature phase is established, only Login Request and > > Login Response PDUs are allowed. ..." > > ----- Forwarded by Julian Satran/Haifa/IBM on 05-11-01 10:37 ----- Paul Koning Sent by: owner-ips@ece.cmu.edu 03-11-01 01:01 Please respond to Paul Koning To: rcohen@eurologic.com cc: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest: Reserved bits Excerpt of message (sent 2 November 2001) by Ron Cohen: > The problem with accepting values in reserved bits is that when the reserved > bit is no longer reserved (evolution in the standard) it gives the > impression to the initiator that a target implements the new feature when it > may actually only be ignoring it. Yes, if you don't do some other things that are needed. You MUST have protocol version numbers in the initial messages. Otherwise you are utterly lost. With that, you know what version the other end has. The key point is that requiring reserved fields to be ignored is that you can rely on the fact that new things can be asked for in reserved fields with the knowledge that old implementations will ignore those things. Quite often this is what you want. Occasionally it is not; for those cases you make use of the version number. In any case, what I was describing is long established practice in many protocols that have successfully navigated the version number migration path. paul From owner-ips@ece.cmu.edu Mon Nov 5 04:03:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA09426 for ; Mon, 5 Nov 2001 04:03:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA58TcJ00075 for ips-outgoing; Mon, 5 Nov 2001 03:29:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA58Tal00071 for ; Mon, 5 Nov 2001 03:29:36 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA108670 for ; Mon, 5 Nov 2001 09:29:29 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA58TS359404 for ; Mon, 5 Nov 2001 09:29:28 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 5 Nov 2001 10:29:21 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 05/11/2001 10:29:29, Serialize complete at 05/11/2001 10:29:29 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, It all depends on what assumptions you make about data buffering for unsolicited data. If you assume (conservatively) that you will have enough buffers for all possible commands in flight and their unsolicited data then ordering is not an issue. If you assume - less conservative - that you have do not have room for all and account for the "flight time" to make up for some of the missing buffers and perhaps keep some of the data in the "TCP buffers" then allowing commands out of order might result in deadlock if the source and sink have different ordering. As shipping commands out of order will hardly get you a real performance boost and keeping things in order might have beneficial effects on the memory sizes required at target I thought that we should require ordering. Regards, Julo Dave Sheehy Sent by: owner-ips@ece.cmu.edu 02-11-01 23:18 Please respond to Dave Sheehy To: ips@ece.cmu.edu (IETF IP SAN Reflector) cc: Subject: Re: iSCSI: current UNH Plugfest > 3. Can commands be sent out of order on the same connection? > > The behavior of targets is clearly specified in Section 2.2.2.3 on > page 25 of draft 8, which says: > "Except for the commands marked for immediate delivery the iSCSI > target layer MUST eliver the commands for execution in the order > specified by CmdSN." > > Section 2.2.2.3 on page 26 of draft 8 also says: > "- CmdSN - the current command Sequence Number advanced by 1 on > each command shipped except for commands marked for immediate > delivery." > but the meaning of the term "shipped" is vague, and does not > necessarily > require that the PDUs arrive on the other end of a TCP connection > in the same order that the CmdSN values were assigned to these PDUs. > > Some initiators have been designed to send commands out of CmdSN > order on one connection. Consider the situation where there is only > one connection and a high-level dispatcher creates a PDU for a SCSI > command that involves writing immediate data to the target. This PDU > is enqueued to a lower-level layer which has to setup, start, and > wait-for a DMA operation to move the immediate data into an onboard > buffer before the PDU can be put onto the wire. While this is > happening, the dispatcher creates another unrelated PDU for a SCSI > read command (for example), and when this PDU is passed to the > lower-level layer it can be sent immediately, ahead of the previous > write PDU and therefore out of order on this connection. > > The standard clearly allows this to happen if the two PDUs were sent > on different connections, and seems to imply that this can also happen > when the two PDUs are sent on the same connection. > > The suggestion is to put in the standard an explicit statement that > this is allowed or not allowed, as appropriate. > > If this is allowed, such a statement would avoid the erroneous > assumption being made by some target implementers that within a single > connection, commands will arrive in order. > > If this is not allowed, such a statement would avoid the erroneous > assumption being made by some initiator implementers that within a > single connection, commands can be put on the wire out of order. > > +++ > > will add an explicit statement saying that this behaviour is forbidden. > 2.2.2.1 will contain: > > On any given connection, the iSCSI initiator MUST send the commands in the > order specified by CmdSN. > > +++ Why do you feel this behavior should be forbidden? Targets already have to order commands across the session. I don't see why it's a problem to extend that to the connection as well. I, for one, believe we should take a liberal stance on this. Dave Sheehy From owner-ips@ece.cmu.edu Mon Nov 5 04:04:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA09437 for ; Mon, 5 Nov 2001 04:04:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA58ZcX00456 for ips-outgoing; Mon, 5 Nov 2001 03:35:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA58Zal00449 for ; Mon, 5 Nov 2001 03:35:36 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id JAA14678 for ; Mon, 5 Nov 2001 09:35:30 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA58ZUv36326 for ; Mon, 5 Nov 2001 09:35:30 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 5 Nov 2001 10:35:28 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 05/11/2001 10:35:29, Serialize complete at 05/11/2001 10:35:29 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Rod, I all examples give the point I find hard to understand is why is the ordering on the wire different from the presentation order to the initiator. You can get as many overlaps as you want by presenting the commands to the initiator in the desired order. What we are considering here is the case in which you want to ship in an order different than the one you present the commands. Julo "Rod Harrison" Sent by: owner-ips@ece.cmu.edu 04-11-01 04:42 Please respond to "Rod Harrison" To: "Barry Reinhold" , "Dave Sheehy" , "IETF IP SAN Reflector" cc: Subject: RE: iSCSI: current UNH Plugfest Barry, In general I agree but I don't think this is as much of a corner case as it at first appears. Targets will have code very similar to that needed to handle out of order commands to deal with digest errors. Targets also need to queue commands whilst waiting for both solicited and unsolicited data to arrive. Queuing out of order commands seems little extra work. From an initiators point of view there are efficiency, and probably performance gains to be had from sending commands out of order. Bob Russell gave the example of a read being sent whilst write data DMA is happening, and a similar situation can arise with DMA for writes overtaking that of earlier writes if the initiator has multiple DMA engines. In this case the initiator might be forced to let the wire go idle if it can't send the data from completed DMAs as soon as possible. We already have a command queue at the target to enforce correct serialisation of commands, doing the same thing at the initiator is redundant. Finally, I don't believe we should be writing a standard to work around poor coding and test coverage, especially at the cost of potential efficiency gains. I agree with Dave and Santosh that commands being sent out of order on a single session should be allowed by the standard. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Barry Reinhold Sent: Friday, November 02, 2001 5:24 PM To: Dave Sheehy; IETF IP SAN Reflector Subject: RE: iSCSI: current UNH Plugfest Using features such as out of order command delivery on a connection tend to be the sort of things that lead to interoperability problems. It is unexpected and probably going to hit poorly tested code paths even if the standard is written to allow it. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Dave Sheehy >Sent: Friday, November 02, 2001 4:19 PM >To: IETF IP SAN Reflector >Subject: Re: iSCSI: current UNH Plugfest > > > >> 3. Can commands be sent out of order on the same connection? >> >> The behavior of targets is clearly specified in Section 2.2.2.3 on >> page 25 of draft 8, which says: >> "Except for the commands marked for immediate delivery the iSCSI >> target layer MUST eliver the commands for execution in the order >> specified by CmdSN." >> >> Section 2.2.2.3 on page 26 of draft 8 also says: >> "- CmdSN - the current command Sequence Number advanced by 1 on >> each command shipped except for commands marked for immediate >> delivery." >> but the meaning of the term "shipped" is vague, and does not >> necessarily >> require that the PDUs arrive on the other end of a TCP connection >> in the same order that the CmdSN values were assigned to these PDUs. >> >> Some initiators have been designed to send commands out of CmdSN >> order on one connection. Consider the situation where there is only >> one connection and a high-level dispatcher creates a PDU for a SCSI >> command that involves writing immediate data to the target. This PDU >> is enqueued to a lower-level layer which has to setup, start, and >> wait-for a DMA operation to move the immediate data into an onboard >> buffer before the PDU can be put onto the wire. While this is >> happening, the dispatcher creates another unrelated PDU for a SCSI >> read command (for example), and when this PDU is passed to the >> lower-level layer it can be sent immediately, ahead of the previous >> write PDU and therefore out of order on this connection. >> >> The standard clearly allows this to happen if the two PDUs were sent >> on different connections, and seems to imply that this can also happen >> when the two PDUs are sent on the same connection. >> >> The suggestion is to put in the standard an explicit statement that >> this is allowed or not allowed, as appropriate. >> >> If this is allowed, such a statement would avoid the erroneous >> assumption being made by some target implementers that within a single >> connection, commands will arrive in order. >> >> If this is not allowed, such a statement would avoid the erroneous >> assumption being made by some initiator implementers that within a >> single connection, commands can be put on the wire out of order. >> >> +++ >> >> will add an explicit statement saying that this behaviour is forbidden. >> 2.2.2.1 will contain: >> >> On any given connection, the iSCSI initiator MUST send the >commands in the >> order specified by CmdSN. >> >> +++ > >Why do you feel this behavior should be forbidden? Targets already have to >order commands across the session. I don't see why it's a problem to extend >that to the connection as well. I, for one, believe we should take >a liberal >stance on this. > >Dave Sheehy > From owner-ips@ece.cmu.edu Mon Nov 5 05:07:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA10316 for ; Mon, 5 Nov 2001 05:07:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA59K1K02804 for ips-outgoing; Mon, 5 Nov 2001 04:20:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA59Jxl02799 for ; Mon, 5 Nov 2001 04:19:59 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id EAA25912 for ; Mon, 5 Nov 2001 04:17:23 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA59Jvt126848 for ; Mon, 5 Nov 2001 02:19:57 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: current UNH Plugfest To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Mon, 5 Nov 2001 01:19:04 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/05/2001 02:19:58 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, The TargetName Must be specified on the Initial Login of all non discovery connections, not just the Leading Login. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/05/2001 12:11:13 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: current UNH Plugfest Bob, Comments in text. Thanks, Julo "Robert D. Russell" Sent by: owner-ips@ece.cmu.edu 02-11-01 01:31 Please respond to "Robert D. Russell" To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: current UNH Plugfest Attached are the new issues that arose during the iSCSI plugfest at UNH on Thursday 1-Nov-2001. Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 ---------------------------------------------------------------------- There were no new issues today! Lots of people were busy sending data and interoperating with each other! There were, however, a few suggestions/requests for clarifications in the standard: 1. Since all Login Requests MUST be issued as immediate requests, the diagram in section 3.12 should show bit 6 of byte 0 as "1", not "I", and section 3.12.2 should simply be eliminated. ++++ OK - Julo ++++ 2. The first paragraph of Appendix D should describe the special role of the first Login request on the first connection of a new session. In particular, there are 3 keys that MUST be sent in that Login PDU (InitiatorName, SessionType if it is discovery, TargetName if the session is normal). This is, in fact, yet another classification that is currently included within LO, but LO is too general, since LO allows keys to appear at any time during the leading login phase, and these keys must appear on the very first login PDU in that leading login phase. +++ There is now a text in Section 5 that reads: The initial Login request of the first connection of a session (primary login) MUST include the InitiatorName key=value pair. The primary Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. +++ 3. A target cannot release its resources for a command until it receives an ExpStatSN back from the initiator that acknowledges receipt of the StatSN carried in the SCSI Response to that command. However, if the initiator does not have any more I/O to perform, that ExpStatSN will not come back to the target, and the target may end up holding on to those resources indefinitely. This is a situation where initiator non-action can impact target performance, and could even lead to denial of service attacks if several initiators were to do this simultaneously to the same target. The standard provides the NOP-In/Out mechanism to deal with this: The last paragraph of section 2.2.2 says: During periods when traffic on a connection is unidirectional, iSCSI NOP-Out/In PDUs may be utilized to synchronize the command and status ordering counters of the target and initiator. and section 3.18 says: A NOP-Out may also be used to confirm a changed ExpStatSN if there is no other PDU to carry it for a long time. The question is, is there a recommended mechanism to trigger these NOP-pings? Clearly the target could set a timer, and if ExpStatSN is not received back at the end of the time interval, the target could send a NOP-in as a ping in order to get the initiator to send back a NOP-out with the updated ExpStatSN. Also clearly the initiator could set a timer, and if it has no traffic to send during the time interval, the initiator could send a NOP-out to deliver the updated ExpStatSN to the target. This approach is probably not as reliable as the first, since a) if the initiator does not do it, only the target gets hurt, and b) if the connection drops, the target will never receive the NOP-out. On the other hand, it may be more efficient for the initiator to do this, since a typical target may be dealing with hundreds of connections, and the initiator only a few. Which is the best procedure? Are there other ways to do this? What is the recommended way, or is there a reason for the standard not to recommend a triggering mechanism, in which case, could the standard suggest methods or provide guidance, especially if there are better, less obvious ways to do it? +++ There are no better ways to do it. However we see this a purely an implementation matter. I assume that a target low on resources will try to recover some and ping the initiators. +++ From owner-ips@ece.cmu.edu Mon Nov 5 10:00:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA21811 for ; Mon, 5 Nov 2001 10:00:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5E0Kg29288 for ips-outgoing; Mon, 5 Nov 2001 09:00:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from auemail2.firewall.lucent.com (auemail2.lucent.com [192.11.223.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5E0Il29281 for ; Mon, 5 Nov 2001 09:00:18 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by auemail2.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA5E0Br15589 for ; Mon, 5 Nov 2001 09:00:11 -0500 (EST) Content-Class: urn:content-classes:message Subject: IPS-ALL: IPS meetings TENTATIVELY set [FW: 52nd IETF Draft Agenda] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Date: Mon, 5 Nov 2001 09:00:10 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2AC@nc8220exchange.ral.lucent.com> Thread-Topic: 52nd IETF Draft Agenda Thread-Index: AcFj41HJS4qR2ohISEmsAZsTZmlM1wCHhr/Q From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA5E0Il29282 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit All, The IETF agenda has been set. Note that the agenda is subject to change up until a couple of weeks prior to the meeting. Currently, the agenda shows IPS meeting between 9-11:30 am on Monday, Dec 10 and Tuesday, Dec 11. Again, these times are subject to change. FYI Elizabeth Rodriguez -----Original Message----- From: agenda@ietf.org [mailto:agenda@ietf.org] Sent: Friday, November 02, 2001 1:39 PM To: IETF-Announce; @loki.ietf.org@ihrh2.emsr.lucent.com Subject: 52nd IETF Draft Agenda Please find below the Draft Agenda of the 52nd IETF Meeting. For more information visit: http://www.ietf.org/meetings/IETF-52.html Also please be advised that the cut-off date for hotel reservations at the special IETF rate is Wednesday, November 7th. Please see: http://www.ietf.org/meetings/hotels_52.html +++ Draft Agenda of the Fifty-second IETF December 9-14, 2001 (As of November 2, 2001) SUNDAY, December 9, 2001 1200-1900 Registration 1530-1600 Newcomer's Orientation 1600-1630 IETF Standards Process Orientation 1700-1900 Welcome Reception MONDAY, December 10, 2001 0800-1930 IETF Registration 0800-0900 Continental Breakfast 0900-1130 Morning Sessions APP apparea & irnss Applications Open Area Meeting & Internet Resource Name Search Service BO F INT dhc Dynamic Host Configuration WG IRTF gsec Group Security Research Group RTG forces Forwarding and Control Element Separation WG SEC kink Kerberized Internet Negotiation of Keys WG SUB ccamp Common Control and Measurement Plane WG TSV ips IP Storage WG TSV rmt Reliable Multicast Transport WG 1130-1300 Break 1300-1500 Afternoon Sessions I APP geopriv Geographic Location/Privacy WG INT idn Internationalized Domain Name WG INT l2tpext Layer Two Tunneling Protocol Extensions WG OPS mboned MBONE Deployment WG SEC inch Extended Incident Handling BOF SEC msec Multicast Security WG USV uswg User Services WG 1500-1530 Break (Refreshments provided) 1530-1730 Afternoon Sessions II APP webdav WWW Distributed Authoring and Versioning WG APP intloc Internationalization and Localization of Internet Protocols BOF INT ipcdn IP over Cable Data Network WG INT magma Multicast & Anycast Group Membership WG OPS aaa Authentication, Authorization and Accounting WG SUB mpls Multiprotocol Label Switching WG TSV mmusic Multiparty Multimedia Session Control WG 1730-1930 Break 1930-2200 Evening Sessions APP fax Internet Fax WG INT dnsext DNS Extensions WG OPS rmonmib Remote Network Monitoring WG RTG manet Mobile Ad-hoc Networks WG SEC ipsp IP Security Policy WG TSV midcom Middlebox Communication WG TSV seamoby Context Transfer, Handoff Candidate Discovery, and Dormant Mode Host Aler ting WG TUESDAY, December 11, 2001 0800-1700 IETF Registration 0800-0900 Continental Breakfast 0900-1130 Morning Sessions APP trade Internet Open Trading Protocol WG INT ipngwg IPNG WG OPS bmwg Benchmarking Methodology WG OPS hubmib Ethernet Interfaces and Hub MIB WG SEC pkix Public-Key Infrastructure (X.509) WG TSV ips IP Storage WG TSV rserpool Reliable Server Pooling WG 1130-1300 Break 1300-1400 Afternoon Sessions I OPS hubmib Ethernet Interfaces and Hub MIB WG & AToM MIB WG &INT &atommib RTG vrrp Virtual Router Redundancy Protocol WG SUB gsmp General Switch Management Protocol WG TSV diffserv Differentiated Services WG TSV pilc Performance Implications of Link Characteristics WG 1415-1515 Afternoon Sessions II APP provreg Provisioning Registry Protocol WG OPS rmonmib Remote Network Monitoring WG SUB ipo IP over Optical WG TSV rmt Reliable Multicast Transport WG 1515-1545 Break (Refreshments provided) 1545-1645 Afternoon Sessions III INT ipoib IP over InfiniBand WG RTG udlr UniDirectional Link Routing WG TSV iptel IP Telephony WG 1700-1800 Afternoon Sessions IV INT pppext Point-to-Point Protocol Extensions WG OPS rap Resource Allocation Protocol WG TSV spirits Service in the PSTN/IN Requesting InTernet Service WG WEDNESDAY, December 12, 2001 0800-1700 IETF Registration 0800-0900 Continental Breakfast 0900-1130 Morning Sessions APP vpim Voice Profile for Internet Mail WG APP ldapbis LDAP (v3) Revision WG OPS dnsop Domain Name Server Operations WG INT mobileip IP Routing for Wireless/Mobile Hosts WG SEC sacred Securely Available Credentials WG SUB ppvpn Provider Provisioned Virtual Private Networks WG TSV avt Audio/Video Transport WG 1130-1300 Break 1300-1500 Afternoon Sessions I APP calsch Calendaring and Scheduling WG OPS bridge Bridge MIB WG OPS ngtrans Next Generation Transition WG RTG pim Protocol Independent Multicast WG SEC smime S/MIME Mail Security WG TSV seamoby Context Transfer, Handoff Candidate Discovery, and Dormant Mode Host Aler ting WG 1500-1530 Break (Refreshments provided) 1530-1730 Afternoon Sessions II APP simple SIP for Instant Messaging and Presence Leveraging Extensions WG INT ipoib IP over InfiniBand WG OPS ipfix IP Flow Information Export WG RTG idr Inter-Domain Routing WG SEC ipsec IP Security Protocol WG TSV nfsv4 Network File System Version 4 WG 1730-1930 Break 1930-2200 Open Plenary Welcome Speakers IESG Report IESG Open Mike 2230 Late Night Session PGP Key Signing THURSDAY, December 13, 2001 0800-1700 IETF Registration 0800-0900 Continental Breakfast 0900-1130 Morning Sessions APP opes Open Pluggable Edge Services BOF INT ipngwg IPNG WG OPS aaa Authentication, Authorization and Accounting WG SEC ipsec IP Security Protocol WG SUB tewg Internet Traffic Engineering WG TSV avt Audio/Video Transport WG 1130-1300 Break 1300-1500 Afternoon Sessions I APP ipp Internet Printing Protocol WG IRTF aaaarch Authentication Authorisation Accounting ARCHitectureResearch Group OPS ngtrans Next Generation Transition WG OPS policy Policy Framework WG SEC krb-wg Kerberos WG 1500-1530 Break (Refreshments provided) 1530-1730 Afternoon Sessions II APP geopriv Geographic Location/Privacy WG APP ldup LDAP Duplication/Replication/Update Protocols WG OPS entmib Entity MIB WG SEC saag Open Security Area Directorate Meeting TSV pwe3 Pseudo Wire Emulation Edge to Edge WG 1730-1930 Break 1930-2200 Open IAB Plenary IAB Report including: RFC Editor Report IANA Report IRTF Report Question Open discussion on architectural principles FRIDAY, December 14, 2001 0800-1000 IETF Registration 0800-0900 Continental Breakfast 0900-1130 Morning Sessions SEC idwg Intrusion Detection Exchange Format WG SUB subarea Sub-IP Area Meeting From owner-ips@ece.cmu.edu Mon Nov 5 10:39:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA24368 for ; Mon, 5 Nov 2001 10:39:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5EhmP02569 for ips-outgoing; Mon, 5 Nov 2001 09:43:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch04.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5Ehkl02559 for ; Mon, 5 Nov 2001 09:43:46 -0500 (EST) Received: by LMOXCH04 with Internet Mail Service (5.5.2653.19) id ; Mon, 5 Nov 2001 09:39:25 -0500 Message-ID: <8BE017CC8923D511A00A0008C78605EE7A814E@LMOXCH04> From: David Dillard To: "'Julian Satran'" Cc: ips@ece.cmu.edu Subject: RE: iSCSI: current UNH Plugfest: Reserved bits Date: Mon, 5 Nov 2001 09:39:22 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This would seem to be keeping with how SCSI acts. From SPC-3: "3.3.9 reserved: A keyword referring to bits, bytes, words, fields and code values that are set aside for future standardization. A reserved bit, byte, word or field shall be set to zero, or in accordance with a future extension to this standard. Recipients are not required to check reserved bits, bytes, words or fields for zero values. Receipt of reserved code values in defined fields shall be reported as error." David -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Monday, November 05, 2001 3:42 AM To: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest: Reserved bits The usual way other standard bodies handle reserved fields are that they are mandated to be 0 but a compliant receiver does not have to check. However most of them don't say this explicitly. However to avoid having "compliance tests" test receivers we might want to say in Section 3: Any compliant sender MUST set to zero all bits not defined and all reserved fields unless specified otherwise. Any compliant receiver MUST ignore any bit not defined and all reserved fields unless specified otherwise. Julo ----- Forwarded by Julian Satran/Haifa/IBM on 05-11-01 10:37 ----- "Ron Cohen" Sent by: owner-ips@ece.cmu.edu 02-11-01 16:59 Please respond to "Ron Cohen" To: "ips" , "Paul Koning" cc: Subject: Re: iSCSI: current UNH Plugfest: Reserved bits The problem with accepting values in reserved bits is that when the reserved bit is no longer reserved (evolution in the standard) it gives the impression to the initiator that a target implements the new feature when it may actually only be ignoring it. Ron ----- Original Message ----- From: "Paul Koning" To: Cc: ; Sent: Thursday, November 01, 2001 6:38 PM Subject: RE: iSCSI: current UNH Plugfest > Excerpt of message (sent 1 November 2001) by Bill Strahm: > > Usually the "Conservative in what you send, Liberal in what you accept" > > policy is used... > > That's a very important principle... > > > In otherwords, The sender MUST set to 0 (or some other value) The receiver > > MUST ignore the value... > > That's the way to express the principle. But the text quoted in the > earlier notes does not express it the same way. > > In particular "... are format errors. This when detected..." implies > that receivers may or may not detect format errors. In other words, > it implies -- but does NOT state explicitly -- that receivers MAY > check reserved fields for zeroness. > > > This allows for some tweaking of the implementation, if I control both ends > > I might set a reserved value to 1, then I know something... If I receive a > > reserve value set to 1 and I don't do anything the other end knows it is not > > talking to itself (this can even be a versioning thing as well) > > > > Now, we need to be VERY careful in defining this, do we plan on having > > Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean... > > is it possible, is it desirable ? will there be extensions ??? > > > > If you can truly answer NO to all of those things, I would argue for > > REMOVING the reserved fields (if possible), if not, the MUST set, MUST > > ignore policy seems better > > I agree strongly. There is a lot of experience in the community on > what helps and what hurts convenient version upgrade. In particular, > there is a LOT of evidence that ANY checking of reserved fields is a > nasty thing. Unless you plan never to go beyond V1, you really need > to require the rule Bill stated, i.e., receivers MUST ignore the > contents of reserved fields -- they are NOT allowed to "verify" them. > > So the spec needs to be clarified to say that random values in > reserved fields are NOT format errors and receivers MUST NOT treat > them as such. > > paul > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Eddy Quicksall > > Sent: Thursday, November 01, 2001 5:15 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > I am reluctant to say this because I think most people think the > > initiator/target must check for correctness ... but, it is my feeling that > > that job should be up to a basher program. The target should not be in the > > business of diagnosing the initiator. The only time a target should check a > > field is when it could crash the system or data. Some format errors may have > > no consequences whatsoever. > > > > Eddy > > > > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Wednesday, October 31, 2001 05:39 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: current UNH Plugfest > > > > > > Attached are the new issues that arose during the iSCSI plugfest > > at UNH on Wednesday 31-Oct-2001. > > > > (Note: these issues do not take into account any modifications or > > clarifications that occured in the standard due to the issues raised > > on Monday or Tuesday.) > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > ------------------------------------------------------------------------ > > ---- > > > > 1. Are receivers (initiator or target) REQUIRED to check that reserved > > bits and/or fields are set to 0? > > > > Section 3 on page 48 of draft 8 says: > > "Any bits not defined MUST be set to zero. Any reserved fields and > > values MUST be 0 unless specified otherwise." > > > > and Section 8.3 on page 127 of draft 8 says: > > "Explicit violations of the PDU layout rules stated in this > > document > > are format errors. This when detected, usually indicates a major > > implementation flaw in one of the parties. > > > > When a target or an initiator receives an iSCSI PDU with a format > > error, it MUST reset all transport connections in the session > > immediately and escalate the format error to session recovery > > (section 8.11.4)." > > > > According to these rules, a PDU with reserved bits and/or fields that > > are not set to 0 violates the PDU layout rules. Therefore, if an > > initiator or target receives such a PDU, it should immediately close > > all connections in the session and go to session recovery. > > > > Clearly a format error has extremely severe consequences! > > > > Although all vendors are setting reserved bits and fields to 0 on > > PDUs they are sending, many are NOT checking PDUs they are receiving > > to see if these bits and fields are set to 0. Basically, vendors are > > saying "who cares if reserved bits and/or fields in incoming PDUs are > > not zero? We do not want to take the time to do this checking, and > > there is no benefit to doing it. As long as the non-reserved bits > > and > > fields are set properly, we can and should proceed. Any time devoted > > to doing this checking is wasted in 99+% of the cases, and in the > > (unlikely) case that a non-zero bit or field is found, the > > consequences are too severe." > > > > There should be some statement in the standard to clarify what > > checking > > is required and what is optional. > > > > 2. A similar situation arises with respect to checking the consistency > > of fields such as Version-max, Version-min and Version-active in > > Login > > Requests and Login Responses. > > > > For example, consider the Version-max field. > > > > Section 3.12.5 says: > > "All Login requests within the Login phase MUST carry the same > > Version-max." > > > > All vendor initiators are setting Version-max correctly on all > > login requests they are sending, but many vendor targets are NOT > > checking received login requests to ensure that this rule is > > enforced. > > In particular, many targets simply use the Version-max and > > Version-min > > on the first login request they receive on a new connection, and then > > they ignore these fields on all subsequent login requests in the same > > login phase. > > > > Strictly speaking, a change in the Version-max field during the login > > phase constitutes a protocol error according to section 8.8 on page > > 130 > > of draft 8: > > > > "All violations of iSCSI PDU exchange sequences specified in this > > draft are also protocol errors. This category of errors can be > > addressed only by fixing the implementations; iSCSI defines Reject > > and response codes to enable this". > > > > Therefore the target should send back a login response with a status > > of 0x0200 and then close the connection. > > > > However, Section 3.12.5 also says: > > "The target MUST use the value presented with the first login > > request." > > > > This rule seems to imply that the value CAN change, because if it > > cannot > > change, then it doesn't matter which one of the login requests it is > > taken from, they are all the same anyway. > > > > The suggestion is to keep the requirement that the target MUST use > > the > > value presented on the first login request, but to allowed the target > > to ignore the value presented on all subsequent login requests in the > > same login phase. A similar rewording should be done for the other > > fields. > > > > 3. Can commands be sent out of order on the same connection? > > > > The behavior of targets is clearly specified in Section 2.2.2.3 on > > page 25 of draft 8, which says: > > "Except for the commands marked for immediate delivery the iSCSI > > target layer MUST eliver the commands for execution in the order > > specified by CmdSN." > > > > Section 2.2.2.3 on page 26 of draft 8 also says: > > "- CmdSN - the current command Sequence Number advanced by 1 on > > each command shipped except for commands marked for immediate > > delivery." > > but the meaning of the term "shipped" is vague, and does not > > necessarily > > require that the PDUs arrive on the other end of a TCP connection > > in the same order that the CmdSN values were assigned to these PDUs. > > > > Some initiators have been designed to send commands out of CmdSN > > order on one connection. Consider the situation where there is only > > one connection and a high-level dispatcher creates a PDU for a SCSI > > command that involves writing immediate data to the target. This PDU > > is enqueued to a lower-level layer which has to setup, start, and > > wait-for a DMA operation to move the immediate data into an onboard > > buffer before the PDU can be put onto the wire. While this is > > happening, the dispatcher creates another unrelated PDU for a SCSI > > read command (for example), and when this PDU is passed to the > > lower-level layer it can be sent immediately, ahead of the previous > > write PDU and therefore out of order on this connection. > > > > The standard clearly allows this to happen if the two PDUs were sent > > on different connections, and seems to imply that this can also > > happen > > when the two PDUs are sent on the same connection. > > > > The suggestion is to put in the standard an explicit statement that > > this is allowed or not allowed, as appropriate. > > > > If this is allowed, such a statement would avoid the erroneous > > assumption being made by some target implementers that within a > > single > > connection, commands will arrive in order. > > > > If this is not allowed, such a statement would avoid the erroneous > > assumption being made by some initiator implementers that within a > > single connection, commands can be put on the wire out of order. > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > now allow: "A value of 0 indicates no limit." > > > > Is this useful? Does it buy anything? > > > > The difficulties implementers are having with this are: > > > > 1) It is a special case. > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > 3) It violates the min/max function normally used for the key. > > 4) There is always a limit anyway. > > > > Consider FirstBurstSize, which can have a value that is described > > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > > is selected. > > > > I one side offers 0 to mean unlimited, and the other side > > has a limit, it will reply with that limit, say 128 Kbytes. > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > The statement in the standard that "the minimum of the 2 numbers is > > selected" is therefore wrong when one of the numbers is 0. > > > > Furthermore, when an initiator or target receives an offer for one > > of these keys, it cannot simply check that the offered value is > > legal by testing it against some minimum and maximum. It must first > > check for 0 and then only if that check shows the value is non-zero > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > Finally, there is always a limit. If nothing else it is the > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > requesting the transfer. It is useless to specify a FirstBurstSize > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > the largest possible DataSegmentLength in any PDU that can use > > this value is 2**24-1. > > > > The suggestion is to just eliminate this special case of 0 and > > require > > that the range 64-to-(2**24-1) be used instead -- it has exactly the > > same effect in all cases, it is easier to describe in the standard > > because it avoids all the extra words, and it is easier to code > > because it avoids all the special cases. > > > > NOTE: the standard should specify the limit in the ranges for > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > DataSegmentLength field and therefore can never be used. > > > > 5. This is a suggestion for a minor rewording in the standard to avoid > > misunderstandings. > > > > In Appendix E on page 188 of draft 8 it says: > > > > "The response to this command is a text response containing a list > > of > > targets and their addresses. Each target is returned as a target > > record. A target record begins with the TargetName text key, > > followed by a list of TargetAddress text keys, ..." > > > > In fact, there are situations where there are no targets and/or no > > addresses. These situations are clearly defined in the draft after > > the sentences quoted above, but it would help if those sentences > > at least hinted at the possibility that the lists could be empty > > or might not contain addresses. A possible rewording would be: > > > > "The response to this command is a text response containing a list > > of > > zero or more targets and, optionally, their addresses. Each target > > is returned as a target record. A target record begins with the > > TargetName text key, followed by a list of zero or more > > TargetAddress text keys, ..." > > > > > > 6. This is a suggestion for another very minor rewording in the > > standard. > > > > At the end of section 2.2.3 on page 29 of draft 8 it says: > > > > "Before full feature phase is established, only Login PDUs are > > allowed. ..." > > > > The suggested rewording is: > > > > "Before full feature phase is established, only Login Request and > > Login Response PDUs are allowed. ..." > > ----- Forwarded by Julian Satran/Haifa/IBM on 05-11-01 10:37 ----- "Bill Strahm" Sent by: owner-ips@ece.cmu.edu 02-11-01 19:23 Please respond to "Bill Strahm" To: "Ron Cohen" , "ips" , "Paul Koning" cc: Subject: RE: iSCSI: current UNH Plugfest: Reserved bits In many cases that is acceptable... Imagine what would happen if Router vendors threw out packets with headers that set reserved bits to anything but 1. Anyone that wanted to play with DiffServ would have to buy new routers that implemented the functionality of the TOS bits... This when it is perfectly acceptable to just ignore them and send packets on without applying any extra functionality - What a pain. Some of the upgrade problem can be handled with a new version number (if you don't speak the new version, I will drop the packet) but there are MANY extensions that I can think of where using a bit or two of a reserved field would allow potentially better performance if the endpoint knows what to do, and no real degradation of service if it doesn't know what to do... Why should I have to rev the version number to handle this case ? Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ron Cohen Sent: Friday, November 02, 2001 6:59 AM To: ips; Paul Koning Subject: Re: iSCSI: current UNH Plugfest: Reserved bits The problem with accepting values in reserved bits is that when the reserved bit is no longer reserved (evolution in the standard) it gives the impression to the initiator that a target implements the new feature when it may actually only be ignoring it. Ron ----- Original Message ----- From: "Paul Koning" To: Cc: ; Sent: Thursday, November 01, 2001 6:38 PM Subject: RE: iSCSI: current UNH Plugfest > Excerpt of message (sent 1 November 2001) by Bill Strahm: > > Usually the "Conservative in what you send, Liberal in what you accept" > > policy is used... > > That's a very important principle... > > > In otherwords, The sender MUST set to 0 (or some other value) The receiver > > MUST ignore the value... > > That's the way to express the principle. But the text quoted in the > earlier notes does not express it the same way. > > In particular "... are format errors. This when detected..." implies > that receivers may or may not detect format errors. In other words, > it implies -- but does NOT state explicitly -- that receivers MAY > check reserved fields for zeroness. > > > This allows for some tweaking of the implementation, if I control both ends > > I might set a reserved value to 1, then I know something... If I receive a > > reserve value set to 1 and I don't do anything the other end knows it is not > > talking to itself (this can even be a versioning thing as well) > > > > Now, we need to be VERY careful in defining this, do we plan on having > > Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean... > > is it possible, is it desirable ? will there be extensions ??? > > > > If you can truly answer NO to all of those things, I would argue for > > REMOVING the reserved fields (if possible), if not, the MUST set, MUST > > ignore policy seems better > > I agree strongly. There is a lot of experience in the community on > what helps and what hurts convenient version upgrade. In particular, > there is a LOT of evidence that ANY checking of reserved fields is a > nasty thing. Unless you plan never to go beyond V1, you really need > to require the rule Bill stated, i.e., receivers MUST ignore the > contents of reserved fields -- they are NOT allowed to "verify" them. > > So the spec needs to be clarified to say that random values in > reserved fields are NOT format errors and receivers MUST NOT treat > them as such. > > paul > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Eddy Quicksall > > Sent: Thursday, November 01, 2001 5:15 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > I am reluctant to say this because I think most people think the > > initiator/target must check for correctness ... but, it is my feeling that > > that job should be up to a basher program. The target should not be in the > > business of diagnosing the initiator. The only time a target should check a > > field is when it could crash the system or data. Some format errors may have > > no consequences whatsoever. > > > > Eddy > > > > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Wednesday, October 31, 2001 05:39 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: current UNH Plugfest > > > > > > Attached are the new issues that arose during the iSCSI plugfest > > at UNH on Wednesday 31-Oct-2001. > > > > (Note: these issues do not take into account any modifications or > > clarifications that occured in the standard due to the issues raised > > on Monday or Tuesday.) > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > ------------------------------------------------------------------------ > > ---- > > > > 1. Are receivers (initiator or target) REQUIRED to check that reserved > > bits and/or fields are set to 0? > > > > Section 3 on page 48 of draft 8 says: > > "Any bits not defined MUST be set to zero. Any reserved fields and > > values MUST be 0 unless specified otherwise." > > > > and Section 8.3 on page 127 of draft 8 says: > > "Explicit violations of the PDU layout rules stated in this > > document > > are format errors. This when detected, usually indicates a major > > implementation flaw in one of the parties. > > > > When a target or an initiator receives an iSCSI PDU with a format > > error, it MUST reset all transport connections in the session > > immediately and escalate the format error to session recovery > > (section 8.11.4)." > > > > According to these rules, a PDU with reserved bits and/or fields that > > are not set to 0 violates the PDU layout rules. Therefore, if an > > initiator or target receives such a PDU, it should immediately close > > all connections in the session and go to session recovery. > > > > Clearly a format error has extremely severe consequences! > > > > Although all vendors are setting reserved bits and fields to 0 on > > PDUs they are sending, many are NOT checking PDUs they are receiving > > to see if these bits and fields are set to 0. Basically, vendors are > > saying "who cares if reserved bits and/or fields in incoming PDUs are > > not zero? We do not want to take the time to do this checking, and > > there is no benefit to doing it. As long as the non-reserved bits > > and > > fields are set properly, we can and should proceed. Any time devoted > > to doing this checking is wasted in 99+% of the cases, and in the > > (unlikely) case that a non-zero bit or field is found, the > > consequences are too severe." > > > > There should be some statement in the standard to clarify what > > checking > > is required and what is optional. > > > > 2. A similar situation arises with respect to checking the consistency > > of fields such as Version-max, Version-min and Version-active in > > Login > > Requests and Login Responses. > > > > For example, consider the Version-max field. > > > > Section 3.12.5 says: > > "All Login requests within the Login phase MUST carry the same > > Version-max." > > > > All vendor initiators are setting Version-max correctly on all > > login requests they are sending, but many vendor targets are NOT > > checking received login requests to ensure that this rule is > > enforced. > > In particular, many targets simply use the Version-max and > > Version-min > > on the first login request they receive on a new connection, and then > > they ignore these fields on all subsequent login requests in the same > > login phase. > > > > Strictly speaking, a change in the Version-max field during the login > > phase constitutes a protocol error according to section 8.8 on page > > 130 > > of draft 8: > > > > "All violations of iSCSI PDU exchange sequences specified in this > > draft are also protocol errors. This category of errors can be > > addressed only by fixing the implementations; iSCSI defines Reject > > and response codes to enable this". > > > > Therefore the target should send back a login response with a status > > of 0x0200 and then close the connection. > > > > However, Section 3.12.5 also says: > > "The target MUST use the value presented with the first login > > request." > > > > This rule seems to imply that the value CAN change, because if it > > cannot > > change, then it doesn't matter which one of the login requests it is > > taken from, they are all the same anyway. > > > > The suggestion is to keep the requirement that the target MUST use > > the > > value presented on the first login request, but to allowed the target > > to ignore the value presented on all subsequent login requests in the > > same login phase. A similar rewording should be done for the other > > fields. > > > > 3. Can commands be sent out of order on the same connection? > > > > The behavior of targets is clearly specified in Section 2.2.2.3 on > > page 25 of draft 8, which says: > > "Except for the commands marked for immediate delivery the iSCSI > > target layer MUST eliver the commands for execution in the order > > specified by CmdSN." > > > > Section 2.2.2.3 on page 26 of draft 8 also says: > > "- CmdSN - the current command Sequence Number advanced by 1 on > > each command shipped except for commands marked for immediate > > delivery." > > but the meaning of the term "shipped" is vague, and does not > > necessarily > > require that the PDUs arrive on the other end of a TCP connection > > in the same order that the CmdSN values were assigned to these PDUs. > > > > Some initiators have been designed to send commands out of CmdSN > > order on one connection. Consider the situation where there is only > > one connection and a high-level dispatcher creates a PDU for a SCSI > > command that involves writing immediate data to the target. This PDU > > is enqueued to a lower-level layer which has to setup, start, and > > wait-for a DMA operation to move the immediate data into an onboard > > buffer before the PDU can be put onto the wire. While this is > > happening, the dispatcher creates another unrelated PDU for a SCSI > > read command (for example), and when this PDU is passed to the > > lower-level layer it can be sent immediately, ahead of the previous > > write PDU and therefore out of order on this connection. > > > > The standard clearly allows this to happen if the two PDUs were sent > > on different connections, and seems to imply that this can also > > happen > > when the two PDUs are sent on the same connection. > > > > The suggestion is to put in the standard an explicit statement that > > this is allowed or not allowed, as appropriate. > > > > If this is allowed, such a statement would avoid the erroneous > > assumption being made by some target implementers that within a > > single > > connection, commands will arrive in order. > > > > If this is not allowed, such a statement would avoid the erroneous > > assumption being made by some initiator implementers that within a > > single connection, commands can be put on the wire out of order. > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > now allow: "A value of 0 indicates no limit." > > > > Is this useful? Does it buy anything? > > > > The difficulties implementers are having with this are: > > > > 1) It is a special case. > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > 3) It violates the min/max function normally used for the key. > > 4) There is always a limit anyway. > > > > Consider FirstBurstSize, which can have a value that is described > > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > > is selected. > > > > I one side offers 0 to mean unlimited, and the other side > > has a limit, it will reply with that limit, say 128 Kbytes. > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > The statement in the standard that "the minimum of the 2 numbers is > > selected" is therefore wrong when one of the numbers is 0. > > > > Furthermore, when an initiator or target receives an offer for one > > of these keys, it cannot simply check that the offered value is > > legal by testing it against some minimum and maximum. It must first > > check for 0 and then only if that check shows the value is non-zero > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > Finally, there is always a limit. If nothing else it is the > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > requesting the transfer. It is useless to specify a FirstBurstSize > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > the largest possible DataSegmentLength in any PDU that can use > > this value is 2**24-1. > > > > The suggestion is to just eliminate this special case of 0 and > > require > > that the range 64-to-(2**24-1) be used instead -- it has exactly the > > same effect in all cases, it is easier to describe in the standard > > because it avoids all the extra words, and it is easier to code > > because it avoids all the special cases. > > > > NOTE: the standard should specify the limit in the ranges for > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > DataSegmentLength field and therefore can never be used. > > > > 5. This is a suggestion for a minor rewording in the standard to avoid > > misunderstandings. > > > > In Appendix E on page 188 of draft 8 it says: > > > > "The response to this command is a text response containing a list > > of > > targets and their addresses. Each target is returned as a target > > record. A target record begins with the TargetName text key, > > followed by a list of TargetAddress text keys, ..." > > > > In fact, there are situations where there are no targets and/or no > > addresses. These situations are clearly defined in the draft after > > the sentences quoted above, but it would help if those sentences > > at least hinted at the possibility that the lists could be empty > > or might not contain addresses. A possible rewording would be: > > > > "The response to this command is a text response containing a list > > of > > zero or more targets and, optionally, their addresses. Each target > > is returned as a target record. A target record begins with the > > TargetName text key, followed by a list of zero or more > > TargetAddress text keys, ..." > > > > > > 6. This is a suggestion for another very minor rewording in the > > standard. > > > > At the end of section 2.2.3 on page 29 of draft 8 it says: > > > > "Before full feature phase is established, only Login PDUs are > > allowed. ..." > > > > The suggested rewording is: > > > > "Before full feature phase is established, only Login Request and > > Login Response PDUs are allowed. ..." > > ----- Forwarded by Julian Satran/Haifa/IBM on 05-11-01 10:37 ----- Paul Koning Sent by: owner-ips@ece.cmu.edu 03-11-01 01:01 Please respond to Paul Koning To: rcohen@eurologic.com cc: ips@ece.cmu.edu Subject: Re: iSCSI: current UNH Plugfest: Reserved bits Excerpt of message (sent 2 November 2001) by Ron Cohen: > The problem with accepting values in reserved bits is that when the reserved > bit is no longer reserved (evolution in the standard) it gives the > impression to the initiator that a target implements the new feature when it > may actually only be ignoring it. Yes, if you don't do some other things that are needed. You MUST have protocol version numbers in the initial messages. Otherwise you are utterly lost. With that, you know what version the other end has. The key point is that requiring reserved fields to be ignored is that you can rely on the fact that new things can be asked for in reserved fields with the knowledge that old implementations will ignore those things. Quite often this is what you want. Occasionally it is not; for those cases you make use of the version number. In any case, what I was describing is long established practice in many protocols that have successfully navigated the version number migration path. paul From owner-ips@ece.cmu.edu Mon Nov 5 10:42:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA24617 for ; Mon, 5 Nov 2001 10:42:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5EoAb03022 for ips-outgoing; Mon, 5 Nov 2001 09:50:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mayfair.vxindia.veritas.com (bay-bridge.veritas.com [143.127.3.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5Eghl02480 for ; Mon, 5 Nov 2001 09:42:43 -0500 (EST) Received: from RAHULB (rahulb.vxindia.veritas.com [10.212.80.59]) by mayfair.vxindia.veritas.com (8.9.3/8.9.3) with SMTP id TAA09353 for ; Mon, 5 Nov 2001 19:00:58 +0530 (IST) Message-ID: <012a01c165fe$2dfedae0$3b50d40a@vxindia.veritas.com> From: "Rahul Bhagwat" To: Subject: Connection Recovery Date: Mon, 5 Nov 2001 19:01:28 +0530 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0127_01C1662C.46283980" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0127_01C1662C.46283980 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, Is there any order in task reassignments and connection logout (implicit = or explicit) during a connection recovery.=20 If these two are not related, what is the use of moving the connection = to CLEANUP_WAIT state? CLEANUP_WAIT state typically means that there are pending tasks = for this connection due to which it cannot be moved to FREE state. That is only = difference=20 betweeen FREE state and CLEANUP_WAIT state. Which probably means that it is mandatory that Task reassigment happens = before logging out a failed connection (in CLEANUP_WAIT state). Once a CSM-E or = a CSM-I drives the connection to free state, all the pending tasks need to be = freed up. Am I correct here? Regards, Rahul ------=_NextPart_000_0127_01C1662C.46283980 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi,
 
Is there any order in task reassignments and connection logout = (implicit or=20 explicit)
during a connection recovery.
 
If these two are not related, what is the use of moving the = connection to=20 CLEANUP_WAIT
state? CLEANUP_WAIT state typically means that there are pending = tasks for=20 this
connection due to which it cannot be moved to FREE state. That is = only=20 difference
betweeen FREE state and CLEANUP_WAIT state.
 
Which probably means that it is mandatory that Task reassigment = happens=20 before
logging out a failed connection (in CLEANUP_WAIT state). Once a = CSM-E or a=20 CSM-I
drives the connection to free state, all the pending tasks need to = be freed=20 up.
 
Am I correct here?
 
Regards,
Rahul
------=_NextPart_000_0127_01C1662C.46283980-- From owner-ips@ece.cmu.edu Mon Nov 5 12:01:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28266 for ; Mon, 5 Nov 2001 12:01:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5FJfM05349 for ips-outgoing; Mon, 5 Nov 2001 10:19:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from exch-connector.netcomsystems.com (mushroom.netcomsystems.com [12.9.24.195]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5FJdl05344 for ; Mon, 5 Nov 2001 10:19:39 -0500 (EST) Received: by exch-connector.netcomsystems.com with Internet Mail Service (5.5.2653.19) id ; Mon, 5 Nov 2001 07:19:33 -0800 Message-ID: <9384475DFC05D2118F9C00805F6F263106DDB91B@exchange1.netcomsystems.com> From: "Cunningham, Howard" To: ips@ece.cmu.edu Subject: Login interoperability versus efficiency Date: Mon, 5 Nov 2001 07:19:29 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C1660D.43B4E0A0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C1660D.43B4E0A0 Content-Type: text/plain; charset="iso-8859-1" I know that this is probably too late but I am concerned that some of iSCSI source code that I have seen indicates some development centers may not be considering all of the possible complexities of the session and connection establishment. Since I am a test equipment vendor, I have been silently waiting as the login process has settled and hoping it would converge on a solution with assured interoperability. It seems that efficiency (minimize the number of PDU exchanges) has been preferred to simplicity. Let me ask a question to higlight my concern: Assume I am an Initiator who would like to get to full-feature phase as quickly as possible but I am willing authenticate. So I issue: I-> Login (CSG=0, NSG=3,T=1) InitiatorName=iqn.1999-07.com.os.hostif.77 TargetName=iqn.1999-07.com.acme.diskarray.sn.88 AuthMethod=SRP,none SessionType=normal FMarker=send,no ImmediateData=yes,no InitiatR2T=yes,no As I read the draft I believe this is a valid PDU. It is not clear to me: 1. Can the operational parameters be negotiated by the target along with the AuthMethod=SRP? T->Login (CSG=0, NSG=1,T=0) AuthMethd=SRP FMarker=receive ImmediateData=no InitialR2T=no 2. Does the target 'defer' the operational parameters until after authentication (so that they are not 'revealed' to an unauthenticated party? 3. Does the target ignore or reject the initiator operational parameters? 4. Is this implementation dependent? Just as authentication messages are rigid in nature (e.g., Intiator MUST send TargetAuth=yes along with SRP-U and not later with SRP_M - why not make two kinds of SRP - unidirectional, bidirectional?), why not make the entire login process more structured? At one time there were four possible values to CSG and NSG (I believe) and I would support a return to them with a more structured process. What I propose is: 1. Stage 0 =Identication stage - Intiator and target identify each other, decide type of session and what kind of authentication 1=Authentication stage - Initiator and target authenticate each other (optional) 2=Operational stage - Initiator and target explicitly agree on operational parameters 3=FullFeature stage 2. The first login request PDU (CSG=0) for any connection (new session or add-on) from the Initiator can only contain ONLY: InitiatorName {no default}, AuthMethod {no default}, SessionType (default normal), TargetName {if SessionType=normal, no default} Arguably not sending a TargetName could imply a discovery session. These keys can ONLY be in this first PDU. 3. The first login response PDU from the target can only contain: AuthMethod=... with NSG=1 OR "login accepted" with NSG=2 OR "login rejected" 4. After the Initiator and Target have agreed to the authentication method if any, they exchange stage 1 authentication PDUs as appropriate and then transition to stage 2. 5. In Stage 2, the Initiator and Target negotiate operational parameters using ONLY TextOut PDUs and then transition to stage 3. I could live with using Login PDUs here instead of TextOut since the CSG has changed. 6. TextOut PDUs in stage 3 could be used ONLY to query operational parameters (which I think could be eliminated entirely). 7. With the more rigid process, the CSG and NSG are not really needed in the PDUs. So for the 'simple' no authentication Discovery session, there would be 1 login PDU exchange the Initiator TextOut PDU (MaxRcvPDULength=..., SendTargets=...) the Target TextOut replies 1 logout PDU exchange For the 'simple' no authentication Normal session, there would be 1 login PDU exchange the Initiator TextOut PDU with negotiated operational parameters the Target TextOut reply negotiation results ----- full feature ------------- 1 logout PDU exchange In this case, there is a mandatory, 'possibly extra' TextOut exchange to get to full-feature phase. I would argue that this is not significant for performance. Structure can be used to make things simpler and ease interoperability. Regards... Howard Cunningham, Senior MTS Spirent Communications 900 Main Campus Drive, Suite 201 Raleigh, NC 27606 Voice: +1 (919) 829-6332 Fax: +1 (919) 829-6400 Email: howard.cunningham@spirentcom.com ------_=_NextPart_001_01C1660D.43B4E0A0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Login interoperability versus efficiency

I know that this is probably too late = but I am concerned that some of iSCSI source code that I have seen = indicates some development centers may not be considering all of the = possible complexities of the session and connection = establishment.

Since I am a test equipment vendor, I = have been silently waiting as the login process has settled and hoping = it would converge on a solution with assured interoperability.  It = seems that efficiency (minimize the number of PDU exchanges) has been = preferred to simplicity. Let me ask a question to higlight my = concern:

Assume I am an Initiator who would = like to get to full-feature phase as quickly as possible but I am = willing authenticate. So I issue:

I-> Login (CSG=3D0, = NSG=3D3,T=3D1)
        InitiatorName=3Diqn.1999-07.com.os.hostif.77
        TargetName=3Diqn.1999-07.com.acme.diskarray.sn.88
        AuthMethod=3DSRP,none
        SessionType=3Dnormal
        FMarker=3Dsend,no
        ImmediateData=3Dyes,no
        InitiatR2T=3Dyes,no
As I read the draft I believe this is = a valid PDU. It is not clear to me:
1. Can the operational parameters be = negotiated by the target along with the AuthMethod=3DSRP?
T->Login (CSG=3D0, = NSG=3D1,T=3D0)
        AuthMethd=3DSRP
        FMarker=3Dreceive
        ImmediateData=3Dno
        InitialR2T=3Dno
2. Does the target 'defer' the = operational parameters until after authentication (so that they are not = 'revealed' to an unauthenticated party?

3. Does the target ignore or reject = the initiator operational parameters?
4. Is this implementation = dependent?

Just as authentication messages are = rigid in nature (e.g., Intiator MUST send TargetAuth=3Dyes along with = SRP-U and not later with SRP_M - why not make two kinds of SRP - = unidirectional, bidirectional?), why not make the entire login process = more structured? At one time there were four possible values to CSG and = NSG (I believe) and I would support a return to them with a more = structured process. What I propose is:

1. Stage
        0 =3DIdentication stage - Intiator and target identify = each other, decide type of session and what kind of = authentication

        1=3DAuthentication stage - Initiator and target = authenticate each other (optional)
        2=3DOperational stage - Initiator and target explicitly = agree on operational parameters
        3=3DFullFeature stage
2. The first login request PDU = (CSG=3D0) for any connection (new session or add-on) from the Initiator = can only contain ONLY:

        InitiatorName {no default}, AuthMethod {no default}, = SessionType (default normal), TargetName {if SessionType=3Dnormal, no = default}

Arguably not sending a TargetName = could imply a discovery session. These keys can ONLY be in this first = PDU.
3. The first login response PDU from = the target can only contain:
        AuthMethod=3D... with NSG=3D1 OR "login = accepted" with NSG=3D2 OR "login rejected"
4. After the Initiator and Target = have agreed to the authentication method if any, they exchange stage 1 = authentication PDUs as appropriate and then transition to stage = 2.

5. In Stage 2, the Initiator and = Target negotiate operational parameters using ONLY TextOut PDUs and = then transition to stage 3. I could live with using Login PDUs here = instead of TextOut since the CSG has changed.

6. TextOut PDUs in stage 3 could be = used ONLY to query operational parameters (which I think could be = eliminated entirely).

7. With the more rigid process, the = CSG and NSG are not really needed in the PDUs.

So for the 'simple' no authentication = Discovery session, there would be
        1 login PDU exchange
        the Initiator TextOut PDU (MaxRcvPDULength=3D..., = SendTargets=3D...)
        the Target TextOut replies
        1 logout PDU exchange

For the 'simple' no authentication = Normal session, there would be
        1 login PDU exchange
        the Initiator TextOut PDU with negotiated operational = parameters
        the Target TextOut reply negotiation results
        ----- full feature -------------
        1 logout PDU exchange
In this case, there is a mandatory, = 'possibly extra' TextOut exchange to get to full-feature phase. I would = argue that this is not significant for performance. Structure can be = used to make things simpler and ease interoperability.

Regards...
 
Howard Cunningham, Senior MTS
Spirent Communications
900 Main Campus Drive, Suite = 201
Raleigh, NC 27606
Voice: +1 (919) 829-6332
Fax: +1 (919) 829-6400
Email: = howard.cunningham@spirentcom.com


------_=_NextPart_001_01C1660D.43B4E0A0-- From owner-ips@ece.cmu.edu Mon Nov 5 13:22:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA01946 for ; Mon, 5 Nov 2001 13:22:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5HQmq16091 for ips-outgoing; Mon, 5 Nov 2001 12:26:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5HQkl16087 for ; Mon, 5 Nov 2001 12:26:46 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA143382 for ; Mon, 5 Nov 2001 18:26:40 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA5HQb374238 for ; Mon, 5 Nov 2001 18:26:37 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: Login interoperability versus efficiency X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 5 Nov 2001 19:26:34 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 05/11/2001 19:26:36, Serialize complete at 05/11/2001 19:26:36 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Howard, Comments in text - Julo "Cunningham, Howard" Sent by: owner-ips@ece.cmu.edu 05-11-01 17:19 Please respond to "Cunningham, Howard" To: ips@ece.cmu.edu cc: Subject: Login interoperability versus efficiency I know that this is probably too late but I am concerned that some of iSCSI source code that I have seen indicates some development centers may not be considering all of the possible complexities of the session and connection establishment. Since I am a test equipment vendor, I have been silently waiting as the login process has settled and hoping it would converge on a solution with assured interoperability. It seems that efficiency (minimize the number of PDU exchanges) has been preferred to simplicity. Let me ask a question to higlight my concern: Assume I am an Initiator who would like to get to full-feature phase as quickly as possible but I am willing authenticate. So I issue: I-> Login (CSG=0, NSG=3,T=1) InitiatorName=iqn.1999-07.com.os.hostif.77 TargetName=iqn.1999-07.com.acme.diskarray.sn.88 AuthMethod=SRP,none SessionType=normal FMarker=send,no ImmediateData=yes,no InitiatR2T=yes,no As I read the draft I believe this is a valid PDU. It is not clear to me: 1. Can the operational parameters be negotiated by the target along with the AuthMethod=SRP? ++++ Operational parameters can't be negotiated together with authentication This sequence will result in the session being dropped. +++ T->Login (CSG=0, NSG=1,T=0) AuthMethd=SRP FMarker=receive ImmediateData=no InitialR2T=no 2. Does the target 'defer' the operational parameters until after authentication (so that they are not 'revealed' to an unauthenticated party? 3. Does the target ignore or reject the initiator operational parameters? 4. Is this implementation dependent? +++ It was the group's will that parameters be segregated in stages. ++++ Just as authentication messages are rigid in nature (e.g., Intiator MUST send TargetAuth=yes along with SRP-U and not later with SRP_M - why not make two kinds of SRP - unidirectional, bidirectional?), why not make the entire login process more structured? At one time there were four possible values to CSG and NSG (I believe) and I would support a return to them with a more structured process. What I propose is: 1. Stage 0 =Identication stage - Intiator and target identify each other, decide type of session and what kind of authentication 1=Authentication stage - Initiator and target authenticate each other (optional) 2=Operational stage - Initiator and target explicitly agree on operational parameters 3=FullFeature stage +++ There are 3 distinct stages - and consesus on them was reached +++ 2. The first login request PDU (CSG=0) for any connection (new session or add-on) from the Initiator can only contain ONLY: InitiatorName {no default}, AuthMethod {no default}, SessionType (default normal), TargetName {if SessionType=normal, no default} Arguably not sending a TargetName could imply a discovery session. These keys can ONLY be in this first PDU. 3. The first login response PDU from the target can only contain: AuthMethod=... with NSG=1 OR "login accepted" with NSG=2 OR "login rejected" 4. After the Initiator and Target have agreed to the authentication method if any, they exchange stage 1 authentication PDUs as appropriate and then transition to stage 2. 5. In Stage 2, the Initiator and Target negotiate operational parameters using ONLY TextOut PDUs and then transition to stage 3. I could live with using Login PDUs here instead of TextOut since the CSG has changed. 6. TextOut PDUs in stage 3 could be used ONLY to query operational parameters (which I think could be eliminated entirely). 7. With the more rigid process, the CSG and NSG are not really needed in the PDUs. So for the 'simple' no authentication Discovery session, there would be 1 login PDU exchange the Initiator TextOut PDU (MaxRcvPDULength=..., SendTargets=...) the Target TextOut replies 1 logout PDU exchange For the 'simple' no authentication Normal session, there would be 1 login PDU exchange the Initiator TextOut PDU with negotiated operational parameters the Target TextOut reply negotiation results ----- full feature ------------- 1 logout PDU exchange In this case, there is a mandatory, 'possibly extra' TextOut exchange to get to full-feature phase. I would argue that this is not significant for performance. Structure can be used to make things simpler and ease interoperability. Regards... ++++ I suggest you reread section 5 +++ Howard Cunningham, Senior MTS Spirent Communications 900 Main Campus Drive, Suite 201 Raleigh, NC 27606 Voice: +1 (919) 829-6332 Fax: +1 (919) 829-6400 Email: howard.cunningham@spirentcom.com From owner-ips@ece.cmu.edu Mon Nov 5 14:53:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05261 for ; Mon, 5 Nov 2001 14:53:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5IkHA22369 for ips-outgoing; Mon, 5 Nov 2001 13:46:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.windriver.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5IkFl22362 for ; Mon, 5 Nov 2001 13:46:16 -0500 (EST) Received: from london ([147.11.45.211]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id KAA02466; Mon, 5 Nov 2001 10:45:38 -0800 (PST) From: "Rod Harrison" To: "Julian Satran" , Subject: iSCSI: Out of order commands, was current UNH Plugfest Date: Mon, 5 Nov 2001 10:48:40 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit [ Subject changed ] Julian, The ordering difference is introduced between the host side driver and the iSCSI HBA. The host side driver must present SCSI commands to the HBA in the order they are received from the OS to prevent read after write dependency failures. The HBA might reorder the commands depending on when DMA completes. The reordering can't be done ahead of time in the host driver since it doesn't know how long each DMA might take. As long as the HBA assigns CmdSN in the order it receives commands the desired host ordering is preserved. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Monday, November 05, 2001 12:35 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: current UNH Plugfest Rod, I all examples give the point I find hard to understand is why is the ordering on the wire different from the presentation order to the initiator. You can get as many overlaps as you want by presenting the commands to the initiator in the desired order. What we are considering here is the case in which you want to ship in an order different than the one you present the commands. Julo "Rod Harrison" Sent by: owner-ips@ece.cmu.edu 04-11-01 04:42 Please respond to "Rod Harrison" To: "Barry Reinhold" , "Dave Sheehy" , "IETF IP SAN Reflector" cc: Subject: RE: iSCSI: current UNH Plugfest Barry, In general I agree but I don't think this is as much of a corner case as it at first appears. Targets will have code very similar to that needed to handle out of order commands to deal with digest errors. Targets also need to queue commands whilst waiting for both solicited and unsolicited data to arrive. Queuing out of order commands seems little extra work. From an initiators point of view there are efficiency, and probably performance gains to be had from sending commands out of order. Bob Russell gave the example of a read being sent whilst write data DMA is happening, and a similar situation can arise with DMA for writes overtaking that of earlier writes if the initiator has multiple DMA engines. In this case the initiator might be forced to let the wire go idle if it can't send the data from completed DMAs as soon as possible. We already have a command queue at the target to enforce correct serialisation of commands, doing the same thing at the initiator is redundant. Finally, I don't believe we should be writing a standard to work around poor coding and test coverage, especially at the cost of potential efficiency gains. I agree with Dave and Santosh that commands being sent out of order on a single session should be allowed by the standard. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Barry Reinhold Sent: Friday, November 02, 2001 5:24 PM To: Dave Sheehy; IETF IP SAN Reflector Subject: RE: iSCSI: current UNH Plugfest Using features such as out of order command delivery on a connection tend to be the sort of things that lead to interoperability problems. It is unexpected and probably going to hit poorly tested code paths even if the standard is written to allow it. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Dave Sheehy >Sent: Friday, November 02, 2001 4:19 PM >To: IETF IP SAN Reflector >Subject: Re: iSCSI: current UNH Plugfest > > > >> 3. Can commands be sent out of order on the same connection? >> >> The behavior of targets is clearly specified in Section 2.2.2.3 on >> page 25 of draft 8, which says: >> "Except for the commands marked for immediate delivery the iSCSI >> target layer MUST eliver the commands for execution in the order >> specified by CmdSN." >> >> Section 2.2.2.3 on page 26 of draft 8 also says: >> "- CmdSN - the current command Sequence Number advanced by 1 on >> each command shipped except for commands marked for immediate >> delivery." >> but the meaning of the term "shipped" is vague, and does not >> necessarily >> require that the PDUs arrive on the other end of a TCP connection >> in the same order that the CmdSN values were assigned to these PDUs. >> >> Some initiators have been designed to send commands out of CmdSN >> order on one connection. Consider the situation where there is only >> one connection and a high-level dispatcher creates a PDU for a SCSI >> command that involves writing immediate data to the target. This PDU >> is enqueued to a lower-level layer which has to setup, start, and >> wait-for a DMA operation to move the immediate data into an onboard >> buffer before the PDU can be put onto the wire. While this is >> happening, the dispatcher creates another unrelated PDU for a SCSI >> read command (for example), and when this PDU is passed to the >> lower-level layer it can be sent immediately, ahead of the previous >> write PDU and therefore out of order on this connection. >> >> The standard clearly allows this to happen if the two PDUs were sent >> on different connections, and seems to imply that this can also happen >> when the two PDUs are sent on the same connection. >> >> The suggestion is to put in the standard an explicit statement that >> this is allowed or not allowed, as appropriate. >> >> If this is allowed, such a statement would avoid the erroneous >> assumption being made by some target implementers that within a single >> connection, commands will arrive in order. >> >> If this is not allowed, such a statement would avoid the erroneous >> assumption being made by some initiator implementers that within a >> single connection, commands can be put on the wire out of order. >> >> +++ >> >> will add an explicit statement saying that this behaviour is forbidden. >> 2.2.2.1 will contain: >> >> On any given connection, the iSCSI initiator MUST send the >commands in the >> order specified by CmdSN. >> >> +++ > >Why do you feel this behavior should be forbidden? Targets already have to >order commands across the session. I don't see why it's a problem to extend >that to the connection as well. I, for one, believe we should take >a liberal >stance on this. > >Dave Sheehy > From owner-ips@ece.cmu.edu Mon Nov 5 16:05:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA07773 for ; Mon, 5 Nov 2001 16:05:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5K8bB29515 for ips-outgoing; Mon, 5 Nov 2001 15:08:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5K8Fl29472 for ; Mon, 5 Nov 2001 15:08:34 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel10.hp.com (Postfix) with ESMTP id 707411F6E1; Mon, 5 Nov 2001 12:08:09 -0800 (PST) Received: from swathi (swathi.rose.hp.com [15.43.213.237]) by core.rose.hp.com with SMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id MAA23966; Mon, 5 Nov 2001 12:09:35 -0800 (PST) Message-ID: <005701c16635$96597730$edd52b0f@rose.hp.com> From: "Mallikarjun C." To: "Rahul Bhagwat" Cc: References: <012a01c165fe$2dfedae0$3b50d40a@vxindia.veritas.com> Subject: Re: Connection Recovery Date: Mon, 5 Nov 2001 12:08:07 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Rahul, A successful connection logout (implicit or explicit) must precede the task reassignments during a connection recovery operation. But please note that the notion of "connection cleanup" (graceful closing of a previously operational iSCSI connection) in the state diagrams goes beyond the connection recovery (in fact, that is the reason I renamed from its previous name, please refer to my email to ips on 11/2/01 with the slide posting announcement). A connection cleanup is highly desirable even in the absence of task reassignment, to quickly reclaim the tags and buffers on either end (or, both sides would have to wait for a connection timeout to happen, symbolized by the M1 transition). >Once a CSM-E or a CSM-I >drives the connection to free state, all the pending tasks need to be freed up. Not correct. The decision to free up the pending tasks is depedent on the operational ErrorRecoveryLevel in the CSM-I case (please look at the discussion in section 3.12.2), or is dependent on the Logout reason code (recovery Vs close) in the CSM-E case. All the FREE state symbolizes really is that the iSCSI connection is gracefully closed with a successful explicit/implit iSCSI Logout. The pending tasks at this point have no connection allegiance, and are loosely "owned" by the session. It is legitimate for the pending tasks to be existent (waiting for reassignment) even when all the connections reported FREE (please look at the discussion under 3.15.2, Time2Retain). Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization Hewlett-Packard MS 5668 Roseville CA 95747 ----- Original Message ----- From: Rahul Bhagwat To: ips@ece.cmu.edu Sent: Monday, November 05, 2001 5:31 AM Subject: Connection Recovery Hi, Is there any order in task reassignments and connection logout (implicit or explicit) during a connection recovery. If these two are not related, what is the use of moving the connection to CLEANUP_WAIT state? CLEANUP_WAIT state typically means that there are pending tasks for this connection due to which it cannot be moved to FREE state. That is only difference betweeen FREE state and CLEANUP_WAIT state. Which probably means that it is mandatory that Task reassigment happens before logging out a failed connection (in CLEANUP_WAIT state). Once a CSM-E or a CSM-I drives the connection to free state, all the pending tasks need to be freed up. Am I correct here? Regards, Rahul From owner-ips@ece.cmu.edu Mon Nov 5 16:58:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA09231 for ; Mon, 5 Nov 2001 16:58:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5KtdJ03588 for ips-outgoing; Mon, 5 Nov 2001 15:55:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e24.nc.us.ibm.com (e24.nc.us.ibm.com [32.97.136.230]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5KtZl03581 for ; Mon, 5 Nov 2001 15:55:36 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e24.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA66238 for ; Mon, 5 Nov 2001 14:52:56 -0600 Received: from d04nms41.raleigh.ibm.com (d04nms41.raleigh.ibm.com [9.67.228.19]) by southrelay02.raleigh.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA5KsCe93992 for ; Mon, 5 Nov 2001 15:54:13 -0500 Subject: spec revs to make TargetName reqd on every connection To: "Julian Satran" Cc: "John Hufferd" , "Jim Hafner" , ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Andre Asselin" Date: Mon, 5 Nov 2001 15:55:17 -0500 X-MIMETrack: Serialize by Router on D04NMS41/04/M/IBM(Release 5.0.8 |June 18, 2001) at 11/05/2001 03:54:12 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Attached is an attempt to pull together all the required spec updates to make InitiatorName required on every login, TargetName required on every normal login, and clarify related text. Does this look good to everyone? Any places I missed? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Appendix D, TargetName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). To This key must be provided by the initiator of the TCP connection to the remote endpoint in the first login request if the initiator is not establishing a discovery session. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). Appendix D, InitiatorName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The Initiator key enables the initiator to identify itself to the remote endpoint. To This key must be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The InitiatorName key enables the initiator to identify itself to the remote endpoint. The current version of the 6th paragraph in chapter 5 reads: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. A suggested rewrite would be (building on the text suggested by Bob Russell): All initial Login requests MUST include the InitiatorName key=value pair. If the initial Login request is also a leading Login (TSID=0) and the new session is to be a discovery session, then the initial Login request MUST also include the SessionType=discovery key=value pair. If the initial Login request is a leading Login and the new session is to be a normal session, then the initial Login request MUST also include the TargetName key=value pair and MAY also include the SessionType=normal key=value pair. All initial Login requests that are not also a leading Login (TSID != 0) MUST include the TargetName key=value pair. Also, this text appears in 2.2.7: The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session. The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. A suggested rewrite would be: The initiator must present its iSCSI Initiator Name in the first login request. If the initiator is not establishing a discovery session (see 2.4), it also must present the iSCSI Target Name to which it wishes to connect in the first login request. The key "SessionType=discovery" is sent by the initiator on the Initial Login request to indicate a discovery session. See chapter 5 for a more detailed description of the Login process. Section 3.12.8 currently reads: The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies the session with that initiator. Suggested rewrite (melding current text w/John's rewrite): The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity). From owner-ips@ece.cmu.edu Mon Nov 5 16:58:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA09245 for ; Mon, 5 Nov 2001 16:58:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA5L1li04114 for ips-outgoing; Mon, 5 Nov 2001 16:01:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e23.nc.us.ibm.com (e23.nc.us.ibm.com [32.97.136.229]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA5L1il04105 for ; Mon, 5 Nov 2001 16:01:45 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e23.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA68562 for ; Mon, 5 Nov 2001 14:59:08 -0600 Received: from d04nms41.raleigh.ibm.com (d04nms41.raleigh.ibm.com [9.67.228.19]) by southrelay02.raleigh.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA5L0Qe152240 for ; Mon, 5 Nov 2001 16:00:26 -0500 Subject: Re: portal group is a component of ___? To: "Jim Hafner" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Andre Asselin" Date: Mon, 5 Nov 2001 16:01:31 -0500 X-MIMETrack: Serialize by Router on D04NMS41/04/M/IBM(Release 5.0.8 |June 18, 2001) at 11/05/2001 04:00:26 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, The thing that bothers me about the current picture is that network portals are not only pictured within a portal group, but also within an iSCSI TARGET. When I see a box within a box, the concept I get is "is-a-component-of", which isn't the case here. Since network portals are a component of a network entity and can be shared between iSCSI targets, I think its clearer to show them pictured within the network entity and connected to targets/portal groups. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: Andre Asselin/Raleigh/IBM@IBMUS 11/02/2001 cc: ips@ece.cmu.edu 04:59 PM From: Jim Hafner/Almaden/IBM@IBMUS Subject: Re: portal group is a component of ___?(Document link: Andre Asselin) Andre, The network portals are components of the network entity (these are possibly shared resources). The portal groups are components of the iSCSI Node. One portal group for one iSCSI Node may span an overlapping but non-equal set of network portals as another portal group for a different iSCSI Node within the same network entity. In effect, the portal groups are possibly logical constructs within a node. This is implied by the current language of the model because session coordination is a function of an iSCSI node and portal groups are collections of network portals over which the session coordinator can handle multi-connection sessions. BUT, I agree that your words: A Portal Group is a component of an iSCSI node that defines a set of Network Portals that collectively supports the capability of coordinating a session with connections spanning these portals. make this more clear. Julo? Can you make this change (in the definition of Portal Groups in the model clause)? In your note attached, you respond to my first comment with "you mean portal groups, not targets". No, I meant targets! One named "bar" and one named "foobar". They share the same network portal. Each may have a different view of how they lay portal groups across the shared (or not shared) network portals. But I think we have this straight now. In the previous thread, and concurred by John Hufferd, I've already made the claim that the draft is ambiguous about the TargetName use -- and have advocated clarification of the language. As for redrawing the picture... It is very difficult in ASCII to draw a picture that shows the complexity of possible ways of putting all these pieces together. It's even difficult (as John will attest) to put them in a fancy graphics picture. Also, the intent of the drawing is to encapsulate what ONE target sees. Perhaps your change that the network portals are "attached to" but not "contained in" portal groups is better and more consistent with the first picture in the model clause. I have no objection to this change either. (Julo?). Thanks for the contributions! Jim Hafner Andre Asselin 11/02/2001 01:36 pm To: Jim Hafner/Almaden/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: portal group is a component of ___? (Document link: Database 'Jim Hafner', View '($Inbox)') Jim, First, let me say thanks-- this thread has educated me immensly on some of the subtle aspects of iSCSI. >Your picture isn't quite right. The portal group tag is relative to the iSCSI target and that target is *uniquely* identified by it's name. So, in your picture, there are two targets (not one). I think you mean portal groups, not targets. >Each can label its target portal group any way it wants (independent of the other). Each has independent control over the TSIDs it uses. Each may *share* use of a network portal, but that's a different issue. The model implies two independent targets (even if they live in the same network entity and share resources) in your scenario. In other words, the portal groups are wrt targets not the network entity. Thanks for the clarification-- I was missing this understanding of the architecture. > In your scenario, whatever layer (you put it in the network code) has to decide what session to add the connection to has the initiator name, the ISID, the TSID *and* the target name (you left this out) in the login request. That fully qualifies both the target and the session as well. I concur that that is how a target * should * identify a session, however, that's not what the spec currently says, and needs to be updated. Now, if you can put up with one more question, is a network portal a component of a network entity or an iSCSI node? 2.5.1 (c) says its a component of a network entity, but 2.5.1(d) says A Portal Group defines a set of Network Portals within an iSCSI Node that collectively supports the capability of coordinating a session with connections spanning these portals. which is kind of ambiguous as to whether it means portal group is within an iSCSI node, or the network portals are within the iSCSI node. I'd suggest this to clarify it: A Portal Group is a component of an iSCSI node that defines a set of Network Portals that collectively supports the capability of coordinating a session with connections spanning these portals. Also if a network portal really is a component of a network entity, shouldn't the diagram on the following page be redrawn? (original) ----------------------------IP Network--------------------- | | | +--------|---------------|--------------------|---------------------+ | +----|---------------|-----+ +----|---------+ | | | +---------+ +---------+ | | +---------+ | | | | | Network | | Network | | | | Network | | | | | | Portal | | Portal | | | | Portal | | | | | +--|------+ +---------+ | | +---------+ | | | | | | | | | | | | | | Portal | | | | Portal | | | | | Group 1 | | | | Group 2 | | | +--------------------------+ +--------------+ | | | | | | | +----------------------------+ +-----------------------------+ | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | +----------------------------+ +-----------------------------+ | | | | iSCSI Target Node | | (within Network Entity, not shown) | +-------------------------------------------------------------------+ (corrected?) ----------------------------IP Network--------------------- | | | +----------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +--------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +------|-------+ | | | | | Portal | | Portal | | | | | | Group 1 | | Group 2 | | | | | +--------------------------+ +--------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-------------------------------------------------------------------+ | | | | Network Entity | +-----------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: Andre Asselin/Raleigh/IBM@IBMUS 11/01/2001 03:39 PM cc: ips@ece.cmu.edu From: Jim Hafner/Almaden/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, Your picture isn't quite right. The portal group tag is relative to the iSCSI target and that target is *uniquely* identified by it's name. So, in your picture, there are two targets (not one). Each can label its target portal group any way it wants (independent of the other). Each has independent control over the TSIDs it uses. Each may *share* use of a network portal, but that's a different issue. The model implies two independent targets (even if they live in the same network entity and share resources) in your scenario. In other words, the portal groups are wrt targets not the network entity. In your scenario, whatever layer (you put it in the network code) has to decide what session to add the connection to has the initiator name, the ISID, the TSID *and* the target name (you left this out) in the login request. That fully qualifies both the target and the session as well. Jim Hafner Andre Asselin 11/01/2001 12:15 pm To: Jim Hafner/Almaden/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? (Document link: Database 'Jim Hafner', View '($Inbox)') Jim, I agree with what you say, except for the part that mapping TSID=target portal group tag will work. Let's assume the following architecture: I have a network entity with one network portal (and thus one portal group). Inside this entity lives two iSCSI targets. Scenerio: An iSCSI initiator opens a connection to the network portal in order to add a connection to an already existing session (the network entity's networking code knows this because TSID in the initial login request is 0). The networking code needs to determine what session the initiator wants to add to, so it compares some items from the initial login request to each of the established sessions until it finds a match. The question is what items should it compare to identify a match? Section 3.12.9 of the spec reads: The TSID is the target assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator. As you say, this is clearly target centric (because, for example, an initiator could have 2 different sessions open to two different targets that have the same TSID). But from a target point of view, what that text means to me that the network entity's networking code should compare ISID + InitiatorName + TSID to determine a match. That implies that TSID must be unique per TargetName and per portal group ID. If TSID is just the target portal group tag, then the comparison wouldn't work. For example, using the architecture above where there is just one portal group, the target could have two sessions open: session A (InitiatorName=foo, ISID=1, TargetName=bar, TSID=0) and session B (InitiatorName=foo, ISID=1, TargetName=foobar, TSID=0). If it receives a login request with (InitiatorName=foo, ISID=1, TSID=0), which session is it for? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Jim Hafner To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 06:33 PM cc: ips@ece.cmu.edu From: Jim Hafner/Almaden/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, First, your comment "SSID + InitiatorName must be enough to uniquely identify a session" is target-centric. It would be different from the initiator's viewpoint. However, from the target's viewpoint, the target name is implicit and from the initiator viewpoint, the initiator name is implicit, so globally, the triple of the two names + SSID (made up of the ISID and TSID) form the identifier of the session. Locally (between two specific guys), the names are implicit so only the SSID is required. It's all a matter of point of view! As for the difference in identifiers, as I mentioned in the private note, the session is an iSCSI construct and is identified by iSCSI things. The nexus is a SCSI thing and is identified by SCSI constructs (based on how we've mapped the iSCSI things to SCSI things). However, you've brought to the fore again a related question: "what value does the TSID provide to the protocol?" I'm not going to answer that, but I will note that one target implementation that (I think) works is that the TSID = target portal group tag. The other thing to ask is "what value is there in the nexus identifier?" This is never really used in SCSI at all, so it's not a critical issue what composes it. However, it is important (IMO) that the SCSI ports have names and the logical derivative of that statement is that the nexus is identified by the concatenation of the SCSI port names (hence the definition we have). Jim Hafner Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 03:00:53 pm Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu Subject: Re: is TargetName always mandatory or not? John, WHOOPS! I was wrong; you are absolutely right that the spec says "TargetName" and not "TSID". When I was reading through it, I saw "TargetName", but read to myself "TSID". Apologies... In my defense (confusion?), however, 3.12.9 says TSID rather than TargetName is used to uniquely identify a session. Going by that, SSID + InitiatorName must be enough to uniquely identify a session. Jim Hafner pointed out to me that the I_T nexus is identified by one thing and the session is identified by another. If the two must have a 1-1 mapping in iSCSI, why are there two different identifiers? Why not just use the current definition of the I_T nexus to uniquely identify both the nexus and session (i.e. get rid of TSID)? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM@IBMUS 10/31/2001 cc: ips@ece.cmu.edu 04:42 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not?(Document link: Andre Asselin) Andre, I looked again through the document and No where could I find a statement that you claimed as "a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag". It is the InitiatorName, ISID, TSID, with the TargetName and PortalGroup. Please point out to me in the Spec (8 or above), where I can find your statement on I_T Nexus. I did find the following (please ignore for this conversation the "i" and 't" stuff): "- Session: The group of TCP connections that link an initiator with a target, form a session (loosely equivalent to a SCSI I-T nexus). TCP connections can be added and removed from a session. Across all connections within a session, an initiator sees one "target image". " " - I_T nexus: According to [SAM2], the I_T nexus is a relationship between a SCSI Initiator Port and a SCSI Target Port. For iSCSI, this relationship is a session, defined as a relationship between an iSCSI Initiator's end of the session (SCSI Initiator Port) and the iSCSI Target's Portal Group. The I_T nexus can be identified by the conjunction of the SCSI port names; that is, the I_T nexus identifier is the tuple (iSCSI Initiator Name + 'i'+ ISID, iSCSI Target Name + 't'+ Portal Group Tag). NOTE: The I_T nexus identifier is not equal to the session identifier (SSID). " I have not found a place where Session ID is tied to the TSID. Hence my comment that we also need to have the TargetName in the Initial Login on all Connections. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Andre Asselin/Raleigh/IBM@IBMUS@ece.cmu.edu on 10/31/2001 10:40:55 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: John Hufferd/San Jose/IBM@IBMUS Subject: Re: is TargetName always mandatory or not? John & Julian, How about this for the section 5 text: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The initial login request of the leading Login MAY also include the SessionType key=value pair, in which case if the SessionType is not "discovery", then the initial login request MUST also include the key=value pair TargetName. John, I disagree with your second statement: I don't see any way you could have 2 different sessions established within the same portal group with the same InitiatorName, ISID, and TSID. The spec. says that a nexus, and therefore an iSCSI session, is uniquely identified by the InitiatorName, ISID, TSID, and portal group tag. There is no mention of TargetName contributing to the identificaiton of a session. In my opinion, a non-leading connection should NOT have the TargetName, since it doesn't contribute anything. Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd/San To: Julian Satran/Haifa/IBM@IBMIL Jose/IBM@IBMUS cc: ips@ece.cmu.edu Sent by: Subject: Re: is TargetName always mandatory or not? owner-ips@ece. cmu.edu 10/31/2001 04:20 AM Julian, I think the TargetName should be included on the Initial Login Request on the Leading Login. It seem strange to permit the Authentication functions to proceed if perhaps the intended Target is different then the one doing the Authentication. The way it currently is written, you could pass all the Security test and then find out just before going into Full Function Phase, that it was intended for some other Target. Seems like a waste. My I think that TargetName should also be on all connections on the Initial Login Request. Here is my thinking: The SSID (ISID+TSID) is unique only with regards to a Specific Initiator and Target Node Pair. It is therefore not clear that just knowing the SSID and InitiatorName is enough to understand what session the subsequent connections are being attached to. And it is possible that the CID could be also overlapped with another session. Therefore, I think it make since to determine all this on the initial Login on every Connection, so you know at the beginning what values can be negotiated, or that are being set to the right Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/30/2001 11:33:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: is TargetName always mandatory or not? It is the leading login: The section 5 paragraph will read: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. Julo Andre Asselin/Raleigh/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 31-10-01 02:08 Please respond to Andre Asselin To: "IPS Reflector (E-mail)" cc: Subject: is TargetName always mandatory or not? In section 5 of the spec, it says "If the SessionType is not "discovery" then the initial Login Request MUST also include the key=value pair TargetName.". However, in Appendix D, the description for TargetName says it is Leading Only. Should TargetName not be Leading Only, or should the text in section 5 say that TargetName must be in the initial Login Request of a leading connection? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Tue Nov 6 00:00:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16487 for ; Tue, 6 Nov 2001 00:00:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64Xj804589 for ips-outgoing; Mon, 5 Nov 2001 23:33:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64Xil04583 for ; Mon, 5 Nov 2001 23:33:44 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64XbI15422 for ; Mon, 5 Nov 2001 23:33:37 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-ifcp-06.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C1667C.33A2FCDA" Date: Mon, 5 Nov 2001 23:33:37 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2CB@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-ifcp-06.txt Thread-Index: AcFa+bEqr59H5ygBQ0CdnB/gIpHESwLgjirQ From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1667C.33A2FCDA Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Monday, October 22, 2001 6:03 AM To: IETF-Announce; @loki.ietf.org@horh1.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-ifcp-06.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iFCP - A Protocol for Internet Fibre Channel Storage=20 Networking Author(s) : C. Monia et al. Filename : draft-ietf-ips-ifcp-06.txt Pages : 91 Date : 19-Oct-01 =09 This document specifies an architecture and gateway-to-gateway protocol for the implementation of Fibre Channel fabric functionality on a network in which TCP/IP switching and routing elements replace Fibre Channel components. The protocol enables the attachment of existing Fibre Channel storage products to an IP network by supporting the fabric services required by such devices. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-ifcp-06.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-ifcp-06.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-ifcp-06.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C1667C.33A2FCDA Content-Type: application/octet-stream; name="ATT32517.TXT" Content-Description: ATT32517.TXT Content-Disposition: attachment; filename="ATT32517.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAxOTE0MTIwOC5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pZmNwLTA2 LnR4dA0K ------_=_NextPart_001_01C1667C.33A2FCDA Content-Type: application/octet-stream; name="draft-ietf-ips-ifcp-06.URL" Content-Description: draft-ietf-ips-ifcp-06.URL Content-Disposition: attachment; filename="draft-ietf-ips-ifcp-06.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pZmNwLTA2LnR4dA0K ------_=_NextPart_001_01C1667C.33A2FCDA-- From owner-ips@ece.cmu.edu Tue Nov 6 00:01:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16509 for ; Tue, 6 Nov 2001 00:01:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64gDn05081 for ips-outgoing; Mon, 5 Nov 2001 23:42:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64gBl05075 for ; Mon, 5 Nov 2001 23:42:11 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64g5I17033 for ; Mon, 5 Nov 2001 23:42:05 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-fcencapsulation-03.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C1667D.6233EB82" Date: Mon, 5 Nov 2001 23:42:04 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2CC@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-fcencapsulation-03.txt Thread-Index: AcFQu02T5ZYmyuYPSae/RI07MQ6+MgVwcKSA From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1667D.6233EB82 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Tuesday, October 09, 2001 5:56 AM To: IETF-Announce; @ece.cmu.edu@ihrh1.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-fcencapsulation-03.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : FC Frame Encapsulation Author(s) : R. Weber, M. Rajagopal, F. Travostino,=20 V. Chau, M. O'Donnell, C. Monia, M. Merhar Filename : draft-ietf-ips-fcencapsulation-03.txt Pages : 15 Date : 08-Oct-01 =09 This is the ips (IP Storage) working group draft describing the common encapsulation format and a procedure for the measurement and calculation of frame transit time through the IP network. This specification is intended for use by any IETF protocol that encapsulates Fibre Channel (FC) frames. This draft describes a frame header containing information mandated for encapsulating, transmitting, de-encapsulating, and calculating the transit times of FC frames. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-fcencapsulation-03.tx t To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-fcencapsulation-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-fcencapsulation-03.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C1667D.6233EB82 Content-Type: application/octet-stream; name="ATT787026.TXT" Content-Description: ATT787026.TXT Content-Disposition: attachment; filename="ATT787026.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAwODEzMTAxNC5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1mY2VuY2Fw c3VsYXRpb24tMDMudHh0DQo= ------_=_NextPart_001_01C1667D.6233EB82 Content-Type: application/octet-stream; name="draft-ietf-ips-fcencapsulation-03.URL" Content-Description: draft-ietf-ips-fcencapsulation-03.URL Content-Disposition: attachment; filename="draft-ietf-ips-fcencapsulation-03.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1mY2VuY2Fwc3VsYXRpb24tMDMudHh0DQo= ------_=_NextPart_001_01C1667D.6233EB82-- From owner-ips@ece.cmu.edu Tue Nov 6 00:01:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16520 for ; Tue, 6 Nov 2001 00:01:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64FA003441 for ips-outgoing; Mon, 5 Nov 2001 23:15:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ihemail1.firewall.lucent.com (ihemail1.lucent.com [192.11.222.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64F9l03437 for ; Mon, 5 Nov 2001 23:15:09 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by ihemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64F7311965 for ; Mon, 5 Nov 2001 23:15:07 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-isns-mib-00.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C16679.9E005C4F" Date: Mon, 5 Nov 2001 23:15:07 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2C2@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-isns-mib-00.txt Thread-Index: AcFWN1Ha1JBG8sjBR7GNjgrXhIblyQQQRaXQ From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C16679.9E005C4F Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Tuesday, October 16, 2001 6:06 AM To: IETF-Announce; @ece.cmu.edu@horh1.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-isns-mib-00.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Definitions of Managed Objects for iSNS (Internet=20 Storage Name Service) Author(s) : K. Gibbons, J. Tseng, C. Monia, T. McSweeney Filename : draft-ietf-ips-isns-mib-00.txt Pages : 45 Date : 15-Oct-01 =09 This memo defines a portion of the Management Information Base (MIB) for use with network management protocols in the Internet community. In particular, it defines a basic set of managed objects for SNMP- based monitoring and management of an internet Storage Name Service (iSNS). A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-mib-00.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-isns-mib-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-isns-mib-00.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C16679.9E005C4F Content-Type: application/octet-stream; name="ATT861731.TXT" Content-Description: ATT861731.TXT Content-Disposition: attachment; filename="ATT861731.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAxNTEzMzQ0NC5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pc25zLW1p Yi0wMC50eHQNCg== ------_=_NextPart_001_01C16679.9E005C4F Content-Type: application/octet-stream; name="draft-ietf-ips-isns-mib-00.URL" Content-Description: draft-ietf-ips-isns-mib-00.URL Content-Disposition: attachment; filename="draft-ietf-ips-isns-mib-00.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pc25zLW1pYi0wMC50eHQNCg== ------_=_NextPart_001_01C16679.9E005C4F-- From owner-ips@ece.cmu.edu Tue Nov 6 00:01:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16531 for ; Tue, 6 Nov 2001 00:01:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64Lal03828 for ips-outgoing; Mon, 5 Nov 2001 23:21:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64LZl03824 for ; Mon, 5 Nov 2001 23:21:35 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64LSI12532 for ; Mon, 5 Nov 2001 23:21:28 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-iscsi-mib-03.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C1667A.80F698B1" Date: Mon, 5 Nov 2001 23:21:27 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2C6@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-iscsi-mib-03.txt Thread-Index: AcFiFBf4Ntng4+AmT/u6niVT9SaKqgEZh3Cw From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1667A.80F698B1 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Wednesday, October 31, 2001 6:11 AM To: IETF-Announce; @loki.ietf.org@ihrh2.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-iscsi-mib-03.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Definitions of Managed Objects for iSCSI Author(s) : M. Bakke, J. Muchow, M. Krueger, T. McSweeney Filename : draft-ietf-ips-iscsi-mib-03.txt Pages : 61 Date : 30-Oct-01 =09 This memo defines a portion of the Management Information Base (MIB) for use with network management protocols in TCP/IP based internets. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-mib-03.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-mib-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-mib-03.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C1667A.80F698B1 Content-Type: application/octet-stream; name="ATT134923.TXT" Content-Description: ATT134923.TXT Content-Disposition: attachment; filename="ATT134923.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAzMDEzMzYzNi5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pc2NzaS1t aWItMDMudHh0DQo= ------_=_NextPart_001_01C1667A.80F698B1 Content-Type: application/octet-stream; name="draft-ietf-ips-iscsi-mib-03.URL" Content-Description: draft-ietf-ips-iscsi-mib-03.URL Content-Disposition: attachment; filename="draft-ietf-ips-iscsi-mib-03.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pc2NzaS1taWItMDMudHh0DQo= ------_=_NextPart_001_01C1667A.80F698B1-- From owner-ips@ece.cmu.edu Tue Nov 6 00:02:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16542 for ; Tue, 6 Nov 2001 00:02:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64O1703998 for ips-outgoing; Mon, 5 Nov 2001 23:24:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64O0l03994 for ; Mon, 5 Nov 2001 23:24:00 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64NsI13106 for ; Mon, 5 Nov 2001 23:23:54 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-isns-05.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C1667A.D80CF9A0" Date: Mon, 5 Nov 2001 23:23:53 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2C8@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-isns-05.txt Thread-Index: AcFi4UejMifcX0diRwGBxV6EgMy0fADmUGiQ From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1667A.D80CF9A0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Thursday, November 01, 2001 6:04 AM To: IETF-Announce; @loki.ietf.org@horh1.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-isns-05.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSNS Internet Storage Name Service Author(s) : K. Gibbons et al. Filename : draft-ietf-ips-isns-05.txt Pages : 79 Date : 31-Oct-01 =09 This document provides a generic framework centering around use of the iSNS for discovery and management of iSCSI and Fibre Channel (FCP) storage devices in an enterprise-scale IP storage network. iSNS is an application that stores iSCSI and FC device attributes and monitors their availability and reachability in an integrated IP storage network. Due to its role as a consolidated information repository, iSNS provides for more efficient and scalable management of storage devices in an IP network. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-05.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-isns-05.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-isns-05.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C1667A.D80CF9A0 Content-Type: application/octet-stream; name="ATT148082.TXT" Content-Description: ATT148082.TXT Content-Disposition: attachment; filename="ATT148082.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAzMTExMjIxNC5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pc25zLTA1 LnR4dA0K ------_=_NextPart_001_01C1667A.D80CF9A0 Content-Type: application/octet-stream; name="draft-ietf-ips-isns-05.URL" Content-Description: draft-ietf-ips-isns-05.URL Content-Disposition: attachment; filename="draft-ietf-ips-isns-05.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pc25zLTA1LnR4dA0K ------_=_NextPart_001_01C1667A.D80CF9A0-- From owner-ips@ece.cmu.edu Tue Nov 6 00:02:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16546 for ; Tue, 6 Nov 2001 00:02:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64kcU05357 for ips-outgoing; Mon, 5 Nov 2001 23:46:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64kbl05352 for ; Mon, 5 Nov 2001 23:46:37 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64kUI17702 for ; Mon, 5 Nov 2001 23:46:30 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: REMINDER - Internet-Draft Cutoff Dates for Salt Lake City, Utah MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Date: Mon, 5 Nov 2001 23:46:30 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2CE@nc8220exchange.ral.lucent.com> Thread-Topic: REMINDER - Internet-Draft Cutoff Dates for Salt Lake City, Utah Thread-Index: AcFjvRCDZbBrgmxXSS+73qIb2PZEBACwC21w From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA64kbl05354 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Reminder on the cutoff dates for Salt Lake City. Next Wed, Nov 14 at 5pm EST is the cutoff for any previously unpublished drafts. That is any draft ending in -00. Wed, Nov 21 at 5PM EST is the cutoff for all other drafts. Although the cutoff is at 5pm EST, please try to send in prior to the last minute. We have had drafts held up in mail queues for several hours, and have had people forget the time zone conversion, and ended up submitting late drafts. Please take this into consideration when submitting. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts Administrator [mailto:internet-drafts@ietf.org] Sent: Friday, November 02, 2001 9:02 AM To: IETF-Announce; @loki.ietf.org@horh1.emsr.lucent.com Subject: REMINDER - Internet-Draft Cutoff Dates for Salt Lake City, Utah NOTE: There are two (2) Internet-Draft Cutoff dates November 14th: Cutoff for Initial Submissions (new documents) All initial submissions(-00) must be submitted by Wednesday, November 14th, 17:00 US-EST. Initial submissions received after this time will NOT be made available in the Internet-Drafts directory, and will have to be resubmitted. As before, all initial submissions (-00.txt) with a filename beginning with a draft-ietf MUST be approved by the appropriate WG Chair prior to processing and announcing. WG Chair approval must be received by Monday, November 19th. Please do NOT wait until the last minute to submit. Be advised: NO placeholders. Updates to initial submissions received the week of November 14th will NOT be accepted. November 21st: FINAL Internet-Draft Cutoff All revised Internet-Draft submissions must be submitted by Wednesday, November 21st, 2001 at 17:00 US-EST. Internet-Drafts received after this time will NOT be announced NOR made available in the Internet-Drafts Directories. We will begin accepting Internet-Draft submissions the week of the meeting, though announcements will NOT be sent until the IETF meeting is over. Thank you for your understanding and cooperation. Please do not hesitate to contact us if you have any questions or concenrs. FYI: These and other significant dates can be found at http://www.ietf.org/meetings/cutoff_dates_52.html From owner-ips@ece.cmu.edu Tue Nov 6 00:02:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16567 for ; Tue, 6 Nov 2001 00:02:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64XLa04565 for ips-outgoing; Mon, 5 Nov 2001 23:33:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from auemail1.firewall.lucent.com (auemail1.lucent.com [192.11.223.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64XJl04560 for ; Mon, 5 Nov 2001 23:33:19 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by auemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64XC224524 for ; Mon, 5 Nov 2001 23:33:13 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-security-04.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C1667C.24CD0109" Date: Mon, 5 Nov 2001 23:33:12 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2CA@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-security-04.txt Thread-Index: AcFdaM2YU8eANQ6OR1aQnITebPxcqQJEwtUg From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1667C.24CD0109 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Thursday, October 25, 2001 6:04 AM To: IETF-Announce; @ece.cmu.edu@horh1.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-security-04.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Securing iSCSI, iFCP and FCIP Author(s) : B. Aboba et al. Filename : draft-ietf-ips-security-04.txt Pages : 56 Date : 24-Oct-01 =09 This document discusses how iSCSI, iFCP, and FCIP utilize IPsec to provide security. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-security-04.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-security-04.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C1667C.24CD0109 Content-Type: application/octet-stream; name="ATT73043.TXT" Content-Description: ATT73043.TXT Content-Disposition: attachment; filename="ATT73043.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAyNDEyNTIzMS5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1zZWN1cml0 eS0wNC50eHQNCg== ------_=_NextPart_001_01C1667C.24CD0109 Content-Type: application/octet-stream; name="draft-ietf-ips-security-04.URL" Content-Description: draft-ietf-ips-security-04.URL Content-Disposition: attachment; filename="draft-ietf-ips-security-04.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1zZWN1cml0eS0wNC50eHQNCg== ------_=_NextPart_001_01C1667C.24CD0109-- From owner-ips@ece.cmu.edu Tue Nov 6 00:04:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16585 for ; Tue, 6 Nov 2001 00:04:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64Kx403782 for ips-outgoing; Mon, 5 Nov 2001 23:20:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from auemail2.firewall.lucent.com (auemail2.lucent.com [192.11.223.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64Kwl03777 for ; Mon, 5 Nov 2001 23:20:58 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by auemail2.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64Kpm07562 for ; Mon, 5 Nov 2001 23:20:51 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-ifcp-mib-00.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C1667A.6AC05988" Date: Mon, 5 Nov 2001 23:20:50 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2C5@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-ifcp-mib-00.txt Thread-Index: AcFi4TeQQcH9Gtj5RFmmUZXIxVeIWQDmNq+g From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1667A.6AC05988 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Thursday, November 01, 2001 6:04 AM To: IETF-Announce; @loki.ietf.org@horh1.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-ifcp-mib-00.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Definitions of Managed Objects For iFCP Author(s) : K. Gibbons, J. Tseng, C. Monia, F. Travostino Filename : draft-ietf-ips-ifcp-mib-00.txt Pages : 19 Date : 31-Oct-01 =09 This memo defines a portion of the Management Information Base (MIB) for use with network management protocols in the Internet community. In particular, it defines a basic set of managed objects for SNMP-based monitoring and management of the Internet Fibre Channel Protocol (iFCP). A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-ifcp-mib-00.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-ifcp-mib-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-ifcp-mib-00.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C1667A.6AC05988 Content-Type: application/octet-stream; name="ATT148086.TXT" Content-Description: ATT148086.TXT Content-Disposition: attachment; filename="ATT148086.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAzMTExMjE1NC5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pZmNwLW1p Yi0wMC50eHQNCg== ------_=_NextPart_001_01C1667A.6AC05988 Content-Type: application/octet-stream; name="draft-ietf-ips-ifcp-mib-00.URL" Content-Description: draft-ietf-ips-ifcp-mib-00.URL Content-Disposition: attachment; filename="draft-ietf-ips-ifcp-mib-00.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pZmNwLW1pYi0wMC50eHQNCg== ------_=_NextPart_001_01C1667A.6AC05988-- From owner-ips@ece.cmu.edu Tue Nov 6 00:04:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16597 for ; Tue, 6 Nov 2001 00:04:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64gNH05095 for ips-outgoing; Mon, 5 Nov 2001 23:42:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64gMl05091 for ; Mon, 5 Nov 2001 23:42:22 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64gFI17060 for ; Mon, 5 Nov 2001 23:42:15 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-iscsi-08.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C1667D.687D3C2D" Date: Mon, 5 Nov 2001 23:42:15 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2CD@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-iscsi-08.txt Thread-Index: AcFLOjnUC6s+PuBqR8yMjfnVkDMiTgbQqoNA From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1667D.687D3C2D Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Tuesday, October 02, 2001 6:02 AM To: IETF-Announce; @ece.cmu.edu@ihrh2.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-iscsi-08.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Author(s) : J. Satran et al. Filename : draft-ietf-ips-iscsi-08.txt Pages : 211 Date : 01-Oct-01 =09 The Small Computer Systems Interface (SCSI) is a popular family of=20 protocols for communicating with I/O devices, especially storage=20 devices. This memo describes a transport protocol for SCSI that=20 operates on top of TCP. The iSCSI protocol aims to be fully=20 compliant with the requirements laid out in the SCSI Architecture=20 Model - 2 [SAM2] document. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-08.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-08.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-08.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C1667D.687D3C2D Content-Type: application/octet-stream; name="ATT713317.TXT" Content-Description: ATT713317.TXT Content-Disposition: attachment; filename="ATT713317.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAwMTExMjczMS5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pc2NzaS0w OC50eHQNCg== ------_=_NextPart_001_01C1667D.687D3C2D Content-Type: application/octet-stream; name="draft-ietf-ips-iscsi-08.URL" Content-Description: draft-ietf-ips-iscsi-08.URL Content-Disposition: attachment; filename="draft-ietf-ips-iscsi-08.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pc2NzaS0wOC50eHQNCg== ------_=_NextPart_001_01C1667D.687D3C2D-- From owner-ips@ece.cmu.edu Tue Nov 6 00:07:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16661 for ; Tue, 6 Nov 2001 00:07:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA64Lux03859 for ips-outgoing; Mon, 5 Nov 2001 23:21:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA64Lsl03852 for ; Mon, 5 Nov 2001 23:21:54 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA64LmI12640 for ; Mon, 5 Nov 2001 23:21:48 -0500 (EST) Content-Class: urn:content-classes:message Subject: FW: I-D ACTION:draft-ietf-ips-isns-05.txt MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C1667A.8CEBDF15" Date: Mon, 5 Nov 2001 23:21:47 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2C7@nc8220exchange.ral.lucent.com> Thread-Topic: I-D ACTION:draft-ietf-ips-isns-05.txt Thread-Index: AcFi4UejMifcX0diRwGBxV6EgMy0fADmPvDA From: "Elizabeth Rodriguez" To: "IPS Mailing List (E-mail)" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1667A.8CEBDF15 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable It has been brought to my attention that for some reason, the I-D Action announcements for recently posted drafts have not been showing up on the IPS reflector. =20 I am trying to determine why these announcements have not made it through, and am forwarding copies to the list for now. Thanks, Elizabeth -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Thursday, November 01, 2001 6:04 AM To: IETF-Announce; @loki.ietf.org@horh1.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-isns-05.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSNS Internet Storage Name Service Author(s) : K. Gibbons et al. Filename : draft-ietf-ips-isns-05.txt Pages : 79 Date : 31-Oct-01 =09 This document provides a generic framework centering around use of the iSNS for discovery and management of iSCSI and Fibre Channel (FCP) storage devices in an enterprise-scale IP storage network. iSNS is an application that stores iSCSI and FC device attributes and monitors their availability and reachability in an integrated IP storage network. Due to its role as a consolidated information repository, iSNS provides for more efficient and scalable management of storage devices in an IP network. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-05.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-isns-05.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-isns-05.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C1667A.8CEBDF15 Content-Type: application/octet-stream; name="ATT148082.TXT" Content-Description: ATT148082.TXT Content-Disposition: attachment; filename="ATT148082.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAzMTExMjIxNC5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pc25zLTA1 LnR4dA0K ------_=_NextPart_001_01C1667A.8CEBDF15 Content-Type: application/octet-stream; name="draft-ietf-ips-isns-05.URL" Content-Description: draft-ietf-ips-isns-05.URL Content-Disposition: attachment; filename="draft-ietf-ips-isns-05.URL" Content-Transfer-Encoding: base64 W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pc25zLTA1LnR4dA0K ------_=_NextPart_001_01C1667A.8CEBDF15-- From owner-ips@ece.cmu.edu Tue Nov 6 00:50:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA17302 for ; Tue, 6 Nov 2001 00:50:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA656jP06542 for ips-outgoing; Tue, 6 Nov 2001 00:06:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ihemail2.firewall.lucent.com (ihemail2.lucent.com [192.11.222.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA656hl06538 for ; Tue, 6 Nov 2001 00:06:43 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by ihemail2.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA656f920374 for ; Tue, 6 Nov 2001 00:06:42 -0500 (EST) Content-Class: urn:content-classes:message Subject: FC Management MIB -- Transfer from IPFC To IPS WG MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Date: Tue, 6 Nov 2001 00:06:40 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D4538236498360408E9@nc8220exchange.ral.lucent.com> Thread-Topic: FC Management MIB -- Transfer from IPFC To IPS WG Thread-Index: AcFmTTTQdJOE6NTpTi6Ai17WID5eKAAMat8A From: "Elizabeth Rodriguez" To: , "IPS Mailing List (E-mail)" Cc: , , , , , , , "Elizabeth Rodriguez" , Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA656il06539 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit All, The IPFC working group has only one item left on its charter. It is that of the FC Management MIB. It has been identified that this MIB does not follow the IETF guidelines for MIBs in its current format, in its lack of focus, and in its overlap with existing MIBs. Rework of this MIB will take some time. The content of this MIB will fit in well with the IPS WG, especially because the subject matter experts participate in this WG. For these reasons, the working group chairs, with the INT and TSV area directors, have determined that this effort should be moved from the IPFC working group to the IPS working group. Upon transferring of this work, the IPFC working group will have completed the items in its charter and the IPFC WG will be closed. The IPS working group has a technical advisor for MIB work -- Keith McCloghrie. Since it has been determined that the current MIB has issues with format, Keith McCloghrie has agreed to become the interim editor of this MIB. As part of the re-architecture, the MIB will be evaluated with respect to fit with other IPS WG MIBs, and may take the form of a single new MIB or multiple MIBs, as appropriate. The IPS working group will also be seeking Fibre Channel expertise to help formulate the new MIB, including an editor with FC and MIB experience. The IPFC working group would also like to thank Steven Blumenau for all his work on the original MIB. Elizabeth Rodriguez & David Black, IPS Working Group Chairs Murali Rajagopal, IPFC Working Group Chair From owner-ips@ece.cmu.edu Tue Nov 6 00:55:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA17372 for ; Tue, 6 Nov 2001 00:55:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA65GZS07190 for ips-outgoing; Tue, 6 Nov 2001 00:16:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mayfair.vxindia.veritas.com (bay-bridge.veritas.com [143.127.3.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA65GMl07169 for ; Tue, 6 Nov 2001 00:16:23 -0500 (EST) Received: from RAHULB (rahulb.vxindia.veritas.com [10.212.80.59]) by mayfair.vxindia.veritas.com (8.9.3/8.9.3) with SMTP id KAA07439; Tue, 6 Nov 2001 10:42:22 +0530 (IST) Message-ID: <007201c16681$b4fc9130$3b50d40a@vxindia.veritas.com> From: "Rahul Bhagwat" To: "Mallikarjun C." Cc: References: <012a01c165fe$2dfedae0$3b50d40a@vxindia.veritas.com> <005701c16635$96597730$edd52b0f@rose.hp.com> Subject: Re: Connection Recovery Date: Tue, 6 Nov 2001 10:42:50 +0530 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi, The only possible state that CLEANUP_WAIT eventually ends up is FREE (on timeout or successful implicit/explicit logout). If task reassignment happens unrelated to connection state, is there any other resource associated with connection that can be still used while it is in CLEANUP_WAIT state. Why do we have to wait for initiator to send logout to clean up things. Regards, Rahul > > A successful connection logout (implicit or explicit) must precede the > task reassignments during a connection recovery operation. > > But please note that the notion of "connection cleanup" (graceful closing of > a > previously operational iSCSI connection) in the state diagrams goes beyond > the connection recovery (in fact, that is the reason I renamed from its > previous > name, please refer to my email to ips on 11/2/01 with the slide posting > announcement). > A connection cleanup is highly desirable even in the absence of task > reassignment, > to quickly reclaim the tags and buffers on either end (or, both sides would > have > to wait for a connection timeout to happen, symbolized by the M1 > transition). > > >Once a CSM-E or a CSM-I > >drives the connection to free state, all the pending tasks need to be freed > up. > > Not correct. The decision to free up the pending tasks is depedent on the > operational ErrorRecoveryLevel in the CSM-I case (please look at the > discussion in section 3.12.2), or is dependent on the Logout reason code > (recovery Vs close) in the CSM-E case. All the FREE state symbolizes > really is that the iSCSI connection is gracefully closed with a successful > explicit/implit iSCSI Logout. The pending tasks at this point have no > connection allegiance, and are loosely "owned" by the session. It is > legitimate > for the pending tasks to be existent (waiting for reassignment) even when > all the connections reported FREE (please look at the discussion under > 3.15.2, Time2Retain). > > Regards. > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > Hewlett-Packard MS 5668 > Roseville CA 95747 > > > ----- Original Message ----- > From: Rahul Bhagwat > To: ips@ece.cmu.edu > Sent: Monday, November 05, 2001 5:31 AM > Subject: Connection Recovery > > > Hi, > > Is there any order in task reassignments and connection logout (implicit or > explicit) > during a connection recovery. > > If these two are not related, what is the use of moving the connection to > CLEANUP_WAIT > state? CLEANUP_WAIT state typically means that there are pending tasks for > this > connection due to which it cannot be moved to FREE state. That is only > difference > betweeen FREE state and CLEANUP_WAIT state. > > Which probably means that it is mandatory that Task reassigment happens > before > logging out a failed connection (in CLEANUP_WAIT state). Once a CSM-E or a > CSM-I > drives the connection to free state, all the pending tasks need to be freed > up. > > Am I correct here? > > Regards, > Rahul > From owner-ips@ece.cmu.edu Tue Nov 6 01:53:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA23959 for ; Tue, 6 Nov 2001 01:53:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA66M7H10924 for ips-outgoing; Tue, 6 Nov 2001 01:22:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA66M4l10916 for ; Tue, 6 Nov 2001 01:22:04 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA117700 for ; Tue, 6 Nov 2001 07:21:58 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA66LvJ32090 for ; Tue, 6 Nov 2001 07:21:58 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: spec revs to make TargetName reqd on every connection X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Tue, 6 Nov 2001 08:21:55 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 06/11/2001 08:21:58, Serialize complete at 06/11/2001 08:21:58 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk thanks - julo Andre Asselin@IBMUS 05-11-01 22:55 To: Julian Satran/Haifa/IBM@IBMIL cc: John Hufferd/San Jose/IBM@IBMUS, Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: spec revs to make TargetName reqd on every connection Julian, Attached is an attempt to pull together all the required spec updates to make InitiatorName required on every login, TargetName required on every normal login, and clarify related text. Does this look good to everyone? Any places I missed? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Appendix D, TargetName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). To This key must be provided by the initiator of the TCP connection to the remote endpoint in the first login request if the initiator is not establishing a discovery session. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). Appendix D, InitiatorName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The Initiator key enables the initiator to identify itself to the remote endpoint. To This key must be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The InitiatorName key enables the initiator to identify itself to the remote endpoint. The current version of the 6th paragraph in chapter 5 reads: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. A suggested rewrite would be (building on the text suggested by Bob Russell): All initial Login requests MUST include the InitiatorName key=value pair. If the initial Login request is also a leading Login (TSID=0) and the new session is to be a discovery session, then the initial Login request MUST also include the SessionType=discovery key=value pair. If the initial Login request is a leading Login and the new session is to be a normal session, then the initial Login request MUST also include the TargetName key=value pair and MAY also include the SessionType=normal key=value pair. All initial Login requests that are not also a leading Login (TSID != 0) MUST include the TargetName key=value pair. Also, this text appears in 2.2.7: The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session. The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. A suggested rewrite would be: The initiator must present its iSCSI Initiator Name in the first login request. If the initiator is not establishing a discovery session (see 2.4), it also must present the iSCSI Target Name to which it wishes to connect in the first login request. The key "SessionType=discovery" is sent by the initiator on the Initial Login request to indicate a discovery session. See chapter 5 for a more detailed description of the Login process. Section 3.12.8 currently reads: The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies the session with that initiator. Suggested rewrite (melding current text w/John's rewrite): The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity). From owner-ips@ece.cmu.edu Tue Nov 6 01:59:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA24670 for ; Tue, 6 Nov 2001 01:59:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA666Ic10026 for ips-outgoing; Tue, 6 Nov 2001 01:06:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA666Gl10018 for ; Tue, 6 Nov 2001 01:06:16 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA22160 for ; Tue, 6 Nov 2001 07:06:10 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA6669138146 for ; Tue, 6 Nov 2001 07:06:09 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Tue, 6 Nov 2001 08:06:06 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 06/11/2001 08:06:09, Serialize complete at 06/11/2001 08:06:09 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Rod, I don't see any reason why DMA operations cant be "multiplexed" with commands. If you have scheduled a long outbound DMA you are doomed regardless of the command ordering. And if you have scheduled DMA operations piecemeal then you can insert your commands in correct order. Julo "Rod Harrison" 05-11-01 20:48 Please respond to "Rod Harrison" To: Julian Satran/Haifa/IBM@IBMIL, cc: Subject: iSCSI: Out of order commands, was current UNH Plugfest [ Subject changed ] Julian, The ordering difference is introduced between the host side driver and the iSCSI HBA. The host side driver must present SCSI commands to the HBA in the order they are received from the OS to prevent read after write dependency failures. The HBA might reorder the commands depending on when DMA completes. The reordering can't be done ahead of time in the host driver since it doesn't know how long each DMA might take. As long as the HBA assigns CmdSN in the order it receives commands the desired host ordering is preserved. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Monday, November 05, 2001 12:35 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: current UNH Plugfest Rod, I all examples give the point I find hard to understand is why is the ordering on the wire different from the presentation order to the initiator. You can get as many overlaps as you want by presenting the commands to the initiator in the desired order. What we are considering here is the case in which you want to ship in an order different than the one you present the commands. Julo "Rod Harrison" Sent by: owner-ips@ece.cmu.edu 04-11-01 04:42 Please respond to "Rod Harrison" To: "Barry Reinhold" , "Dave Sheehy" , "IETF IP SAN Reflector" cc: Subject: RE: iSCSI: current UNH Plugfest Barry, In general I agree but I don't think this is as much of a corner case as it at first appears. Targets will have code very similar to that needed to handle out of order commands to deal with digest errors. Targets also need to queue commands whilst waiting for both solicited and unsolicited data to arrive. Queuing out of order commands seems little extra work. From an initiators point of view there are efficiency, and probably performance gains to be had from sending commands out of order. Bob Russell gave the example of a read being sent whilst write data DMA is happening, and a similar situation can arise with DMA for writes overtaking that of earlier writes if the initiator has multiple DMA engines. In this case the initiator might be forced to let the wire go idle if it can't send the data from completed DMAs as soon as possible. We already have a command queue at the target to enforce correct serialisation of commands, doing the same thing at the initiator is redundant. Finally, I don't believe we should be writing a standard to work around poor coding and test coverage, especially at the cost of potential efficiency gains. I agree with Dave and Santosh that commands being sent out of order on a single session should be allowed by the standard. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Barry Reinhold Sent: Friday, November 02, 2001 5:24 PM To: Dave Sheehy; IETF IP SAN Reflector Subject: RE: iSCSI: current UNH Plugfest Using features such as out of order command delivery on a connection tend to be the sort of things that lead to interoperability problems. It is unexpected and probably going to hit poorly tested code paths even if the standard is written to allow it. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Dave Sheehy >Sent: Friday, November 02, 2001 4:19 PM >To: IETF IP SAN Reflector >Subject: Re: iSCSI: current UNH Plugfest > > > >> 3. Can commands be sent out of order on the same connection? >> >> The behavior of targets is clearly specified in Section 2.2.2.3 on >> page 25 of draft 8, which says: >> "Except for the commands marked for immediate delivery the iSCSI >> target layer MUST eliver the commands for execution in the order >> specified by CmdSN." >> >> Section 2.2.2.3 on page 26 of draft 8 also says: >> "- CmdSN - the current command Sequence Number advanced by 1 on >> each command shipped except for commands marked for immediate >> delivery." >> but the meaning of the term "shipped" is vague, and does not >> necessarily >> require that the PDUs arrive on the other end of a TCP connection >> in the same order that the CmdSN values were assigned to these PDUs. >> >> Some initiators have been designed to send commands out of CmdSN >> order on one connection. Consider the situation where there is only >> one connection and a high-level dispatcher creates a PDU for a SCSI >> command that involves writing immediate data to the target. This PDU >> is enqueued to a lower-level layer which has to setup, start, and >> wait-for a DMA operation to move the immediate data into an onboard >> buffer before the PDU can be put onto the wire. While this is >> happening, the dispatcher creates another unrelated PDU for a SCSI >> read command (for example), and when this PDU is passed to the >> lower-level layer it can be sent immediately, ahead of the previous >> write PDU and therefore out of order on this connection. >> >> The standard clearly allows this to happen if the two PDUs were sent >> on different connections, and seems to imply that this can also happen >> when the two PDUs are sent on the same connection. >> >> The suggestion is to put in the standard an explicit statement that >> this is allowed or not allowed, as appropriate. >> >> If this is allowed, such a statement would avoid the erroneous >> assumption being made by some target implementers that within a single >> connection, commands will arrive in order. >> >> If this is not allowed, such a statement would avoid the erroneous >> assumption being made by some initiator implementers that within a >> single connection, commands can be put on the wire out of order. >> >> +++ >> >> will add an explicit statement saying that this behaviour is forbidden. >> 2.2.2.1 will contain: >> >> On any given connection, the iSCSI initiator MUST send the >commands in the >> order specified by CmdSN. >> >> +++ > >Why do you feel this behavior should be forbidden? Targets already have to >order commands across the session. I don't see why it's a problem to extend >that to the connection as well. I, for one, believe we should take >a liberal >stance on this. > >Dave Sheehy > From owner-ips@ece.cmu.edu Tue Nov 6 07:30:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA07730 for ; Tue, 6 Nov 2001 07:30:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6BX3b08274 for ips-outgoing; Tue, 6 Nov 2001 06:33:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from opus.pdl.cmu.edu (root@OPUS.PDL.CMU.EDU [128.2.134.91]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6BX2l08270 for ; Tue, 6 Nov 2001 06:33:02 -0500 (EST) Received: from opus.pdl.cmu.edu (bassoon@localhost [127.0.0.1]) by opus.pdl.cmu.edu (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) with ESMTP id GAA01102 for ; Tue, 6 Nov 2001 06:33:02 -0500 Message-Id: <200111061133.GAA01102@opus.pdl.cmu.edu> To: ips@ece.cmu.edu Subject: FW: I-D ACTION:draft-ietf-ips-isns-mib-00.txt Date: Tue, 06 Nov 2001 06:33:01 -0500 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Definitions of Managed Objects for iSNS (Internet=20 Storage Name Service) Author(s) : K. Gibbons, J. Tseng, C. Monia, T. McSweeney Filename : draft-ietf-ips-isns-mib-00.txt Pages : 45 Date : 15-Oct-01 =09 This memo defines a portion of the Management Information Base (MIB) for use with network management protocols in the Internet community. In particular, it defines a basic set of managed objects for SNMP- based monitoring and management of an internet Storage Name Service (iSNS). A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-mib-00.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-isns-mib-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-isns-mib-00.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. - ------_=_NextPart_001_01C16679.9E005C4F Content-Type: application/octet-stream; name="ATT861731.TXT" Content-Transfer-Encoding: base64 Content-Description: ATT861731.TXT Content-Disposition: attachment; filename="ATT861731.TXT" Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAxNTEzMzQ0NC5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pc25zLW1p Yi0wMC50eHQNCg== - ------_=_NextPart_001_01C16679.9E005C4F Content-Type: application/octet-stream; name="draft-ietf-ips-isns-mib-00.URL" Content-Transfer-Encoding: base64 Content-Description: draft-ietf-ips-isns-mib-00.URL Content-Disposition: attachment; filename="draft-ietf-ips-isns-mib-00.URL" W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pc25zLW1pYi0wMC50eHQNCg== - ------_=_NextPart_001_01C16679.9E005C4F-- ------- End of Forwarded Message From owner-ips@ece.cmu.edu Tue Nov 6 10:13:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16940 for ; Tue, 6 Nov 2001 10:13:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6ETow18798 for ips-outgoing; Tue, 6 Nov 2001 09:29:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6ETnl18794 for ; Tue, 6 Nov 2001 09:29:49 -0500 (EST) Received: from cisco.com (58509@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA07853; Tue, 6 Nov 2001 09:29:31 -0500 (EST) Message-ID: <3BE7F16C.49D71692@cisco.com> Date: Tue, 06 Nov 2001 08:19:24 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Michele Hallak - Stamler CC: IPS Subject: Re: iSCSI: New iSCSI MIB draft References: <838D8D2617300146B7F47E4D9AE7FF10114A0E@SANSRV1.SAN-RAD.CO.IL> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Michele- Answers inline. Basically, we have discussed making parts of the MIB writable, and need to start working on it some more. Please keep posting anything that you think might be useful as a writable attribute or a creatable/deletable row. -- Mark Michele Hallak - Stamler wrote: > > Hi, > 1.I see that the target access list is still read-only. How do you think > that it should be configured? I had originally assumed that these would be configured by other means than SNMP, but it is certainly something that makes sense to configure. I will post another message to start a thread on this. > 2. Who should configure the aliases of the iscsi targets and initiators? These could easily be writable as well. > 3. Same question for the portals. Have to think about this one. Any ideas? > Thanks, > Michele > > -----Original Message----- > From: Mark Bakke [mailto:mbakke@cisco.com] > Sent: Wednesday, October 31, 2001 9:01 PM > To: IPS > Subject: Re: iSCSI: New iSCSI MIB draft > > For those who are more pictorially inclined when looking at > MIBs, I have an updated MIB tree drawing available at: > > ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-mib-structure-03.pd > f > > -- > Mark > > Mark Bakke wrote: > > > > We have submitted a new iSCSI MIB draft to the repository. > > Until it becomes available, it may be retrieved as either > > a gzipped or text version from: > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/draft-ietf-ips-iscsi-mib-03.txt.gz > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/draft-ietf-ips-iscsi-mib-03.txt > > > > A matching UML drawing is available at: > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-uml-model-03.pdf > > > > The table hierarchy has been significantly flattened. This does > > not change the object model, but does reduce the number of redundant > > levels of OIDs when address individual objects. I will send a > > new table structure drawing soon. > > > > We will send a list of open issues to the IPS list shortly. > > > > Thanks, > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > -- > Mark A. Bakke > Cisco Systems > mbakke@cisco.com > 763.398.1054 -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Nov 6 10:15:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA17057 for ; Tue, 6 Nov 2001 10:15:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6BWKS08239 for ips-outgoing; Tue, 6 Nov 2001 06:32:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from opus.pdl.cmu.edu (root@OPUS.PDL.CMU.EDU [128.2.134.91]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6BWJl08234 for ; Tue, 6 Nov 2001 06:32:19 -0500 (EST) Received: from opus.pdl.cmu.edu (bassoon@localhost [127.0.0.1]) by opus.pdl.cmu.edu (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) with ESMTP id GAA01080 for ; Tue, 6 Nov 2001 06:32:18 -0500 Message-Id: <200111061132.GAA01080@opus.pdl.cmu.edu> To: ips@ece.cmu.edu Subject: FW: I-D ACTION:draft-ietf-ips-ifcp-mib-00.txt Date: Tue, 06 Nov 2001 06:32:18 -0500 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Thursday, November 01, 2001 6:04 AM To: IETF-Announce; @loki.ietf.org@horh1.emsr.lucent.com Cc: ips@ece.cmu.edu Subject: I-D ACTION:draft-ietf-ips-ifcp-mib-00.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Definitions of Managed Objects For iFCP Author(s) : K. Gibbons, J. Tseng, C. Monia, F. Travostino Filename : draft-ietf-ips-ifcp-mib-00.txt Pages : 19 Date : 31-Oct-01 =09 This memo defines a portion of the Management Information Base (MIB) for use with network management protocols in the Internet community. In particular, it defines a basic set of managed objects for SNMP-based monitoring and management of the Internet Fibre Channel Protocol (iFCP). A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-ifcp-mib-00.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-ifcp-mib-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-ifcp-mib-00.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. - ------_=_NextPart_001_01C16679.EAB4639B Content-Type: application/octet-stream; name="ATT148086.TXT" Content-Transfer-Encoding: base64 Content-Description: ATT148086.TXT Content-Disposition: attachment; filename="ATT148086.TXT" Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTAzMTExMjE1NC5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1pZXRmLWlwcy1pZmNwLW1p Yi0wMC50eHQNCg== - ------_=_NextPart_001_01C16679.EAB4639B Content-Type: application/octet-stream; name="draft-ietf-ips-ifcp-mib-00.URL" Content-Transfer-Encoding: base64 Content-Description: draft-ietf-ips-ifcp-mib-00.URL Content-Disposition: attachment; filename="draft-ietf-ips-ifcp-mib-00.URL" W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1pZXRmLWlwcy1pZmNwLW1pYi0wMC50eHQNCg== - ------_=_NextPart_001_01C16679.EAB4639B-- ------- End of Forwarded Message From owner-ips@ece.cmu.edu Tue Nov 6 10:23:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA17391 for ; Tue, 6 Nov 2001 10:23:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6EPuK18552 for ips-outgoing; Tue, 6 Nov 2001 09:25:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6EPsl18544 for ; Tue, 6 Nov 2001 09:25:54 -0500 (EST) Received: from cisco.com (58509@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id JAA04173; Tue, 6 Nov 2001 09:25:45 -0500 (EST) Message-ID: <3BE7F08A.7697354C@cisco.com> Date: Tue, 06 Nov 2001 08:15:38 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Peter Mellquist CC: IPS Subject: Re: Comments: New iSCSI MIB draft References: <3BDEE304.B89C3782@cisco.com> <3BE04A51.199E8E20@cisco.com> <008801c1625f$c3ed0470$1001010a@blekinge> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Peter- Sorry for the delay. Some answers below. -- Mark Peter Mellquist wrote: > > Mark, > > Everything looks well. I appreciate the UML diagrams. > > I have a general question in regard to the iscsiSessionAttributeTable. > - What is the life of a row in this table? Do they persist after the session > is gone? Session rows are not kept after a session is gone. > - Do rows come and go with each iSCSI session? I assume that this is so > since iscsiConnectionNumber must have a value from (1..65535). Yes. We had discussed the lifespan of some of these objects (especially session), and while they could provide some value if we kept them after termination, we didn't see a compelling enough reason to have implementations do the extra work of maintaining them after they are gone. > I am considering session statistics (iscsiSessionStats ) in which a SNMP > manager would like to access stats after the session has been terminated. > With the current design, is it possible for this to work? Or... does this That was not the intent, although there's nothing that says an implementation can't keep them around for a while. If we decided that we needed this functionality, it would be better to require it. However, that gets us into the dilemma of how long to keep dead sessions before destroying them. > information exists only as long as a session is active. It would be nice to > have historical iSCSI stats in the same manner as the RMON MIB does through > the history group. Otherwise one would have to have an active SNMP manager > running all the time gathering session info. Consider a session in which a > number of errors occur to the point the session is terminated. How does an > SNMP manager determine what the actual errors were? The instance object contains counters for errors that cause session termination; it would not show you which sessions were terminated by which errors, but you would be able to see what types of errors are causing sessions to terminate. > > Thank you, > > -peter > > Peter Mellquist > Seven-Systems Technolgies > peterm@seven-systems.com > > ----- Original Message ----- > From: "Mark Bakke" > To: "IPS" > Sent: Wednesday, October 31, 2001 11:00 AM > Subject: Re: iSCSI: New iSCSI MIB draft > > > For those who are more pictorially inclined when looking at > > MIBs, I have an updated MIB tree drawing available at: > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-mib-structure-03.pdf > > > > -- > > Mark > > > > Mark Bakke wrote: > > > > > > We have submitted a new iSCSI MIB draft to the repository. > > > Until it becomes available, it may be retrieved as either > > > a gzipped or text version from: > > > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/draft-ietf-ips-iscsi-mib-03.txt.gz > > > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/draft-ietf-ips-iscsi-mib-03.txt > > > > > > A matching UML drawing is available at: > > > > > > ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-uml-model-03.pdf > > > > > > The table hierarchy has been significantly flattened. This does > > > not change the object model, but does reduce the number of redundant > > > levels of OIDs when address individual objects. I will send a > > > new table structure drawing soon. > > > > > > We will send a list of open issues to the IPS list shortly. > > > > > > Thanks, > > > > > > -- > > > Mark A. Bakke > > > Cisco Systems > > > mbakke@cisco.com > > > 763.398.1054 > > > > -- > > Mark A. Bakke > > Cisco Systems > > mbakke@cisco.com > > 763.398.1054 > > -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Tue Nov 6 11:57:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA21355 for ; Tue, 6 Nov 2001 11:57:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6Fbj823649 for ips-outgoing; Tue, 6 Nov 2001 10:37:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e1.ny.us.ibm.com (e1.ny.us.ibm.com [32.97.182.101]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6Fbfl23634 for ; Tue, 6 Nov 2001 10:37:41 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e1.ny.us.ibm.com (8.9.3/8.9.3) with ESMTP id KAA36590 for ; Tue, 6 Nov 2001 10:34:59 -0500 Received: from d04nms41.raleigh.ibm.com (d04nms41.raleigh.ibm.com [9.67.228.19]) by southrelay02.raleigh.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA6FbX0274190 for ; Tue, 6 Nov 2001 10:37:33 -0500 Subject: iSCSI: resend of diagram from 2.5.1 To: "John Hufferd" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Andre Asselin" Date: Tue, 6 Nov 2001 10:38:38 -0500 X-MIMETrack: Serialize by Router on D04NMS41/04/M/IBM(Release 5.0.8 |June 18, 2001) at 11/06/2001 10:37:33 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk It looks like my original post with an updated diagram for section 2.5.1 got wrapped badly. Here's a second attempt that hopefully will make it through. (original) ----------------------------IP Network--------------------- | | | +--------|---------------|--------------------|---------------------+ | +----|---------------|-----+ +----|---------+ | | | +---------+ +---------+ | | +---------+ | | | | | Network | | Network | | | | Network | | | | | | Portal | | Portal | | | | Portal | | | | | +--|------+ +---------+ | | +---------+ | | | | | | | | | | | | | | Portal | | | | Portal | | | | | Group 1 | | | | Group 2 | | | +--------------------------+ +--------------+ | | | | | | | +----------------------------+ +-----------------------------+ | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | +----------------------------+ +-----------------------------+ | | | | iSCSI Target Node | | (within Network Entity, not shown) | +-------------------------------------------------------------------+ (updated) ----------------------------IP Network--------------------- | | | +--------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +------|-------+ | | | | | Portal | | Portal | | | | | | Group 1 | | Group 2 | | | | | +--------------------------+ +--------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-----------------------------------------------------------------+ | | | | Network Entity | +---------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Tue Nov 6 13:46:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26566 for ; Tue, 6 Nov 2001 13:46:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6I0xL05043 for ips-outgoing; Tue, 6 Nov 2001 13:00:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1aa.compuserve.com (siaar1aa.compuserve.com [149.174.40.9]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6I0ul05033 for ; Tue, 6 Nov 2001 13:00:56 -0500 (EST) Received: (from mailgate@localhost) by siaar1aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id NAA29535 for ips@ece.cmu.edu; Tue, 6 Nov 2001 13:00:50 -0500 (EST) Received: from compuserve.com (sfr-tgn-sfu-vty91.as.wcom.net [216.192.39.91]) by siaar1aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id NAA29487 for ; Tue, 6 Nov 2001 13:00:44 -0500 (EST) Message-ID: <3BE825DE.12B11EE9@compuserve.com> Date: Tue, 06 Nov 2001 12:03:10 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win98; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCencap: Missing SOF/EOF characters References: <9410A0F67DFE7F4D998D45382364983617B29B@nc8220exchange.ral.lucent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I am becoming convinced that the FC Encapsulation document is fully ready for last call, since the only topic we can find to talk about is a detailed discussion of what the Encapsulation cannot do. That seems like a strong indication that what the Encapsulation does do is well defined. As for the idea of adding an Annex to describe what the Encapsulation cannot do ... I would rather kill trees describing what the FC Encapsulation can do, and leave the large body of what it cannot do unspecified. Ralph... Elizabeth Rodriguez wrote: > As mentioned in previous email, we may also want to include the > 'why not appropriate' in annex, for reader's information. > > Elizabeth From owner-ips@ece.cmu.edu Tue Nov 6 13:50:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26738 for ; Tue, 6 Nov 2001 13:50:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6Hxib04927 for ips-outgoing; Tue, 6 Nov 2001 12:59:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6HxJl04887 for ; Tue, 6 Nov 2001 12:59:42 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel1.hp.com (Postfix) with ESMTP id 00197C1D; Tue, 6 Nov 2001 09:59:17 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id KAA20171; Tue, 6 Nov 2001 10:00:44 -0800 (PST) Message-Id: <200111061800.KAA20171@core.rose.hp.com> Date: Tue, 06 Nov 2001 10:11:36 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: Rahul Bhagwat Cc: ips@ece.cmu.edu Subject: Re: Connection Recovery References: <012a01c165fe$2dfedae0$3b50d40a@vxindia.veritas.com> <005701c16635$96597730$edd52b0f@rose.hp.com> <007201c16681$b4fc9130$3b50d40a@vxindia.veritas.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Rahul Bhagwat wrote: > > Hi, > > The only possible state that CLEANUP_WAIT eventually ends up is > FREE (on timeout or successful implicit/explicit logout). > > If task reassignment happens unrelated to connection state, As I already said - " A successful connection logout (implicit or explicit) must precede the task reassignments during a connection recovery operation." Task reassignment *is* related to connection state FREE. >is there any > other resource associated with connection that can be still used while it is > in CLEANUP_WAIT state. > > Why do we have to wait for initiator to send logout to clean up things. I am not sure I quite understood what you're saying here. There's an architected timeout (M1 transition) to reclaim the resources, I only pointed out Logout as "highly desirable" if all an implementation wants is to reclaim connection and task resources. OTOH, Logout is mandatory if task reassignment is sought. A CLEANUP_WAIT state from initiator's perspective (let's say due to an initiator NIC failure) does not imply a CLEANUP_WAIT on the target (for the same connection). Logout is a mechanism that forces both sides to sync up on the FREE state, and is the architected mechanism to formally decouple the connection allegiance of the task. It is to avoid cases where target may receive stale PDUs on one connection, while receiving the PDUs also on a (reassigned) different connection. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com > Regards, > Rahul > > > > A successful connection logout (implicit or explicit) must precede the > > task reassignments during a connection recovery operation. > > > > But please note that the notion of "connection cleanup" (graceful closing > of > > a > > previously operational iSCSI connection) in the state diagrams goes beyond > > the connection recovery (in fact, that is the reason I renamed from its > > previous > > name, please refer to my email to ips on 11/2/01 with the slide posting > > announcement). > > A connection cleanup is highly desirable even in the absence of task > > reassignment, > > to quickly reclaim the tags and buffers on either end (or, both sides > would > > have > > to wait for a connection timeout to happen, symbolized by the M1 > > transition). > > > > >Once a CSM-E or a CSM-I > > >drives the connection to free state, all the pending tasks need to be > freed > > up. > > > > Not correct. The decision to free up the pending tasks is depedent on the > > operational ErrorRecoveryLevel in the CSM-I case (please look at the > > discussion in section 3.12.2), or is dependent on the Logout reason code > > (recovery Vs close) in the CSM-E case. All the FREE state symbolizes > > really is that the iSCSI connection is gracefully closed with a successful > > explicit/implit iSCSI Logout. The pending tasks at this point have no > > connection allegiance, and are loosely "owned" by the session. It is > > legitimate > > for the pending tasks to be existent (waiting for reassignment) even when > > all the connections reported FREE (please look at the discussion under > > 3.15.2, Time2Retain). > > > > Regards. > > -- > > Mallikarjun > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > Hewlett-Packard MS 5668 > > Roseville CA 95747 > > > > > > ----- Original Message ----- > > From: Rahul Bhagwat > > To: ips@ece.cmu.edu > > Sent: Monday, November 05, 2001 5:31 AM > > Subject: Connection Recovery > > > > > > Hi, > > > > Is there any order in task reassignments and connection logout (implicit > or > > explicit) > > during a connection recovery. > > > > If these two are not related, what is the use of moving the connection to > > CLEANUP_WAIT > > state? CLEANUP_WAIT state typically means that there are pending tasks for > > this > > connection due to which it cannot be moved to FREE state. That is only > > difference > > betweeen FREE state and CLEANUP_WAIT state. > > > > Which probably means that it is mandatory that Task reassigment happens > > before > > logging out a failed connection (in CLEANUP_WAIT state). Once a CSM-E or a > > CSM-I > > drives the connection to free state, all the pending tasks need to be > freed > > up. > > > > Am I correct here? > > > > Regards, > > Rahul > > From owner-ips@ece.cmu.edu Tue Nov 6 13:51:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26769 for ; Tue, 6 Nov 2001 13:51:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6I0x705042 for ips-outgoing; Tue, 6 Nov 2001 13:00:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1aa.compuserve.com (siaar1aa.compuserve.com [149.174.40.9]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6I0ul05032 for ; Tue, 6 Nov 2001 13:00:56 -0500 (EST) Received: (from mailgate@localhost) by siaar1aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id NAA29521 for ips@ece.cmu.edu; Tue, 6 Nov 2001 13:00:50 -0500 (EST) Received: from compuserve.com (sfr-tgn-sfu-vty91.as.wcom.net [216.192.39.91]) by siaar1aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id NAA29474 for ; Tue, 6 Nov 2001 13:00:40 -0500 (EST) Message-ID: <3BE823BC.D382C1E2@compuserve.com> Date: Tue, 06 Nov 2001 11:54:04 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win98; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCencap: Missing SOF/EOF characters References: <5.1.0.14.2.20011101111644.02e8e500@zbl6c002.corpeast.baynetworks.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I would be willing to change the first paragraph of the scope from: "This document describes common mechanisms for the transport of Fibre Channel frames over an IP network, including the encapsulation format and a mechanism for enforcing the Fibre Channel frame lifetime limits." to: "This document describes common mechanisms for the transport of Fibre Channel frames over an IP network within the limitations of service provided by an IP network. The topics described in this document include the encapsulation format and a mechanism for enforcing the Fibre Channel frame lifetime limits." I am NOT willing to discuss Fibre Channel Classes in the draft because to do so would require adding a definition of the term. Fibre Channel Classes are not discussed in the draft as it is currently written. Ralph... Franco Travostino wrote: > Ralph, > why should we be waiting until 5.3 (last page) to break this news (e.g., Class 1 isn't supported)? > Can we move your text up to section 1, aptly titled "Scope". Alternately, we could narrow down the very title of the document. > > 0.02 > -franco From owner-ips@ece.cmu.edu Tue Nov 6 15:27:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA02262 for ; Tue, 6 Nov 2001 15:27:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6JU3M12181 for ips-outgoing; Tue, 6 Nov 2001 14:30:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars0m9.nortelnetworks.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6JU2l12175 for ; Tue, 6 Nov 2001 14:30:02 -0500 (EST) Received: from zcars04f.ca.nortel.com (zcars04f.ca.nortel.com [47.129.242.57]) by zcars0m9.nortelnetworks.com (8.11.0/8.11.0) with ESMTP id fA6JTHS29619 for ; Tue, 6 Nov 2001 14:29:17 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04f.ca.nortel.com; Tue, 6 Nov 2001 14:28:48 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id 40YK363B; Tue, 6 Nov 2001 14:28:02 -0500 Received: from cse-ns-laptop.nortelnetworks.com (cse-ns-laptop.us.nortel.com [47.16.69.109]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id VW7FBRK2; Tue, 6 Nov 2001 14:28:03 -0500 Message-Id: <5.1.0.14.2.20011106140630.020fdca8@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Tue, 06 Nov 2001 14:30:42 -0500 To: ENDL_TX@computer.org, IPS Reflector From: "Franco Travostino" Subject: Re: FCencap: Missing SOF/EOF characters In-Reply-To: <3BE823BC.D382C1E2@compuserve.com> References: <5.1.0.14.2.20011101111644.02e8e500@zbl6c002.corpeast.baynetworks.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_13654213==_.ALT" X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_13654213==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed At 12:54 PM 11/6/2001, Ralph Weber wrote: > "This document describes common mechanisms for the transport > of Fibre Channel frames over an IP network within the limitations > of service provided by an IP network. The topics described in > this document include the encapsulation format and a mechanism > for enforcing the Fibre Channel frame lifetime limits." This proposed text comes a tad closer, but it's cryptic still. >I am NOT willing to discuss Fibre Channel Classes in the draft >because to do so would require adding a definition of the term. >Fibre Channel Classes are not discussed in the draft as it is >currently written. I agree that a definition/discussion would be way inappropriate here. A T11 citation should do the trick instead. We already have a sentence like "The format and content of an FC frame is described in the FC standards (e.g., FC-FS [3], FC-SW-2 [4], and FC-PI [5])." I don't see a problem with writing "Scope is limited to FC Class 2 and 3, which are described in the FC standard ([x]), and then adding [x] and the authoritative T11 document to Section 7 References. This direct and simple. -franco >Ralph... > >Franco Travostino wrote: > > > Ralph, > > why should we be waiting until 5.3 (last page) to break this news > (e.g., Class 1 isn't supported)? > > Can we move your text up to section 1, aptly titled "Scope". > Alternately, we could narrow down the very title of the document. > > > > 0.02 > > -franco --=====================_13654213==_.ALT Content-Type: text/html; charset="us-ascii" At 12:54 PM 11/6/2001, Ralph Weber wrote:
  "This document describes common mechanisms for the transport
  of Fibre Channel frames over an IP network within the limitations
  of service provided by an IP network. The topics described in
  this document include the encapsulation format and a mechanism
  for enforcing the Fibre Channel frame lifetime limits."

This proposed text comes a tad closer, but it's cryptic still.

I am NOT willing to discuss Fibre Channel Classes in the draft
because to do so would require adding a definition of the term.
Fibre Channel Classes are not discussed in the draft as it is
currently written.

I agree that a definition/discussion would be way inappropriate here. A T11 citation should do the trick instead. We already have a sentence like "The format and content of an FC frame is described in the FC standards (e.g., FC-FS [3], FC-SW-2 [4], and FC-PI [5])." I don't see a problem with writing  "Scope is limited to FC Class 2 and 3,  which are described in the FC standard ([x]), and then adding [x] and the authoritative T11 document to Section 7 References. This direct and simple.

-franco


Ralph...

Franco Travostino wrote:

> Ralph,
> why should we be waiting until 5.3 (last page) to break this news (e.g., Class 1 isn't supported)?
> Can we move your text up to section 1, aptly titled "Scope". Alternately, we could narrow down the very title of the document.
>
> 0.02
> -franco
--=====================_13654213==_.ALT-- From owner-ips@ece.cmu.edu Tue Nov 6 15:29:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA02341 for ; Tue, 6 Nov 2001 15:29:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6JqiH14097 for ips-outgoing; Tue, 6 Nov 2001 14:52:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6Jqfl14089 for ; Tue, 6 Nov 2001 14:52:41 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id UAA49402; Tue, 6 Nov 2001 20:52:21 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA6JptO115794; Tue, 6 Nov 2001 20:52:09 +0100 Importance: Normal Subject: RE: iSCSI: IPsec tunnel / transport mode decision To: "Bill Strahm" Cc: , X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "Ofer Biran" Date: Tue, 6 Nov 2001 21:53:05 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 06/11/2001 21:52:21 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, I agree that you can make external devices that support transport mode, but it seems that most of those existing today do not support it. Anyway for our required decision... you also said you prefer tunnel mode, right ? Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Bill Strahm" on 04/11/2001 21:39:22 Please respond to "Bill Strahm" To: Ofer Biran/Haifa/IBM@IBMIL, cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision Ok, How does mandatory Transport mode remove the possibility of external IPsec... I have said before I can make IPsec transport & tunnel mode work in external devices, just like you can do SSL/TLS accelerators both internally and externally Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Sunday, November 04, 2001 4:27 AM To: saqibj@margallacomm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Saqib, Mandatory transport mode would make bundling of external IPSec impossible, while tunnel mode is not more difficult to implement within the iSCSI endpoint than transport mode. "Cost of ownership and complexity of deploying a stand-alone IPsec gateway" might be among the considerations of vendors and customers, but I don't think the standard should block such solutions (and it blocks more than just stand-alone IPsec gateway). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 02/11/2001 20:59:03 Please respond to To: "Bill Strahm" , "CAVANNA,VICENTE V (A-Roseville,ex1)" cc: "SHEEHY,DAVE (A-Americas,unix1)" , Ofer Biran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: IPsec tunnel / transport mode decision What about the cost of ownership and complexity of deploying a stand-alone IPsec gateway for use with IPsec end-points? If transport-mode IPsec is a must-to-implement capability in iSCSI end-points there is an opportunity to have much more coherent security for iSCSI. Saqib From owner-ips@ece.cmu.edu Tue Nov 6 15:32:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA02388 for ; Tue, 6 Nov 2001 15:32:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6Jhru13338 for ips-outgoing; Tue, 6 Nov 2001 14:43:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel13.hp.com (palrel13.hp.com [156.153.255.238]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6Jhkl13324 for ; Tue, 6 Nov 2001 14:43:46 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel13.hp.com (Postfix) with ESMTP id 49EF21F87F; Tue, 6 Nov 2001 11:43:40 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA10840; Tue, 6 Nov 2001 11:43:34 -0800 (PST) Message-ID: <3BE83F6C.D54FE175@cup.hp.com> Date: Tue, 06 Nov 2001 11:52:12 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Julian Satran Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, There's further reasons why iscsi must not mandate that the initiator issue commands in order within a given connection : 1) The cmdsn assignment may be done in a host resident session manager in an architecture that allows for multi-connection sessions that span HBAs. In such a model, the initiator may issue commands from the host driver to the HBA in cmdsn order. However, within the HBA firmware, some command[s] may be errored back to the host either due to resource reasons or link down or mailbox command incorrectly initialized. Requiring that the initiator issue commands in order implies that the initiator HBA firmware needs to implement some sort of sequencing scheme to ensure that it does not issue a lower cmdsn. There's no need to add any of the above complexity into initiator HBA firmware, since command ordering is already being implemented at the target end. What we've ended up doing is inserting sequencing logic at both ends in attempting to enforce compliance to this rule ! 2) On a connection recovery, re-issued commands [on a cmdsn ack timeout] may be issued on a different connection, thus, causing out of order cmdsn's to be issued on a specific connection. There's no way for the initiator HBA firmware to know that this is a re-issued command and detect that this specific out of order cmdsn is to be allowed. 3) Initiator driver algorithms are quite flexible in their ordering schemes today and imposing strict ordering at the initiator is beyond what is required for the wire protocol. The layer that hands out cmdsn's to outgoing commands may be a higher layer driver in the iscsi stack (common across multiple vendor specific drivers) and re-ordering of commands may occur in the lower layers of the iscsi stack. For all of the above reasons, in addition to HBA optimizations that Rod has voiced, I suggest that NO ordering requirement be imposed on initiators within a specific connection. The cmdsn ordering scheme is mandatory to implement/use and that is more than sufficient. Regards, Santosh Julian Satran wrote: > > Rod, > > I don't see any reason why DMA operations cant be "multiplexed" with > commands. > If you have scheduled a long outbound DMA you are doomed regardless of the > command ordering. > And if you have scheduled DMA operations piecemeal then you can insert > your commands in correct order. > > Julo > > "Rod Harrison" > 05-11-01 20:48 > Please respond to "Rod Harrison" > > > To: Julian Satran/Haifa/IBM@IBMIL, > cc: > Subject: iSCSI: Out of order commands, was current UNH Plugfest > > > > [ Subject changed ] > > Julian, > > The ordering difference is introduced between the host > side driver > and the iSCSI HBA. The host side driver must present SCSI commands to > the HBA in the order they are received from the OS to prevent read > after write dependency failures. The HBA might reorder the commands > depending on when DMA completes. The reordering can't be done ahead of > time in the host driver since it doesn't know how long each DMA might > take. As long as the HBA assigns CmdSN in the order it receives > commands the desired host ordering is preserved. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Monday, November 05, 2001 12:35 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: current UNH Plugfest > > Rod, > > I all examples give the point I find hard to understand is why is the > ordering on the wire different from the presentation order to the > initiator. You can get as many overlaps as you want by presenting the > commands to the initiator in the desired order. > What we are considering here is the case in which you want to ship in > an > order different than the one you present the commands. > > Julo > > "Rod Harrison" > Sent by: owner-ips@ece.cmu.edu > 04-11-01 04:42 > Please respond to "Rod Harrison" > > To: "Barry Reinhold" , "Dave > Sheehy" > , "IETF IP SAN Reflector" > > cc: > Subject: RE: iSCSI: current UNH Plugfest > > Barry, > > In general I agree but I don't think this is as much > of a > corner case > as it at first appears. Targets will have code very similar to that > needed to handle out of order commands to deal with digest errors. > Targets also need to queue commands whilst waiting for both solicited > and unsolicited data to arrive. Queuing out of order commands seems > little extra work. > > From an initiators point of view there are > efficiency, > and probably > performance gains to be had from sending commands out of order. Bob > Russell gave the example of a read being sent whilst write data DMA is > happening, and a similar situation can arise with DMA for writes > overtaking that of earlier writes if the initiator has multiple DMA > engines. In this case the initiator might be forced to let the wire go > idle if it can't send the data from completed DMAs as soon as > possible. > > We already have a command queue at the target to > enforce > correct > serialisation of commands, doing the same thing at the initiator is > redundant. > > Finally, I don't believe we should be writing a > standard > to work > around poor coding and test coverage, especially at the cost of > potential efficiency gains. > > I agree with Dave and Santosh that commands being > sent > out of order > on a single session should be allowed by the standard. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Barry Reinhold > Sent: Friday, November 02, 2001 5:24 PM > To: Dave Sheehy; IETF IP SAN Reflector > Subject: RE: iSCSI: current UNH Plugfest > > Using features such as out of order command delivery on a connection > tend to > be the sort of things that lead to interoperability problems. It is > unexpected and probably going to hit poorly tested code paths even if > the > standard is written to allow it. > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > >Dave Sheehy > >Sent: Friday, November 02, 2001 4:19 PM > >To: IETF IP SAN Reflector > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > >> 3. Can commands be sent out of order on the same connection? > >> > >> The behavior of targets is clearly specified in Section 2.2.2.3 > on > >> page 25 of draft 8, which says: > >> "Except for the commands marked for immediate delivery the > iSCSI > >> target layer MUST eliver the commands for execution in the > order > >> specified by CmdSN." > >> > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >> "- CmdSN - the current command Sequence Number advanced by 1 > on > >> each command shipped except for commands marked for immediate > >> delivery." > >> but the meaning of the term "shipped" is vague, and does not > >> necessarily > >> require that the PDUs arrive on the other end of a TCP > connection > >> in the same order that the CmdSN values were assigned to these > PDUs. > >> > >> Some initiators have been designed to send commands out of CmdSN > >> order on one connection. Consider the situation where there is > only > >> one connection and a high-level dispatcher creates a PDU for a > SCSI > >> command that involves writing immediate data to the target. > This PDU > >> is enqueued to a lower-level layer which has to setup, start, > and > >> wait-for a DMA operation to move the immediate data into an > onboard > >> buffer before the PDU can be put onto the wire. While this is > >> happening, the dispatcher creates another unrelated PDU for a > SCSI > >> read command (for example), and when this PDU is passed to the > >> lower-level layer it can be sent immediately, ahead of the > previous > >> write PDU and therefore out of order on this connection. > >> > >> The standard clearly allows this to happen if the two PDUs were > sent > >> on different connections, and seems to imply that this can also > happen > >> when the two PDUs are sent on the same connection. > >> > >> The suggestion is to put in the standard an explicit statement > that > >> this is allowed or not allowed, as appropriate. > >> > >> If this is allowed, such a statement would avoid the erroneous > >> assumption being made by some target implementers that within a > single > >> connection, commands will arrive in order. > >> > >> If this is not allowed, such a statement would avoid the > erroneous > >> assumption being made by some initiator implementers that within > a > >> single connection, commands can be put on the wire out of order. > >> > >> +++ > >> > >> will add an explicit statement saying that this behaviour is > forbidden. > >> 2.2.2.1 will contain: > >> > >> On any given connection, the iSCSI initiator MUST send the > >commands in the > >> order specified by CmdSN. > >> > >> +++ > > > >Why do you feel this behavior should be forbidden? Targets already > have to > >order commands across the session. I don't see why it's a problem to > extend > >that to the connection as well. I, for one, believe we should take > >a liberal > >stance on this. > > > >Dave Sheehy > > -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Tue Nov 6 15:33:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA02420 for ; Tue, 6 Nov 2001 15:33:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6JPmu11817 for ips-outgoing; Tue, 6 Nov 2001 14:25:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.windriver.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6JPkl11810 for ; Tue, 6 Nov 2001 14:25:46 -0500 (EST) Received: from london ([147.11.45.211]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id LAA25499; Tue, 6 Nov 2001 11:25:05 -0800 (PST) From: "Rod Harrison" To: "Julian Satran" , Subject: RE: iSCSI: Out of order commands, was current UNH Plugfest Date: Tue, 6 Nov 2001 11:28:06 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, I don't understand what you are proposing here, what do you mean by "multiplexed" DMA? The problem is that the DMAs take some time, the more there are queued the longer the last DMAs queued take to complete. Some commands require DMAs to complete before they can be sent, i.e. Writes with immediate data, some commands do not, i.e. Reads and writes with no immediate data. The iSCSI HBA wants to be able to send commands as soon a possible, which for a read after a write can be before the write's DMA has completed. Maintaining an ordered queue for commands to be sent on the HBA is expensive and redundant since the target already knows how to queue commands before committing them to its SCSI layer. The iSCSI HBA and its host driver are not at liberty to change the order of commands from the OS, but the DMAs those commands need are unlikely to complete in the same order, and as I mentioned some commands need no DMA. If the HBA can't send commands out of CmdSN order it has to maintain an ordered queue of commands waiting to be sent, and potentially buffer a lot of data. For an HBA this makes immediate data almost impossible to support. I don't see the problem with allowing out of order commands given that the target already has to deal with very similar problems. I think we are getting in to the area of implementation choices here, which is inappropriate for a specification. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Monday, November 05, 2001 10:06 PM To: ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest Rod, I don't see any reason why DMA operations cant be "multiplexed" with commands. If you have scheduled a long outbound DMA you are doomed regardless of the command ordering. And if you have scheduled DMA operations piecemeal then you can insert your commands in correct order. Julo "Rod Harrison" 05-11-01 20:48 Please respond to "Rod Harrison" To: Julian Satran/Haifa/IBM@IBMIL, cc: Subject: iSCSI: Out of order commands, was current UNH Plugfest [ Subject changed ] Julian, The ordering difference is introduced between the host side driver and the iSCSI HBA. The host side driver must present SCSI commands to the HBA in the order they are received from the OS to prevent read after write dependency failures. The HBA might reorder the commands depending on when DMA completes. The reordering can't be done ahead of time in the host driver since it doesn't know how long each DMA might take. As long as the HBA assigns CmdSN in the order it receives commands the desired host ordering is preserved. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Monday, November 05, 2001 12:35 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: current UNH Plugfest Rod, I all examples give the point I find hard to understand is why is the ordering on the wire different from the presentation order to the initiator. You can get as many overlaps as you want by presenting the commands to the initiator in the desired order. What we are considering here is the case in which you want to ship in an order different than the one you present the commands. Julo "Rod Harrison" Sent by: owner-ips@ece.cmu.edu 04-11-01 04:42 Please respond to "Rod Harrison" To: "Barry Reinhold" , "Dave Sheehy" , "IETF IP SAN Reflector" cc: Subject: RE: iSCSI: current UNH Plugfest Barry, In general I agree but I don't think this is as much of a corner case as it at first appears. Targets will have code very similar to that needed to handle out of order commands to deal with digest errors. Targets also need to queue commands whilst waiting for both solicited and unsolicited data to arrive. Queuing out of order commands seems little extra work. From an initiators point of view there are efficiency, and probably performance gains to be had from sending commands out of order. Bob Russell gave the example of a read being sent whilst write data DMA is happening, and a similar situation can arise with DMA for writes overtaking that of earlier writes if the initiator has multiple DMA engines. In this case the initiator might be forced to let the wire go idle if it can't send the data from completed DMAs as soon as possible. We already have a command queue at the target to enforce correct serialisation of commands, doing the same thing at the initiator is redundant. Finally, I don't believe we should be writing a standard to work around poor coding and test coverage, especially at the cost of potential efficiency gains. I agree with Dave and Santosh that commands being sent out of order on a single session should be allowed by the standard. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Barry Reinhold Sent: Friday, November 02, 2001 5:24 PM To: Dave Sheehy; IETF IP SAN Reflector Subject: RE: iSCSI: current UNH Plugfest Using features such as out of order command delivery on a connection tend to be the sort of things that lead to interoperability problems. It is unexpected and probably going to hit poorly tested code paths even if the standard is written to allow it. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Dave Sheehy >Sent: Friday, November 02, 2001 4:19 PM >To: IETF IP SAN Reflector >Subject: Re: iSCSI: current UNH Plugfest > > > >> 3. Can commands be sent out of order on the same connection? >> >> The behavior of targets is clearly specified in Section 2.2.2.3 on >> page 25 of draft 8, which says: >> "Except for the commands marked for immediate delivery the iSCSI >> target layer MUST eliver the commands for execution in the order >> specified by CmdSN." >> >> Section 2.2.2.3 on page 26 of draft 8 also says: >> "- CmdSN - the current command Sequence Number advanced by 1 on >> each command shipped except for commands marked for immediate >> delivery." >> but the meaning of the term "shipped" is vague, and does not >> necessarily >> require that the PDUs arrive on the other end of a TCP connection >> in the same order that the CmdSN values were assigned to these PDUs. >> >> Some initiators have been designed to send commands out of CmdSN >> order on one connection. Consider the situation where there is only >> one connection and a high-level dispatcher creates a PDU for a SCSI >> command that involves writing immediate data to the target. This PDU >> is enqueued to a lower-level layer which has to setup, start, and >> wait-for a DMA operation to move the immediate data into an onboard >> buffer before the PDU can be put onto the wire. While this is >> happening, the dispatcher creates another unrelated PDU for a SCSI >> read command (for example), and when this PDU is passed to the >> lower-level layer it can be sent immediately, ahead of the previous >> write PDU and therefore out of order on this connection. >> >> The standard clearly allows this to happen if the two PDUs were sent >> on different connections, and seems to imply that this can also happen >> when the two PDUs are sent on the same connection. >> >> The suggestion is to put in the standard an explicit statement that >> this is allowed or not allowed, as appropriate. >> >> If this is allowed, such a statement would avoid the erroneous >> assumption being made by some target implementers that within a single >> connection, commands will arrive in order. >> >> If this is not allowed, such a statement would avoid the erroneous >> assumption being made by some initiator implementers that within a >> single connection, commands can be put on the wire out of order. >> >> +++ >> >> will add an explicit statement saying that this behaviour is forbidden. >> 2.2.2.1 will contain: >> >> On any given connection, the iSCSI initiator MUST send the >commands in the >> order specified by CmdSN. >> >> +++ > >Why do you feel this behavior should be forbidden? Targets already have to >order commands across the session. I don't see why it's a problem to extend >that to the connection as well. I, for one, believe we should take >a liberal >stance on this. > >Dave Sheehy > From owner-ips@ece.cmu.edu Tue Nov 6 16:29:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA04094 for ; Tue, 6 Nov 2001 16:29:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6KZLc17532 for ips-outgoing; Tue, 6 Nov 2001 15:35:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6KZGl17525 for ; Tue, 6 Nov 2001 15:35:16 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA49034 for ; Tue, 6 Nov 2001 15:32:39 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA6KYT090682 for ; Tue, 6 Nov 2001 13:34:29 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: resend of diagram from 2.5.1 To: "Andre Asselin" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Tue, 6 Nov 2001 12:33:35 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/06/2001 01:34:28 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Andre, Now that I can see the picture that you sent. It probably now needs to have the Names of the Target Side SCSI Ports clearly called out. The picture might imply to some folks that the SCSI Port Name is associated with "iSCSI Name + TSID" where as the SCSI Port Name should be "iSCSI Name + PortalGroupTag". The picture is, I think, technically correct, the Target End of the Session can be associated with iSCSI Name+TSID, but that may cause others to go through the same set of discussions that we have held recently, and they might think that the SCSI Port Name includes the TSID. I wonder if it would be clearer to Align, in the pictures, the concept of Target End of a Session with SCSI Port Name and give it the identification of "iSCSI Name + PortalGroupTag" and leave TSID to the various text in the Draft and for it be understood as just a Handle which the implementation can use in any way it wants. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Andre Asselin 11/06/2001 07:38 AM To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: iSCSI: resend of diagram from 2.5.1 It looks like my original post with an updated diagram for section 2.5.1 got wrapped badly. Here's a second attempt that hopefully will make it through. (original) ----------------------------IP Network--------------------- | | | +--------|---------------|--------------------|---------------------+ | +----|---------------|-----+ +----|---------+ | | | +---------+ +---------+ | | +---------+ | | | | | Network | | Network | | | | Network | | | | | | Portal | | Portal | | | | Portal | | | | | +--|------+ +---------+ | | +---------+ | | | | | | | | | | | | | | Portal | | | | Portal | | | | | Group 1 | | | | Group 2 | | | +--------------------------+ +--------------+ | | | | | | | +----------------------------+ +-----------------------------+ | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | +----------------------------+ +-----------------------------+ | | | | iSCSI Target Node | | (within Network Entity, not shown) | +-------------------------------------------------------------------+ (updated) ----------------------------IP Network--------------------- | | | +--------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +------|-------+ | | | | | Portal | | Portal | | | | | | Group 1 | | Group 2 | | | | | +--------------------------+ +--------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-----------------------------------------------------------------+ | | | | Network Entity | +---------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Tue Nov 6 17:37:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA05291 for ; Tue, 6 Nov 2001 17:37:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6LYDp22071 for ips-outgoing; Tue, 6 Nov 2001 16:34:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6LYBl22066 for ; Tue, 6 Nov 2001 16:34:11 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA6LY2u03009; Tue, 6 Nov 2001 13:34:02 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA6LXu610144; Tue, 6 Nov 2001 13:33:56 -0800 From: "Bill Strahm" To: "Ofer Biran" Cc: , Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Tue, 6 Nov 2001 13:33:38 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I actually would prefer if we didn't say anything other than a statement saying "Here is a policy that will cover IPS traffic" from there it is up to compliant IPsec implementations to utilize this policy... Being that the WG feels that just specifying a coverage policy is adequate, but must get into specifying portions of the IPsec functionality, I would prefer Tunnel mode, because as far as I can tell, no one has shown a functional e-e transport mode implementation in the wild... Can anyone point to one ? Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: Ofer Biran [mailto:BIRAN@il.ibm.com] Sent: Tuesday, November 06, 2001 11:53 AM To: Bill Strahm Cc: saqibj@margallacomm.com; ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Bill, I agree that you can make external devices that support transport mode, but it seems that most of those existing today do not support it. Anyway for our required decision... you also said you prefer tunnel mode, right ? Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Bill Strahm" on 04/11/2001 21:39:22 Please respond to "Bill Strahm" To: Ofer Biran/Haifa/IBM@IBMIL, cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision Ok, How does mandatory Transport mode remove the possibility of external IPsec... I have said before I can make IPsec transport & tunnel mode work in external devices, just like you can do SSL/TLS accelerators both internally and externally Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Sunday, November 04, 2001 4:27 AM To: saqibj@margallacomm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Saqib, Mandatory transport mode would make bundling of external IPSec impossible, while tunnel mode is not more difficult to implement within the iSCSI endpoint than transport mode. "Cost of ownership and complexity of deploying a stand-alone IPsec gateway" might be among the considerations of vendors and customers, but I don't think the standard should block such solutions (and it blocks more than just stand-alone IPsec gateway). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 02/11/2001 20:59:03 Please respond to To: "Bill Strahm" , "CAVANNA,VICENTE V (A-Roseville,ex1)" cc: "SHEEHY,DAVE (A-Americas,unix1)" , Ofer Biran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: IPsec tunnel / transport mode decision What about the cost of ownership and complexity of deploying a stand-alone IPsec gateway for use with IPsec end-points? If transport-mode IPsec is a must-to-implement capability in iSCSI end-points there is an opportunity to have much more coherent security for iSCSI. Saqib From owner-ips@ece.cmu.edu Tue Nov 6 19:20:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA06877 for ; Tue, 6 Nov 2001 19:20:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA6N9oU28940 for ips-outgoing; Tue, 6 Nov 2001 18:09:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA6N9ml28935 for ; Tue, 6 Nov 2001 18:09:48 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id PAA00101; Tue, 6 Nov 2001 15:09:30 -0800 (PST) From: "Murali Rajagopal" To: "Franco Travostino" , , "IPS Reflector" Subject: RE: FCencap: Missing SOF/EOF characters Date: Tue, 6 Nov 2001 15:10:32 -0800 Message-ID: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0050_01C166D5.2DF0AF60" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 Importance: Normal In-Reply-To: <5.1.0.14.2.20011106140630.020fdca8@zbl6c002.corpeast.baynetworks.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0050_01C166D5.2DF0AF60 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit How about stating something that we do support in the Scope section: "This document only considers encapsulation for only FC Classes 2 and 3." -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Franco Travostino Sent: Tuesday, November 06, 2001 11:31 AM To: ENDL_TX@computer.org; IPS Reflector Subject: Re: FCencap: Missing SOF/EOF characters At 12:54 PM 11/6/2001, Ralph Weber wrote: "This document describes common mechanisms for the transport of Fibre Channel frames over an IP network within the limitations of service provided by an IP network. The topics described in this document include the encapsulation format and a mechanism for enforcing the Fibre Channel frame lifetime limits." This proposed text comes a tad closer, but it's cryptic still. I am NOT willing to discuss Fibre Channel Classes in the draft because to do so would require adding a definition of the term. Fibre Channel Classes are not discussed in the draft as it is currently written. I agree that a definition/discussion would be way inappropriate here. A T11 citation should do the trick instead. We already have a sentence like "The format and content of an FC frame is described in the FC standards (e.g., FC-FS [3], FC-SW-2 [4], and FC-PI [5])." I don't see a problem with writing "Scope is limited to FC Class 2 and 3, which are described in the FC standard ([x]), and then adding [x] and the authoritative T11 document to Section 7 References. This direct and simple. -franco Ralph... Franco Travostino wrote: > Ralph, > why should we be waiting until 5.3 (last page) to break this news (e.g., Class 1 isn't supported)? > Can we move your text up to section 1, aptly titled "Scope". Alternately, we could narrow down the very title of the document. > > 0.02 > -franco ------=_NextPart_000_0050_01C166D5.2DF0AF60 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Ho= w about stating something that we do support in the Scope = section:

 

&q= uot;This document only considers encapsulation for only FC Classes 2 and = 3."

 

-M= urali

 

-----Original Message-----
From: = owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On = Behalf Of Franco Travostino
Sent: Tuesday, November = 06, 2001 11:31 AM
To: ENDL_TX@computer.org; = IPS Reflector
Subject: Re: FCencap: = Missing SOF/EOF characters

 

At = 12:54 PM 11/6/2001, Ralph Weber wrote:

  "This document = describes common mechanisms for the transport
  of Fibre Channel frames over an IP network within the = limitations
  of service provided by an IP network. The topics described in
  this document include the encapsulation format and a = mechanism
  for enforcing the Fibre Channel frame lifetime = limits."<= /p>


This proposed text comes a tad closer, but it's cryptic still.


I am NOT willing to discuss Fibre = Channel Classes in the draft
because to do so would require adding a definition of the term.
Fibre Channel Classes are not discussed in the draft as it is
currently written.


I agree that a definition/discussion would be way inappropriate here. A = T11 citation should do the trick instead. We already have a sentence like = "The format and content of an FC frame is described in the FC standards = (e.g., FC-FS [3], FC-SW-2 [4], and FC-PI [5])." I don't see a problem with writing  "Scope is limited to FC Class 2 and 3,  which = are described in the FC standard ([x]), and then adding [x] and the = authoritative T11 document to Section 7 References. This direct and simple.

-franco



Ralph...

Franco Travostino wrote:

> Ralph,
> why should we be waiting until 5.3 (last page) to break this news = (e.g., Class 1 isn't supported)?
> Can we move your text up to section 1, aptly titled = "Scope". Alternately, we could narrow down the very title of the document.
>
> 0.02
> -franco

------=_NextPart_000_0050_01C166D5.2DF0AF60-- From owner-ips@ece.cmu.edu Tue Nov 6 20:27:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA07626 for ; Tue, 6 Nov 2001 20:27:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA70GRE03096 for ips-outgoing; Tue, 6 Nov 2001 19:16:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from rodney.xo.com (rodney.xo.com [207.155.252.48]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA70GPl03092 for ; Tue, 6 Nov 2001 19:16:25 -0500 (EST) Received: from blekinge ([66.89.96.162]) by rodney.xo.com id TAA05332; Tue, 6 Nov 2001 19:16:24 -0500 (EST) [ConcentricHost SMTP Relay 1.14] Message-ID: <020001c16721$536f4f70$1001010a@blekinge> From: "Peter Mellquist" To: Subject: iSCSI over TLS Date: Tue, 6 Nov 2001 16:15:17 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I am aware that the ips group is leaning toward IPSEC as for the security solution but I am interested if anyone is also considering using Transport Layer Security (TLS)? I am concerned that the requirement for IPSEC might make TOEs more complex than they need to be. Can TLS be optionally used as well as defined by the specification? This could allow TOE vendors to only be concerned with providing normal IPv4 / ipv6 and leave the security to a higher layer. A TLS stack sitting above the TOE could then handle security very well. Also, I anticipate that the first generation of TOEs will not support IPSEC. With a iSCSI/TLS we could enable security solutions with the first generation of TOEs and get speed and security. Are any TOE vendors planning to support IPSEC? Can TLS or IPSEC be supported? -peter Peter Mellquist Seven Systems Technologies 575 Menlo Drive Suite 2 Rocklin CA 916-577-1275 peterm@seven-systems.com From owner-ips@ece.cmu.edu Tue Nov 6 21:17:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA08296 for ; Tue, 6 Nov 2001 21:17:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA715de06220 for ips-outgoing; Tue, 6 Nov 2001 20:05:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA715bl06215 for ; Tue, 6 Nov 2001 20:05:37 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel12.hp.com (Postfix) with ESMTP id 78B981F837; Tue, 6 Nov 2001 17:05:31 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id RAA07404; Tue, 6 Nov 2001 17:06:58 -0800 (PST) Message-Id: <200111070106.RAA07404@core.rose.hp.com> Date: Tue, 06 Nov 2001 17:17:50 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: Rod Harrison Cc: Julian Satran , ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Rod and Julian, This has been an interesting thread of discussion. Some comments - 1.My first reaction was - allowing out-of-order command transmission on the same connection deprives targets of an implementation choice. Targets which support only single-connection sessions and only support session recovery (reasonable assumptions in my mind) can no longer afford *not to* implement a command scoreboard. 2.Any end-node efficiency that is sought to be achieved by transmitting CmdSNs out-of-order from the initiator would be lost on the other end-node, since the target now must wait for re-ordering the commands. 3.The flipside is that out-of-order transmission saves link badwidth (albeit at the expense of end-node efficiency), compared to idling the link waiting for outbound DMA. We have to determine if this is a reasonable trade-off. 4.I can see Rod's point that prefetching all immediate data can be a burden on the NIC resources. But, two questions - - could the NIC not use unsolicited separate data PDUs in these cases? [ I realize that InitialR2T has to be "no" to let it happen... ] - could the NIC have a memory architecture that allows data prefetching for the next command (so this is a non-issue from the protocol perspective)? This scheme incurs one DMA delay for every new burst of commands. 5.Another (perhaps radical at this point) option is to do away with immediate unsolicited data, to stick only with separate unsolicited data. I would personally be okay with the choice, particularly if this feature (that helps software implementations) starts making hardware design complicated/expensive. So, to summarize - option immediate allow data in spec? out-of-order? (A) (5) above no no (B) No real reason to do this. no yes (C) (4) above yes no (D) pros & cons (1), (2) & (3) yes yes From the arguments I heard so far, I am leaning towards option A, and option C in that order. Comments? -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Rod Harrison wrote: > > Julian, > > I don't understand what you are proposing here, what do you mean by > "multiplexed" DMA? > > The problem is that the DMAs take some time, the more there are > queued the longer the last DMAs queued take to complete. Some commands > require DMAs to complete before they can be sent, i.e. Writes with > immediate data, some commands do not, i.e. Reads and writes with no > immediate data. The iSCSI HBA wants to be able to send commands as > soon a possible, which for a read after a write can be before the > write's DMA has completed. Maintaining an ordered queue for commands > to be sent on the HBA is expensive and redundant since the target > already knows how to queue commands before committing them to its SCSI > layer. > > The iSCSI HBA and its host driver are not at liberty to change the > order of commands from the OS, but the DMAs those commands need are > unlikely to complete in the same order, and as I mentioned some > commands need no DMA. If the HBA can't send commands out of CmdSN > order it has to maintain an ordered queue of commands waiting to be > sent, and potentially buffer a lot of data. For an HBA this makes > immediate data almost impossible to support. > > I don't see the problem with allowing out of order commands given > that the target already has to deal with very similar problems. I > think we are getting in to the area of implementation choices here, > which is inappropriate for a specification. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Monday, November 05, 2001 10:06 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > Rod, > > I don't see any reason why DMA operations cant be "multiplexed" with > commands. > If you have scheduled a long outbound DMA you are doomed regardless of > the > command ordering. > And if you have scheduled DMA operations piecemeal then you can insert > your commands in correct order. > > Julo > > "Rod Harrison" > 05-11-01 20:48 > Please respond to "Rod Harrison" > > To: Julian Satran/Haifa/IBM@IBMIL, > cc: > Subject: iSCSI: Out of order commands, was current UNH > Plugfest > > [ Subject changed ] > > Julian, > > The ordering difference is introduced between the > host > side driver > and the iSCSI HBA. The host side driver must present SCSI commands to > the HBA in the order they are received from the OS to prevent read > after write dependency failures. The HBA might reorder the commands > depending on when DMA completes. The reordering can't be done ahead of > time in the host driver since it doesn't know how long each DMA might > take. As long as the HBA assigns CmdSN in the order it receives > commands the desired host ordering is preserved. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Monday, November 05, 2001 12:35 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: current UNH Plugfest > > Rod, > > I all examples give the point I find hard to understand is why is the > ordering on the wire different from the presentation order to the > initiator. You can get as many overlaps as you want by presenting the > commands to the initiator in the desired order. > What we are considering here is the case in which you want to ship in > an > order different than the one you present the commands. > > Julo > > "Rod Harrison" > Sent by: owner-ips@ece.cmu.edu > 04-11-01 04:42 > Please respond to "Rod Harrison" > > To: "Barry Reinhold" , "Dave > Sheehy" > , "IETF IP SAN Reflector" > > cc: > Subject: RE: iSCSI: current UNH Plugfest > > Barry, > > In general I agree but I don't think this is as much > of a > corner case > as it at first appears. Targets will have code very similar to that > needed to handle out of order commands to deal with digest errors. > Targets also need to queue commands whilst waiting for both solicited > and unsolicited data to arrive. Queuing out of order commands seems > little extra work. > > From an initiators point of view there are > efficiency, > and probably > performance gains to be had from sending commands out of order. Bob > Russell gave the example of a read being sent whilst write data DMA is > happening, and a similar situation can arise with DMA for writes > overtaking that of earlier writes if the initiator has multiple DMA > engines. In this case the initiator might be forced to let the wire go > idle if it can't send the data from completed DMAs as soon as > possible. > > We already have a command queue at the target to > enforce > correct > serialisation of commands, doing the same thing at the initiator is > redundant. > > Finally, I don't believe we should be writing a > standard > to work > around poor coding and test coverage, especially at the cost of > potential efficiency gains. > > I agree with Dave and Santosh that commands being > sent > out of order > on a single session should be allowed by the standard. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Barry Reinhold > Sent: Friday, November 02, 2001 5:24 PM > To: Dave Sheehy; IETF IP SAN Reflector > Subject: RE: iSCSI: current UNH Plugfest > > Using features such as out of order command delivery on a connection > tend to > be the sort of things that lead to interoperability problems. It is > unexpected and probably going to hit poorly tested code paths even if > the > standard is written to allow it. > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > >Dave Sheehy > >Sent: Friday, November 02, 2001 4:19 PM > >To: IETF IP SAN Reflector > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > >> 3. Can commands be sent out of order on the same connection? > >> > >> The behavior of targets is clearly specified in Section 2.2.2.3 > on > >> page 25 of draft 8, which says: > >> "Except for the commands marked for immediate delivery the > iSCSI > >> target layer MUST eliver the commands for execution in the > order > >> specified by CmdSN." > >> > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >> "- CmdSN - the current command Sequence Number advanced by 1 > on > >> each command shipped except for commands marked for immediate > >> delivery." > >> but the meaning of the term "shipped" is vague, and does not > >> necessarily > >> require that the PDUs arrive on the other end of a TCP > connection > >> in the same order that the CmdSN values were assigned to these > PDUs. > >> > >> Some initiators have been designed to send commands out of CmdSN > >> order on one connection. Consider the situation where there is > only > >> one connection and a high-level dispatcher creates a PDU for a > SCSI > >> command that involves writing immediate data to the target. > This PDU > >> is enqueued to a lower-level layer which has to setup, start, > and > >> wait-for a DMA operation to move the immediate data into an > onboard > >> buffer before the PDU can be put onto the wire. While this is > >> happening, the dispatcher creates another unrelated PDU for a > SCSI > >> read command (for example), and when this PDU is passed to the > >> lower-level layer it can be sent immediately, ahead of the > previous > >> write PDU and therefore out of order on this connection. > >> > >> The standard clearly allows this to happen if the two PDUs were > sent > >> on different connections, and seems to imply that this can also > happen > >> when the two PDUs are sent on the same connection. > >> > >> The suggestion is to put in the standard an explicit statement > that > >> this is allowed or not allowed, as appropriate. > >> > >> If this is allowed, such a statement would avoid the erroneous > >> assumption being made by some target implementers that within a > single > >> connection, commands will arrive in order. > >> > >> If this is not allowed, such a statement would avoid the > erroneous > >> assumption being made by some initiator implementers that within > a > >> single connection, commands can be put on the wire out of order. > >> > >> +++ > >> > >> will add an explicit statement saying that this behaviour is > forbidden. > >> 2.2.2.1 will contain: > >> > >> On any given connection, the iSCSI initiator MUST send the > >commands in the > >> order specified by CmdSN. > >> > >> +++ > > > >Why do you feel this behavior should be forbidden? Targets already > have to > >order commands across the session. I don't see why it's a problem to > extend > >that to the connection as well. I, for one, believe we should take > >a liberal > >stance on this. > > > >Dave Sheehy > > From owner-ips@ece.cmu.edu Tue Nov 6 22:10:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA09940 for ; Tue, 6 Nov 2001 22:10:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA727ob10196 for ips-outgoing; Tue, 6 Nov 2001 21:07:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel2.hp.com (palrel2.hp.com [156.153.255.234]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA727Pl10158 for ; Tue, 6 Nov 2001 21:07:25 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel2.hp.com (Postfix) with ESMTP id EBCD153B; Tue, 6 Nov 2001 18:07:23 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA13138; Tue, 6 Nov 2001 18:07:18 -0800 (PST) Message-ID: <3BE8995C.892512E8@cup.hp.com> Date: Tue, 06 Nov 2001 18:15:56 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: cbm@rose.hp.com Cc: Rod Harrison , Julian Satran , ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands References: <200111070106.RAA07404@core.rose.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mallikarjun, Some comments below. Regards, Santosh "Mallikarjun C." wrote: > > Rod and Julian, > > This has been an interesting thread of discussion. Some > comments - > > 1.My first reaction was - allowing out-of-order command > transmission on the same connection deprives targets of > an implementation choice. Targets which support only > single-connection sessions and only support session > recovery (reasonable assumptions in my mind) can no > longer afford *not to* implement a command scoreboard. Even a single connection target *MUST* implement a scoreboard. The reason being that it can see out-of-order arrival of commands due to commands being dropped on digest errors. In such a case, it must block further command processing until holes are filled. Thus, there is no getting away from implementing a sequencer at the target. Given this, I think it is unreasonable to restrict initiator implementation flexibility by imposing a strict ordering requirement within the connection. > 2.Any end-node efficiency that is sought to be achieved > by transmitting CmdSNs out-of-order from the initiator > would be lost on the other end-node, since the target > now must wait for re-ordering the commands. It has to handle this situation anyway to deal with holes caused by digest errors. This scenario occurs even with initiators that issue commands in order. > > 3.The flipside is that out-of-order transmission saves > link badwidth (albeit at the expense of end-node efficiency), > compared to idling the link waiting for outbound DMA. > We have to determine if this is a reasonable trade-off. > > 4.I can see Rod's point that prefetching all immediate > data can be a burden on the NIC resources. But, two > questions - > - could the NIC not use unsolicited separate data > PDUs in these cases? [ I realize that InitialR2T > has to be "no" to let it happen... ] > - could the NIC have a memory architecture that > allows data prefetching for the next command (so > this is a non-issue from the protocol perspective)? > This scheme incurs one DMA delay for every new > burst of commands. > > 5.Another (perhaps radical at this point) option is to do > away with immediate unsolicited data, to stick only with > separate unsolicited data. I would personally be okay > with the choice, particularly if this feature (that > helps software implementations) starts making hardware > design complicated/expensive. > > So, to summarize - > > option immediate allow > data in spec? out-of-order? > > (A) (5) above no no > (B) No real reason to do this. no yes > (C) (4) above yes no > (D) pros & cons (1), (2) & (3) yes yes > > >From the arguments I heard so far, I am leaning towards > option A, and option C in that order. > > Comments? > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > Rod Harrison wrote: > > > > Julian, > > > > I don't understand what you are proposing here, what do you mean by > > "multiplexed" DMA? > > > > The problem is that the DMAs take some time, the more there are > > queued the longer the last DMAs queued take to complete. Some commands > > require DMAs to complete before they can be sent, i.e. Writes with > > immediate data, some commands do not, i.e. Reads and writes with no > > immediate data. The iSCSI HBA wants to be able to send commands as > > soon a possible, which for a read after a write can be before the > > write's DMA has completed. Maintaining an ordered queue for commands > > to be sent on the HBA is expensive and redundant since the target > > already knows how to queue commands before committing them to its SCSI > > layer. > > > > The iSCSI HBA and its host driver are not at liberty to change the > > order of commands from the OS, but the DMAs those commands need are > > unlikely to complete in the same order, and as I mentioned some > > commands need no DMA. If the HBA can't send commands out of CmdSN > > order it has to maintain an ordered queue of commands waiting to be > > sent, and potentially buffer a lot of data. For an HBA this makes > > immediate data almost impossible to support. > > > > I don't see the problem with allowing out of order commands given > > that the target already has to deal with very similar problems. I > > think we are getting in to the area of implementation choices here, > > which is inappropriate for a specification. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Monday, November 05, 2001 10:06 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > Rod, > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > commands. > > If you have scheduled a long outbound DMA you are doomed regardless of > > the > > command ordering. > > And if you have scheduled DMA operations piecemeal then you can insert > > your commands in correct order. > > > > Julo > > > > "Rod Harrison" > > 05-11-01 20:48 > > Please respond to "Rod Harrison" > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > cc: > > Subject: iSCSI: Out of order commands, was current UNH > > Plugfest > > > > [ Subject changed ] > > > > Julian, > > > > The ordering difference is introduced between the > > host > > side driver > > and the iSCSI HBA. The host side driver must present SCSI commands to > > the HBA in the order they are received from the OS to prevent read > > after write dependency failures. The HBA might reorder the commands > > depending on when DMA completes. The reordering can't be done ahead of > > time in the host driver since it doesn't know how long each DMA might > > take. As long as the HBA assigns CmdSN in the order it receives > > commands the desired host ordering is preserved. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Monday, November 05, 2001 12:35 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > Rod, > > > > I all examples give the point I find hard to understand is why is the > > ordering on the wire different from the presentation order to the > > initiator. You can get as many overlaps as you want by presenting the > > commands to the initiator in the desired order. > > What we are considering here is the case in which you want to ship in > > an > > order different than the one you present the commands. > > > > Julo > > > > "Rod Harrison" > > Sent by: owner-ips@ece.cmu.edu > > 04-11-01 04:42 > > Please respond to "Rod Harrison" > > > > To: "Barry Reinhold" , "Dave > > Sheehy" > > , "IETF IP SAN Reflector" > > > > cc: > > Subject: RE: iSCSI: current UNH Plugfest > > > > Barry, > > > > In general I agree but I don't think this is as much > > of a > > corner case > > as it at first appears. Targets will have code very similar to that > > needed to handle out of order commands to deal with digest errors. > > Targets also need to queue commands whilst waiting for both solicited > > and unsolicited data to arrive. Queuing out of order commands seems > > little extra work. > > > > From an initiators point of view there are > > efficiency, > > and probably > > performance gains to be had from sending commands out of order. Bob > > Russell gave the example of a read being sent whilst write data DMA is > > happening, and a similar situation can arise with DMA for writes > > overtaking that of earlier writes if the initiator has multiple DMA > > engines. In this case the initiator might be forced to let the wire go > > idle if it can't send the data from completed DMAs as soon as > > possible. > > > > We already have a command queue at the target to > > enforce > > correct > > serialisation of commands, doing the same thing at the initiator is > > redundant. > > > > Finally, I don't believe we should be writing a > > standard > > to work > > around poor coding and test coverage, especially at the cost of > > potential efficiency gains. > > > > I agree with Dave and Santosh that commands being > > sent > > out of order > > on a single session should be allowed by the standard. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Barry Reinhold > > Sent: Friday, November 02, 2001 5:24 PM > > To: Dave Sheehy; IETF IP SAN Reflector > > Subject: RE: iSCSI: current UNH Plugfest > > > > Using features such as out of order command delivery on a connection > > tend to > > be the sort of things that lead to interoperability problems. It is > > unexpected and probably going to hit poorly tested code paths even if > > the > > standard is written to allow it. > > > > >-----Original Message----- > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > >Dave Sheehy > > >Sent: Friday, November 02, 2001 4:19 PM > > >To: IETF IP SAN Reflector > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > >> > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > on > > >> page 25 of draft 8, which says: > > >> "Except for the commands marked for immediate delivery the > > iSCSI > > >> target layer MUST eliver the commands for execution in the > > order > > >> specified by CmdSN." > > >> > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > on > > >> each command shipped except for commands marked for immediate > > >> delivery." > > >> but the meaning of the term "shipped" is vague, and does not > > >> necessarily > > >> require that the PDUs arrive on the other end of a TCP > > connection > > >> in the same order that the CmdSN values were assigned to these > > PDUs. > > >> > > >> Some initiators have been designed to send commands out of CmdSN > > >> order on one connection. Consider the situation where there is > > only > > >> one connection and a high-level dispatcher creates a PDU for a > > SCSI > > >> command that involves writing immediate data to the target. > > This PDU > > >> is enqueued to a lower-level layer which has to setup, start, > > and > > >> wait-for a DMA operation to move the immediate data into an > > onboard > > >> buffer before the PDU can be put onto the wire. While this is > > >> happening, the dispatcher creates another unrelated PDU for a > > SCSI > > >> read command (for example), and when this PDU is passed to the > > >> lower-level layer it can be sent immediately, ahead of the > > previous > > >> write PDU and therefore out of order on this connection. > > >> > > >> The standard clearly allows this to happen if the two PDUs were > > sent > > >> on different connections, and seems to imply that this can also > > happen > > >> when the two PDUs are sent on the same connection. > > >> > > >> The suggestion is to put in the standard an explicit statement > > that > > >> this is allowed or not allowed, as appropriate. > > >> > > >> If this is allowed, such a statement would avoid the erroneous > > >> assumption being made by some target implementers that within a > > single > > >> connection, commands will arrive in order. > > >> > > >> If this is not allowed, such a statement would avoid the > > erroneous > > >> assumption being made by some initiator implementers that within > > a > > >> single connection, commands can be put on the wire out of order. > > >> > > >> +++ > > >> > > >> will add an explicit statement saying that this behaviour is > > forbidden. > > >> 2.2.2.1 will contain: > > >> > > >> On any given connection, the iSCSI initiator MUST send the > > >commands in the > > >> order specified by CmdSN. > > >> > > >> +++ > > > > > >Why do you feel this behavior should be forbidden? Targets already > > have to > > >order commands across the session. I don't see why it's a problem to > > extend > > >that to the connection as well. I, for one, believe we should take > > >a liberal > > >stance on this. > > > > > >Dave Sheehy > > > -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Tue Nov 6 23:04:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA11581 for ; Tue, 6 Nov 2001 23:04:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA72tYH13111 for ips-outgoing; Tue, 6 Nov 2001 21:55:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from svldns02.veritas.com (bay-bridge.veritas.com [143.127.3.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA72tWl13106 for ; Tue, 6 Nov 2001 21:55:32 -0500 (EST) Received: from megami.veritas.com (megami.veritas.com [10.182.128.180]) by svldns02.veritas.com (8.11.6/8.11.6) with SMTP id fA72o9320298; Tue, 6 Nov 2001 18:50:09 -0800 (PST) Received: from rahul([10.212.86.214]) (15171 bytes) by megami.veritas.com via sendmail with P:smtp/R:smart_host/T:smtp (sender: ) id for ; Tue, 6 Nov 2001 18:51:28 -0800 (PST) (Smail-3.2.0.101 1997-Dec-17 #15 built 2001-Aug-30) Message-ID: <002301c16736$0d3c38f0$d656d40a@rahul> From: "Rahul Bhagwat" To: "Santosh Rao" , Cc: "Rod Harrison" , "Julian Satran" , References: <200111070106.RAA07404@core.rose.hp.com> <3BE8995C.892512E8@cup.hp.com> Subject: Re: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 08:13:54 +0530 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi, Comment below > Mallikarjun, > > Some comments below. > > Regards, > Santosh > > "Mallikarjun C." wrote: > > > > Rod and Julian, > > > > This has been an interesting thread of discussion. Some > > comments - > > > > 1.My first reaction was - allowing out-of-order command > > transmission on the same connection deprives targets of > > an implementation choice. Targets which support only > > single-connection sessions and only support session > > recovery (reasonable assumptions in my mind) can no > > longer afford *not to* implement a command scoreboard. > > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. > > Thus, there is no getting away from implementing a sequencer at the > target. Given this, I think it is unreasonable to restrict initiator > implementation flexibility by imposing a strict ordering requirement > within the connection. If a target supports only session recovery, then a digest (header) error means we have to drop the connection (Such a simple target would not even probably support markers). I think for data digest errors command can be treated as received (Command SN window moves). > > > > 2.Any end-node efficiency that is sought to be achieved > > by transmitting CmdSNs out-of-order from the initiator > > would be lost on the other end-node, since the target > > now must wait for re-ordering the commands. > > It has to handle this situation anyway to deal with holes caused by > digest errors. This scenario occurs even with initiators that issue > commands in order. > > > > > > 3.The flipside is that out-of-order transmission saves > > link badwidth (albeit at the expense of end-node efficiency), > > compared to idling the link waiting for outbound DMA. > > We have to determine if this is a reasonable trade-off. > > > > 4.I can see Rod's point that prefetching all immediate > > data can be a burden on the NIC resources. But, two > > questions - > > - could the NIC not use unsolicited separate data > > PDUs in these cases? [ I realize that InitialR2T > > has to be "no" to let it happen... ] > > - could the NIC have a memory architecture that > > allows data prefetching for the next command (so > > this is a non-issue from the protocol perspective)? > > This scheme incurs one DMA delay for every new > > burst of commands. > > > > 5.Another (perhaps radical at this point) option is to do > > away with immediate unsolicited data, to stick only with > > separate unsolicited data. I would personally be okay > > with the choice, particularly if this feature (that > > helps software implementations) starts making hardware > > design complicated/expensive. > > > > So, to summarize - > > > > option immediate allow > > data in spec? out-of-order? > > > > (A) (5) above no no > > (B) No real reason to do this. no yes > > (C) (4) above yes no > > (D) pros & cons (1), (2) & (3) yes yes > > > > >From the arguments I heard so far, I am leaning towards > > option A, and option C in that order. > > > > Comments? > > -- > > Mallikarjun > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > Rod Harrison wrote: > > > > > > Julian, > > > > > > I don't understand what you are proposing here, what do you mean by > > > "multiplexed" DMA? > > > > > > The problem is that the DMAs take some time, the more there are > > > queued the longer the last DMAs queued take to complete. Some commands > > > require DMAs to complete before they can be sent, i.e. Writes with > > > immediate data, some commands do not, i.e. Reads and writes with no > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > soon a possible, which for a read after a write can be before the > > > write's DMA has completed. Maintaining an ordered queue for commands > > > to be sent on the HBA is expensive and redundant since the target > > > already knows how to queue commands before committing them to its SCSI > > > layer. > > > > > > The iSCSI HBA and its host driver are not at liberty to change the > > > order of commands from the OS, but the DMAs those commands need are > > > unlikely to complete in the same order, and as I mentioned some > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > order it has to maintain an ordered queue of commands waiting to be > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > immediate data almost impossible to support. > > > > > > I don't see the problem with allowing out of order commands given > > > that the target already has to deal with very similar problems. I > > > think we are getting in to the area of implementation choices here, > > > which is inappropriate for a specification. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > Rod, > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > commands. > > > If you have scheduled a long outbound DMA you are doomed regardless of > > > the > > > command ordering. > > > And if you have scheduled DMA operations piecemeal then you can insert > > > your commands in correct order. > > > > > > Julo > > > > > > "Rod Harrison" > > > 05-11-01 20:48 > > > Please respond to "Rod Harrison" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > cc: > > > Subject: iSCSI: Out of order commands, was current UNH > > > Plugfest > > > > > > [ Subject changed ] > > > > > > Julian, > > > > > > The ordering difference is introduced between the > > > host > > > side driver > > > and the iSCSI HBA. The host side driver must present SCSI commands to > > > the HBA in the order they are received from the OS to prevent read > > > after write dependency failures. The HBA might reorder the commands > > > depending on when DMA completes. The reordering can't be done ahead of > > > time in the host driver since it doesn't know how long each DMA might > > > take. As long as the HBA assigns CmdSN in the order it receives > > > commands the desired host ordering is preserved. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 12:35 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Rod, > > > > > > I all examples give the point I find hard to understand is why is the > > > ordering on the wire different from the presentation order to the > > > initiator. You can get as many overlaps as you want by presenting the > > > commands to the initiator in the desired order. > > > What we are considering here is the case in which you want to ship in > > > an > > > order different than the one you present the commands. > > > > > > Julo > > > > > > "Rod Harrison" > > > Sent by: owner-ips@ece.cmu.edu > > > 04-11-01 04:42 > > > Please respond to "Rod Harrison" > > > > > > To: "Barry Reinhold" , "Dave > > > Sheehy" > > > , "IETF IP SAN Reflector" > > > > > > cc: > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Barry, > > > > > > In general I agree but I don't think this is as much > > > of a > > > corner case > > > as it at first appears. Targets will have code very similar to that > > > needed to handle out of order commands to deal with digest errors. > > > Targets also need to queue commands whilst waiting for both solicited > > > and unsolicited data to arrive. Queuing out of order commands seems > > > little extra work. > > > > > > From an initiators point of view there are > > > efficiency, > > > and probably > > > performance gains to be had from sending commands out of order. Bob > > > Russell gave the example of a read being sent whilst write data DMA is > > > happening, and a similar situation can arise with DMA for writes > > > overtaking that of earlier writes if the initiator has multiple DMA > > > engines. In this case the initiator might be forced to let the wire go > > > idle if it can't send the data from completed DMAs as soon as > > > possible. > > > > > > We already have a command queue at the target to > > > enforce > > > correct > > > serialisation of commands, doing the same thing at the initiator is > > > redundant. > > > > > > Finally, I don't believe we should be writing a > > > standard > > > to work > > > around poor coding and test coverage, especially at the cost of > > > potential efficiency gains. > > > > > > I agree with Dave and Santosh that commands being > > > sent > > > out of order > > > on a single session should be allowed by the standard. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Barry Reinhold > > > Sent: Friday, November 02, 2001 5:24 PM > > > To: Dave Sheehy; IETF IP SAN Reflector > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Using features such as out of order command delivery on a connection > > > tend to > > > be the sort of things that lead to interoperability problems. It is > > > unexpected and probably going to hit poorly tested code paths even if > > > the > > > standard is written to allow it. > > > > > > >-----Original Message----- > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > >Dave Sheehy > > > >Sent: Friday, November 02, 2001 4:19 PM > > > >To: IETF IP SAN Reflector > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > >> > > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > > on > > > >> page 25 of draft 8, which says: > > > >> "Except for the commands marked for immediate delivery the > > > iSCSI > > > >> target layer MUST eliver the commands for execution in the > > > order > > > >> specified by CmdSN." > > > >> > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > on > > > >> each command shipped except for commands marked for immediate > > > >> delivery." > > > >> but the meaning of the term "shipped" is vague, and does not > > > >> necessarily > > > >> require that the PDUs arrive on the other end of a TCP > > > connection > > > >> in the same order that the CmdSN values were assigned to these > > > PDUs. > > > >> > > > >> Some initiators have been designed to send commands out of CmdSN > > > >> order on one connection. Consider the situation where there is > > > only > > > >> one connection and a high-level dispatcher creates a PDU for a > > > SCSI > > > >> command that involves writing immediate data to the target. > > > This PDU > > > >> is enqueued to a lower-level layer which has to setup, start, > > > and > > > >> wait-for a DMA operation to move the immediate data into an > > > onboard > > > >> buffer before the PDU can be put onto the wire. While this is > > > >> happening, the dispatcher creates another unrelated PDU for a > > > SCSI > > > >> read command (for example), and when this PDU is passed to the > > > >> lower-level layer it can be sent immediately, ahead of the > > > previous > > > >> write PDU and therefore out of order on this connection. > > > >> > > > >> The standard clearly allows this to happen if the two PDUs were > > > sent > > > >> on different connections, and seems to imply that this can also > > > happen > > > >> when the two PDUs are sent on the same connection. > > > >> > > > >> The suggestion is to put in the standard an explicit statement > > > that > > > >> this is allowed or not allowed, as appropriate. > > > >> > > > >> If this is allowed, such a statement would avoid the erroneous > > > >> assumption being made by some target implementers that within a > > > single > > > >> connection, commands will arrive in order. > > > >> > > > >> If this is not allowed, such a statement would avoid the > > > erroneous > > > >> assumption being made by some initiator implementers that within > > > a > > > >> single connection, commands can be put on the wire out of order. > > > >> > > > >> +++ > > > >> > > > >> will add an explicit statement saying that this behaviour is > > > forbidden. > > > >> 2.2.2.1 will contain: > > > >> > > > >> On any given connection, the iSCSI initiator MUST send the > > > >commands in the > > > >> order specified by CmdSN. > > > >> > > > >> +++ > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > have to > > > >order commands across the session. I don't see why it's a problem to > > > extend > > > >that to the connection as well. I, for one, believe we should take > > > >a liberal > > > >stance on this. > > > > > > > >Dave Sheehy > > > > > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## > From owner-ips@ece.cmu.edu Wed Nov 7 00:18:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA12799 for ; Wed, 7 Nov 2001 00:18:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA74EiV17955 for ips-outgoing; Tue, 6 Nov 2001 23:14:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.windriver.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA74Egl17950 for ; Tue, 6 Nov 2001 23:14:42 -0500 (EST) Received: from london ([147.11.45.211]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id UAA13945; Tue, 6 Nov 2001 20:13:28 -0800 (PST) From: "Rod Harrison" To: "Santosh Rao" , Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Tue, 6 Nov 2001 20:16:26 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 In-Reply-To: <3BE8995C.892512E8@cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit It seems to me that if a target offers a command window greater than one it is buying into the complexity associated with supporting that window. There is very little difference at the target between buffering a command that arrives out of order and a buffering a write command that arrives without all the payload. In both cases other commands may arrive which cannot be committed to the SCSI layer. If a target doesn't want to be in the business of command queuing it has the option of offering a command window of one. Implementing a command queue is a much simpler proposition at the target than at the initiator. The target is in complete control of the command window and can therefore simply use a static array to hold command descriptors. The initiator has no such luxury since it has no a priori knowledge of the command window size, indeed the command window size can change dynamically. For the initiator to support ordered command queuing it must use an ordered list which can be expensive, especially when we consider the CPU power that will typically be available to an iSCSI HBA. Negotiating the command window size as part of login would make this more palatable for an initiator, but I suspect targets wouldn't want to commit to an unchanging command window. We've been debating the merits of an initiator sending out of order commands which is perhaps beyond the scope of where we should be. The cost to a target implementer is negligible and there is a potential benefit to an initiator implementer, so why should we prohibit this behaviour? - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Santosh Rao Sent: Tuesday, November 06, 2001 6:16 PM To: cbm@rose.hp.com Cc: Rod Harrison; Julian Satran; ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands Mallikarjun, Some comments below. Regards, Santosh "Mallikarjun C." wrote: > > Rod and Julian, > > This has been an interesting thread of discussion. Some > comments - > > 1.My first reaction was - allowing out-of-order command > transmission on the same connection deprives targets of > an implementation choice. Targets which support only > single-connection sessions and only support session > recovery (reasonable assumptions in my mind) can no > longer afford *not to* implement a command scoreboard. Even a single connection target *MUST* implement a scoreboard. The reason being that it can see out-of-order arrival of commands due to commands being dropped on digest errors. In such a case, it must block further command processing until holes are filled. Thus, there is no getting away from implementing a sequencer at the target. Given this, I think it is unreasonable to restrict initiator implementation flexibility by imposing a strict ordering requirement within the connection. > 2.Any end-node efficiency that is sought to be achieved > by transmitting CmdSNs out-of-order from the initiator > would be lost on the other end-node, since the target > now must wait for re-ordering the commands. It has to handle this situation anyway to deal with holes caused by digest errors. This scenario occurs even with initiators that issue commands in order. > > 3.The flipside is that out-of-order transmission saves > link badwidth (albeit at the expense of end-node efficiency), > compared to idling the link waiting for outbound DMA. > We have to determine if this is a reasonable trade-off. > > 4.I can see Rod's point that prefetching all immediate > data can be a burden on the NIC resources. But, two > questions - > - could the NIC not use unsolicited separate data > PDUs in these cases? [ I realize that InitialR2T > has to be "no" to let it happen... ] > - could the NIC have a memory architecture that > allows data prefetching for the next command (so > this is a non-issue from the protocol perspective)? > This scheme incurs one DMA delay for every new > burst of commands. > > 5.Another (perhaps radical at this point) option is to do > away with immediate unsolicited data, to stick only with > separate unsolicited data. I would personally be okay > with the choice, particularly if this feature (that > helps software implementations) starts making hardware > design complicated/expensive. > > So, to summarize - > > option immediate allow > data in spec? out-of-order? > > (A) (5) above no no > (B) No real reason to do this. no yes > (C) (4) above yes no > (D) pros & cons (1), (2) & (3) yes yes > > >From the arguments I heard so far, I am leaning towards > option A, and option C in that order. > > Comments? > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > Rod Harrison wrote: > > > > Julian, > > > > I don't understand what you are proposing here, what do you mean by > > "multiplexed" DMA? > > > > The problem is that the DMAs take some time, the more there are > > queued the longer the last DMAs queued take to complete. Some commands > > require DMAs to complete before they can be sent, i.e. Writes with > > immediate data, some commands do not, i.e. Reads and writes with no > > immediate data. The iSCSI HBA wants to be able to send commands as > > soon a possible, which for a read after a write can be before the > > write's DMA has completed. Maintaining an ordered queue for commands > > to be sent on the HBA is expensive and redundant since the target > > already knows how to queue commands before committing them to its SCSI > > layer. > > > > The iSCSI HBA and its host driver are not at liberty to change the > > order of commands from the OS, but the DMAs those commands need are > > unlikely to complete in the same order, and as I mentioned some > > commands need no DMA. If the HBA can't send commands out of CmdSN > > order it has to maintain an ordered queue of commands waiting to be > > sent, and potentially buffer a lot of data. For an HBA this makes > > immediate data almost impossible to support. > > > > I don't see the problem with allowing out of order commands given > > that the target already has to deal with very similar problems. I > > think we are getting in to the area of implementation choices here, > > which is inappropriate for a specification. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Monday, November 05, 2001 10:06 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > Rod, > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > commands. > > If you have scheduled a long outbound DMA you are doomed regardless of > > the > > command ordering. > > And if you have scheduled DMA operations piecemeal then you can insert > > your commands in correct order. > > > > Julo > > > > "Rod Harrison" > > 05-11-01 20:48 > > Please respond to "Rod Harrison" > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > cc: > > Subject: iSCSI: Out of order commands, was current UNH > > Plugfest > > > > [ Subject changed ] > > > > Julian, > > > > The ordering difference is introduced between the > > host > > side driver > > and the iSCSI HBA. The host side driver must present SCSI commands to > > the HBA in the order they are received from the OS to prevent read > > after write dependency failures. The HBA might reorder the commands > > depending on when DMA completes. The reordering can't be done ahead of > > time in the host driver since it doesn't know how long each DMA might > > take. As long as the HBA assigns CmdSN in the order it receives > > commands the desired host ordering is preserved. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Monday, November 05, 2001 12:35 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > Rod, > > > > I all examples give the point I find hard to understand is why is the > > ordering on the wire different from the presentation order to the > > initiator. You can get as many overlaps as you want by presenting the > > commands to the initiator in the desired order. > > What we are considering here is the case in which you want to ship in > > an > > order different than the one you present the commands. > > > > Julo > > > > "Rod Harrison" > > Sent by: owner-ips@ece.cmu.edu > > 04-11-01 04:42 > > Please respond to "Rod Harrison" > > > > To: "Barry Reinhold" , "Dave > > Sheehy" > > , "IETF IP SAN Reflector" > > > > cc: > > Subject: RE: iSCSI: current UNH Plugfest > > > > Barry, > > > > In general I agree but I don't think this is as much > > of a > > corner case > > as it at first appears. Targets will have code very similar to that > > needed to handle out of order commands to deal with digest errors. > > Targets also need to queue commands whilst waiting for both solicited > > and unsolicited data to arrive. Queuing out of order commands seems > > little extra work. > > > > From an initiators point of view there are > > efficiency, > > and probably > > performance gains to be had from sending commands out of order. Bob > > Russell gave the example of a read being sent whilst write data DMA is > > happening, and a similar situation can arise with DMA for writes > > overtaking that of earlier writes if the initiator has multiple DMA > > engines. In this case the initiator might be forced to let the wire go > > idle if it can't send the data from completed DMAs as soon as > > possible. > > > > We already have a command queue at the target to > > enforce > > correct > > serialisation of commands, doing the same thing at the initiator is > > redundant. > > > > Finally, I don't believe we should be writing a > > standard > > to work > > around poor coding and test coverage, especially at the cost of > > potential efficiency gains. > > > > I agree with Dave and Santosh that commands being > > sent > > out of order > > on a single session should be allowed by the standard. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Barry Reinhold > > Sent: Friday, November 02, 2001 5:24 PM > > To: Dave Sheehy; IETF IP SAN Reflector > > Subject: RE: iSCSI: current UNH Plugfest > > > > Using features such as out of order command delivery on a connection > > tend to > > be the sort of things that lead to interoperability problems. It is > > unexpected and probably going to hit poorly tested code paths even if > > the > > standard is written to allow it. > > > > >-----Original Message----- > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > >Dave Sheehy > > >Sent: Friday, November 02, 2001 4:19 PM > > >To: IETF IP SAN Reflector > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > >> > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > on > > >> page 25 of draft 8, which says: > > >> "Except for the commands marked for immediate delivery the > > iSCSI > > >> target layer MUST eliver the commands for execution in the > > order > > >> specified by CmdSN." > > >> > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > on > > >> each command shipped except for commands marked for immediate > > >> delivery." > > >> but the meaning of the term "shipped" is vague, and does not > > >> necessarily > > >> require that the PDUs arrive on the other end of a TCP > > connection > > >> in the same order that the CmdSN values were assigned to these > > PDUs. > > >> > > >> Some initiators have been designed to send commands out of CmdSN > > >> order on one connection. Consider the situation where there is > > only > > >> one connection and a high-level dispatcher creates a PDU for a > > SCSI > > >> command that involves writing immediate data to the target. > > This PDU > > >> is enqueued to a lower-level layer which has to setup, start, > > and > > >> wait-for a DMA operation to move the immediate data into an > > onboard > > >> buffer before the PDU can be put onto the wire. While this is > > >> happening, the dispatcher creates another unrelated PDU for a > > SCSI > > >> read command (for example), and when this PDU is passed to the > > >> lower-level layer it can be sent immediately, ahead of the > > previous > > >> write PDU and therefore out of order on this connection. > > >> > > >> The standard clearly allows this to happen if the two PDUs were > > sent > > >> on different connections, and seems to imply that this can also > > happen > > >> when the two PDUs are sent on the same connection. > > >> > > >> The suggestion is to put in the standard an explicit statement > > that > > >> this is allowed or not allowed, as appropriate. > > >> > > >> If this is allowed, such a statement would avoid the erroneous > > >> assumption being made by some target implementers that within a > > single > > >> connection, commands will arrive in order. > > >> > > >> If this is not allowed, such a statement would avoid the > > erroneous > > >> assumption being made by some initiator implementers that within > > a > > >> single connection, commands can be put on the wire out of order. > > >> > > >> +++ > > >> > > >> will add an explicit statement saying that this behaviour is > > forbidden. > > >> 2.2.2.1 will contain: > > >> > > >> On any given connection, the iSCSI initiator MUST send the > > >commands in the > > >> order specified by CmdSN. > > >> > > >> +++ > > > > > >Why do you feel this behavior should be forbidden? Targets already > > have to > > >order commands across the session. I don't see why it's a problem to > > extend > > >that to the connection as well. I, for one, believe we should take > > >a liberal > > >stance on this. > > > > > >Dave Sheehy > > > -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Wed Nov 7 01:48:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA18339 for ; Wed, 7 Nov 2001 01:48:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7600Q25214 for ips-outgoing; Wed, 7 Nov 2001 01:00:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA75xwl25210 for ; Wed, 7 Nov 2001 00:59:58 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id GAA41692 for ; Wed, 7 Nov 2001 06:59:52 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA75xp154876 for ; Wed, 7 Nov 2001 06:59:51 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 7 Nov 2001 07:59:48 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 07/11/2001 07:59:50, Serialize complete at 07/11/2001 07:59:50 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Rod, I stated repeatedly that this causes deadlock and adds nothing in terms of performance. It is explicitly stated for unsolicited data and it was just an overlook that it was never made explicit for commands. Julo "Rod Harrison" 07-11-01 06:16 Please respond to "Rod Harrison" To: "Santosh Rao" , cc: Julian Satran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: Out of order commands It seems to me that if a target offers a command window greater than one it is buying into the complexity associated with supporting that window. There is very little difference at the target between buffering a command that arrives out of order and a buffering a write command that arrives without all the payload. In both cases other commands may arrive which cannot be committed to the SCSI layer. If a target doesn't want to be in the business of command queuing it has the option of offering a command window of one. Implementing a command queue is a much simpler proposition at the target than at the initiator. The target is in complete control of the command window and can therefore simply use a static array to hold command descriptors. The initiator has no such luxury since it has no a priori knowledge of the command window size, indeed the command window size can change dynamically. For the initiator to support ordered command queuing it must use an ordered list which can be expensive, especially when we consider the CPU power that will typically be available to an iSCSI HBA. Negotiating the command window size as part of login would make this more palatable for an initiator, but I suspect targets wouldn't want to commit to an unchanging command window. We've been debating the merits of an initiator sending out of order commands which is perhaps beyond the scope of where we should be. The cost to a target implementer is negligible and there is a potential benefit to an initiator implementer, so why should we prohibit this behaviour? - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Santosh Rao Sent: Tuesday, November 06, 2001 6:16 PM To: cbm@rose.hp.com Cc: Rod Harrison; Julian Satran; ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands Mallikarjun, Some comments below. Regards, Santosh "Mallikarjun C." wrote: > > Rod and Julian, > > This has been an interesting thread of discussion. Some > comments - > > 1.My first reaction was - allowing out-of-order command > transmission on the same connection deprives targets of > an implementation choice. Targets which support only > single-connection sessions and only support session > recovery (reasonable assumptions in my mind) can no > longer afford *not to* implement a command scoreboard. Even a single connection target *MUST* implement a scoreboard. The reason being that it can see out-of-order arrival of commands due to commands being dropped on digest errors. In such a case, it must block further command processing until holes are filled. Thus, there is no getting away from implementing a sequencer at the target. Given this, I think it is unreasonable to restrict initiator implementation flexibility by imposing a strict ordering requirement within the connection. > 2.Any end-node efficiency that is sought to be achieved > by transmitting CmdSNs out-of-order from the initiator > would be lost on the other end-node, since the target > now must wait for re-ordering the commands. It has to handle this situation anyway to deal with holes caused by digest errors. This scenario occurs even with initiators that issue commands in order. > > 3.The flipside is that out-of-order transmission saves > link badwidth (albeit at the expense of end-node efficiency), > compared to idling the link waiting for outbound DMA. > We have to determine if this is a reasonable trade-off. > > 4.I can see Rod's point that prefetching all immediate > data can be a burden on the NIC resources. But, two > questions - > - could the NIC not use unsolicited separate data > PDUs in these cases? [ I realize that InitialR2T > has to be "no" to let it happen... ] > - could the NIC have a memory architecture that > allows data prefetching for the next command (so > this is a non-issue from the protocol perspective)? > This scheme incurs one DMA delay for every new > burst of commands. > > 5.Another (perhaps radical at this point) option is to do > away with immediate unsolicited data, to stick only with > separate unsolicited data. I would personally be okay > with the choice, particularly if this feature (that > helps software implementations) starts making hardware > design complicated/expensive. > > So, to summarize - > > option immediate allow > data in spec? out-of-order? > > (A) (5) above no no > (B) No real reason to do this. no yes > (C) (4) above yes no > (D) pros & cons (1), (2) & (3) yes yes > > >From the arguments I heard so far, I am leaning towards > option A, and option C in that order. > > Comments? > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > Rod Harrison wrote: > > > > Julian, > > > > I don't understand what you are proposing here, what do you mean by > > "multiplexed" DMA? > > > > The problem is that the DMAs take some time, the more there are > > queued the longer the last DMAs queued take to complete. Some commands > > require DMAs to complete before they can be sent, i.e. Writes with > > immediate data, some commands do not, i.e. Reads and writes with no > > immediate data. The iSCSI HBA wants to be able to send commands as > > soon a possible, which for a read after a write can be before the > > write's DMA has completed. Maintaining an ordered queue for commands > > to be sent on the HBA is expensive and redundant since the target > > already knows how to queue commands before committing them to its SCSI > > layer. > > > > The iSCSI HBA and its host driver are not at liberty to change the > > order of commands from the OS, but the DMAs those commands need are > > unlikely to complete in the same order, and as I mentioned some > > commands need no DMA. If the HBA can't send commands out of CmdSN > > order it has to maintain an ordered queue of commands waiting to be > > sent, and potentially buffer a lot of data. For an HBA this makes > > immediate data almost impossible to support. > > > > I don't see the problem with allowing out of order commands given > > that the target already has to deal with very similar problems. I > > think we are getting in to the area of implementation choices here, > > which is inappropriate for a specification. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Monday, November 05, 2001 10:06 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > Rod, > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > commands. > > If you have scheduled a long outbound DMA you are doomed regardless of > > the > > command ordering. > > And if you have scheduled DMA operations piecemeal then you can insert > > your commands in correct order. > > > > Julo > > > > "Rod Harrison" > > 05-11-01 20:48 > > Please respond to "Rod Harrison" > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > cc: > > Subject: iSCSI: Out of order commands, was current UNH > > Plugfest > > > > [ Subject changed ] > > > > Julian, > > > > The ordering difference is introduced between the > > host > > side driver > > and the iSCSI HBA. The host side driver must present SCSI commands to > > the HBA in the order they are received from the OS to prevent read > > after write dependency failures. The HBA might reorder the commands > > depending on when DMA completes. The reordering can't be done ahead of > > time in the host driver since it doesn't know how long each DMA might > > take. As long as the HBA assigns CmdSN in the order it receives > > commands the desired host ordering is preserved. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Monday, November 05, 2001 12:35 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: current UNH Plugfest > > > > Rod, > > > > I all examples give the point I find hard to understand is why is the > > ordering on the wire different from the presentation order to the > > initiator. You can get as many overlaps as you want by presenting the > > commands to the initiator in the desired order. > > What we are considering here is the case in which you want to ship in > > an > > order different than the one you present the commands. > > > > Julo > > > > "Rod Harrison" > > Sent by: owner-ips@ece.cmu.edu > > 04-11-01 04:42 > > Please respond to "Rod Harrison" > > > > To: "Barry Reinhold" , "Dave > > Sheehy" > > , "IETF IP SAN Reflector" > > > > cc: > > Subject: RE: iSCSI: current UNH Plugfest > > > > Barry, > > > > In general I agree but I don't think this is as much > > of a > > corner case > > as it at first appears. Targets will have code very similar to that > > needed to handle out of order commands to deal with digest errors. > > Targets also need to queue commands whilst waiting for both solicited > > and unsolicited data to arrive. Queuing out of order commands seems > > little extra work. > > > > From an initiators point of view there are > > efficiency, > > and probably > > performance gains to be had from sending commands out of order. Bob > > Russell gave the example of a read being sent whilst write data DMA is > > happening, and a similar situation can arise with DMA for writes > > overtaking that of earlier writes if the initiator has multiple DMA > > engines. In this case the initiator might be forced to let the wire go > > idle if it can't send the data from completed DMAs as soon as > > possible. > > > > We already have a command queue at the target to > > enforce > > correct > > serialisation of commands, doing the same thing at the initiator is > > redundant. > > > > Finally, I don't believe we should be writing a > > standard > > to work > > around poor coding and test coverage, especially at the cost of > > potential efficiency gains. > > > > I agree with Dave and Santosh that commands being > > sent > > out of order > > on a single session should be allowed by the standard. > > > > - Rod > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Barry Reinhold > > Sent: Friday, November 02, 2001 5:24 PM > > To: Dave Sheehy; IETF IP SAN Reflector > > Subject: RE: iSCSI: current UNH Plugfest > > > > Using features such as out of order command delivery on a connection > > tend to > > be the sort of things that lead to interoperability problems. It is > > unexpected and probably going to hit poorly tested code paths even if > > the > > standard is written to allow it. > > > > >-----Original Message----- > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > >Dave Sheehy > > >Sent: Friday, November 02, 2001 4:19 PM > > >To: IETF IP SAN Reflector > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > >> > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > on > > >> page 25 of draft 8, which says: > > >> "Except for the commands marked for immediate delivery the > > iSCSI > > >> target layer MUST eliver the commands for execution in the > > order > > >> specified by CmdSN." > > >> > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > on > > >> each command shipped except for commands marked for immediate > > >> delivery." > > >> but the meaning of the term "shipped" is vague, and does not > > >> necessarily > > >> require that the PDUs arrive on the other end of a TCP > > connection > > >> in the same order that the CmdSN values were assigned to these > > PDUs. > > >> > > >> Some initiators have been designed to send commands out of CmdSN > > >> order on one connection. Consider the situation where there is > > only > > >> one connection and a high-level dispatcher creates a PDU for a > > SCSI > > >> command that involves writing immediate data to the target. > > This PDU > > >> is enqueued to a lower-level layer which has to setup, start, > > and > > >> wait-for a DMA operation to move the immediate data into an > > onboard > > >> buffer before the PDU can be put onto the wire. While this is > > >> happening, the dispatcher creates another unrelated PDU for a > > SCSI > > >> read command (for example), and when this PDU is passed to the > > >> lower-level layer it can be sent immediately, ahead of the > > previous > > >> write PDU and therefore out of order on this connection. > > >> > > >> The standard clearly allows this to happen if the two PDUs were > > sent > > >> on different connections, and seems to imply that this can also > > happen > > >> when the two PDUs are sent on the same connection. > > >> > > >> The suggestion is to put in the standard an explicit statement > > that > > >> this is allowed or not allowed, as appropriate. > > >> > > >> If this is allowed, such a statement would avoid the erroneous > > >> assumption being made by some target implementers that within a > > single > > >> connection, commands will arrive in order. > > >> > > >> If this is not allowed, such a statement would avoid the > > erroneous > > >> assumption being made by some initiator implementers that within > > a > > >> single connection, commands can be put on the wire out of order. > > >> > > >> +++ > > >> > > >> will add an explicit statement saying that this behaviour is > > forbidden. > > >> 2.2.2.1 will contain: > > >> > > >> On any given connection, the iSCSI initiator MUST send the > > >commands in the > > >> order specified by CmdSN. > > >> > > >> +++ > > > > > >Why do you feel this behavior should be forbidden? Targets already > > have to > > >order commands across the session. I don't see why it's a problem to > > extend > > >that to the connection as well. I, for one, believe we should take > > >a liberal > > >stance on this. > > > > > >Dave Sheehy > > > -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Wed Nov 7 01:49:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA18384 for ; Wed, 7 Nov 2001 01:49:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA76YIs27246 for ips-outgoing; Wed, 7 Nov 2001 01:34:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from intens2.pdcint ([194.90.167.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA76YGl27241 for ; Wed, 7 Nov 2001 01:34:17 -0500 (EST) Received: by INTENS2 with Internet Mail Service (5.5.2650.21) id ; Wed, 7 Nov 2001 08:36:19 +0200 Message-ID: <8B578C8302B3D411811200D0B7B64CD1149120@INTENS2> From: Ron Grinfeld To: "'ips@ece.cmu.edu'" Subject: iSCSI acknowledgment mechanism Date: Wed, 7 Nov 2001 08:36:15 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="windows-1255" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello, After monitoring the Q&A resulting from the UNH I'm a little puzzled and in need for some clarifications: 1. Does the ExpStatSn acknowledges the status received on a task and all its data PDUs (not only the receive of the status itself) ? If so then the ExpStatSn is incremented only after receiving all the Data-In PDUs and the related status. 2. Does the 'A' bit in the Data-In PDUs relates only to data sent on a specific task (triggered by the target when sending long bursts of Data-In PDUs relating to the same task) and the SNACK response with type DataACK (positive acknowledge of Data-In PDUs) acknowledges this specific task as well ? _________________________ Ron Grinfeld, Siliquent rong@siliquent.com Israel office: tel: +972 3 6948671 e-fax +972 3 6953736 USA office: 2929 Campus Drive, suite #400 San Mateo, CA 94403 tel: 650-358-2854 From owner-ips@ece.cmu.edu Wed Nov 7 01:49:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA18399 for ; Wed, 7 Nov 2001 01:49:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA76H1W26224 for ips-outgoing; Wed, 7 Nov 2001 01:17:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA76H0l26218 for ; Wed, 7 Nov 2001 01:17:00 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA56092 for ; Wed, 7 Nov 2001 07:16:54 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA76GrW25386 for ; Wed, 7 Nov 2001 07:16:53 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI over TLS X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 7 Nov 2001 08:16:50 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 07/11/2001 08:16:53, Serialize complete at 07/11/2001 08:16:53 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, A group of us seriously considered TLS. The main reason for dropping it was that it would interfere with any mechanism we could think of doing framing and steering and we thought that framing and steering are essential at 10Gbps and over. Julo "Peter Mellquist" Sent by: owner-ips@ece.cmu.edu 07-11-01 02:15 Please respond to "Peter Mellquist" To: cc: Subject: iSCSI over TLS I am aware that the ips group is leaning toward IPSEC as for the security solution but I am interested if anyone is also considering using Transport Layer Security (TLS)? I am concerned that the requirement for IPSEC might make TOEs more complex than they need to be. Can TLS be optionally used as well as defined by the specification? This could allow TOE vendors to only be concerned with providing normal IPv4 / ipv6 and leave the security to a higher layer. A TLS stack sitting above the TOE could then handle security very well. Also, I anticipate that the first generation of TOEs will not support IPSEC. With a iSCSI/TLS we could enable security solutions with the first generation of TOEs and get speed and security. Are any TOE vendors planning to support IPSEC? Can TLS or IPSEC be supported? -peter Peter Mellquist Seven Systems Technologies 575 Menlo Drive Suite 2 Rocklin CA 916-577-1275 peterm@seven-systems.com From owner-ips@ece.cmu.edu Wed Nov 7 01:57:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA19414 for ; Wed, 7 Nov 2001 01:57:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA76EQ026083 for ips-outgoing; Wed, 7 Nov 2001 01:14:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from host32.cedant.com (host32.cedant.com [209.239.32.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA76EOl26079 for ; Wed, 7 Nov 2001 01:14:24 -0500 (EST) Received: from sahara (153.sanjose-11-12rs16rt.ca.dial-access.att.net [12.81.4.153]) by host32.cedant.com (8.10.2/8.10.2) with ESMTP id fA76ELD03227; Wed, 7 Nov 2001 01:14:21 -0500 Message-ID: <200111062227010305.0AE8F00C@opulentsystems.com> In-Reply-To: <020001c16721$536f4f70$1001010a@blekinge> References: <020001c16721$536f4f70$1001010a@blekinge> X-Mailer: Calypso Version 3.30.00.00 (4) Date: Tue, 06 Nov 2001 22:27:01 -0800 Reply-To: sganguly@opulentsystems.com From: "Sukanta Ganguly" To: "Peter Mellquist" , "IPS" Subject: Re: iSCSI over TLS Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA76EPl26080 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Peter, A very good point. I am not sure if the TOE vendors have plans of implementing IPSec and/or TLS. But allowing TLS as another mechanism is also going to increase the complexity on the TOE side. The more logic that is applied to the TOE the more expensive and difficult it is going to get. The TOE vendors take over the packet processing at layer 4 and hence is already fairly restrictive scale-wise. Adding TLS will make it more difficult. However, a good mix of TLS on software and a synergistic TOE can make a good combination. Hence I like the idea. I am not sure if any TOE vendors have any comment of this ??? SG *********** REPLY SEPARATOR *********** On 11/6/2001 at 4:15 PM Peter Mellquist wrote: >I am aware that the ips group is leaning toward IPSEC as for the security >solution but I am interested if anyone is also considering using Transport >Layer Security (TLS)? > >I am concerned that the requirement for IPSEC might make TOEs more complex >than they need to be. Can TLS be optionally used as well as defined by the >specification? This could allow TOE vendors to only be concerned with >providing normal IPv4 / ipv6 and leave the security to a higher layer. A >TLS >stack sitting above the TOE could then handle security very well. Also, I >anticipate that the first generation of TOEs will not support IPSEC. With a >iSCSI/TLS we could enable security solutions with the first generation of >TOEs and get speed and security. > >Are any TOE vendors planning to support IPSEC? > >Can TLS or IPSEC be supported? > >-peter > > > >Peter Mellquist >Seven Systems Technologies >575 Menlo Drive Suite 2 >Rocklin CA >916-577-1275 >peterm@seven-systems.com From owner-ips@ece.cmu.edu Wed Nov 7 02:37:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA28897 for ; Wed, 7 Nov 2001 02:37:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA76iV827877 for ips-outgoing; Wed, 7 Nov 2001 01:44:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from host32.cedant.com (host32.cedant.com [209.239.32.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA76iTl27872 for ; Wed, 7 Nov 2001 01:44:29 -0500 (EST) Received: from sahara (153.sanjose-11-12rs16rt.ca.dial-access.att.net [12.81.4.153]) by host32.cedant.com (8.10.2/8.10.2) with ESMTP id fA763pD27311; Wed, 7 Nov 2001 01:03:54 -0500 Message-ID: <200111062217090314.0ADFE795@opulentsystems.com> In-Reply-To: References: X-Mailer: Calypso Version 3.30.00.00 (4) Date: Tue, 06 Nov 2001 22:17:09 -0800 Reply-To: sganguly@opulentsystems.com From: "Sukanta Ganguly" To: "Rod Harrison" , "Santosh Rao" , cbm@rose.hp.com Cc: "Julian Satran" , "IPS" Subject: RE: iSCSI: Out of order commands Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA76iTl27873 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Rod, I agree with the philosophy of letting the target have the capability to decide on the command window. The initiator just follows suit. Stating that the initiator will not generate holes on its own. The holes will only occur due to loss of packets in which case TCP retransmission will handle the resends and hence the initiaTOR is not even aware of the holes being generated during the transport mechanism. Even if the initiator decides to handle multiple command windows, the target will still have to do it. So loading the initiator with extra work does not buy any benefits. SG *********** REPLY SEPARATOR *********** On 11/6/2001 at 8:16 PM Rod Harrison wrote: >It seems to me that if a target offers a command window greater than >one it is buying into the complexity associated with supporting that >window. > > There is very little difference at the target between buffering a >command that arrives out of order and a buffering a write command that >arrives without all the payload. In both cases other commands may >arrive which cannot be committed to the SCSI layer. If a target >doesn't want to be in the business of command queuing it has the >option of offering a command window of one. > > Implementing a command queue is a much simpler proposition at the >target than at the initiator. The target is in complete control of the >command window and can therefore simply use a static array to hold >command descriptors. The initiator has no such luxury since it has no >a priori knowledge of the command window size, indeed the command >window size can change dynamically. For the initiator to support >ordered command queuing it must use an ordered list which can be >expensive, especially when we consider the CPU power that will >typically be available to an iSCSI HBA. Negotiating the command window >size as part of login would make this more palatable for an initiator, >but I suspect targets wouldn't want to commit to an unchanging command >window. > > We've been debating the merits of an initiator sending out of order >commands which is perhaps beyond the scope of where we should be. The >cost to a target implementer is negligible and there is a potential >benefit to an initiator implementer, so why should we prohibit this >behaviour? > > - Rod > >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Santosh Rao >Sent: Tuesday, November 06, 2001 6:16 PM >To: cbm@rose.hp.com >Cc: Rod Harrison; Julian Satran; ips@ece.cmu.edu >Subject: Re: iSCSI: Out of order commands > > >Mallikarjun, > >Some comments below. > >Regards, >Santosh > >"Mallikarjun C." wrote: >> >> Rod and Julian, >> >> This has been an interesting thread of discussion. Some >> comments - >> >> 1.My first reaction was - allowing out-of-order command >> transmission on the same connection deprives targets of >> an implementation choice. Targets which support only >> single-connection sessions and only support session >> recovery (reasonable assumptions in my mind) can no >> longer afford *not to* implement a command scoreboard. > >Even a single connection target *MUST* implement a scoreboard. The >reason being that it can see out-of-order arrival of commands due to >commands being dropped on digest errors. In such a case, it must block >further command processing until holes are filled. > >Thus, there is no getting away from implementing a sequencer at the >target. Given this, I think it is unreasonable to restrict initiator >implementation flexibility by imposing a strict ordering requirement >within the connection. > > > >> 2.Any end-node efficiency that is sought to be achieved >> by transmitting CmdSNs out-of-order from the initiator >> would be lost on the other end-node, since the target >> now must wait for re-ordering the commands. > >It has to handle this situation anyway to deal with holes caused by >digest errors. This scenario occurs even with initiators that issue >commands in order. > > >> >> 3.The flipside is that out-of-order transmission saves >> link badwidth (albeit at the expense of end-node efficiency), >> compared to idling the link waiting for outbound DMA. >> We have to determine if this is a reasonable trade-off. >> >> 4.I can see Rod's point that prefetching all immediate >> data can be a burden on the NIC resources. But, two >> questions - >> - could the NIC not use unsolicited separate data >> PDUs in these cases? [ I realize that InitialR2T >> has to be "no" to let it happen... ] >> - could the NIC have a memory architecture that >> allows data prefetching for the next command (so >> this is a non-issue from the protocol perspective)? >> This scheme incurs one DMA delay for every new >> burst of commands. >> >> 5.Another (perhaps radical at this point) option is to do >> away with immediate unsolicited data, to stick only with >> separate unsolicited data. I would personally be okay >> with the choice, particularly if this feature (that >> helps software implementations) starts making hardware >> design complicated/expensive. >> >> So, to summarize - >> >> option immediate allow >> data in spec? out-of-order? >> >> (A) (5) above no no >> (B) No real reason to do this. no yes >> (C) (4) above yes no >> (D) pros & cons (1), (2) & (3) yes yes >> >> >From the arguments I heard so far, I am leaning towards >> option A, and option C in that order. >> >> Comments? >> -- >> Mallikarjun >> >> Mallikarjun Chadalapaka >> Networked Storage Architecture >> Network Storage Solutions Organization >> MS 5668 Hewlett-Packard, Roseville. >> cbm@rose.hp.com >> >> Rod Harrison wrote: >> > >> > Julian, >> > >> > I don't understand what you are proposing here, what do >you mean by >> > "multiplexed" DMA? >> > >> > The problem is that the DMAs take some time, the more >there are >> > queued the longer the last DMAs queued take to complete. Some >commands >> > require DMAs to complete before they can be sent, i.e. Writes with >> > immediate data, some commands do not, i.e. Reads and writes with >no >> > immediate data. The iSCSI HBA wants to be able to send commands as >> > soon a possible, which for a read after a write can be before the >> > write's DMA has completed. Maintaining an ordered queue for >commands >> > to be sent on the HBA is expensive and redundant since the target >> > already knows how to queue commands before committing them to its >SCSI >> > layer. >> > >> > The iSCSI HBA and its host driver are not at liberty to >change the >> > order of commands from the OS, but the DMAs those commands need >are >> > unlikely to complete in the same order, and as I mentioned some >> > commands need no DMA. If the HBA can't send commands out of CmdSN >> > order it has to maintain an ordered queue of commands waiting to >be >> > sent, and potentially buffer a lot of data. For an HBA this makes >> > immediate data almost impossible to support. >> > >> > I don't see the problem with allowing out of order >commands given >> > that the target already has to deal with very similar problems. I >> > think we are getting in to the area of implementation choices >here, >> > which is inappropriate for a specification. >> > >> > - Rod >> > >> > -----Original Message----- >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On >Behalf Of >> > Julian Satran >> > Sent: Monday, November 05, 2001 10:06 PM >> > To: ips@ece.cmu.edu >> > Subject: Re: iSCSI: Out of order commands, was current UNH >Plugfest >> > >> > Rod, >> > >> > I don't see any reason why DMA operations cant be "multiplexed" >with >> > commands. >> > If you have scheduled a long outbound DMA you are doomed >regardless of >> > the >> > command ordering. >> > And if you have scheduled DMA operations piecemeal then you can >insert >> > your commands in correct order. >> > >> > Julo >> > >> > "Rod Harrison" >> > 05-11-01 20:48 >> > Please respond to "Rod Harrison" >> > >> > To: Julian Satran/Haifa/IBM@IBMIL, >> > cc: >> > Subject: iSCSI: Out of order commands, was current >UNH >> > Plugfest >> > >> > [ Subject changed ] >> > >> > Julian, >> > >> > The ordering difference is introduced between the >> > host >> > side driver >> > and the iSCSI HBA. The host side driver must present SCSI commands >to >> > the HBA in the order they are received from the OS to prevent read >> > after write dependency failures. The HBA might reorder the >commands >> > depending on when DMA completes. The reordering can't be done >ahead of >> > time in the host driver since it doesn't know how long each DMA >might >> > take. As long as the HBA assigns CmdSN in the order it receives >> > commands the desired host ordering is preserved. >> > >> > - Rod >> > >> > -----Original Message----- >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On >Behalf Of >> > Julian Satran >> > Sent: Monday, November 05, 2001 12:35 AM >> > To: ips@ece.cmu.edu >> > Subject: RE: iSCSI: current UNH Plugfest >> > >> > Rod, >> > >> > I all examples give the point I find hard to understand is why is >the >> > ordering on the wire different from the presentation order to the >> > initiator. You can get as many overlaps as you want by presenting >the >> > commands to the initiator in the desired order. >> > What we are considering here is the case in which you want to ship >in >> > an >> > order different than the one you present the commands. >> > >> > Julo >> > >> > "Rod Harrison" >> > Sent by: owner-ips@ece.cmu.edu >> > 04-11-01 04:42 >> > Please respond to "Rod Harrison" >> > >> > To: "Barry Reinhold" , "Dave >> > Sheehy" >> > , "IETF IP SAN Reflector" >> > >> > cc: >> > Subject: RE: iSCSI: current UNH Plugfest >> > >> > Barry, >> > >> > In general I agree but I don't think this is as >much >> > of a >> > corner case >> > as it at first appears. Targets will have code very similar to >that >> > needed to handle out of order commands to deal with digest errors. >> > Targets also need to queue commands whilst waiting for both >solicited >> > and unsolicited data to arrive. Queuing out of order commands >seems >> > little extra work. >> > >> > From an initiators point of view there are >> > efficiency, >> > and probably >> > performance gains to be had from sending commands out of order. >Bob >> > Russell gave the example of a read being sent whilst write data >DMA is >> > happening, and a similar situation can arise with DMA for writes >> > overtaking that of earlier writes if the initiator has multiple >DMA >> > engines. In this case the initiator might be forced to let the >wire go >> > idle if it can't send the data from completed DMAs as soon as >> > possible. >> > >> > We already have a command queue at the target to >> > enforce >> > correct >> > serialisation of commands, doing the same thing at the initiator >is >> > redundant. >> > >> > Finally, I don't believe we should be writing a >> > standard >> > to work >> > around poor coding and test coverage, especially at the cost of >> > potential efficiency gains. >> > >> > I agree with Dave and Santosh that commands being >> > sent >> > out of order >> > on a single session should be allowed by the standard. >> > >> > - Rod >> > >> > -----Original Message----- >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On >Behalf Of >> > Barry Reinhold >> > Sent: Friday, November 02, 2001 5:24 PM >> > To: Dave Sheehy; IETF IP SAN Reflector >> > Subject: RE: iSCSI: current UNH Plugfest >> > >> > Using features such as out of order command delivery on a >connection >> > tend to >> > be the sort of things that lead to interoperability problems. It >is >> > unexpected and probably going to hit poorly tested code paths even >if >> > the >> > standard is written to allow it. >> > >> > >-----Original Message----- >> > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On >Behalf >> > Of >> > >Dave Sheehy >> > >Sent: Friday, November 02, 2001 4:19 PM >> > >To: IETF IP SAN Reflector >> > >Subject: Re: iSCSI: current UNH Plugfest >> > > >> > > >> > > >> > >> 3. Can commands be sent out of order on the same connection? >> > >> >> > >> The behavior of targets is clearly specified in Section >2.2.2.3 >> > on >> > >> page 25 of draft 8, which says: >> > >> "Except for the commands marked for immediate delivery the >> > iSCSI >> > >> target layer MUST eliver the commands for execution in the >> > order >> > >> specified by CmdSN." >> > >> >> > >> Section 2.2.2.3 on page 26 of draft 8 also says: >> > >> "- CmdSN - the current command Sequence Number advanced by >1 >> > on >> > >> each command shipped except for commands marked for >immediate >> > >> delivery." >> > >> but the meaning of the term "shipped" is vague, and does not >> > >> necessarily >> > >> require that the PDUs arrive on the other end of a TCP >> > connection >> > >> in the same order that the CmdSN values were assigned to >these >> > PDUs. >> > >> >> > >> Some initiators have been designed to send commands out of >CmdSN >> > >> order on one connection. Consider the situation where there >is >> > only >> > >> one connection and a high-level dispatcher creates a PDU for >a >> > SCSI >> > >> command that involves writing immediate data to the target. >> > This PDU >> > >> is enqueued to a lower-level layer which has to setup, >start, >> > and >> > >> wait-for a DMA operation to move the immediate data into an >> > onboard >> > >> buffer before the PDU can be put onto the wire. While this >is >> > >> happening, the dispatcher creates another unrelated PDU for >a >> > SCSI >> > >> read command (for example), and when this PDU is passed to >the >> > >> lower-level layer it can be sent immediately, ahead of the >> > previous >> > >> write PDU and therefore out of order on this connection. >> > >> >> > >> The standard clearly allows this to happen if the two PDUs >were >> > sent >> > >> on different connections, and seems to imply that this can >also >> > happen >> > >> when the two PDUs are sent on the same connection. >> > >> >> > >> The suggestion is to put in the standard an explicit >statement >> > that >> > >> this is allowed or not allowed, as appropriate. >> > >> >> > >> If this is allowed, such a statement would avoid the >erroneous >> > >> assumption being made by some target implementers that >within a >> > single >> > >> connection, commands will arrive in order. >> > >> >> > >> If this is not allowed, such a statement would avoid the >> > erroneous >> > >> assumption being made by some initiator implementers that >within >> > a >> > >> single connection, commands can be put on the wire out of >order. >> > >> >> > >> +++ >> > >> >> > >> will add an explicit statement saying that this behaviour is >> > forbidden. >> > >> 2.2.2.1 will contain: >> > >> >> > >> On any given connection, the iSCSI initiator MUST send the >> > >commands in the >> > >> order specified by CmdSN. >> > >> >> > >> +++ >> > > >> > >Why do you feel this behavior should be forbidden? Targets >already >> > have to >> > >order commands across the session. I don't see why it's a problem >to >> > extend >> > >that to the connection as well. I, for one, believe we should >take >> > >a liberal >> > >stance on this. >> > > >> > >Dave Sheehy >> > > > >-- >################################## >Santosh Rao >Software Design Engineer, >HP-UX iSCSI Driver Team, >Hewlett Packard, Cupertino. >email : santoshr@cup.hp.com >Phone : 408-447-3751 >################################## From owner-ips@ece.cmu.edu Wed Nov 7 03:43:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA29476 for ; Wed, 7 Nov 2001 03:43:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA77sGU01597 for ips-outgoing; Wed, 7 Nov 2001 02:54:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA77sFl01592 for ; Wed, 7 Nov 2001 02:54:15 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA86004 for ; Wed, 7 Nov 2001 08:54:08 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA77s8123322 for ; Wed, 7 Nov 2001 08:54:08 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI acknowledgment mechanism X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 7 Nov 2001 09:54:01 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 07/11/2001 09:54:08, Serialize complete at 07/11/2001 09:54:08 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ron, ExpStaSN acknowledges status. This is per connection and since status and data for a task have allegiance to one connection acknowledging status implicitly acknowledges data. Data are counted per command - and A refers only to a given command - and so does the dataACK. Julo Ron Grinfeld Sent by: owner-ips@ece.cmu.edu 07-11-01 08:36 Please respond to Ron Grinfeld To: "'ips@ece.cmu.edu'" cc: Subject: iSCSI acknowledgment mechanism Hello, After monitoring the Q&A resulting from the UNH I'm a little puzzled and in need for some clarifications: 1. Does the ExpStatSn acknowledges the status received on a task and all its data PDUs (not only the receive of the status itself) ? If so then the ExpStatSn is incremented only after receiving all the Data-In PDUs and the related status. 2. Does the 'A' bit in the Data-In PDUs relates only to data sent on a specific task (triggered by the target when sending long bursts of Data-In PDUs relating to the same task) and the SNACK response with type DataACK (positive acknowledge of Data-In PDUs) acknowledges this specific task as well ? _________________________ Ron Grinfeld, Siliquent rong@siliquent.com Israel office: tel: +972 3 6948671 e-fax +972 3 6953736 USA office: 2929 Campus Drive, suite #400 San Mateo, CA 94403 tel: 650-358-2854 From owner-ips@ece.cmu.edu Wed Nov 7 05:33:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA00456 for ; Wed, 7 Nov 2001 05:33:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA79deX17837 for ips-outgoing; Wed, 7 Nov 2001 04:39:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sansrv1.san-rad.co.il ([212.199.104.228]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA79dHl17821 for ; Wed, 7 Nov 2001 04:39:18 -0500 (EST) Subject: RE: iSCSI: New iSCSI MIB draft MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Date: Wed, 7 Nov 2001 11:40:46 +0200 Disposition-Notification-To: "Michele Hallak - Stamler" Message-ID: <838D8D2617300146B7F47E4D9AE7FF10114A15@SANSRV1.SAN-RAD.CO.IL> content-class: urn:content-classes:message X-MimeOLE: Produced By Microsoft Exchange V6.0.4417.0 Thread-Topic: iSCSI: New iSCSI MIB draft Thread-Index: AcFmz7d6uQBQNUOKSHC/8ncmaYJlDwAmtmnA From: "Michele Hallak - Stamler" To: "Mark Bakke" Cc: "IPS" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA79dMl17827 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit My answers in-line also: -----Original Message----- From: Mark Bakke [mailto:mbakke@cisco.com] Sent: Tuesday, November 06, 2001 4:19 PM To: Michele Hallak - Stamler Cc: IPS Subject: Re: iSCSI: New iSCSI MIB draft Michele- Answers inline. Basically, we have discussed making parts of the MIB writable, and need to start working on it some more. Please keep posting anything that you think might be useful as a writable attribute or a creatable/deletable row. -- Mark Michele Hallak - Stamler wrote: > > Hi, > 1.I see that the target access list is still read-only. How do you think > that it should be configured? I had originally assumed that these would be configured by other means than SNMP, but it is certainly something that makes sense to configure. I will post another message to start a thread on this. Michele>> I think also that we should add an additional field describing the kind of access (read-only / read-write) I think also that it will be a good idea to have and enable/disable field as you and Keith suggested. > 2. Who should configure the aliases of the iscsi targets and initiators? These could easily be writable as well. > 3. Same question for the portals. Have to think about this one. Any ideas? Michele >> For the portals, I have to tell that I have some problems: 1. it seems correct to allow an administrator to define iSCSI portals from the iSCSI MIB. 2. in another way, a portal has some relation to a specific interface (IP Address/MAC Address) and for that relation we need the IP-MIB (if I'm not wrong) 3. so it should be specified clearly: can we define "new ip addresses" via this MIB or only use the existing IP defined in ipNetToMediaTable? 4. anyway, IMHO, it should be the only standard way to define a new iSCSI TCP Port. Additional comments: Are there no needs to configure: 1. Max Number of Sessions (per instance? per target? per portal?) 2. Max Number of connections per session? Thanks, Michele From owner-ips@ece.cmu.edu Wed Nov 7 08:48:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA05773 for ; Wed, 7 Nov 2001 08:48:18 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7DDNT29220 for ips-outgoing; Wed, 7 Nov 2001 08:13:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar2ab.compuserve.com (siaar2ab.compuserve.com [149.174.40.138]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7DDKl29214 for ; Wed, 7 Nov 2001 08:13:20 -0500 (EST) Received: (from mailgate@localhost) by siaar2ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id IAA26887 for ips@ece.cmu.edu; Wed, 7 Nov 2001 08:13:15 -0500 (EST) Received: from compuserve.com (sfr-tgn-sfo-vty10.as.wcom.net [216.192.34.10]) by siaar2ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id IAA26837; Wed, 7 Nov 2001 08:13:07 -0500 (EST) Message-ID: <3BE933F7.3A7268F1@compuserve.com> Date: Wed, 07 Nov 2001 07:15:35 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win98; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector CC: Murali Rajagopal , Franco Travostino , "Rodriguez, Elizabeth" Subject: FCencap: List ALL SOF/EOF codes Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Upon further reflection, I think the right thing to do is to list all the SOF/EOF codes defined in FC-BB in the FC Encapsulation draft. FIRST There is nothing in the FC Encapsulation draft other than to omission of Class 1 SOF/EOF codes that prevents encapsulating FC Class 1 frames for TCP transport. Sure, a TCP ULP that is smarter than anything anybody has thought about will be required to do it. BUT there is (or should be) nothing the the FC Encapsulation draft that prevents such a protocol from being invented. AND the FC Encapsulation draft specifically says that you need the wisdom of some other protocol document in order to get any use out of the FC Encapsulation draft. Why force the mad man that devises a way to transport Class 1 over TCP/IP to revise the FC Encapsulation SOF/EOF tables? SECOND It is conceivable that a future version of iFCP (or maybe even FCIP) might want to support Class 4. Again, there is nothing in the FC Encapsulation draft that prevents this, except the omission of the SOF/EOF codes. FINALLY I believe that the elimination of all SOF/EOF codes other than Class 2, Class 3, AND CLASS F is a hold over from the early FCIP work, before the FC Encapsulation was split into a separate draft. I believe that decision was right for FCIP but wrong for an FC Encapsulation intended to be used by ALL FC protocols running over TCP/IP. Thanks for your consideration. Ralph... From owner-ips@ece.cmu.edu Wed Nov 7 08:55:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA06111 for ; Wed, 7 Nov 2001 08:55:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7Cio127586 for ips-outgoing; Wed, 7 Nov 2001 07:44:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1ab.compuserve.com (siaar1ab.compuserve.com [149.174.40.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7Cill27582 for ; Wed, 7 Nov 2001 07:44:47 -0500 (EST) Received: (from mailgate@localhost) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id HAA22565 for ips@ece.cmu.edu; Wed, 7 Nov 2001 07:44:42 -0500 (EST) Received: from compuserve.com (sfr-tgn-sff-vty8.as.wcom.net [216.192.8.8]) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id HAA22533; Wed, 7 Nov 2001 07:44:35 -0500 (EST) Message-ID: <3BE92DB5.64B663E@compuserve.com> Date: Wed, 07 Nov 2001 06:48:53 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win98; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector CC: Murali Rajagopal , Franco Travostino Subject: Re: FCencap: Missing SOF/EOF characters References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Murali, Franco, I thank you both for pointing out just how slippery a slope this is. Until receiving your e-mail messages I had been under the misimpression that Class F was an important part of the FCIP mission. :-) Vague may be vague but it also is not unnecessarily constraining. All the best. Ralph... Murali Rajagopal wrote: > How about stating something that we do support in the Scope section: > > "This document only considers encapsulation for only FC Classes 2 and 3." > > -Murali > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Franco Travostino > Sent: Tuesday, November 06, 2001 11:31 AM > To: ENDL_TX@computer.org; IPS Reflector > Subject: Re: FCencap: Missing SOF/EOF characters > > At 12:54 PM 11/6/2001, Ralph Weber wrote: > >> "This document describes common mechanisms for the transport >> of Fibre Channel frames over an IP network within the limitations >> of service provided by an IP network. The topics described in >> this document include the encapsulation format and a mechanism >> for enforcing the Fibre Channel frame lifetime limits." > > > This proposed text comes a tad closer, but it's cryptic still. > > >> I am NOT willing to discuss Fibre Channel Classes in the draft >> because to do so would require adding a definition of the term. >> Fibre Channel Classes are not discussed in the draft as it is >> currently written. > > > I agree that a definition/discussion would be way inappropriate here. > A T11 citation should do the trick instead. We already have a sentence > like "The format and content of an FC frame is described in the FC > standards (e.g., FC-FS [3], FC-SW-2 [4], and FC-PI [5])." I don't see > a problem with writing "Scope is limited to FC Class 2 and 3, which > are described in the FC standard ([x]), and then adding [x] and the > authoritative T11 document to Section 7 References. This direct and > simple. > > -franco From owner-ips@ece.cmu.edu Wed Nov 7 11:07:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA13750 for ; Wed, 7 Nov 2001 11:07:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7FJMY07730 for ips-outgoing; Wed, 7 Nov 2001 10:19:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7FJKl07724 for ; Wed, 7 Nov 2001 10:19:20 -0500 (EST) Received: from cisco.com (58509@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id KAA19521; Wed, 7 Nov 2001 10:18:39 -0500 (EST) Message-ID: <3BE94E6E.E71B0184@cisco.com> Date: Wed, 07 Nov 2001 09:08:30 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Michele Hallak - Stamler CC: IPS Subject: Re: iSCSI: New iSCSI MIB draft References: <838D8D2617300146B7F47E4D9AE7FF10114A15@SANSRV1.SAN-RAD.CO.IL> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Michele Hallak - Stamler wrote: > > My answers in-line also: > > -----Original Message----- > From: Mark Bakke [mailto:mbakke@cisco.com] > Sent: Tuesday, November 06, 2001 4:19 PM > To: Michele Hallak - Stamler > Cc: IPS > Subject: Re: iSCSI: New iSCSI MIB draft > > Michele- > > Answers inline. Basically, we have discussed making parts of the > MIB writable, and need to start working on it some more. Please keep > posting anything that you think might be useful as a writable attribute > or a creatable/deletable row. > > -- > Mark > > Michele Hallak - Stamler wrote: > > > > Hi, > > 1.I see that the target access list is still read-only. How do you > think > > that it should be configured? > > I had originally assumed that these would be configured by other > means than SNMP, but it is certainly something that makes sense to > configure. I will post another message to start a thread on this. > > Michele>> I think also that we should add an additional field describing > the kind of access (read-only / read-write) > I think also that it will be a good idea to have and enable/disable > field as you and Keith suggested. > > > 2. Who should configure the aliases of the iscsi targets and > initiators? > > These could easily be writable as well. > > > 3. Same question for the portals. > > Have to think about this one. Any ideas? > Michele >> For the portals, I have to tell that I have some problems: > 1. it seems correct to allow an administrator to define iSCSI portals > from the iSCSI MIB. That should be possible; the iSCSI MIB would likely be the right place to do this for many implementations. This could work one of two ways, depending on the implementation: - The management station could add the IP address via the iSCSI MIB, and have it automatically be added on the host or device. - The managment station could add an existing IP address (one already configured on the host or device); adding it here would make it usable by the iSCSI implementation. Incidentally, should we add something like iscsiTgtPortalSource to these structures? This would be an enumerated type indicating whether an address was statically configured, or discovered through DHCP. > 2. in another way, a portal has some relation to a specific interface > (IP Address/MAC Address) and for that relation we need the IP-MIB (if > I'm not wrong) Technically, we don't need to do this; the relationship between an IP address and a physical interface is typically an indirect one that makes use of the host or device's routing tables. However, anything supporting iSCSI ought to have all of the TCP and IP MIBs anyway. > 3. so it should be specified clearly: can we define "new ip addresses" > via this MIB or only use the existing IP defined in > ipNetToMediaTable? Good question for debate; it can be done either way. > 4. anyway, IMHO, it should be the only standard way to define a new > iSCSI TCP Port. I would guess that most target implementations will assume an iSCSI TCP port as the default, and the user or management station will only have to specify one if they want something different. > Additional comments: > Are there no needs to configure: > 1. Max Number of Sessions (per instance? per target? per portal?) > 2. Max Number of connections per session? I don't think so. These are generally limited by the implementation, rather than by the user. > > Thanks, > Michele -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Nov 7 11:08:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA13764 for ; Wed, 7 Nov 2001 11:08:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7FU9N08419 for ips-outgoing; Wed, 7 Nov 2001 10:30:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars0m9.nortelnetworks.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7FU6l08410 for ; Wed, 7 Nov 2001 10:30:06 -0500 (EST) Received: from zcars04e.ca.nortel.com (zcars04e.ca.nortel.com [47.129.242.56]) by zcars0m9.nortelnetworks.com (8.11.0/8.11.0) with ESMTP id fA7FTMS03273 for ; Wed, 7 Nov 2001 10:29:22 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04e.ca.nortel.com; Wed, 7 Nov 2001 10:29:43 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id WMPW6NBC; Wed, 7 Nov 2001 10:28:58 -0500 Received: from cse-ns-laptop.nortelnetworks.com (cse-ns-laptop.us.nortel.com [47.16.69.109]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id VW7FBSNR; Wed, 7 Nov 2001 10:28:59 -0500 Message-Id: <5.1.0.14.2.20011107095945.02325d78@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Wed, 07 Nov 2001 10:31:42 -0500 To: ENDL_TX@computer.org, IPS Reflector From: "Franco Travostino" Subject: Re: FCencap: List ALL SOF/EOF codes Cc: Murali Rajagopal , "Rodriguez, Elizabeth" In-Reply-To: <3BE933F7.3A7268F1@compuserve.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_85714270==_.ALT" X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_85714270==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed I agree with this motion. You wrote under another heading: "Vague may be vague but it also is not unnecessarily constraining." What a troubling statement was this. I'm glad we now appear to be moving towards a constructive end after all. thanks -franco At 08:15 AM 11/7/2001, Ralph Weber wrote: >Upon further reflection, I think the right thing to do >is to list all the SOF/EOF codes defined in FC-BB in >the FC Encapsulation draft. > >FIRST > >There is nothing in the FC Encapsulation draft other >than to omission of Class 1 SOF/EOF codes that prevents >encapsulating FC Class 1 frames for TCP transport. >Sure, a TCP ULP that is smarter than anything anybody >has thought about will be required to do it. BUT >there is (or should be) nothing the the FC Encapsulation >draft that prevents such a protocol from being invented. >AND the FC Encapsulation draft specifically says that >you need the wisdom of some other protocol document in >order to get any use out of the FC Encapsulation draft. >Why force the mad man that devises a way to transport >Class 1 over TCP/IP to revise the FC Encapsulation >SOF/EOF tables? > >SECOND > >It is conceivable that a future version of iFCP >(or maybe even FCIP) might want to support Class 4. >Again, there is nothing in the FC Encapsulation >draft that prevents this, except the omission >of the SOF/EOF codes. > >FINALLY > >I believe that the elimination of all SOF/EOF >codes other than Class 2, Class 3, AND CLASS F >is a hold over from the early FCIP work, before >the FC Encapsulation was split into a separate >draft. I believe that decision was right for >FCIP but wrong for an FC Encapsulation intended >to be used by ALL FC protocols running over >TCP/IP. > >Thanks for your consideration. > >Ralph... --=====================_85714270==_.ALT Content-Type: text/html; charset="us-ascii"
I agree with this motion.

You wrote under another heading: "Vague may be vague but it also is not unnecessarily constraining." What a troubling statement was this. I'm glad we now appear to be moving towards a constructive end after all.

thanks
-franco

At 08:15 AM 11/7/2001, Ralph Weber wrote:
Upon further reflection, I think the right thing to do
is to list all the SOF/EOF codes defined in FC-BB in
the FC Encapsulation draft.

FIRST

There is nothing in the FC Encapsulation draft other
than to omission of Class 1 SOF/EOF codes that prevents
encapsulating FC Class 1 frames for TCP transport.
Sure, a TCP ULP that is smarter than anything anybody
has thought about will be required to do it.  BUT
there is (or should be) nothing the the FC Encapsulation
draft that prevents such a protocol from being invented.
AND the FC Encapsulation draft specifically says that
you need the wisdom of some other protocol document in
order to get any use out of the FC Encapsulation draft.
Why force the mad man that devises a way to transport
Class 1 over TCP/IP to revise the FC Encapsulation
SOF/EOF tables?

SECOND

It is conceivable that a future version of iFCP
(or maybe even FCIP) might want to support Class 4.
Again, there is nothing in the FC Encapsulation
draft that prevents this, except the omission
of the SOF/EOF codes.

FINALLY

I believe that the elimination of all SOF/EOF
codes other than Class 2, Class 3, AND CLASS F
is a hold over from the early FCIP work, before
the FC Encapsulation was split into a separate
draft. I believe that decision was right for
FCIP but wrong for an FC Encapsulation intended
to be used by ALL FC protocols running over
TCP/IP.

Thanks for your consideration.

Ralph...
--=====================_85714270==_.ALT-- From owner-ips@ece.cmu.edu Wed Nov 7 11:08:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA13821 for ; Wed, 7 Nov 2001 11:08:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7FA1L07060 for ips-outgoing; Wed, 7 Nov 2001 10:10:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dogwood.cisco.com (dogwood.cisco.com [161.44.11.19]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7FA0l07056 for ; Wed, 7 Nov 2001 10:10:00 -0500 (EST) Received: from cisco.com (58509@mbakke-lnx.cisco.com [161.44.68.87]) by dogwood.cisco.com (8.8.6 (PHNE_14041)/CISCO.SERVER.1.2) with ESMTP id KAA10420 for ; Wed, 7 Nov 2001 10:09:54 -0500 (EST) Message-ID: <3BE94C60.DF3E0708@cisco.com> Date: Wed, 07 Nov 2001 08:59:44 -0600 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3.uid32 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI: SLP MIB draft-00 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit A draft of the new SLP MIB is now available at: http://search.ietf.org/internet-drafts/draft-mcdonald-svrloc-mib-00.txt And an object model drawing is at: ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-slp-mib-uml-model-00.pdf This MIB is discussed on the srvloc-discuss@lists.sourceforge.net, but I thought it might be of interest to iSCSI and FCIP folks. Also, Ira has put quite a bit of work into making this MIB writable, and it deserves a look as we do the same with the SCSI, iSCSI, and FCIP MIBs. -- Mark A. Bakke Cisco Systems mbakke@cisco.com 763.398.1054 From owner-ips@ece.cmu.edu Wed Nov 7 11:14:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA14109 for ; Wed, 7 Nov 2001 11:14:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7Ex9T06286 for ips-outgoing; Wed, 7 Nov 2001 09:59:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7Ex7l06282 for ; Wed, 7 Nov 2001 09:59:07 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id GAA00239; Wed, 7 Nov 2001 06:58:38 -0800 (PST) From: Keith McCloghrie Message-Id: <200111071458.GAA00239@cisco.com> Subject: Re: iSCSI: New iSCSI MIB draft To: michele@sanrad.com (Michele Hallak - Stamler) Date: Wed, 7 Nov 2001 06:58:38 -0800 (PST) Cc: mbakke@cisco.com (Mark Bakke), ips@ece.cmu.edu (IPS) In-Reply-To: <838D8D2617300146B7F47E4D9AE7FF10114A15@SANSRV1.SAN-RAD.CO.IL> from "Michele Hallak - Stamler" at Nov 07, 2001 11:40:46 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > > 3. Same question for the portals. > > Have to think about this one. Any ideas? > > Michele >> For the portals, I have to tell that I have some problems: > 1. it seems correct to allow an administrator to define iSCSI portals > from the iSCSI MIB. > 2. in another way, a portal has some relation to a specific interface > (IP Address/MAC Address) and for that relation we need the IP-MIB (if > I'm not wrong) > 3. so it should be specified clearly: can we define "new ip addresses" > via this MIB or only use the existing IP defined in ipNetToMediaTable? This MIB should use IP address(es) that a device already has; it should not attempt to provide the ability to configure additional IP addresses. Keith. From owner-ips@ece.cmu.edu Wed Nov 7 12:22:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA17200 for ; Wed, 7 Nov 2001 12:22:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7GROb12772 for ips-outgoing; Wed, 7 Nov 2001 11:27:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7GRMl12767 for ; Wed, 7 Nov 2001 11:27:22 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id IAA16519; Wed, 7 Nov 2001 08:26:45 -0800 (PST) From: Keith McCloghrie Message-Id: <200111071626.IAA16519@cisco.com> Subject: Re: iSCSI: New iSCSI MIB draft To: mbakke@cisco.com (Mark Bakke) Date: Wed, 7 Nov 2001 08:26:45 -0800 (PST) Cc: michele@sanrad.com (Michele Hallak - Stamler), ips@ece.cmu.edu (IPS) In-Reply-To: <3BE94E6E.E71B0184@cisco.com> from "Mark Bakke" at Nov 07, 2001 09:08:30 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mark, > > Michele >> For the portals, I have to tell that I have some problems: > > 1. it seems correct to allow an administrator to define iSCSI portals > > from the iSCSI MIB. > > That should be possible; the iSCSI MIB would likely be the > right place to do this for many implementations. This could > work one of two ways, depending on the implementation: > > - The management station could add the IP address via the iSCSI > MIB, and have it automatically be added on the host or device. As I intimated in my previous message, this is out of scope for this MIB because assigning new IP addresses to a host is not an iSCSI-specific function. > - The managment station could add an existing IP address (one already > configured on the host or device); adding it here would make it > usable by the iSCSI implementation. > > Incidentally, should we add something like iscsiTgtPortalSource > to these structures? This would be an enumerated type indicating > whether an address was statically configured, or discovered through > DHCP. Actually, your mention of DHCP raises an interesting problem. The MIB currently allows an iSCSI instance to have two (say, target) portals, one at (I1,P1) and the other at (I2,P2), where I1 and I2 are IP-addresses and P1 and P2 are TCP/transport port numbers. Now, if P1 and P2 are different, and the host/device uses DHCP to obtain the two IP addresses. How do you know which port number goes with which address ?? My reaction is that you don't/can't. To solve this problem the MIB must not statically bind local IP addresses to portals. None of the IP-based applications that I can think of (e.g., FTP, HTTP, SNMP, SMTP, BGP, etc.) have a static binding of a local TCP port number to a subset of the local host's IP-addresses. Rather, they have a static TCP/UDP port on which the server listens, and the client has a port dynamically assigned on a per-connection/session basis. The server listens on the static port on all local IP addresses; the client dynamically picks a local IP address on a per-connection/session basis. So, does iSCSI work this way also ?? If so, then the MIB is wrong; if not, I think iSCSI has a problem with DHCP. Also note that you have used InetAddressType (from RFC 2851), and one of the enumerated values of InetAddressType is 'dns'. As and when 'dns' is the value of iscsiTgtPortalAddrType or iscsiIntrPortalAddrType, then you would typically not have a static binding of a local TCP port number to one of the local host's IP-addresses. Keith. From owner-ips@ece.cmu.edu Wed Nov 7 12:32:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA17469 for ; Wed, 7 Nov 2001 12:32:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7GQvL12734 for ips-outgoing; Wed, 7 Nov 2001 11:26:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e21.nc.us.ibm.com (e21.nc.us.ibm.com [32.97.136.227]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7GQll12715 for ; Wed, 7 Nov 2001 11:26:47 -0500 (EST) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e21.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id KAA59936 for ; Wed, 7 Nov 2001 10:24:08 -0600 Received: from d04nms41.raleigh.ibm.com (d04nms41.raleigh.ibm.com [9.67.228.19]) by southrelay02.raleigh.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA7GQi3110310 for ; Wed, 7 Nov 2001 11:26:44 -0500 Subject: Re: iSCSI: resend of diagram from 2.5.1 To: "John Hufferd" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Andre Asselin" Date: Wed, 7 Nov 2001 11:27:51 -0500 X-MIMETrack: Serialize by Router on D04NMS41/04/M/IBM(Release 5.0.8 |June 18, 2001) at 11/07/2001 11:26:44 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk John, How's this look? ----------------------------IP Network--------------------- | | | +--------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +-----------|-----------------+ | | | | | Portal Group 1 | | Portal Group 2 | | | | | | | | | | | | | | SCSI Target Port | | SCSI Target Port | | | | | | (iSCSI Target Name + | | (iSCSI Target Name + | | | | | | 't' + 1) | | 't' + 2) | | | | | +--------------------------+ +-----------------------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session | | iSCSI Session | | | | | | | | | | | | | | (iSCSI Initiator Name + | | (iSCSI Initiator Name + | | | | | | ISID + iSCSI Target Name + | | ISID + iSCSI Target Name + | | | | | | Portal Group ID) | | Portal Group ID) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-----------------------------------------------------------------+ | | | | Network Entity | +---------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM 11/06/2001 cc: ips@ece.cmu.edu 03:33 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: iSCSI: resend of diagram from 2.5.1(Document link: Andre Asselin) Andre, Now that I can see the picture that you sent. It probably now needs to have the Names of the Target Side SCSI Ports clearly called out. The picture might imply to some folks that the SCSI Port Name is associated with "iSCSI Name + TSID" where as the SCSI Port Name should be "iSCSI Name + PortalGroupTag". The picture is, I think, technically correct, the Target End of the Session can be associated with iSCSI Name+TSID, but that may cause others to go through the same set of discussions that we have held recently, and they might think that the SCSI Port Name includes the TSID. I wonder if it would be clearer to Align, in the pictures, the concept of Target End of a Session with SCSI Port Name and give it the identification of "iSCSI Name + PortalGroupTag" and leave TSID to the various text in the Draft and for it be understood as just a Handle which the implementation can use in any way it wants. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Andre Asselin 11/06/2001 07:38 AM To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: iSCSI: resend of diagram from 2.5.1 It looks like my original post with an updated diagram for section 2.5.1 got wrapped badly. Here's a second attempt that hopefully will make it through. (original) ----------------------------IP Network--------------------- | | | +--------|---------------|--------------------|---------------------+ | +----|---------------|-----+ +----|---------+ | | | +---------+ +---------+ | | +---------+ | | | | | Network | | Network | | | | Network | | | | | | Portal | | Portal | | | | Portal | | | | | +--|------+ +---------+ | | +---------+ | | | | | | | | | | | | | | Portal | | | | Portal | | | | | Group 1 | | | | Group 2 | | | +--------------------------+ +--------------+ | | | | | | | +----------------------------+ +-----------------------------+ | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | +----------------------------+ +-----------------------------+ | | | | iSCSI Target Node | | (within Network Entity, not shown) | +-------------------------------------------------------------------+ (updated) ----------------------------IP Network--------------------- | | | +--------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +------|-------+ | | | | | Portal | | Portal | | | | | | Group 1 | | Group 2 | | | | | +--------------------------+ +--------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-----------------------------------------------------------------+ | | | | Network Entity | +---------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Wed Nov 7 13:25:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA17210 for ; Wed, 7 Nov 2001 12:22:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7GTga12950 for ips-outgoing; Wed, 7 Nov 2001 11:29:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7GTfl12943 for ; Wed, 7 Nov 2001 11:29:41 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id IAA11486; Wed, 7 Nov 2001 08:29:29 -0800 (PST) From: "Murali Rajagopal" To: , "IPS Reflector" Cc: "Franco Travostino" Subject: RE: FCencap: Missing SOF/EOF characters Date: Wed, 7 Nov 2001 08:30:35 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 Importance: Normal In-Reply-To: <3BE92DB5.64B663E@compuserve.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ralph: Missed Class F in the slippery slopes of the wonderful world of FC Classes! -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ralph Weber Sent: Wednesday, November 07, 2001 4:49 AM To: IPS Reflector Cc: Murali Rajagopal; Franco Travostino Subject: Re: FCencap: Missing SOF/EOF characters Murali, Franco, I thank you both for pointing out just how slippery a slope this is. Until receiving your e-mail messages I had been under the misimpression that Class F was an important part of the FCIP mission. :-) Vague may be vague but it also is not unnecessarily constraining. All the best. Ralph... Murali Rajagopal wrote: > How about stating something that we do support in the Scope section: > > "This document only considers encapsulation for only FC Classes 2 and 3." > > -Murali > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Franco Travostino > Sent: Tuesday, November 06, 2001 11:31 AM > To: ENDL_TX@computer.org; IPS Reflector > Subject: Re: FCencap: Missing SOF/EOF characters > > At 12:54 PM 11/6/2001, Ralph Weber wrote: > >> "This document describes common mechanisms for the transport >> of Fibre Channel frames over an IP network within the limitations >> of service provided by an IP network. The topics described in >> this document include the encapsulation format and a mechanism >> for enforcing the Fibre Channel frame lifetime limits." > > > This proposed text comes a tad closer, but it's cryptic still. > > >> I am NOT willing to discuss Fibre Channel Classes in the draft >> because to do so would require adding a definition of the term. >> Fibre Channel Classes are not discussed in the draft as it is >> currently written. > > > I agree that a definition/discussion would be way inappropriate here. > A T11 citation should do the trick instead. We already have a sentence > like "The format and content of an FC frame is described in the FC > standards (e.g., FC-FS [3], FC-SW-2 [4], and FC-PI [5])." I don't see > a problem with writing "Scope is limited to FC Class 2 and 3, which > are described in the FC standard ([x]), and then adding [x] and the > authoritative T11 document to Section 7 References. This direct and > simple. > > -franco From owner-ips@ece.cmu.edu Wed Nov 7 13:31:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA19498 for ; Wed, 7 Nov 2001 13:31:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7I0xF20003 for ips-outgoing; Wed, 7 Nov 2001 13:00:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7I0vl19995 for ; Wed, 7 Nov 2001 13:00:57 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id KAA09409; Wed, 7 Nov 2001 10:00:45 -0800 (PST) From: Keith McCloghrie Message-Id: <200111071800.KAA09409@cisco.com> Subject: Re: iSCSI: New iSCSI MIB draft To: bill@sanera.net (Bill Strahm) Date: Wed, 7 Nov 2001 10:00:45 -0800 (PST) Cc: kzm@cisco.com, ips@ece.cmu.edu In-Reply-To: from "Bill Strahm" at Nov 07, 2001 09:39:35 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi Bill, (I'll post your message and my response to the list as you said I could.) Yes, a socket interface typically provides the ability to listen for incoming connection requests on one or all addresses. That is, a common implementation of the IP-layer allows for what iSCSI is proposing. However, few if any applications do it. So, just because a lower-layer implementation allows for it to be done, should the application's arhcitecture require it be done ? Binding to an interface does seem better, but I think it causes iSCSI to enter the murky world of multi-homing, see the descriptions of the Strong ES model and the Weak ES model in RFC 1122 (pages 62-63). (Or perhaps, iSCSI is already in that world?) Keith. > Keith, > > I want to pick a nit with your description of how a TCP/UDP Server works... > I'll do this privately, if you think it is worth expanding on feel free to > post to the IPS mailing list > > Comments inline > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Keith McCloghrie > Sent: Wednesday, November 07, 2001 8:27 AM > To: Mark Bakke > Cc: Michele Hallak - Stamler; IPS > Subject: Re: iSCSI: New iSCSI MIB draft > > None of the IP-based applications that I can think of (e.g., FTP, HTTP, > SNMP, SMTP, BGP, etc.) have a static binding of a local TCP port number > to a subset of the local host's IP-addresses. Rather, they have a > static TCP/UDP port on which the server listens, and the client has a > port dynamically assigned on a per-connection/session basis. The > server listens on the static port on all local IP addresses; the client > dynamically picks a local IP address on a per-connection/session > basis. So, does iSCSI work this way also ?? If so, then the MIB > is wrong; if not, I think iSCSI has a problem with DHCP. > > > Actually the server may bind to ANY local address, in fact in many > situations you want the service to bind to one interface but not the other > (let say providing DHCP internally, but not externally, or providing > internal DNS, vs. external DNS). This binding is done based on IP address > (the parameter provided to the bind call), you are correct most times a > default address is provided as 0.0.0.0 which means bind to all interfaces. > > What you want to do (and I think will solve the DHCP problem) is to bind to > an InterfaceIndex, allowing the application to determine which IP address to > use based on entries in the IP table (what interface is that address bound > too) > > > Also note that you have used InetAddressType (from RFC 2851), and one > of the enumerated values of InetAddressType is 'dns'. As and when 'dns' > is the value of iscsiTgtPortalAddrType or iscsiIntrPortalAddrType, then > you would typically not have a static binding of a local TCP port > number to one of the local host's IP-addresses. > > I think you can still statically bind to a single interface in this case > > > Bill Strahm > +========+=========+=========+=========+=========+=========+=========+ > Bill Strahm Software Development is a race between Programmers > Member of the trying to build bigger and better idiot proof software > Technical Staff and the Universe trying to produce bigger and better > bill@sanera.net idiots. > (503) 601-0263 So far the Universe is winning --- Rich Cook > > > From owner-ips@ece.cmu.edu Wed Nov 7 13:35:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA19647 for ; Wed, 7 Nov 2001 13:35:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7HuLn19648 for ips-outgoing; Wed, 7 Nov 2001 12:56:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA7HuJl19643 for ; Wed, 7 Nov 2001 12:56:19 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by crufty; Wed Nov 7 12:50:54 EST 2001 Received: from aura.research.bell-labs.com (aura.research.bell-labs.com [135.104.46.10]) by grubby.research.bell-labs.com (8.11.6/8.11.6) with ESMTP id fA7HtAo61153; Wed, 7 Nov 2001 12:55:10 -0500 (EST) Received: (from sandeepj@localhost) by aura.research.bell-labs.com (8.9.1/8.9.1) id MAA25322; Wed, 7 Nov 2001 12:55:09 -0500 (EST) Date: Wed, 7 Nov 2001 12:55:09 -0500 (EST) Message-Id: <200111071755.MAA25322@aura.research.bell-labs.com> From: sandeepj@research.bell-labs.com (Sandeep Joshi) To: bill@sanera.net, ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, > I actually would prefer if we didn't say anything other than a statement > saying "Here is a policy that will cover IPS traffic" from there it is up to > compliant IPsec implementations to utilize this policy... > > Being that the WG feels that just specifying a coverage policy is adequate, > but must get into specifying portions of the IPsec functionality, I would > prefer Tunnel mode, because as far as I can tell, no one has shown a > functional e-e transport mode implementation in the wild... > > Can anyone point to one ? I think Borderware has a bridge proxy which supports transport mode but you would need to double-check. You must be aware of OpenBSD's bridging which came close but did not do transport mode, since RFC 2401 requires that transport mode must not be applied to IP fragments. Anything In the wild..? dunno -Sandeep > > Bill > +========+=========+=========+=========+=========+=========+=========+ > Bill Strahm Software Development is a race between Programmers > Member of the trying to build bigger and better idiot proof software > Technical Staff and the Universe trying to produce bigger and better > bill@sanera.net idiots. > (503) 601-0263 So far the Universe is winning --- Rich Cook > > > -----Original Message----- > > From: Ofer Biran [mailto:BIRAN@il.ibm.com] > > Sent: Tuesday, November 06, 2001 11:53 AM > > To: Bill Strahm > > Cc: saqibj@margallacomm.com; ips@ece.cmu.edu > > Subject: RE: iSCSI: IPsec tunnel / transport mode decision > > > > Bill, > > > > I agree that you can make external devices that support transport mode, > > but it seems that most of those existing today do not support it. > > > > Anyway for our required decision... you also said you prefer tunnel mode, > > right ? > > > > Regards, > > Ofer > > > > Ofer Biran > > Storage and Systems Technology > > IBM Research Lab in Haifa > > biran@il.ibm.com 972-4-8296253 > > > > > "Bill Strahm" on 04/11/2001 21:39:22 > > > > > > Please respond to "Bill Strahm" > > > > > > To: Ofer Biran/Haifa/IBM@IBMIL, > > > cc: > > > Subject: RE: iSCSI: IPsec tunnel / transport mode decision > > > > > > Ok, > > > > > > How does mandatory Transport mode remove the possibility of external > > > IPsec... > > > > > > I have said before I can make IPsec transport & tunnel mode work in > > > external > > > devices, just like you can do SSL/TLS accelerators both internally and > > > externally > > > > > > Bill > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Ofer Biran > > > Sent: Sunday, November 04, 2001 4:27 AM > > > To: saqibj@margallacomm.com > > > Cc: ips@ece.cmu.edu > > > Subject: RE: iSCSI: IPsec tunnel / transport mode decision > > > > > > Saqib, > > > > > > Mandatory transport mode would make bundling of external IPSec > > > impossible, while tunnel mode is not more difficult to implement > > > within the iSCSI endpoint than transport mode. > > > > > > "Cost of ownership and complexity of deploying a stand-alone > > > IPsec gateway" might be among the considerations of vendors and > > > customers, but I don't think the standard should block such > > > solutions (and it blocks more than just stand-alone IPsec > > > gateway). > > > > > > Regards, > > > Ofer > > > > > > Ofer Biran > > > Storage and Systems Technology > > > IBM Research Lab in Haifa > > > biran@il.ibm.com 972-4-8296253 > > > > > > "Saqib Jang" on 02/11/2001 20:59:03 > > > > > > Please respond to > > > > > > To: "Bill Strahm" , "CAVANNA,VICENTE V > > > (A-Roseville,ex1)" > > > cc: "SHEEHY,DAVE (A-Americas,unix1)" , Ofer > > > Biran/Haifa/IBM@IBMIL, > > > Subject: RE: iSCSI: IPsec tunnel / transport mode decision > > > > > > What about the cost of ownership and complexity of deploying > > > a stand-alone IPsec gateway for use with IPsec end-points? > > > If transport-mode IPsec is a must-to-implement capability in > > > iSCSI end-points there is an opportunity to have much > > > more coherent security for iSCSI. > > > > > > Saqib > > > From owner-ips@ece.cmu.edu Wed Nov 7 13:38:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA19810 for ; Wed, 7 Nov 2001 13:38:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7HSrZ17460 for ips-outgoing; Wed, 7 Nov 2001 12:28:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7HSql17455 for ; Wed, 7 Nov 2001 12:28:52 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel12.hp.com (Postfix) with ESMTP id 63CEF1F6BA for ; Wed, 7 Nov 2001 09:28:46 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id JAA18255; Wed, 7 Nov 2001 09:30:12 -0800 (PST) Message-Id: <200111071730.JAA18255@core.rose.hp.com> Date: Wed, 07 Nov 2001 09:41:05 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: Santosh Rao , ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands References: <200111070106.RAA07404@core.rose.hp.com> <3BE8995C.892512E8@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh, I have only one comment on your responses. > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. I made two convenient assumptions if you noticed, :-), one of which is that target forces session recovery on *any* error that it sees (ErrorRecoveryLevel=0) - including a dropped command due to a digest error. With that assumption, a target can afford not to implement a scoreboard. As I said in a private note, I guess what primarily bothers me about OOO commands on a connection is that it requires the receiver to undo this "optimization" on its end - most notably on a single connection. TCP experts may comment on how/if they dealt with a similar issue. OTOH, you had some valid comments on exceptions to ordering during connection recovery. Perhaps we can move on by making Julian's proposed stipulation a SHOULD.... -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Santosh Rao wrote: > > Mallikarjun, > > Some comments below. > > Regards, > Santosh > > "Mallikarjun C." wrote: > > > > Rod and Julian, > > > > This has been an interesting thread of discussion. Some > > comments - > > > > 1.My first reaction was - allowing out-of-order command > > transmission on the same connection deprives targets of > > an implementation choice. Targets which support only > > single-connection sessions and only support session > > recovery (reasonable assumptions in my mind) can no > > longer afford *not to* implement a command scoreboard. > > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. > > Thus, there is no getting away from implementing a sequencer at the > target. Given this, I think it is unreasonable to restrict initiator > implementation flexibility by imposing a strict ordering requirement > within the connection. > > > 2.Any end-node efficiency that is sought to be achieved > > by transmitting CmdSNs out-of-order from the initiator > > would be lost on the other end-node, since the target > > now must wait for re-ordering the commands. > > It has to handle this situation anyway to deal with holes caused by > digest errors. This scenario occurs even with initiators that issue > commands in order. > > > > > 3.The flipside is that out-of-order transmission saves > > link badwidth (albeit at the expense of end-node efficiency), > > compared to idling the link waiting for outbound DMA. > > We have to determine if this is a reasonable trade-off. > > > > 4.I can see Rod's point that prefetching all immediate > > data can be a burden on the NIC resources. But, two > > questions - > > - could the NIC not use unsolicited separate data > > PDUs in these cases? [ I realize that InitialR2T > > has to be "no" to let it happen... ] > > - could the NIC have a memory architecture that > > allows data prefetching for the next command (so > > this is a non-issue from the protocol perspective)? > > This scheme incurs one DMA delay for every new > > burst of commands. > > > > 5.Another (perhaps radical at this point) option is to do > > away with immediate unsolicited data, to stick only with > > separate unsolicited data. I would personally be okay > > with the choice, particularly if this feature (that > > helps software implementations) starts making hardware > > design complicated/expensive. > > > > So, to summarize - > > > > option immediate allow > > data in spec? out-of-order? > > > > (A) (5) above no no > > (B) No real reason to do this. no yes > > (C) (4) above yes no > > (D) pros & cons (1), (2) & (3) yes yes > > > > >From the arguments I heard so far, I am leaning towards > > option A, and option C in that order. > > > > Comments? > > -- > > Mallikarjun > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > Rod Harrison wrote: > > > > > > Julian, > > > > > > I don't understand what you are proposing here, what do you mean by > > > "multiplexed" DMA? > > > > > > The problem is that the DMAs take some time, the more there are > > > queued the longer the last DMAs queued take to complete. Some commands > > > require DMAs to complete before they can be sent, i.e. Writes with > > > immediate data, some commands do not, i.e. Reads and writes with no > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > soon a possible, which for a read after a write can be before the > > > write's DMA has completed. Maintaining an ordered queue for commands > > > to be sent on the HBA is expensive and redundant since the target > > > already knows how to queue commands before committing them to its SCSI > > > layer. > > > > > > The iSCSI HBA and its host driver are not at liberty to change the > > > order of commands from the OS, but the DMAs those commands need are > > > unlikely to complete in the same order, and as I mentioned some > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > order it has to maintain an ordered queue of commands waiting to be > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > immediate data almost impossible to support. > > > > > > I don't see the problem with allowing out of order commands given > > > that the target already has to deal with very similar problems. I > > > think we are getting in to the area of implementation choices here, > > > which is inappropriate for a specification. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > Rod, > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > commands. > > > If you have scheduled a long outbound DMA you are doomed regardless of > > > the > > > command ordering. > > > And if you have scheduled DMA operations piecemeal then you can insert > > > your commands in correct order. > > > > > > Julo > > > > > > "Rod Harrison" > > > 05-11-01 20:48 > > > Please respond to "Rod Harrison" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > cc: > > > Subject: iSCSI: Out of order commands, was current UNH > > > Plugfest > > > > > > [ Subject changed ] > > > > > > Julian, > > > > > > The ordering difference is introduced between the > > > host > > > side driver > > > and the iSCSI HBA. The host side driver must present SCSI commands to > > > the HBA in the order they are received from the OS to prevent read > > > after write dependency failures. The HBA might reorder the commands > > > depending on when DMA completes. The reordering can't be done ahead of > > > time in the host driver since it doesn't know how long each DMA might > > > take. As long as the HBA assigns CmdSN in the order it receives > > > commands the desired host ordering is preserved. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 12:35 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Rod, > > > > > > I all examples give the point I find hard to understand is why is the > > > ordering on the wire different from the presentation order to the > > > initiator. You can get as many overlaps as you want by presenting the > > > commands to the initiator in the desired order. > > > What we are considering here is the case in which you want to ship in > > > an > > > order different than the one you present the commands. > > > > > > Julo > > > > > > "Rod Harrison" > > > Sent by: owner-ips@ece.cmu.edu > > > 04-11-01 04:42 > > > Please respond to "Rod Harrison" > > > > > > To: "Barry Reinhold" , "Dave > > > Sheehy" > > > , "IETF IP SAN Reflector" > > > > > > cc: > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Barry, > > > > > > In general I agree but I don't think this is as much > > > of a > > > corner case > > > as it at first appears. Targets will have code very similar to that > > > needed to handle out of order commands to deal with digest errors. > > > Targets also need to queue commands whilst waiting for both solicited > > > and unsolicited data to arrive. Queuing out of order commands seems > > > little extra work. > > > > > > From an initiators point of view there are > > > efficiency, > > > and probably > > > performance gains to be had from sending commands out of order. Bob > > > Russell gave the example of a read being sent whilst write data DMA is > > > happening, and a similar situation can arise with DMA for writes > > > overtaking that of earlier writes if the initiator has multiple DMA > > > engines. In this case the initiator might be forced to let the wire go > > > idle if it can't send the data from completed DMAs as soon as > > > possible. > > > > > > We already have a command queue at the target to > > > enforce > > > correct > > > serialisation of commands, doing the same thing at the initiator is > > > redundant. > > > > > > Finally, I don't believe we should be writing a > > > standard > > > to work > > > around poor coding and test coverage, especially at the cost of > > > potential efficiency gains. > > > > > > I agree with Dave and Santosh that commands being > > > sent > > > out of order > > > on a single session should be allowed by the standard. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Barry Reinhold > > > Sent: Friday, November 02, 2001 5:24 PM > > > To: Dave Sheehy; IETF IP SAN Reflector > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Using features such as out of order command delivery on a connection > > > tend to > > > be the sort of things that lead to interoperability problems. It is > > > unexpected and probably going to hit poorly tested code paths even if > > > the > > > standard is written to allow it. > > > > > > >-----Original Message----- > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > >Dave Sheehy > > > >Sent: Friday, November 02, 2001 4:19 PM > > > >To: IETF IP SAN Reflector > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > >> > > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > > on > > > >> page 25 of draft 8, which says: > > > >> "Except for the commands marked for immediate delivery the > > > iSCSI > > > >> target layer MUST eliver the commands for execution in the > > > order > > > >> specified by CmdSN." > > > >> > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > on > > > >> each command shipped except for commands marked for immediate > > > >> delivery." > > > >> but the meaning of the term "shipped" is vague, and does not > > > >> necessarily > > > >> require that the PDUs arrive on the other end of a TCP > > > connection > > > >> in the same order that the CmdSN values were assigned to these > > > PDUs. > > > >> > > > >> Some initiators have been designed to send commands out of CmdSN > > > >> order on one connection. Consider the situation where there is > > > only > > > >> one connection and a high-level dispatcher creates a PDU for a > > > SCSI > > > >> command that involves writing immediate data to the target. > > > This PDU > > > >> is enqueued to a lower-level layer which has to setup, start, > > > and > > > >> wait-for a DMA operation to move the immediate data into an > > > onboard > > > >> buffer before the PDU can be put onto the wire. While this is > > > >> happening, the dispatcher creates another unrelated PDU for a > > > SCSI > > > >> read command (for example), and when this PDU is passed to the > > > >> lower-level layer it can be sent immediately, ahead of the > > > previous > > > >> write PDU and therefore out of order on this connection. > > > >> > > > >> The standard clearly allows this to happen if the two PDUs were > > > sent > > > >> on different connections, and seems to imply that this can also > > > happen > > > >> when the two PDUs are sent on the same connection. > > > >> > > > >> The suggestion is to put in the standard an explicit statement > > > that > > > >> this is allowed or not allowed, as appropriate. > > > >> > > > >> If this is allowed, such a statement would avoid the erroneous > > > >> assumption being made by some target implementers that within a > > > single > > > >> connection, commands will arrive in order. > > > >> > > > >> If this is not allowed, such a statement would avoid the > > > erroneous > > > >> assumption being made by some initiator implementers that within > > > a > > > >> single connection, commands can be put on the wire out of order. > > > >> > > > >> +++ > > > >> > > > >> will add an explicit statement saying that this behaviour is > > > forbidden. > > > >> 2.2.2.1 will contain: > > > >> > > > >> On any given connection, the iSCSI initiator MUST send the > > > >commands in the > > > >> order specified by CmdSN. > > > >> > > > >> +++ > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > have to > > > >order commands across the session. I don't see why it's a problem to > > > extend > > > >that to the connection as well. I, for one, believe we should take > > > >a liberal > > > >stance on this. > > > > > > > >Dave Sheehy > > > > > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## From owner-ips@ece.cmu.edu Wed Nov 7 13:47:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA20082 for ; Wed, 7 Nov 2001 13:47:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7HxmM19905 for ips-outgoing; Wed, 7 Nov 2001 12:59:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7Hxkl19901 for ; Wed, 7 Nov 2001 12:59:46 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA50620 for ; Wed, 7 Nov 2001 18:59:39 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA7HxcW93688 for ; Wed, 7 Nov 2001 18:59:38 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 7 Nov 2001 19:59:36 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 07/11/2001 19:59:38, Serialize complete at 07/11/2001 19:59:38 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, I did not see a SINGLE performance improvement that results from OOO shipping. I would be bad engineering to give away the "no-deadlock" mechanism we have now for nothing. I have also the impression that the point about deadlock that I keep repeating is ignored or not understood. As we stand today commands can be shipped with Immediate data or without and an implementer determined to squeeze maximum bandwidth and overlap command start with delivery will choose not to work with immediate data (as you have pointed out) while a low performance software implementation will use immediate data to minimize CPU cycles consumed. However both will be guaranteed to work without deadlock as source and sink use the same ordering. Recovery is still a low probability event and should be handled with a different set of considerations in mind. As for the strictness of the recommendation - yes we could settle on SHOULD. Julo "Mallikarjun C." Sent by: owner-ips@ece.cmu.edu 07-11-01 19:41 Please respond to cbm To: Santosh Rao , ips@ece.cmu.edu cc: Subject: Re: iSCSI: Out of order commands Santosh, I have only one comment on your responses. > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. I made two convenient assumptions if you noticed, :-), one of which is that target forces session recovery on *any* error that it sees (ErrorRecoveryLevel=0) - including a dropped command due to a digest error. With that assumption, a target can afford not to implement a scoreboard. As I said in a private note, I guess what primarily bothers me about OOO commands on a connection is that it requires the receiver to undo this "optimization" on its end - most notably on a single connection. TCP experts may comment on how/if they dealt with a similar issue. OTOH, you had some valid comments on exceptions to ordering during connection recovery. Perhaps we can move on by making Julian's proposed stipulation a SHOULD.... -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Santosh Rao wrote: > > Mallikarjun, > > Some comments below. > > Regards, > Santosh > > "Mallikarjun C." wrote: > > > > Rod and Julian, > > > > This has been an interesting thread of discussion. Some > > comments - > > > > 1.My first reaction was - allowing out-of-order command > > transmission on the same connection deprives targets of > > an implementation choice. Targets which support only > > single-connection sessions and only support session > > recovery (reasonable assumptions in my mind) can no > > longer afford *not to* implement a command scoreboard. > > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. > > Thus, there is no getting away from implementing a sequencer at the > target. Given this, I think it is unreasonable to restrict initiator > implementation flexibility by imposing a strict ordering requirement > within the connection. > > > 2.Any end-node efficiency that is sought to be achieved > > by transmitting CmdSNs out-of-order from the initiator > > would be lost on the other end-node, since the target > > now must wait for re-ordering the commands. > > It has to handle this situation anyway to deal with holes caused by > digest errors. This scenario occurs even with initiators that issue > commands in order. > > > > > 3.The flipside is that out-of-order transmission saves > > link badwidth (albeit at the expense of end-node efficiency), > > compared to idling the link waiting for outbound DMA. > > We have to determine if this is a reasonable trade-off. > > > > 4.I can see Rod's point that prefetching all immediate > > data can be a burden on the NIC resources. But, two > > questions - > > - could the NIC not use unsolicited separate data > > PDUs in these cases? [ I realize that InitialR2T > > has to be "no" to let it happen... ] > > - could the NIC have a memory architecture that > > allows data prefetching for the next command (so > > this is a non-issue from the protocol perspective)? > > This scheme incurs one DMA delay for every new > > burst of commands. > > > > 5.Another (perhaps radical at this point) option is to do > > away with immediate unsolicited data, to stick only with > > separate unsolicited data. I would personally be okay > > with the choice, particularly if this feature (that > > helps software implementations) starts making hardware > > design complicated/expensive. > > > > So, to summarize - > > > > option immediate allow > > data in spec? out-of-order? > > > > (A) (5) above no no > > (B) No real reason to do this. no yes > > (C) (4) above yes no > > (D) pros & cons (1), (2) & (3) yes yes > > > > >From the arguments I heard so far, I am leaning towards > > option A, and option C in that order. > > > > Comments? > > -- > > Mallikarjun > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > Rod Harrison wrote: > > > > > > Julian, > > > > > > I don't understand what you are proposing here, what do you mean by > > > "multiplexed" DMA? > > > > > > The problem is that the DMAs take some time, the more there are > > > queued the longer the last DMAs queued take to complete. Some commands > > > require DMAs to complete before they can be sent, i.e. Writes with > > > immediate data, some commands do not, i.e. Reads and writes with no > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > soon a possible, which for a read after a write can be before the > > > write's DMA has completed. Maintaining an ordered queue for commands > > > to be sent on the HBA is expensive and redundant since the target > > > already knows how to queue commands before committing them to its SCSI > > > layer. > > > > > > The iSCSI HBA and its host driver are not at liberty to change the > > > order of commands from the OS, but the DMAs those commands need are > > > unlikely to complete in the same order, and as I mentioned some > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > order it has to maintain an ordered queue of commands waiting to be > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > immediate data almost impossible to support. > > > > > > I don't see the problem with allowing out of order commands given > > > that the target already has to deal with very similar problems. I > > > think we are getting in to the area of implementation choices here, > > > which is inappropriate for a specification. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > Rod, > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > commands. > > > If you have scheduled a long outbound DMA you are doomed regardless of > > > the > > > command ordering. > > > And if you have scheduled DMA operations piecemeal then you can insert > > > your commands in correct order. > > > > > > Julo > > > > > > "Rod Harrison" > > > 05-11-01 20:48 > > > Please respond to "Rod Harrison" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > cc: > > > Subject: iSCSI: Out of order commands, was current UNH > > > Plugfest > > > > > > [ Subject changed ] > > > > > > Julian, > > > > > > The ordering difference is introduced between the > > > host > > > side driver > > > and the iSCSI HBA. The host side driver must present SCSI commands to > > > the HBA in the order they are received from the OS to prevent read > > > after write dependency failures. The HBA might reorder the commands > > > depending on when DMA completes. The reordering can't be done ahead of > > > time in the host driver since it doesn't know how long each DMA might > > > take. As long as the HBA assigns CmdSN in the order it receives > > > commands the desired host ordering is preserved. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 12:35 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Rod, > > > > > > I all examples give the point I find hard to understand is why is the > > > ordering on the wire different from the presentation order to the > > > initiator. You can get as many overlaps as you want by presenting the > > > commands to the initiator in the desired order. > > > What we are considering here is the case in which you want to ship in > > > an > > > order different than the one you present the commands. > > > > > > Julo > > > > > > "Rod Harrison" > > > Sent by: owner-ips@ece.cmu.edu > > > 04-11-01 04:42 > > > Please respond to "Rod Harrison" > > > > > > To: "Barry Reinhold" , "Dave > > > Sheehy" > > > , "IETF IP SAN Reflector" > > > > > > cc: > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Barry, > > > > > > In general I agree but I don't think this is as much > > > of a > > > corner case > > > as it at first appears. Targets will have code very similar to that > > > needed to handle out of order commands to deal with digest errors. > > > Targets also need to queue commands whilst waiting for both solicited > > > and unsolicited data to arrive. Queuing out of order commands seems > > > little extra work. > > > > > > From an initiators point of view there are > > > efficiency, > > > and probably > > > performance gains to be had from sending commands out of order. Bob > > > Russell gave the example of a read being sent whilst write data DMA is > > > happening, and a similar situation can arise with DMA for writes > > > overtaking that of earlier writes if the initiator has multiple DMA > > > engines. In this case the initiator might be forced to let the wire go > > > idle if it can't send the data from completed DMAs as soon as > > > possible. > > > > > > We already have a command queue at the target to > > > enforce > > > correct > > > serialisation of commands, doing the same thing at the initiator is > > > redundant. > > > > > > Finally, I don't believe we should be writing a > > > standard > > > to work > > > around poor coding and test coverage, especially at the cost of > > > potential efficiency gains. > > > > > > I agree with Dave and Santosh that commands being > > > sent > > > out of order > > > on a single session should be allowed by the standard. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Barry Reinhold > > > Sent: Friday, November 02, 2001 5:24 PM > > > To: Dave Sheehy; IETF IP SAN Reflector > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Using features such as out of order command delivery on a connection > > > tend to > > > be the sort of things that lead to interoperability problems. It is > > > unexpected and probably going to hit poorly tested code paths even if > > > the > > > standard is written to allow it. > > > > > > >-----Original Message----- > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > >Dave Sheehy > > > >Sent: Friday, November 02, 2001 4:19 PM > > > >To: IETF IP SAN Reflector > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > >> > > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > > on > > > >> page 25 of draft 8, which says: > > > >> "Except for the commands marked for immediate delivery the > > > iSCSI > > > >> target layer MUST eliver the commands for execution in the > > > order > > > >> specified by CmdSN." > > > >> > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > on > > > >> each command shipped except for commands marked for immediate > > > >> delivery." > > > >> but the meaning of the term "shipped" is vague, and does not > > > >> necessarily > > > >> require that the PDUs arrive on the other end of a TCP > > > connection > > > >> in the same order that the CmdSN values were assigned to these > > > PDUs. > > > >> > > > >> Some initiators have been designed to send commands out of CmdSN > > > >> order on one connection. Consider the situation where there is > > > only > > > >> one connection and a high-level dispatcher creates a PDU for a > > > SCSI > > > >> command that involves writing immediate data to the target. > > > This PDU > > > >> is enqueued to a lower-level layer which has to setup, start, > > > and > > > >> wait-for a DMA operation to move the immediate data into an > > > onboard > > > >> buffer before the PDU can be put onto the wire. While this is > > > >> happening, the dispatcher creates another unrelated PDU for a > > > SCSI > > > >> read command (for example), and when this PDU is passed to the > > > >> lower-level layer it can be sent immediately, ahead of the > > > previous > > > >> write PDU and therefore out of order on this connection. > > > >> > > > >> The standard clearly allows this to happen if the two PDUs were > > > sent > > > >> on different connections, and seems to imply that this can also > > > happen > > > >> when the two PDUs are sent on the same connection. > > > >> > > > >> The suggestion is to put in the standard an explicit statement > > > that > > > >> this is allowed or not allowed, as appropriate. > > > >> > > > >> If this is allowed, such a statement would avoid the erroneous > > > >> assumption being made by some target implementers that within a > > > single > > > >> connection, commands will arrive in order. > > > >> > > > >> If this is not allowed, such a statement would avoid the > > > erroneous > > > >> assumption being made by some initiator implementers that within > > > a > > > >> single connection, commands can be put on the wire out of order. > > > >> > > > >> +++ > > > >> > > > >> will add an explicit statement saying that this behaviour is > > > forbidden. > > > >> 2.2.2.1 will contain: > > > >> > > > >> On any given connection, the iSCSI initiator MUST send the > > > >commands in the > > > >> order specified by CmdSN. > > > >> > > > >> +++ > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > have to > > > >order commands across the session. I don't see why it's a problem to > > > extend > > > >that to the connection as well. I, for one, believe we should take > > > >a liberal > > > >stance on this. > > > > > > > >Dave Sheehy > > > > > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## From owner-ips@ece.cmu.edu Wed Nov 7 14:27:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA21243 for ; Wed, 7 Nov 2001 14:27:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7IH0521380 for ips-outgoing; Wed, 7 Nov 2001 13:17:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from philotas.hosting.pacbell.net (philotas.hosting.pacbell.net [216.100.99.24]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7IGvl21376 for ; Wed, 7 Nov 2001 13:16:58 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by philotas.hosting.pacbell.net id NAA25130; Wed, 7 Nov 2001 13:05:06 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "Santosh Rao" , Cc: "Rod Harrison" , "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 10:02:19 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: <3BE8995C.892512E8@cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Santosh, Considering what the likelyhood (or lack thereof) of digest errors, its occurance can be treated in quite a different way, then if out of order command transmission is allowed by the initiator. This premise allows a completely different set of design choices. Of course, the ultimate would be to do away with ordering at either end, and let the upper layer deal with this issue. The upper layer may or may not want any ordering at either end, and this is quite a reasonable scenario in the normal case of a computer communicating with a disk drive (array). Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Santosh Rao > Sent: Tuesday, November 06, 2001 6:16 PM > To: cbm@rose.hp.com > Cc: Rod Harrison; Julian Satran; ips@ece.cmu.edu > Subject: Re: iSCSI: Out of order commands > > > Mallikarjun, > > Some comments below. > > Regards, > Santosh > > "Mallikarjun C." wrote: > > > > Rod and Julian, > > > > This has been an interesting thread of discussion. Some > > comments - > > > > 1.My first reaction was - allowing out-of-order command > > transmission on the same connection deprives targets of > > an implementation choice. Targets which support only > > single-connection sessions and only support session > > recovery (reasonable assumptions in my mind) can no > > longer afford *not to* implement a command scoreboard. > > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. > > Thus, there is no getting away from implementing a sequencer at the > target. Given this, I think it is unreasonable to restrict initiator > implementation flexibility by imposing a strict ordering requirement > within the connection. > > > > > 2.Any end-node efficiency that is sought to be achieved > > by transmitting CmdSNs out-of-order from the initiator > > would be lost on the other end-node, since the target > > now must wait for re-ordering the commands. > > It has to handle this situation anyway to deal with holes caused by > digest errors. This scenario occurs even with initiators that issue > commands in order. > > > > > > 3.The flipside is that out-of-order transmission saves > > link badwidth (albeit at the expense of end-node efficiency), > > compared to idling the link waiting for outbound DMA. > > We have to determine if this is a reasonable trade-off. > > > > 4.I can see Rod's point that prefetching all immediate > > data can be a burden on the NIC resources. But, two > > questions - > > - could the NIC not use unsolicited separate data > > PDUs in these cases? [ I realize that InitialR2T > > has to be "no" to let it happen... ] > > - could the NIC have a memory architecture that > > allows data prefetching for the next command (so > > this is a non-issue from the protocol perspective)? > > This scheme incurs one DMA delay for every new > > burst of commands. > > > > 5.Another (perhaps radical at this point) option is to do > > away with immediate unsolicited data, to stick only with > > separate unsolicited data. I would personally be okay > > with the choice, particularly if this feature (that > > helps software implementations) starts making hardware > > design complicated/expensive. > > > > So, to summarize - > > > > option immediate allow > > data in spec? out-of-order? > > > > (A) (5) above no no > > (B) No real reason to do this. no yes > > (C) (4) above yes no > > (D) pros & cons (1), (2) & (3) yes yes > > > > >From the arguments I heard so far, I am leaning towards > > option A, and option C in that order. > > > > Comments? > > -- > > Mallikarjun > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > Rod Harrison wrote: > > > > > > Julian, > > > > > > I don't understand what you are proposing here, what > do you mean by > > > "multiplexed" DMA? > > > > > > The problem is that the DMAs take some time, the more > there are > > > queued the longer the last DMAs queued take to complete. Some commands > > > require DMAs to complete before they can be sent, i.e. Writes with > > > immediate data, some commands do not, i.e. Reads and writes with no > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > soon a possible, which for a read after a write can be before the > > > write's DMA has completed. Maintaining an ordered queue for commands > > > to be sent on the HBA is expensive and redundant since the target > > > already knows how to queue commands before committing them to its SCSI > > > layer. > > > > > > The iSCSI HBA and its host driver are not at liberty > to change the > > > order of commands from the OS, but the DMAs those commands need are > > > unlikely to complete in the same order, and as I mentioned some > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > order it has to maintain an ordered queue of commands waiting to be > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > immediate data almost impossible to support. > > > > > > I don't see the problem with allowing out of order > commands given > > > that the target already has to deal with very similar problems. I > > > think we are getting in to the area of implementation choices here, > > > which is inappropriate for a specification. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > Rod, > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > commands. > > > If you have scheduled a long outbound DMA you are doomed regardless of > > > the > > > command ordering. > > > And if you have scheduled DMA operations piecemeal then you can insert > > > your commands in correct order. > > > > > > Julo > > > > > > "Rod Harrison" > > > 05-11-01 20:48 > > > Please respond to "Rod Harrison" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > cc: > > > Subject: iSCSI: Out of order commands, was current UNH > > > Plugfest > > > > > > [ Subject changed ] > > > > > > Julian, > > > > > > The ordering difference is introduced between the > > > host > > > side driver > > > and the iSCSI HBA. The host side driver must present SCSI commands to > > > the HBA in the order they are received from the OS to prevent read > > > after write dependency failures. The HBA might reorder the commands > > > depending on when DMA completes. The reordering can't be done ahead of > > > time in the host driver since it doesn't know how long each DMA might > > > take. As long as the HBA assigns CmdSN in the order it receives > > > commands the desired host ordering is preserved. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 12:35 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Rod, > > > > > > I all examples give the point I find hard to understand is why is the > > > ordering on the wire different from the presentation order to the > > > initiator. You can get as many overlaps as you want by presenting the > > > commands to the initiator in the desired order. > > > What we are considering here is the case in which you want to ship in > > > an > > > order different than the one you present the commands. > > > > > > Julo > > > > > > "Rod Harrison" > > > Sent by: owner-ips@ece.cmu.edu > > > 04-11-01 04:42 > > > Please respond to "Rod Harrison" > > > > > > To: "Barry Reinhold" , "Dave > > > Sheehy" > > > , "IETF IP SAN Reflector" > > > > > > cc: > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Barry, > > > > > > In general I agree but I don't think this is as much > > > of a > > > corner case > > > as it at first appears. Targets will have code very similar to that > > > needed to handle out of order commands to deal with digest errors. > > > Targets also need to queue commands whilst waiting for both solicited > > > and unsolicited data to arrive. Queuing out of order commands seems > > > little extra work. > > > > > > From an initiators point of view there are > > > efficiency, > > > and probably > > > performance gains to be had from sending commands out of order. Bob > > > Russell gave the example of a read being sent whilst write data DMA is > > > happening, and a similar situation can arise with DMA for writes > > > overtaking that of earlier writes if the initiator has multiple DMA > > > engines. In this case the initiator might be forced to let the wire go > > > idle if it can't send the data from completed DMAs as soon as > > > possible. > > > > > > We already have a command queue at the target to > > > enforce > > > correct > > > serialisation of commands, doing the same thing at the initiator is > > > redundant. > > > > > > Finally, I don't believe we should be writing a > > > standard > > > to work > > > around poor coding and test coverage, especially at the cost of > > > potential efficiency gains. > > > > > > I agree with Dave and Santosh that commands being > > > sent > > > out of order > > > on a single session should be allowed by the standard. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Barry Reinhold > > > Sent: Friday, November 02, 2001 5:24 PM > > > To: Dave Sheehy; IETF IP SAN Reflector > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Using features such as out of order command delivery on a connection > > > tend to > > > be the sort of things that lead to interoperability problems. It is > > > unexpected and probably going to hit poorly tested code paths even if > > > the > > > standard is written to allow it. > > > > > > >-----Original Message----- > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > >Dave Sheehy > > > >Sent: Friday, November 02, 2001 4:19 PM > > > >To: IETF IP SAN Reflector > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > >> > > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > > on > > > >> page 25 of draft 8, which says: > > > >> "Except for the commands marked for immediate delivery the > > > iSCSI > > > >> target layer MUST eliver the commands for execution in the > > > order > > > >> specified by CmdSN." > > > >> > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > on > > > >> each command shipped except for commands marked for immediate > > > >> delivery." > > > >> but the meaning of the term "shipped" is vague, and does not > > > >> necessarily > > > >> require that the PDUs arrive on the other end of a TCP > > > connection > > > >> in the same order that the CmdSN values were assigned to these > > > PDUs. > > > >> > > > >> Some initiators have been designed to send commands out of CmdSN > > > >> order on one connection. Consider the situation where there is > > > only > > > >> one connection and a high-level dispatcher creates a PDU for a > > > SCSI > > > >> command that involves writing immediate data to the target. > > > This PDU > > > >> is enqueued to a lower-level layer which has to setup, start, > > > and > > > >> wait-for a DMA operation to move the immediate data into an > > > onboard > > > >> buffer before the PDU can be put onto the wire. While this is > > > >> happening, the dispatcher creates another unrelated PDU for a > > > SCSI > > > >> read command (for example), and when this PDU is passed to the > > > >> lower-level layer it can be sent immediately, ahead of the > > > previous > > > >> write PDU and therefore out of order on this connection. > > > >> > > > >> The standard clearly allows this to happen if the two PDUs were > > > sent > > > >> on different connections, and seems to imply that this can also > > > happen > > > >> when the two PDUs are sent on the same connection. > > > >> > > > >> The suggestion is to put in the standard an explicit statement > > > that > > > >> this is allowed or not allowed, as appropriate. > > > >> > > > >> If this is allowed, such a statement would avoid the erroneous > > > >> assumption being made by some target implementers that within a > > > single > > > >> connection, commands will arrive in order. > > > >> > > > >> If this is not allowed, such a statement would avoid the > > > erroneous > > > >> assumption being made by some initiator implementers that within > > > a > > > >> single connection, commands can be put on the wire out of order. > > > >> > > > >> +++ > > > >> > > > >> will add an explicit statement saying that this behaviour is > > > forbidden. > > > >> 2.2.2.1 will contain: > > > >> > > > >> On any given connection, the iSCSI initiator MUST send the > > > >commands in the > > > >> order specified by CmdSN. > > > >> > > > >> +++ > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > have to > > > >order commands across the session. I don't see why it's a problem to > > > extend > > > >that to the connection as well. I, for one, believe we should take > > > >a liberal > > > >stance on this. > > > > > > > >Dave Sheehy > > > > > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## > From owner-ips@ece.cmu.edu Wed Nov 7 15:15:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA22526 for ; Wed, 7 Nov 2001 15:15:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7J8le25516 for ips-outgoing; Wed, 7 Nov 2001 14:08:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (gordanmail.gadzoox.com [216.52.31.45]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7J8jl25508 for ; Wed, 7 Nov 2001 14:08:45 -0500 (EST) Received: by gordan.pl.gadzoox.com with Internet Mail Service (5.5.2653.19) id ; Wed, 7 Nov 2001 11:08:39 -0800 Message-ID: From: Vi Chau To: "'ENDL_TX@computer.org'" , IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Date: Wed, 7 Nov 2001 11:08:38 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, The first FCIP draft included the entire SOF/EOF table from FC-BB. It was during the Orlando meeting when we decided to drop class 1 support because of a lack of applications that use class 1 and thus the table in FCIP was trimmed. I agree that listing all codes from FC-BB is the right thing to do. Vi -----Original Message----- From: Ralph Weber [mailto:ralphoweber@compuserve.com] Sent: Wednesday, November 07, 2001 5:16 AM To: IPS Reflector Cc: Murali Rajagopal; Franco Travostino; Rodriguez, Elizabeth Subject: FCencap: List ALL SOF/EOF codes Upon further reflection, I think the right thing to do is to list all the SOF/EOF codes defined in FC-BB in the FC Encapsulation draft. FIRST There is nothing in the FC Encapsulation draft other than to omission of Class 1 SOF/EOF codes that prevents encapsulating FC Class 1 frames for TCP transport. Sure, a TCP ULP that is smarter than anything anybody has thought about will be required to do it. BUT there is (or should be) nothing the the FC Encapsulation draft that prevents such a protocol from being invented. AND the FC Encapsulation draft specifically says that you need the wisdom of some other protocol document in order to get any use out of the FC Encapsulation draft. Why force the mad man that devises a way to transport Class 1 over TCP/IP to revise the FC Encapsulation SOF/EOF tables? SECOND It is conceivable that a future version of iFCP (or maybe even FCIP) might want to support Class 4. Again, there is nothing in the FC Encapsulation draft that prevents this, except the omission of the SOF/EOF codes. FINALLY I believe that the elimination of all SOF/EOF codes other than Class 2, Class 3, AND CLASS F is a hold over from the early FCIP work, before the FC Encapsulation was split into a separate draft. I believe that decision was right for FCIP but wrong for an FC Encapsulation intended to be used by ALL FC protocols running over TCP/IP. Thanks for your consideration. Ralph... From owner-ips@ece.cmu.edu Wed Nov 7 15:44:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA23170 for ; Wed, 7 Nov 2001 15:44:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7K1Nf29748 for ips-outgoing; Wed, 7 Nov 2001 15:01:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7K1Ll29739 for ; Wed, 7 Nov 2001 15:01:21 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id MAA15667; Wed, 7 Nov 2001 12:01:06 -0800 (PST) From: "Murali Rajagopal" To: "Vi Chau" , , "IPS Reflector" Subject: RE: FCencap: List ALL SOF/EOF codes Date: Wed, 7 Nov 2001 12:02:12 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Vi: I also recollect that there was some discussion about some illegal codes in that table either in that meeting or in one of the ensuing T11 meetings. -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Vi Chau Sent: Wednesday, November 07, 2001 11:09 AM To: 'ENDL_TX@computer.org'; IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Ralph, The first FCIP draft included the entire SOF/EOF table from FC-BB. It was during the Orlando meeting when we decided to drop class 1 support because of a lack of applications that use class 1 and thus the table in FCIP was trimmed. I agree that listing all codes from FC-BB is the right thing to do. Vi -----Original Message----- From: Ralph Weber [mailto:ralphoweber@compuserve.com] Sent: Wednesday, November 07, 2001 5:16 AM To: IPS Reflector Cc: Murali Rajagopal; Franco Travostino; Rodriguez, Elizabeth Subject: FCencap: List ALL SOF/EOF codes Upon further reflection, I think the right thing to do is to list all the SOF/EOF codes defined in FC-BB in the FC Encapsulation draft. FIRST There is nothing in the FC Encapsulation draft other than to omission of Class 1 SOF/EOF codes that prevents encapsulating FC Class 1 frames for TCP transport. Sure, a TCP ULP that is smarter than anything anybody has thought about will be required to do it. BUT there is (or should be) nothing the the FC Encapsulation draft that prevents such a protocol from being invented. AND the FC Encapsulation draft specifically says that you need the wisdom of some other protocol document in order to get any use out of the FC Encapsulation draft. Why force the mad man that devises a way to transport Class 1 over TCP/IP to revise the FC Encapsulation SOF/EOF tables? SECOND It is conceivable that a future version of iFCP (or maybe even FCIP) might want to support Class 4. Again, there is nothing in the FC Encapsulation draft that prevents this, except the omission of the SOF/EOF codes. FINALLY I believe that the elimination of all SOF/EOF codes other than Class 2, Class 3, AND CLASS F is a hold over from the early FCIP work, before the FC Encapsulation was split into a separate draft. I believe that decision was right for FCIP but wrong for an FC Encapsulation intended to be used by ALL FC protocols running over TCP/IP. Thanks for your consideration. Ralph... From owner-ips@ece.cmu.edu Wed Nov 7 15:49:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA23257 for ; Wed, 7 Nov 2001 15:49:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7JNkr26668 for ips-outgoing; Wed, 7 Nov 2001 14:23:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from philotas.hosting.pacbell.net (philotas.hosting.pacbell.net [216.100.99.24]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7JNil26663 for ; Wed, 7 Nov 2001 14:23:44 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by philotas.hosting.pacbell.net id OAA04484; Wed, 7 Nov 2001 14:23:25 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 11:20:39 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I think we should either have it as a MUST or not require it (at both ends to get the real benefit). SHOULD is one of those things that leads to implementation burden and confusion, without perhaps the feature being used. There are implementation as well as protocol considerations mixed in here. If we are to remove the restriction, we should (SHOULD) get the maximum benefit from it, rather than to accomodate an implementation choice. Out of sequence commands, combined with the possibility of digest errors, will add substantial complexity on the target side, without corrosponding benefit in performance. If we change this to SHOULD, we should also relax the requirement to present commands on the target side to a SHOULD. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Wednesday, November 07, 2001 10:00 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Out of order commands > > > Mallikarjun, > > I did not see a SINGLE performance improvement that results from OOO > shipping. > I would be bad engineering to give away the "no-deadlock" mechanism we > have now for nothing. > I have also the impression that the point about deadlock that I keep > repeating is ignored or not understood. > As we stand today commands can be shipped with Immediate data or without > and an implementer determined > to squeeze maximum bandwidth and overlap command start with delivery will > choose not to work with immediate data > (as you have pointed out) while a low performance software implementation > will use immediate data to minimize CPU cycles consumed. However both > will be guaranteed to work without deadlock as source and sink use the > same ordering. > Recovery is still a low probability event and should be handled with a > different set of considerations in mind. > As for the strictness of the recommendation - yes we could settle on > SHOULD. > > Julo > > > > > "Mallikarjun C." > Sent by: owner-ips@ece.cmu.edu > 07-11-01 19:41 > Please respond to cbm > > > To: Santosh Rao , ips@ece.cmu.edu > cc: > Subject: Re: iSCSI: Out of order commands > > > > Santosh, > > I have only one comment on your responses. > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > I made two convenient assumptions if you noticed, :-), one of which > is that target forces session recovery on *any* error that it sees > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > error. With that assumption, a target can afford not to implement > a scoreboard. > > As I said in a private note, I guess what primarily bothers me about > OOO commands on a connection is that it requires the receiver to > undo this "optimization" on its end - most notably on a single > connection. TCP experts may comment on how/if they dealt with a > similar issue. > > OTOH, you had some valid comments on exceptions to ordering during > connection recovery. Perhaps we can move on by making Julian's > proposed stipulation a SHOULD.... > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > Santosh Rao wrote: > > > > Mallikarjun, > > > > Some comments below. > > > > Regards, > > Santosh > > > > "Mallikarjun C." wrote: > > > > > > Rod and Julian, > > > > > > This has been an interesting thread of discussion. Some > > > comments - > > > > > > 1.My first reaction was - allowing out-of-order command > > > transmission on the same connection deprives targets of > > > an implementation choice. Targets which support only > > > single-connection sessions and only support session > > > recovery (reasonable assumptions in my mind) can no > > > longer afford *not to* implement a command scoreboard. > > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > > > Thus, there is no getting away from implementing a sequencer at the > > target. Given this, I think it is unreasonable to restrict initiator > > implementation flexibility by imposing a strict ordering requirement > > within the connection. > > > > > 2.Any end-node efficiency that is sought to be achieved > > > by transmitting CmdSNs out-of-order from the initiator > > > would be lost on the other end-node, since the target > > > now must wait for re-ordering the commands. > > > > It has to handle this situation anyway to deal with holes caused by > > digest errors. This scenario occurs even with initiators that issue > > commands in order. > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > link badwidth (albeit at the expense of end-node efficiency), > > > compared to idling the link waiting for outbound DMA. > > > We have to determine if this is a reasonable trade-off. > > > > > > 4.I can see Rod's point that prefetching all immediate > > > data can be a burden on the NIC resources. But, two > > > questions - > > > - could the NIC not use unsolicited separate data > > > PDUs in these cases? [ I realize that InitialR2T > > > has to be "no" to let it happen... ] > > > - could the NIC have a memory architecture that > > > allows data prefetching for the next command (so > > > this is a non-issue from the protocol perspective)? > > > This scheme incurs one DMA delay for every new > > > burst of commands. > > > > > > 5.Another (perhaps radical at this point) option is to do > > > away with immediate unsolicited data, to stick only with > > > separate unsolicited data. I would personally be okay > > > with the choice, particularly if this feature (that > > > helps software implementations) starts making hardware > > > design complicated/expensive. > > > > > > So, to summarize - > > > > > > option immediate allow > > > data in spec? out-of-order? > > > > > > (A) (5) above no no > > > (B) No real reason to do this. no yes > > > (C) (4) above yes no > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > >From the arguments I heard so far, I am leaning towards > > > option A, and option C in that order. > > > > > > Comments? > > > -- > > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > Rod Harrison wrote: > > > > > > > > Julian, > > > > > > > > I don't understand what you are proposing here, what do you > mean by > > > > "multiplexed" DMA? > > > > > > > > The problem is that the DMAs take some time, the more there > are > > > > queued the longer the last DMAs queued take to complete. Some > commands > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > soon a possible, which for a read after a write can be before the > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > to be sent on the HBA is expensive and redundant since the target > > > > already knows how to queue commands before committing them to its > SCSI > > > > layer. > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > change the > > > > order of commands from the OS, but the DMAs those commands need are > > > > unlikely to complete in the same order, and as I mentioned some > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > order it has to maintain an ordered queue of commands waiting to be > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > immediate data almost impossible to support. > > > > > > > > I don't see the problem with allowing out of order commands > given > > > > that the target already has to deal with very similar problems. I > > > > think we are getting in to the area of implementation choices here, > > > > which is inappropriate for a specification. > > > > > > > > - Rod > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > Julian Satran > > > > Sent: Monday, November 05, 2001 10:06 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > > > Rod, > > > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > > commands. > > > > If you have scheduled a long outbound DMA you are doomed regardless > of > > > > the > > > > command ordering. > > > > And if you have scheduled DMA operations piecemeal then you can > insert > > > > your commands in correct order. > > > > > > > > Julo > > > > > > > > "Rod Harrison" > > > > 05-11-01 20:48 > > > > Please respond to "Rod Harrison" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > cc: > > > > Subject: iSCSI: Out of order commands, was current > UNH > > > > Plugfest > > > > > > > > [ Subject changed ] > > > > > > > > Julian, > > > > > > > > The ordering difference is introduced between the > > > > host > > > > side driver > > > > and the iSCSI HBA. The host side driver must present SCSI commands > to > > > > the HBA in the order they are received from the OS to prevent read > > > > after write dependency failures. The HBA might reorder the commands > > > > depending on when DMA completes. The reordering can't be done ahead > of > > > > time in the host driver since it doesn't know how long each DMA > might > > > > take. As long as the HBA assigns CmdSN in the order it receives > > > > commands the desired host ordering is preserved. > > > > > > > > - Rod > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > Julian Satran > > > > Sent: Monday, November 05, 2001 12:35 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > Rod, > > > > > > > > I all examples give the point I find hard to understand is why is > the > > > > ordering on the wire different from the presentation order to the > > > > initiator. You can get as many overlaps as you want by presenting > the > > > > commands to the initiator in the desired order. > > > > What we are considering here is the case in which you want to ship > in > > > > an > > > > order different than the one you present the commands. > > > > > > > > Julo > > > > > > > > "Rod Harrison" > > > > Sent by: owner-ips@ece.cmu.edu > > > > 04-11-01 04:42 > > > > Please respond to "Rod Harrison" > > > > > > > > To: "Barry Reinhold" , "Dave > > > > Sheehy" > > > > , "IETF IP SAN Reflector" > > > > > > > > cc: > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > Barry, > > > > > > > > In general I agree but I don't think this is as > much > > > > of a > > > > corner case > > > > as it at first appears. Targets will have code very similar to that > > > > needed to handle out of order commands to deal with digest errors. > > > > Targets also need to queue commands whilst waiting for both > solicited > > > > and unsolicited data to arrive. Queuing out of order commands seems > > > > little extra work. > > > > > > > > From an initiators point of view there are > > > > efficiency, > > > > and probably > > > > performance gains to be had from sending commands out of order. Bob > > > > Russell gave the example of a read being sent whilst write data DMA > is > > > > happening, and a similar situation can arise with DMA for writes > > > > overtaking that of earlier writes if the initiator has multiple DMA > > > > engines. In this case the initiator might be forced to let the wire > go > > > > idle if it can't send the data from completed DMAs as soon as > > > > possible. > > > > > > > > We already have a command queue at the target to > > > > enforce > > > > correct > > > > serialisation of commands, doing the same thing at the initiator is > > > > redundant. > > > > > > > > Finally, I don't believe we should be writing a > > > > standard > > > > to work > > > > around poor coding and test coverage, especially at the cost of > > > > potential efficiency gains. > > > > > > > > I agree with Dave and Santosh that commands being > > > > sent > > > > out of order > > > > on a single session should be allowed by the standard. > > > > > > > > - Rod > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > Barry Reinhold > > > > Sent: Friday, November 02, 2001 5:24 PM > > > > To: Dave Sheehy; IETF IP SAN Reflector > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > Using features such as out of order command delivery on a connection > > > > tend to > > > > be the sort of things that lead to interoperability problems. It is > > > > unexpected and probably going to hit poorly tested code paths even > if > > > > the > > > > standard is written to allow it. > > > > > > > > >-----Original Message----- > > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > Of > > > > >Dave Sheehy > > > > >Sent: Friday, November 02, 2001 4:19 PM > > > > >To: IETF IP SAN Reflector > > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > > >> > > > > >> The behavior of targets is clearly specified in Section > 2.2.2.3 > > > > on > > > > >> page 25 of draft 8, which says: > > > > >> "Except for the commands marked for immediate delivery the > > > > iSCSI > > > > >> target layer MUST eliver the commands for execution in the > > > > order > > > > >> specified by CmdSN." > > > > >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > > on > > > > >> each command shipped except for commands marked for > immediate > > > > >> delivery." > > > > >> but the meaning of the term "shipped" is vague, and does not > > > > >> necessarily > > > > >> require that the PDUs arrive on the other end of a TCP > > > > connection > > > > >> in the same order that the CmdSN values were assigned to these > > > > PDUs. > > > > >> > > > > >> Some initiators have been designed to send commands out of > CmdSN > > > > >> order on one connection. Consider the situation where there > is > > > > only > > > > >> one connection and a high-level dispatcher creates a PDU for a > > > > SCSI > > > > >> command that involves writing immediate data to the target. > > > > This PDU > > > > >> is enqueued to a lower-level layer which has to setup, start, > > > > and > > > > >> wait-for a DMA operation to move the immediate data into an > > > > onboard > > > > >> buffer before the PDU can be put onto the wire. While this is > > > > >> happening, the dispatcher creates another unrelated PDU for a > > > > SCSI > > > > >> read command (for example), and when this PDU is passed to the > > > > >> lower-level layer it can be sent immediately, ahead of the > > > > previous > > > > >> write PDU and therefore out of order on this connection. > > > > >> > > > > >> The standard clearly allows this to happen if the two PDUs > were > > > > sent > > > > >> on different connections, and seems to imply that this can > also > > > > happen > > > > >> when the two PDUs are sent on the same connection. > > > > >> > > > > >> The suggestion is to put in the standard an explicit statement > > > > that > > > > >> this is allowed or not allowed, as appropriate. > > > > >> > > > > >> If this is allowed, such a statement would avoid the erroneous > > > > >> assumption being made by some target implementers that within > a > > > > single > > > > >> connection, commands will arrive in order. > > > > >> > > > > >> If this is not allowed, such a statement would avoid the > > > > erroneous > > > > >> assumption being made by some initiator implementers that > within > > > > a > > > > >> single connection, commands can be put on the wire out of > order. > > > > >> > > > > >> +++ > > > > >> > > > > >> will add an explicit statement saying that this behaviour is > > > > forbidden. > > > > >> 2.2.2.1 will contain: > > > > >> > > > > >> On any given connection, the iSCSI initiator MUST send the > > > > >commands in the > > > > >> order specified by CmdSN. > > > > >> > > > > >> +++ > > > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > > have to > > > > >order commands across the session. I don't see why it's a problem > to > > > > extend > > > > >that to the connection as well. I, for one, believe we should take > > > > >a liberal > > > > >stance on this. > > > > > > > > > >Dave Sheehy > > > > > > > > > -- > > ################################## > > Santosh Rao > > Software Design Engineer, > > HP-UX iSCSI Driver Team, > > Hewlett Packard, Cupertino. > > email : santoshr@cup.hp.com > > Phone : 408-447-3751 > > ################################## > > > > From owner-ips@ece.cmu.edu Wed Nov 7 16:57:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA24516 for ; Wed, 7 Nov 2001 16:57:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7L3Bo04714 for ips-outgoing; Wed, 7 Nov 2001 16:03:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7L39l04709 for ; Wed, 7 Nov 2001 16:03:10 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id NAA04316; Wed, 7 Nov 2001 13:03:04 -0800 (PST) From: Keith McCloghrie Message-Id: <200111072103.NAA04316@cisco.com> Subject: FC Management MIB - issues To: ips@ece.cmu.edu Date: Wed, 7 Nov 2001 13:03:04 -0800 (PST) Cc: kzm@cisco.com (Keith McCloghrie) X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi, As indicated by Elizabeth's message (below), I've agreed to be the interim editor for the FC Management MIB. So, first, I'd like to list a set of issues that have been raised concerning the most recent draft: draft-ietf-ipfc-fcmgmt-int-mib-07.txt. They are: 1. The use of SMIv2 is mandatory. 2. This MIB seems to have been defined with the notion that it will be the only MIB that a Fibre Channel product will require. The notion of an agent implementing just a single MIB was abandoned by the IETF in 1992 as being non-scaleable. Rather, this is another MIB in the continuing series of MIBs defined by the IETF, and thus, it needs to be consistent with its predecessors. In other words, there are existing MIBs which all SNMP agents must support, even if the support of Fibre Channel interfaces is the only functionality that they have. Thus, as the Fibre Channel Integration MIB, it is essential that this MIB contain only objects for information which is specific to Fibre Channel. All objects which apply in non-Fibre Channel environments need to be removed. (This applies even it were to require the definition of other new MIBs to contain information which is not already defined in other existing MIBs, but needed by Fibre Channel products.) Note that this issue applies to a large fraction of the objects defined in this MIB. 3. The text needs to include an explanation of the relationship between this MIB and RFC 2837. Is it a replacement or is it complementary ? If complementary, which agents implement which MIB; do some agents implement both ? 4. Every SNMP agent implements the ifTable. The ifTable counters are undoubtedly the MIB objects most well-used by administrators in SNMP management. SNMP agents need to implement a row in the ifTable for each of their network interfaces, including their Fibre Channel interfaces. The IF-MIB requires a media-specific MIB to specify how that type of interface uses the ifTable (see section 4 in RFC 2863). RFC 2837 doesn't do that, and nor (currently) does this MIB. That needs to be fixed. This will likely result in many tables in this MIB being indexed by ifIndex. 5. There are a number of objects related to the "Simple Name Service", and the definitions refer to Fibre Channel's GS-3 spec. However, GS-3 defines two Name Services: a Zoned Name Service and a Unzoned Name Service, but GS-3 does NOT use the term "Simple" for either of them !! This ambiguity needs to be resolved. 6. It is essential that the Counter32 or Counter64 (not OCTET STRINGs) syntax be used for counters. Thanks, Keith. -------------- Forwarded message: > From: "Elizabeth Rodriguez" > To: , > "IPS Mailing List (E-mail)" > Subject: FC Management MIB -- Transfer from IPFC To IPS WG > Cc: , , , > , , , > , > "Elizabeth Rodriguez" , > > > All, > > The IPFC working group has only one item left on its charter. > It is that of the FC Management MIB. It has been identified > that this MIB does not follow the IETF guidelines for MIBs > in its current format, in its lack of focus, and in its overlap > with existing MIBs. Rework of this MIB will take some time. > The content of this MIB will fit in well with the IPS WG, > especially because the subject matter experts participate in > this WG. > > For these reasons, the working group chairs, with the INT and TSV area > directors, have determined that this effort should be moved from the > IPFC working group to the IPS working group. Upon transferring of this > work, the IPFC working group will have completed the items in its > charter and the IPFC WG will be closed. > > The IPS working group has a technical advisor for MIB work -- Keith > McCloghrie. Since it has been determined that the current MIB has > issues with format, Keith McCloghrie has agreed to become the interim > editor of this MIB. As part of the re-architecture, the MIB will be > evaluated with respect to fit with other IPS WG MIBs, and may take the > form of a single new MIB or multiple MIBs, as appropriate. The IPS > working group will also be seeking Fibre Channel expertise to help > formulate the new MIB, including an editor with FC and MIB experience. > > The IPFC working group would also like to thank Steven Blumenau for all > his work on the original MIB. > > Elizabeth Rodriguez & David Black, IPS Working Group Chairs > Murali Rajagopal, IPFC Working Group Chair > From owner-ips@ece.cmu.edu Wed Nov 7 17:06:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA24733 for ; Wed, 7 Nov 2001 17:06:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7LR9206422 for ips-outgoing; Wed, 7 Nov 2001 16:27:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7LQpl06396 for ; Wed, 7 Nov 2001 16:26:52 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel10.hp.com (Postfix) with ESMTP id 5A1321F8C7; Wed, 7 Nov 2001 13:26:45 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id NAA09152; Wed, 7 Nov 2001 13:28:12 -0800 (PST) Message-Id: <200111072128.NAA09152@core.rose.hp.com> Date: Wed, 07 Nov 2001 13:39:05 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: somesh_gupta@silverbacksystems.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh, I see a SHOULD encouraging efficient implementations in this case, and I think it's desirable to do that. As you yourself argued in your last email, this allows implementations to optimize for the most-likely case. IMHO, a MUST here would be going too far (connection recovery scenario pointed out by Santosh) unless we start special casing error recovery cases/multi-connection cases. In effect, that again comes down to a SHOULD for a general n-connection session case. Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Somesh Gupta wrote: > > I think we should either have it as a MUST or not require > it (at both ends to get the real benefit). SHOULD is one > of those things that leads to implementation > burden and confusion, without perhaps the feature being > used. There are implementation as well as protocol > considerations mixed in here. > > If we are to remove the restriction, we should (SHOULD) > get the maximum benefit from it, rather than to > accomodate an implementation choice. Out of sequence > commands, combined with the possibility of digest errors, > will add substantial complexity on the target side, > without corrosponding benefit in performance. If we change > this to SHOULD, we should also relax the requirement > to present commands on the target side to a SHOULD. > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Wednesday, November 07, 2001 10:00 AM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Out of order commands > > > > > > Mallikarjun, > > > > I did not see a SINGLE performance improvement that results from OOO > > shipping. > > I would be bad engineering to give away the "no-deadlock" mechanism we > > have now for nothing. > > I have also the impression that the point about deadlock that I keep > > repeating is ignored or not understood. > > As we stand today commands can be shipped with Immediate data or without > > and an implementer determined > > to squeeze maximum bandwidth and overlap command start with delivery will > > choose not to work with immediate data > > (as you have pointed out) while a low performance software implementation > > will use immediate data to minimize CPU cycles consumed. However both > > will be guaranteed to work without deadlock as source and sink use the > > same ordering. > > Recovery is still a low probability event and should be handled with a > > different set of considerations in mind. > > As for the strictness of the recommendation - yes we could settle on > > SHOULD. > > > > Julo > > > > > > > > > > "Mallikarjun C." > > Sent by: owner-ips@ece.cmu.edu > > 07-11-01 19:41 > > Please respond to cbm > > > > > > To: Santosh Rao , ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: Out of order commands > > > > > > > > Santosh, > > > > I have only one comment on your responses. > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > I made two convenient assumptions if you noticed, :-), one of which > > is that target forces session recovery on *any* error that it sees > > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > > error. With that assumption, a target can afford not to implement > > a scoreboard. > > > > As I said in a private note, I guess what primarily bothers me about > > OOO commands on a connection is that it requires the receiver to > > undo this "optimization" on its end - most notably on a single > > connection. TCP experts may comment on how/if they dealt with a > > similar issue. > > > > OTOH, you had some valid comments on exceptions to ordering during > > connection recovery. Perhaps we can move on by making Julian's > > proposed stipulation a SHOULD.... > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > Santosh Rao wrote: > > > > > > Mallikarjun, > > > > > > Some comments below. > > > > > > Regards, > > > Santosh > > > > > > "Mallikarjun C." wrote: > > > > > > > > Rod and Julian, > > > > > > > > This has been an interesting thread of discussion. Some > > > > comments - > > > > > > > > 1.My first reaction was - allowing out-of-order command > > > > transmission on the same connection deprives targets of > > > > an implementation choice. Targets which support only > > > > single-connection sessions and only support session > > > > recovery (reasonable assumptions in my mind) can no > > > > longer afford *not to* implement a command scoreboard. > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > > > Thus, there is no getting away from implementing a sequencer at the > > > target. Given this, I think it is unreasonable to restrict initiator > > > implementation flexibility by imposing a strict ordering requirement > > > within the connection. > > > > > > > 2.Any end-node efficiency that is sought to be achieved > > > > by transmitting CmdSNs out-of-order from the initiator > > > > would be lost on the other end-node, since the target > > > > now must wait for re-ordering the commands. > > > > > > It has to handle this situation anyway to deal with holes caused by > > > digest errors. This scenario occurs even with initiators that issue > > > commands in order. > > > > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > > link badwidth (albeit at the expense of end-node efficiency), > > > > compared to idling the link waiting for outbound DMA. > > > > We have to determine if this is a reasonable trade-off. > > > > > > > > 4.I can see Rod's point that prefetching all immediate > > > > data can be a burden on the NIC resources. But, two > > > > questions - > > > > - could the NIC not use unsolicited separate data > > > > PDUs in these cases? [ I realize that InitialR2T > > > > has to be "no" to let it happen... ] > > > > - could the NIC have a memory architecture that > > > > allows data prefetching for the next command (so > > > > this is a non-issue from the protocol perspective)? > > > > This scheme incurs one DMA delay for every new > > > > burst of commands. > > > > > > > > 5.Another (perhaps radical at this point) option is to do > > > > away with immediate unsolicited data, to stick only with > > > > separate unsolicited data. I would personally be okay > > > > with the choice, particularly if this feature (that > > > > helps software implementations) starts making hardware > > > > design complicated/expensive. > > > > > > > > So, to summarize - > > > > > > > > option immediate allow > > > > data in spec? out-of-order? > > > > > > > > (A) (5) above no no > > > > (B) No real reason to do this. no yes > > > > (C) (4) above yes no > > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > > > >From the arguments I heard so far, I am leaning towards > > > > option A, and option C in that order. > > > > > > > > Comments? > > > > -- > > > > Mallikarjun > > > > > > > > Mallikarjun Chadalapaka > > > > Networked Storage Architecture > > > > Network Storage Solutions Organization > > > > MS 5668 Hewlett-Packard, Roseville. > > > > cbm@rose.hp.com > > > > > > > > Rod Harrison wrote: > > > > > > > > > > Julian, > > > > > > > > > > I don't understand what you are proposing here, what do you > > mean by > > > > > "multiplexed" DMA? > > > > > > > > > > The problem is that the DMAs take some time, the more there > > are > > > > > queued the longer the last DMAs queued take to complete. Some > > commands > > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > > soon a possible, which for a read after a write can be before the > > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > > to be sent on the HBA is expensive and redundant since the target > > > > > already knows how to queue commands before committing them to its > > SCSI > > > > > layer. > > > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > > change the > > > > > order of commands from the OS, but the DMAs those commands need are > > > > > unlikely to complete in the same order, and as I mentioned some > > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > > order it has to maintain an ordered queue of commands waiting to be > > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > > immediate data almost impossible to support. > > > > > > > > > > I don't see the problem with allowing out of order commands > > given > > > > > that the target already has to deal with very similar problems. I > > > > > think we are getting in to the area of implementation choices here, > > > > > which is inappropriate for a specification. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Julian Satran > > > > > Sent: Monday, November 05, 2001 10:06 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > > > > > Rod, > > > > > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > > > commands. > > > > > If you have scheduled a long outbound DMA you are doomed regardless > > of > > > > > the > > > > > command ordering. > > > > > And if you have scheduled DMA operations piecemeal then you can > > insert > > > > > your commands in correct order. > > > > > > > > > > Julo > > > > > > > > > > "Rod Harrison" > > > > > 05-11-01 20:48 > > > > > Please respond to "Rod Harrison" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > > cc: > > > > > Subject: iSCSI: Out of order commands, was current > > UNH > > > > > Plugfest > > > > > > > > > > [ Subject changed ] > > > > > > > > > > Julian, > > > > > > > > > > The ordering difference is introduced between the > > > > > host > > > > > side driver > > > > > and the iSCSI HBA. The host side driver must present SCSI commands > > to > > > > > the HBA in the order they are received from the OS to prevent read > > > > > after write dependency failures. The HBA might reorder the commands > > > > > depending on when DMA completes. The reordering can't be done ahead > > of > > > > > time in the host driver since it doesn't know how long each DMA > > might > > > > > take. As long as the HBA assigns CmdSN in the order it receives > > > > > commands the desired host ordering is preserved. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Julian Satran > > > > > Sent: Monday, November 05, 2001 12:35 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Rod, > > > > > > > > > > I all examples give the point I find hard to understand is why is > > the > > > > > ordering on the wire different from the presentation order to the > > > > > initiator. You can get as many overlaps as you want by presenting > > the > > > > > commands to the initiator in the desired order. > > > > > What we are considering here is the case in which you want to ship > > in > > > > > an > > > > > order different than the one you present the commands. > > > > > > > > > > Julo > > > > > > > > > > "Rod Harrison" > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > 04-11-01 04:42 > > > > > Please respond to "Rod Harrison" > > > > > > > > > > To: "Barry Reinhold" , "Dave > > > > > Sheehy" > > > > > , "IETF IP SAN Reflector" > > > > > > > > > > cc: > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Barry, > > > > > > > > > > In general I agree but I don't think this is as > > much > > > > > of a > > > > > corner case > > > > > as it at first appears. Targets will have code very similar to that > > > > > needed to handle out of order commands to deal with digest errors. > > > > > Targets also need to queue commands whilst waiting for both > > solicited > > > > > and unsolicited data to arrive. Queuing out of order commands seems > > > > > little extra work. > > > > > > > > > > From an initiators point of view there are > > > > > efficiency, > > > > > and probably > > > > > performance gains to be had from sending commands out of order. Bob > > > > > Russell gave the example of a read being sent whilst write data DMA > > is > > > > > happening, and a similar situation can arise with DMA for writes > > > > > overtaking that of earlier writes if the initiator has multiple DMA > > > > > engines. In this case the initiator might be forced to let the wire > > go > > > > > idle if it can't send the data from completed DMAs as soon as > > > > > possible. > > > > > > > > > > We already have a command queue at the target to > > > > > enforce > > > > > correct > > > > > serialisation of commands, doing the same thing at the initiator is > > > > > redundant. > > > > > > > > > > Finally, I don't believe we should be writing a > > > > > standard > > > > > to work > > > > > around poor coding and test coverage, especially at the cost of > > > > > potential efficiency gains. > > > > > > > > > > I agree with Dave and Santosh that commands being > > > > > sent > > > > > out of order > > > > > on a single session should be allowed by the standard. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Barry Reinhold > > > > > Sent: Friday, November 02, 2001 5:24 PM > > > > > To: Dave Sheehy; IETF IP SAN Reflector > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Using features such as out of order command delivery on a connection > > > > > tend to > > > > > be the sort of things that lead to interoperability problems. It is > > > > > unexpected and probably going to hit poorly tested code paths even > > if > > > > > the > > > > > standard is written to allow it. > > > > > > > > > > >-----Original Message----- > > > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > > Of > > > > > >Dave Sheehy > > > > > >Sent: Friday, November 02, 2001 4:19 PM > > > > > >To: IETF IP SAN Reflector > > > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > > > >> > > > > > >> The behavior of targets is clearly specified in Section > > 2.2.2.3 > > > > > on > > > > > >> page 25 of draft 8, which says: > > > > > >> "Except for the commands marked for immediate delivery the > > > > > iSCSI > > > > > >> target layer MUST eliver the commands for execution in the > > > > > order > > > > > >> specified by CmdSN." > > > > > >> > > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > > > on > > > > > >> each command shipped except for commands marked for > > immediate > > > > > >> delivery." > > > > > >> but the meaning of the term "shipped" is vague, and does not > > > > > >> necessarily > > > > > >> require that the PDUs arrive on the other end of a TCP > > > > > connection > > > > > >> in the same order that the CmdSN values were assigned to these > > > > > PDUs. > > > > > >> > > > > > >> Some initiators have been designed to send commands out of > > CmdSN > > > > > >> order on one connection. Consider the situation where there > > is > > > > > only > > > > > >> one connection and a high-level dispatcher creates a PDU for a > > > > > SCSI > > > > > >> command that involves writing immediate data to the target. > > > > > This PDU > > > > > >> is enqueued to a lower-level layer which has to setup, start, > > > > > and > > > > > >> wait-for a DMA operation to move the immediate data into an > > > > > onboard > > > > > >> buffer before the PDU can be put onto the wire. While this is > > > > > >> happening, the dispatcher creates another unrelated PDU for a > > > > > SCSI > > > > > >> read command (for example), and when this PDU is passed to the > > > > > >> lower-level layer it can be sent immediately, ahead of the > > > > > previous > > > > > >> write PDU and therefore out of order on this connection. > > > > > >> > > > > > >> The standard clearly allows this to happen if the two PDUs > > were > > > > > sent > > > > > >> on different connections, and seems to imply that this can > > also > > > > > happen > > > > > >> when the two PDUs are sent on the same connection. > > > > > >> > > > > > >> The suggestion is to put in the standard an explicit statement > > > > > that > > > > > >> this is allowed or not allowed, as appropriate. > > > > > >> > > > > > >> If this is allowed, such a statement would avoid the erroneous > > > > > >> assumption being made by some target implementers that within > > a > > > > > single > > > > > >> connection, commands will arrive in order. > > > > > >> > > > > > >> If this is not allowed, such a statement would avoid the > > > > > erroneous > > > > > >> assumption being made by some initiator implementers that > > within > > > > > a > > > > > >> single connection, commands can be put on the wire out of > > order. > > > > > >> > > > > > >> +++ > > > > > >> > > > > > >> will add an explicit statement saying that this behaviour is > > > > > forbidden. > > > > > >> 2.2.2.1 will contain: > > > > > >> > > > > > >> On any given connection, the iSCSI initiator MUST send the > > > > > >commands in the > > > > > >> order specified by CmdSN. > > > > > >> > > > > > >> +++ > > > > > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > > > have to > > > > > >order commands across the session. I don't see why it's a problem > > to > > > > > extend > > > > > >that to the connection as well. I, for one, believe we should take > > > > > >a liberal > > > > > >stance on this. > > > > > > > > > > > >Dave Sheehy > > > > > > > > > > > > -- > > > ################################## > > > Santosh Rao > > > Software Design Engineer, > > > HP-UX iSCSI Driver Team, > > > Hewlett Packard, Cupertino. > > > email : santoshr@cup.hp.com > > > Phone : 408-447-3751 > > > ################################## > > > > > > > > From owner-ips@ece.cmu.edu Wed Nov 7 17:06:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA24761 for ; Wed, 7 Nov 2001 17:06:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7La0E07171 for ips-outgoing; Wed, 7 Nov 2001 16:36:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from philotas.hosting.pacbell.net (philotas.hosting.pacbell.net [216.100.99.24]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7LZvl07162 for ; Wed, 7 Nov 2001 16:35:57 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by philotas.hosting.pacbell.net id QAA04754; Wed, 7 Nov 2001 16:33:36 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "Robert D. Russell" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 13:30:48 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > -----Original Message----- > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > Sent: Wednesday, November 07, 2001 1:13 PM > To: Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Somesh, Julian: > > You state that dealing with OOO commands on the target > will add substantial complexity on the target. > Do you have any basis for that claim? My impression from the > plugfest is that most targets are already dealing with > it. Perhaps we need to hear from someone who is actually > building a target for which this would be a real problem. Since we are making related products, I can assure you that this adds complexity. Mallikarjun also added the target perspecitive. > > If anything, what we are hearing from people who really > are building initiators is that dealing with the requirement > to send commands in order will introduce substantial complexity > on the initiator. I think you heard from one (WindRiver). We don't have a problem and it does not add "substantial complexity". > > So why should we be saving complexity on (hypothetically) simple > targets yet requiring complexity on real initiators? > > As far as the deadlock issue is concerned, if the only way > that deadlock can occur with OOO commands on the same > connection is during the use of immediate data (which is I > think what Julian was saying), then shouldn't we change > the standard to just say that -- if the initiator sends > commands out of order on a single connection, then immediate > data MUST NOT be used on that connection in order to avoid deadlock. > > This gives everybody what they want, since initiators who find > it beneficial to deliver commands OOO will just negotiate > immediate data off. Those who really want to use immediate data > will have to ensure that commands are sent in order. > The tradeoff then becomes an implementation issue, not a > standards issue, which is where it belongs. An initiator that has a problem with this model can avoid sending immediate data. > > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 Bob, Sorry to be so blunt, but I say this because of your insinuations here. The only one in the debate not building a product seems to be you. But I do not disagree with your right to add meaningful points as you have done in the past. Somesh > > > On Wed, 7 Nov 2001, Somesh Gupta wrote: > > > I think we should either have it as a MUST or not require > > it (at both ends to get the real benefit). SHOULD is one > > of those things that leads to implementation > > burden and confusion, without perhaps the feature being > > used. There are implementation as well as protocol > > considerations mixed in here. > > > > If we are to remove the restriction, we should (SHOULD) > > get the maximum benefit from it, rather than to > > accomodate an implementation choice. Out of sequence > > commands, combined with the possibility of digest errors, > > will add substantial complexity on the target side, > > without corrosponding benefit in performance. If we change > > this to SHOULD, we should also relax the requirement > > to present commands on the target side to a SHOULD. > > > > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Wednesday, November 07, 2001 10:00 AM > > > To: ips@ece.cmu.edu > > > Subject: Re: iSCSI: Out of order commands > > > > > > > > > Mallikarjun, > > > > > > I did not see a SINGLE performance improvement that results from OOO > > > shipping. > > > I would be bad engineering to give away the "no-deadlock" mechanism we > > > have now for nothing. > > > I have also the impression that the point about deadlock that I keep > > > repeating is ignored or not understood. > > > As we stand today commands can be shipped with Immediate data > or without > > > and an implementer determined > > > to squeeze maximum bandwidth and overlap command start with > delivery will > > > choose not to work with immediate data > > > (as you have pointed out) while a low performance software > implementation > > > will use immediate data to minimize CPU cycles consumed. However both > > > will be guaranteed to work without deadlock as source and sink use the > > > same ordering. > > > Recovery is still a low probability event and should be handled with a > > > different set of considerations in mind. > > > As for the strictness of the recommendation - yes we could settle on > > > SHOULD. > > > > > > Julo > > > > > > > > > > > > > > > "Mallikarjun C." > > > Sent by: owner-ips@ece.cmu.edu > > > 07-11-01 19:41 > > > Please respond to cbm > > > > > > > > > To: Santosh Rao , ips@ece.cmu.edu > > > cc: > > > Subject: Re: iSCSI: Out of order commands > > > > > > > > > > > > Santosh, > > > > > > I have only one comment on your responses. > > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > > reason being that it can see out-of-order arrival of commands due to > > > > commands being dropped on digest errors. In such a case, it > must block > > > > further command processing until holes are filled. > > > > > > I made two convenient assumptions if you noticed, :-), one of which > > > is that target forces session recovery on *any* error that it sees > > > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > > > error. With that assumption, a target can afford not to implement > > > a scoreboard. > > > > > > As I said in a private note, I guess what primarily bothers me about > > > OOO commands on a connection is that it requires the receiver to > > > undo this "optimization" on its end - most notably on a single > > > connection. TCP experts may comment on how/if they dealt with a > > > similar issue. > > > > > > OTOH, you had some valid comments on exceptions to ordering during > > > connection recovery. Perhaps we can move on by making Julian's > > > proposed stipulation a SHOULD.... > > > -- > > > Mallikarjun > > > > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > > > > Santosh Rao wrote: > > > > > > > > Mallikarjun, > > > > > > > > Some comments below. > > > > > > > > Regards, > > > > Santosh > > > > > > > > "Mallikarjun C." wrote: > > > > > > > > > > Rod and Julian, > > > > > > > > > > This has been an interesting thread of discussion. Some > > > > > comments - > > > > > > > > > > 1.My first reaction was - allowing out-of-order command > > > > > transmission on the same connection deprives targets of > > > > > an implementation choice. Targets which support only > > > > > single-connection sessions and only support session > > > > > recovery (reasonable assumptions in my mind) can no > > > > > longer afford *not to* implement a command scoreboard. > > > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > > reason being that it can see out-of-order arrival of commands due to > > > > commands being dropped on digest errors. In such a case, it > must block > > > > further command processing until holes are filled. > > > > > > > > Thus, there is no getting away from implementing a sequencer at the > > > > target. Given this, I think it is unreasonable to restrict initiator > > > > implementation flexibility by imposing a strict ordering requirement > > > > within the connection. > > > > > > > > > 2.Any end-node efficiency that is sought to be achieved > > > > > by transmitting CmdSNs out-of-order from the initiator > > > > > would be lost on the other end-node, since the target > > > > > now must wait for re-ordering the commands. > > > > > > > > It has to handle this situation anyway to deal with holes caused by > > > > digest errors. This scenario occurs even with initiators that issue > > > > commands in order. > > > > > > > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > > > link badwidth (albeit at the expense of end-node efficiency), > > > > > compared to idling the link waiting for outbound DMA. > > > > > We have to determine if this is a reasonable trade-off. > > > > > > > > > > 4.I can see Rod's point that prefetching all immediate > > > > > data can be a burden on the NIC resources. But, two > > > > > questions - > > > > > - could the NIC not use unsolicited separate data > > > > > PDUs in these cases? [ I realize that InitialR2T > > > > > has to be "no" to let it happen... ] > > > > > - could the NIC have a memory architecture that > > > > > allows data prefetching for the next command (so > > > > > this is a non-issue from the protocol perspective)? > > > > > This scheme incurs one DMA delay for every new > > > > > burst of commands. > > > > > > > > > > 5.Another (perhaps radical at this point) option is to do > > > > > away with immediate unsolicited data, to stick only with > > > > > separate unsolicited data. I would personally be okay > > > > > with the choice, particularly if this feature (that > > > > > helps software implementations) starts making hardware > > > > > design complicated/expensive. > > > > > > > > > > So, to summarize - > > > > > > > > > > option immediate allow > > > > > data in spec? out-of-order? > > > > > > > > > > (A) (5) above no no > > > > > (B) No real reason to do this. no yes > > > > > (C) (4) above yes no > > > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > > > > > >From the arguments I heard so far, I am leaning towards > > > > > option A, and option C in that order. > > > > > > > > > > Comments? > > > > > -- > > > > > Mallikarjun > > > > > > > > > > Mallikarjun Chadalapaka > > > > > Networked Storage Architecture > > > > > Network Storage Solutions Organization > > > > > MS 5668 Hewlett-Packard, Roseville. > > > > > cbm@rose.hp.com > > > > > > > > > > Rod Harrison wrote: > > > > > > > > > > > > Julian, > > > > > > > > > > > > I don't understand what you are proposing here, > what do you > > > mean by > > > > > > "multiplexed" DMA? > > > > > > > > > > > > The problem is that the DMAs take some time, > the more there > > > are > > > > > > queued the longer the last DMAs queued take to complete. Some > > > commands > > > > > > require DMAs to complete before they can be sent, i.e. > Writes with > > > > > > immediate data, some commands do not, i.e. Reads and > writes with no > > > > > > immediate data. The iSCSI HBA wants to be able to send > commands as > > > > > > soon a possible, which for a read after a write can be > before the > > > > > > write's DMA has completed. Maintaining an ordered queue > for commands > > > > > > to be sent on the HBA is expensive and redundant since > the target > > > > > > already knows how to queue commands before committing > them to its > > > SCSI > > > > > > layer. > > > > > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > > > change the > > > > > > order of commands from the OS, but the DMAs those > commands need are > > > > > > unlikely to complete in the same order, and as I mentioned some > > > > > > commands need no DMA. If the HBA can't send commands > out of CmdSN > > > > > > order it has to maintain an ordered queue of commands > waiting to be > > > > > > sent, and potentially buffer a lot of data. For an HBA > this makes > > > > > > immediate data almost impossible to support. > > > > > > > > > > > > I don't see the problem with allowing out of > order commands > > > given > > > > > > that the target already has to deal with very similar > problems. I > > > > > > think we are getting in to the area of implementation > choices here, > > > > > > which is inappropriate for a specification. > > > > > > > > > > > > - Rod > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Julian Satran > > > > > > Sent: Monday, November 05, 2001 10:06 PM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: Re: iSCSI: Out of order commands, was current > UNH Plugfest > > > > > > > > > > > > Rod, > > > > > > > > > > > > I don't see any reason why DMA operations cant be > "multiplexed" with > > > > > > commands. > > > > > > If you have scheduled a long outbound DMA you are > doomed regardless > > > of > > > > > > the > > > > > > command ordering. > > > > > > And if you have scheduled DMA operations piecemeal then you can > > > insert > > > > > > your commands in correct order. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Rod Harrison" > > > > > > 05-11-01 20:48 > > > > > > Please respond to "Rod Harrison" > > > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > > > cc: > > > > > > Subject: iSCSI: Out of order commands, > was current > > > UNH > > > > > > Plugfest > > > > > > > > > > > > [ Subject changed ] > > > > > > > > > > > > Julian, > > > > > > > > > > > > The ordering difference is introduced > between the > > > > > > host > > > > > > side driver > > > > > > and the iSCSI HBA. The host side driver must present > SCSI commands > > > to > > > > > > the HBA in the order they are received from the OS to > prevent read > > > > > > after write dependency failures. The HBA might reorder > the commands > > > > > > depending on when DMA completes. The reordering can't > be done ahead > > > of > > > > > > time in the host driver since it doesn't know how long each DMA > > > might > > > > > > take. As long as the HBA assigns CmdSN in the order it receives > > > > > > commands the desired host ordering is preserved. > > > > > > > > > > > > - Rod > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Julian Satran > > > > > > Sent: Monday, November 05, 2001 12:35 AM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > > > Rod, > > > > > > > > > > > > I all examples give the point I find hard to understand > is why is > > > the > > > > > > ordering on the wire different from the presentation > order to the > > > > > > initiator. You can get as many overlaps as you want by > presenting > > > the > > > > > > commands to the initiator in the desired order. > > > > > > What we are considering here is the case in which you > want to ship > > > in > > > > > > an > > > > > > order different than the one you present the commands. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Rod Harrison" > > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > 04-11-01 04:42 > > > > > > Please respond to "Rod Harrison" > > > > > > > > > > > > To: "Barry Reinhold" , "Dave > > > > > > Sheehy" > > > > > > , "IETF IP SAN Reflector" > > > > > > > > > > > > cc: > > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > > > Barry, > > > > > > > > > > > > In general I agree but I don't think this is as > > > much > > > > > > of a > > > > > > corner case > > > > > > as it at first appears. Targets will have code very > similar to that > > > > > > needed to handle out of order commands to deal with > digest errors. > > > > > > Targets also need to queue commands whilst waiting for both > > > solicited > > > > > > and unsolicited data to arrive. Queuing out of order > commands seems > > > > > > little extra work. > > > > > > > > > > > > From an initiators point of view there are > > > > > > efficiency, > > > > > > and probably > > > > > > performance gains to be had from sending commands out > of order. Bob > > > > > > Russell gave the example of a read being sent whilst > write data DMA > > > is > > > > > > happening, and a similar situation can arise with DMA for writes > > > > > > overtaking that of earlier writes if the initiator has > multiple DMA > > > > > > engines. In this case the initiator might be forced to > let the wire > > > go > > > > > > idle if it can't send the data from completed DMAs as soon as > > > > > > possible. > > > > > > > > > > > > We already have a command queue at the > target to > > > > > > enforce > > > > > > correct > > > > > > serialisation of commands, doing the same thing at the > initiator is > > > > > > redundant. > > > > > > > > > > > > Finally, I don't believe we should be writing a > > > > > > standard > > > > > > to work > > > > > > around poor coding and test coverage, especially at the cost of > > > > > > potential efficiency gains. > > > > > > > > > > > > I agree with Dave and Santosh that > commands being > > > > > > sent > > > > > > out of order > > > > > > on a single session should be allowed by the standard. > > > > > > > > > > > > - Rod > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Barry Reinhold > > > > > > Sent: Friday, November 02, 2001 5:24 PM > > > > > > To: Dave Sheehy; IETF IP SAN Reflector > > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > > > Using features such as out of order command delivery on > a connection > > > > > > tend to > > > > > > be the sort of things that lead to interoperability > problems. It is > > > > > > unexpected and probably going to hit poorly tested code > paths even > > > if > > > > > > the > > > > > > standard is written to allow it. > > > > > > > > > > > > >-----Original Message----- > > > > > > >From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > > > Of > > > > > > >Dave Sheehy > > > > > > >Sent: Friday, November 02, 2001 4:19 PM > > > > > > >To: IETF IP SAN Reflector > > > > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > > > > >> > > > > > > >> The behavior of targets is clearly specified in Section > > > 2.2.2.3 > > > > > > on > > > > > > >> page 25 of draft 8, which says: > > > > > > >> "Except for the commands marked for immediate > delivery the > > > > > > iSCSI > > > > > > >> target layer MUST eliver the commands for > execution in the > > > > > > order > > > > > > >> specified by CmdSN." > > > > > > >> > > > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > > > > >> "- CmdSN - the current command Sequence Number > advanced by 1 > > > > > > on > > > > > > >> each command shipped except for commands marked for > > > immediate > > > > > > >> delivery." > > > > > > >> but the meaning of the term "shipped" is vague, > and does not > > > > > > >> necessarily > > > > > > >> require that the PDUs arrive on the other end of a TCP > > > > > > connection > > > > > > >> in the same order that the CmdSN values were > assigned to these > > > > > > PDUs. > > > > > > >> > > > > > > >> Some initiators have been designed to send commands out of > > > CmdSN > > > > > > >> order on one connection. Consider the situation > where there > > > is > > > > > > only > > > > > > >> one connection and a high-level dispatcher > creates a PDU for a > > > > > > SCSI > > > > > > >> command that involves writing immediate data to > the target. > > > > > > This PDU > > > > > > >> is enqueued to a lower-level layer which has to > setup, start, > > > > > > and > > > > > > >> wait-for a DMA operation to move the immediate > data into an > > > > > > onboard > > > > > > >> buffer before the PDU can be put onto the wire. > While this is > > > > > > >> happening, the dispatcher creates another > unrelated PDU for a > > > > > > SCSI > > > > > > >> read command (for example), and when this PDU is > passed to the > > > > > > >> lower-level layer it can be sent immediately, ahead of the > > > > > > previous > > > > > > >> write PDU and therefore out of order on this connection. > > > > > > >> > > > > > > >> The standard clearly allows this to happen if the two PDUs > > > were > > > > > > sent > > > > > > >> on different connections, and seems to imply that this can > > > also > > > > > > happen > > > > > > >> when the two PDUs are sent on the same connection. > > > > > > >> > > > > > > >> The suggestion is to put in the standard an > explicit statement > > > > > > that > > > > > > >> this is allowed or not allowed, as appropriate. > > > > > > >> > > > > > > >> If this is allowed, such a statement would avoid > the erroneous > > > > > > >> assumption being made by some target implementers > that within > > > a > > > > > > single > > > > > > >> connection, commands will arrive in order. > > > > > > >> > > > > > > >> If this is not allowed, such a statement would avoid the > > > > > > erroneous > > > > > > >> assumption being made by some initiator implementers that > > > within > > > > > > a > > > > > > >> single connection, commands can be put on the wire out of > > > order. > > > > > > >> > > > > > > >> +++ > > > > > > >> > > > > > > >> will add an explicit statement saying that this behaviour is > > > > > > forbidden. > > > > > > >> 2.2.2.1 will contain: > > > > > > >> > > > > > > >> On any given connection, the iSCSI initiator MUST send the > > > > > > >commands in the > > > > > > >> order specified by CmdSN. > > > > > > >> > > > > > > >> +++ > > > > > > > > > > > > > >Why do you feel this behavior should be forbidden? > Targets already > > > > > > have to > > > > > > >order commands across the session. I don't see why > it's a problem > > > to > > > > > > extend > > > > > > >that to the connection as well. I, for one, believe we > should take > > > > > > >a liberal > > > > > > >stance on this. > > > > > > > > > > > > > >Dave Sheehy > > > > > > > > > > > > > > > -- > > > > ################################## > > > > Santosh Rao > > > > Software Design Engineer, > > > > HP-UX iSCSI Driver Team, > > > > Hewlett Packard, Cupertino. > > > > email : santoshr@cup.hp.com > > > > Phone : 408-447-3751 > > > > ################################## > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Wed Nov 7 17:08:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA24788 for ; Wed, 7 Nov 2001 17:08:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7LDjU05476 for ips-outgoing; Wed, 7 Nov 2001 16:13:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7LDhl05470 for ; Wed, 7 Nov 2001 16:13:43 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id QAA18382; Wed, 7 Nov 2001 16:13:26 -0500 (EST) Date: Wed, 7 Nov 2001 16:13:26 -0500 From: "Robert D. Russell" To: Somesh Gupta cc: Julian Satran , ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, Julian: You state that dealing with OOO commands on the target will add substantial complexity on the target. Do you have any basis for that claim? My impression from the plugfest is that most targets are already dealing with it. Perhaps we need to hear from someone who is actually building a target for which this would be a real problem. If anything, what we are hearing from people who really are building initiators is that dealing with the requirement to send commands in order will introduce substantial complexity on the initiator. So why should we be saving complexity on (hypothetically) simple targets yet requiring complexity on real initiators? As far as the deadlock issue is concerned, if the only way that deadlock can occur with OOO commands on the same connection is during the use of immediate data (which is I think what Julian was saying), then shouldn't we change the standard to just say that -- if the initiator sends commands out of order on a single connection, then immediate data MUST NOT be used on that connection in order to avoid deadlock. This gives everybody what they want, since initiators who find it beneficial to deliver commands OOO will just negotiate immediate data off. Those who really want to use immediate data will have to ensure that commands are sent in order. The tradeoff then becomes an implementation issue, not a standards issue, which is where it belongs. Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Wed, 7 Nov 2001, Somesh Gupta wrote: > I think we should either have it as a MUST or not require > it (at both ends to get the real benefit). SHOULD is one > of those things that leads to implementation > burden and confusion, without perhaps the feature being > used. There are implementation as well as protocol > considerations mixed in here. > > If we are to remove the restriction, we should (SHOULD) > get the maximum benefit from it, rather than to > accomodate an implementation choice. Out of sequence > commands, combined with the possibility of digest errors, > will add substantial complexity on the target side, > without corrosponding benefit in performance. If we change > this to SHOULD, we should also relax the requirement > to present commands on the target side to a SHOULD. > > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Wednesday, November 07, 2001 10:00 AM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Out of order commands > > > > > > Mallikarjun, > > > > I did not see a SINGLE performance improvement that results from OOO > > shipping. > > I would be bad engineering to give away the "no-deadlock" mechanism we > > have now for nothing. > > I have also the impression that the point about deadlock that I keep > > repeating is ignored or not understood. > > As we stand today commands can be shipped with Immediate data or without > > and an implementer determined > > to squeeze maximum bandwidth and overlap command start with delivery will > > choose not to work with immediate data > > (as you have pointed out) while a low performance software implementation > > will use immediate data to minimize CPU cycles consumed. However both > > will be guaranteed to work without deadlock as source and sink use the > > same ordering. > > Recovery is still a low probability event and should be handled with a > > different set of considerations in mind. > > As for the strictness of the recommendation - yes we could settle on > > SHOULD. > > > > Julo > > > > > > > > > > "Mallikarjun C." > > Sent by: owner-ips@ece.cmu.edu > > 07-11-01 19:41 > > Please respond to cbm > > > > > > To: Santosh Rao , ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: Out of order commands > > > > > > > > Santosh, > > > > I have only one comment on your responses. > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > I made two convenient assumptions if you noticed, :-), one of which > > is that target forces session recovery on *any* error that it sees > > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > > error. With that assumption, a target can afford not to implement > > a scoreboard. > > > > As I said in a private note, I guess what primarily bothers me about > > OOO commands on a connection is that it requires the receiver to > > undo this "optimization" on its end - most notably on a single > > connection. TCP experts may comment on how/if they dealt with a > > similar issue. > > > > OTOH, you had some valid comments on exceptions to ordering during > > connection recovery. Perhaps we can move on by making Julian's > > proposed stipulation a SHOULD.... > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > Santosh Rao wrote: > > > > > > Mallikarjun, > > > > > > Some comments below. > > > > > > Regards, > > > Santosh > > > > > > "Mallikarjun C." wrote: > > > > > > > > Rod and Julian, > > > > > > > > This has been an interesting thread of discussion. Some > > > > comments - > > > > > > > > 1.My first reaction was - allowing out-of-order command > > > > transmission on the same connection deprives targets of > > > > an implementation choice. Targets which support only > > > > single-connection sessions and only support session > > > > recovery (reasonable assumptions in my mind) can no > > > > longer afford *not to* implement a command scoreboard. > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > > > Thus, there is no getting away from implementing a sequencer at the > > > target. Given this, I think it is unreasonable to restrict initiator > > > implementation flexibility by imposing a strict ordering requirement > > > within the connection. > > > > > > > 2.Any end-node efficiency that is sought to be achieved > > > > by transmitting CmdSNs out-of-order from the initiator > > > > would be lost on the other end-node, since the target > > > > now must wait for re-ordering the commands. > > > > > > It has to handle this situation anyway to deal with holes caused by > > > digest errors. This scenario occurs even with initiators that issue > > > commands in order. > > > > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > > link badwidth (albeit at the expense of end-node efficiency), > > > > compared to idling the link waiting for outbound DMA. > > > > We have to determine if this is a reasonable trade-off. > > > > > > > > 4.I can see Rod's point that prefetching all immediate > > > > data can be a burden on the NIC resources. But, two > > > > questions - > > > > - could the NIC not use unsolicited separate data > > > > PDUs in these cases? [ I realize that InitialR2T > > > > has to be "no" to let it happen... ] > > > > - could the NIC have a memory architecture that > > > > allows data prefetching for the next command (so > > > > this is a non-issue from the protocol perspective)? > > > > This scheme incurs one DMA delay for every new > > > > burst of commands. > > > > > > > > 5.Another (perhaps radical at this point) option is to do > > > > away with immediate unsolicited data, to stick only with > > > > separate unsolicited data. I would personally be okay > > > > with the choice, particularly if this feature (that > > > > helps software implementations) starts making hardware > > > > design complicated/expensive. > > > > > > > > So, to summarize - > > > > > > > > option immediate allow > > > > data in spec? out-of-order? > > > > > > > > (A) (5) above no no > > > > (B) No real reason to do this. no yes > > > > (C) (4) above yes no > > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > > > >From the arguments I heard so far, I am leaning towards > > > > option A, and option C in that order. > > > > > > > > Comments? > > > > -- > > > > Mallikarjun > > > > > > > > Mallikarjun Chadalapaka > > > > Networked Storage Architecture > > > > Network Storage Solutions Organization > > > > MS 5668 Hewlett-Packard, Roseville. > > > > cbm@rose.hp.com > > > > > > > > Rod Harrison wrote: > > > > > > > > > > Julian, > > > > > > > > > > I don't understand what you are proposing here, what do you > > mean by > > > > > "multiplexed" DMA? > > > > > > > > > > The problem is that the DMAs take some time, the more there > > are > > > > > queued the longer the last DMAs queued take to complete. Some > > commands > > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > > soon a possible, which for a read after a write can be before the > > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > > to be sent on the HBA is expensive and redundant since the target > > > > > already knows how to queue commands before committing them to its > > SCSI > > > > > layer. > > > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > > change the > > > > > order of commands from the OS, but the DMAs those commands need are > > > > > unlikely to complete in the same order, and as I mentioned some > > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > > order it has to maintain an ordered queue of commands waiting to be > > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > > immediate data almost impossible to support. > > > > > > > > > > I don't see the problem with allowing out of order commands > > given > > > > > that the target already has to deal with very similar problems. I > > > > > think we are getting in to the area of implementation choices here, > > > > > which is inappropriate for a specification. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Julian Satran > > > > > Sent: Monday, November 05, 2001 10:06 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > > > > > Rod, > > > > > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > > > commands. > > > > > If you have scheduled a long outbound DMA you are doomed regardless > > of > > > > > the > > > > > command ordering. > > > > > And if you have scheduled DMA operations piecemeal then you can > > insert > > > > > your commands in correct order. > > > > > > > > > > Julo > > > > > > > > > > "Rod Harrison" > > > > > 05-11-01 20:48 > > > > > Please respond to "Rod Harrison" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > > cc: > > > > > Subject: iSCSI: Out of order commands, was current > > UNH > > > > > Plugfest > > > > > > > > > > [ Subject changed ] > > > > > > > > > > Julian, > > > > > > > > > > The ordering difference is introduced between the > > > > > host > > > > > side driver > > > > > and the iSCSI HBA. The host side driver must present SCSI commands > > to > > > > > the HBA in the order they are received from the OS to prevent read > > > > > after write dependency failures. The HBA might reorder the commands > > > > > depending on when DMA completes. The reordering can't be done ahead > > of > > > > > time in the host driver since it doesn't know how long each DMA > > might > > > > > take. As long as the HBA assigns CmdSN in the order it receives > > > > > commands the desired host ordering is preserved. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Julian Satran > > > > > Sent: Monday, November 05, 2001 12:35 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Rod, > > > > > > > > > > I all examples give the point I find hard to understand is why is > > the > > > > > ordering on the wire different from the presentation order to the > > > > > initiator. You can get as many overlaps as you want by presenting > > the > > > > > commands to the initiator in the desired order. > > > > > What we are considering here is the case in which you want to ship > > in > > > > > an > > > > > order different than the one you present the commands. > > > > > > > > > > Julo > > > > > > > > > > "Rod Harrison" > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > 04-11-01 04:42 > > > > > Please respond to "Rod Harrison" > > > > > > > > > > To: "Barry Reinhold" , "Dave > > > > > Sheehy" > > > > > , "IETF IP SAN Reflector" > > > > > > > > > > cc: > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Barry, > > > > > > > > > > In general I agree but I don't think this is as > > much > > > > > of a > > > > > corner case > > > > > as it at first appears. Targets will have code very similar to that > > > > > needed to handle out of order commands to deal with digest errors. > > > > > Targets also need to queue commands whilst waiting for both > > solicited > > > > > and unsolicited data to arrive. Queuing out of order commands seems > > > > > little extra work. > > > > > > > > > > From an initiators point of view there are > > > > > efficiency, > > > > > and probably > > > > > performance gains to be had from sending commands out of order. Bob > > > > > Russell gave the example of a read being sent whilst write data DMA > > is > > > > > happening, and a similar situation can arise with DMA for writes > > > > > overtaking that of earlier writes if the initiator has multiple DMA > > > > > engines. In this case the initiator might be forced to let the wire > > go > > > > > idle if it can't send the data from completed DMAs as soon as > > > > > possible. > > > > > > > > > > We already have a command queue at the target to > > > > > enforce > > > > > correct > > > > > serialisation of commands, doing the same thing at the initiator is > > > > > redundant. > > > > > > > > > > Finally, I don't believe we should be writing a > > > > > standard > > > > > to work > > > > > around poor coding and test coverage, especially at the cost of > > > > > potential efficiency gains. > > > > > > > > > > I agree with Dave and Santosh that commands being > > > > > sent > > > > > out of order > > > > > on a single session should be allowed by the standard. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Barry Reinhold > > > > > Sent: Friday, November 02, 2001 5:24 PM > > > > > To: Dave Sheehy; IETF IP SAN Reflector > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Using features such as out of order command delivery on a connection > > > > > tend to > > > > > be the sort of things that lead to interoperability problems. It is > > > > > unexpected and probably going to hit poorly tested code paths even > > if > > > > > the > > > > > standard is written to allow it. > > > > > > > > > > >-----Original Message----- > > > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > > Of > > > > > >Dave Sheehy > > > > > >Sent: Friday, November 02, 2001 4:19 PM > > > > > >To: IETF IP SAN Reflector > > > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > > > >> > > > > > >> The behavior of targets is clearly specified in Section > > 2.2.2.3 > > > > > on > > > > > >> page 25 of draft 8, which says: > > > > > >> "Except for the commands marked for immediate delivery the > > > > > iSCSI > > > > > >> target layer MUST eliver the commands for execution in the > > > > > order > > > > > >> specified by CmdSN." > > > > > >> > > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > > > on > > > > > >> each command shipped except for commands marked for > > immediate > > > > > >> delivery." > > > > > >> but the meaning of the term "shipped" is vague, and does not > > > > > >> necessarily > > > > > >> require that the PDUs arrive on the other end of a TCP > > > > > connection > > > > > >> in the same order that the CmdSN values were assigned to these > > > > > PDUs. > > > > > >> > > > > > >> Some initiators have been designed to send commands out of > > CmdSN > > > > > >> order on one connection. Consider the situation where there > > is > > > > > only > > > > > >> one connection and a high-level dispatcher creates a PDU for a > > > > > SCSI > > > > > >> command that involves writing immediate data to the target. > > > > > This PDU > > > > > >> is enqueued to a lower-level layer which has to setup, start, > > > > > and > > > > > >> wait-for a DMA operation to move the immediate data into an > > > > > onboard > > > > > >> buffer before the PDU can be put onto the wire. While this is > > > > > >> happening, the dispatcher creates another unrelated PDU for a > > > > > SCSI > > > > > >> read command (for example), and when this PDU is passed to the > > > > > >> lower-level layer it can be sent immediately, ahead of the > > > > > previous > > > > > >> write PDU and therefore out of order on this connection. > > > > > >> > > > > > >> The standard clearly allows this to happen if the two PDUs > > were > > > > > sent > > > > > >> on different connections, and seems to imply that this can > > also > > > > > happen > > > > > >> when the two PDUs are sent on the same connection. > > > > > >> > > > > > >> The suggestion is to put in the standard an explicit statement > > > > > that > > > > > >> this is allowed or not allowed, as appropriate. > > > > > >> > > > > > >> If this is allowed, such a statement would avoid the erroneous > > > > > >> assumption being made by some target implementers that within > > a > > > > > single > > > > > >> connection, commands will arrive in order. > > > > > >> > > > > > >> If this is not allowed, such a statement would avoid the > > > > > erroneous > > > > > >> assumption being made by some initiator implementers that > > within > > > > > a > > > > > >> single connection, commands can be put on the wire out of > > order. > > > > > >> > > > > > >> +++ > > > > > >> > > > > > >> will add an explicit statement saying that this behaviour is > > > > > forbidden. > > > > > >> 2.2.2.1 will contain: > > > > > >> > > > > > >> On any given connection, the iSCSI initiator MUST send the > > > > > >commands in the > > > > > >> order specified by CmdSN. > > > > > >> > > > > > >> +++ > > > > > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > > > have to > > > > > >order commands across the session. I don't see why it's a problem > > to > > > > > extend > > > > > >that to the connection as well. I, for one, believe we should take > > > > > >a liberal > > > > > >stance on this. > > > > > > > > > > > >Dave Sheehy > > > > > > > > > > > > -- > > > ################################## > > > Santosh Rao > > > Software Design Engineer, > > > HP-UX iSCSI Driver Team, > > > Hewlett Packard, Cupertino. > > > email : santoshr@cup.hp.com > > > Phone : 408-447-3751 > > > ################################## > > > > > > > > > From owner-ips@ece.cmu.edu Wed Nov 7 17:35:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA25135 for ; Wed, 7 Nov 2001 17:35:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7LdHe07454 for ips-outgoing; Wed, 7 Nov 2001 16:39:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.windriver.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7LdFl07450 for ; Wed, 7 Nov 2001 16:39:15 -0500 (EST) Received: from london ([147.11.45.211]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id NAA16490; Wed, 7 Nov 2001 13:08:11 -0800 (PST) From: "Rod Harrison" To: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 13:11:08 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, The deadlock situation you outlined is not related to out of order commands, it arises because the target implementation is broken. A target that advertises a command window larger than the number of commands it can support with the intent of leveraging TCP buffering is making big, and invalid assumptions about the initiator. The target has no idea how long it might take to receive / generate the data associated with commands. For example, if a target offers a command window of 10 with the intent of processing 8 commands at a time and having the other two in flight it will deadlock if the oldest command needs an R2T. This happens even if the commands are sent in order. It is never acceptable for a target to stop reading its TCP stream unless its command window is full since there may be unread DATA-OUTs needed to allow commands to be committed to SCSI. There are very real performance gains associated with sending out of order commands, the simplest of which was outlined by Bob Russell when this thread started. We do not have a zero latency network so queuing as close as possible the ultimate data sink will always be a win. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Wednesday, November 07, 2001 10:00 AM To: ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands Mallikarjun, I did not see a SINGLE performance improvement that results from OOO shipping. I would be bad engineering to give away the "no-deadlock" mechanism we have now for nothing. I have also the impression that the point about deadlock that I keep repeating is ignored or not understood. As we stand today commands can be shipped with Immediate data or without and an implementer determined to squeeze maximum bandwidth and overlap command start with delivery will choose not to work with immediate data (as you have pointed out) while a low performance software implementation will use immediate data to minimize CPU cycles consumed. However both will be guaranteed to work without deadlock as source and sink use the same ordering. Recovery is still a low probability event and should be handled with a different set of considerations in mind. As for the strictness of the recommendation - yes we could settle on SHOULD. Julo "Mallikarjun C." Sent by: owner-ips@ece.cmu.edu 07-11-01 19:41 Please respond to cbm To: Santosh Rao , ips@ece.cmu.edu cc: Subject: Re: iSCSI: Out of order commands Santosh, I have only one comment on your responses. > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. I made two convenient assumptions if you noticed, :-), one of which is that target forces session recovery on *any* error that it sees (ErrorRecoveryLevel=0) - including a dropped command due to a digest error. With that assumption, a target can afford not to implement a scoreboard. As I said in a private note, I guess what primarily bothers me about OOO commands on a connection is that it requires the receiver to undo this "optimization" on its end - most notably on a single connection. TCP experts may comment on how/if they dealt with a similar issue. OTOH, you had some valid comments on exceptions to ordering during connection recovery. Perhaps we can move on by making Julian's proposed stipulation a SHOULD.... -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Santosh Rao wrote: > > Mallikarjun, > > Some comments below. > > Regards, > Santosh > > "Mallikarjun C." wrote: > > > > Rod and Julian, > > > > This has been an interesting thread of discussion. Some > > comments - > > > > 1.My first reaction was - allowing out-of-order command > > transmission on the same connection deprives targets of > > an implementation choice. Targets which support only > > single-connection sessions and only support session > > recovery (reasonable assumptions in my mind) can no > > longer afford *not to* implement a command scoreboard. > > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. > > Thus, there is no getting away from implementing a sequencer at the > target. Given this, I think it is unreasonable to restrict initiator > implementation flexibility by imposing a strict ordering requirement > within the connection. > > > 2.Any end-node efficiency that is sought to be achieved > > by transmitting CmdSNs out-of-order from the initiator > > would be lost on the other end-node, since the target > > now must wait for re-ordering the commands. > > It has to handle this situation anyway to deal with holes caused by > digest errors. This scenario occurs even with initiators that issue > commands in order. > > > > > 3.The flipside is that out-of-order transmission saves > > link badwidth (albeit at the expense of end-node efficiency), > > compared to idling the link waiting for outbound DMA. > > We have to determine if this is a reasonable trade-off. > > > > 4.I can see Rod's point that prefetching all immediate > > data can be a burden on the NIC resources. But, two > > questions - > > - could the NIC not use unsolicited separate data > > PDUs in these cases? [ I realize that InitialR2T > > has to be "no" to let it happen... ] > > - could the NIC have a memory architecture that > > allows data prefetching for the next command (so > > this is a non-issue from the protocol perspective)? > > This scheme incurs one DMA delay for every new > > burst of commands. > > > > 5.Another (perhaps radical at this point) option is to do > > away with immediate unsolicited data, to stick only with > > separate unsolicited data. I would personally be okay > > with the choice, particularly if this feature (that > > helps software implementations) starts making hardware > > design complicated/expensive. > > > > So, to summarize - > > > > option immediate allow > > data in spec? out-of-order? > > > > (A) (5) above no no > > (B) No real reason to do this. no yes > > (C) (4) above yes no > > (D) pros & cons (1), (2) & (3) yes yes > > > > >From the arguments I heard so far, I am leaning towards > > option A, and option C in that order. > > > > Comments? > > -- > > Mallikarjun > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > Rod Harrison wrote: > > > > > > Julian, > > > > > > I don't understand what you are proposing here, what do you mean by > > > "multiplexed" DMA? > > > > > > The problem is that the DMAs take some time, the more there are > > > queued the longer the last DMAs queued take to complete. Some commands > > > require DMAs to complete before they can be sent, i.e. Writes with > > > immediate data, some commands do not, i.e. Reads and writes with no > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > soon a possible, which for a read after a write can be before the > > > write's DMA has completed. Maintaining an ordered queue for commands > > > to be sent on the HBA is expensive and redundant since the target > > > already knows how to queue commands before committing them to its SCSI > > > layer. > > > > > > The iSCSI HBA and its host driver are not at liberty to change the > > > order of commands from the OS, but the DMAs those commands need are > > > unlikely to complete in the same order, and as I mentioned some > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > order it has to maintain an ordered queue of commands waiting to be > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > immediate data almost impossible to support. > > > > > > I don't see the problem with allowing out of order commands given > > > that the target already has to deal with very similar problems. I > > > think we are getting in to the area of implementation choices here, > > > which is inappropriate for a specification. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > Rod, > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > commands. > > > If you have scheduled a long outbound DMA you are doomed regardless of > > > the > > > command ordering. > > > And if you have scheduled DMA operations piecemeal then you can insert > > > your commands in correct order. > > > > > > Julo > > > > > > "Rod Harrison" > > > 05-11-01 20:48 > > > Please respond to "Rod Harrison" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > cc: > > > Subject: iSCSI: Out of order commands, was current UNH > > > Plugfest > > > > > > [ Subject changed ] > > > > > > Julian, > > > > > > The ordering difference is introduced between the > > > host > > > side driver > > > and the iSCSI HBA. The host side driver must present SCSI commands to > > > the HBA in the order they are received from the OS to prevent read > > > after write dependency failures. The HBA might reorder the commands > > > depending on when DMA completes. The reordering can't be done ahead of > > > time in the host driver since it doesn't know how long each DMA might > > > take. As long as the HBA assigns CmdSN in the order it receives > > > commands the desired host ordering is preserved. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 12:35 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Rod, > > > > > > I all examples give the point I find hard to understand is why is the > > > ordering on the wire different from the presentation order to the > > > initiator. You can get as many overlaps as you want by presenting the > > > commands to the initiator in the desired order. > > > What we are considering here is the case in which you want to ship in > > > an > > > order different than the one you present the commands. > > > > > > Julo > > > > > > "Rod Harrison" > > > Sent by: owner-ips@ece.cmu.edu > > > 04-11-01 04:42 > > > Please respond to "Rod Harrison" > > > > > > To: "Barry Reinhold" , "Dave > > > Sheehy" > > > , "IETF IP SAN Reflector" > > > > > > cc: > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Barry, > > > > > > In general I agree but I don't think this is as much > > > of a > > > corner case > > > as it at first appears. Targets will have code very similar to that > > > needed to handle out of order commands to deal with digest errors. > > > Targets also need to queue commands whilst waiting for both solicited > > > and unsolicited data to arrive. Queuing out of order commands seems > > > little extra work. > > > > > > From an initiators point of view there are > > > efficiency, > > > and probably > > > performance gains to be had from sending commands out of order. Bob > > > Russell gave the example of a read being sent whilst write data DMA is > > > happening, and a similar situation can arise with DMA for writes > > > overtaking that of earlier writes if the initiator has multiple DMA > > > engines. In this case the initiator might be forced to let the wire go > > > idle if it can't send the data from completed DMAs as soon as > > > possible. > > > > > > We already have a command queue at the target to > > > enforce > > > correct > > > serialisation of commands, doing the same thing at the initiator is > > > redundant. > > > > > > Finally, I don't believe we should be writing a > > > standard > > > to work > > > around poor coding and test coverage, especially at the cost of > > > potential efficiency gains. > > > > > > I agree with Dave and Santosh that commands being > > > sent > > > out of order > > > on a single session should be allowed by the standard. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Barry Reinhold > > > Sent: Friday, November 02, 2001 5:24 PM > > > To: Dave Sheehy; IETF IP SAN Reflector > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Using features such as out of order command delivery on a connection > > > tend to > > > be the sort of things that lead to interoperability problems. It is > > > unexpected and probably going to hit poorly tested code paths even if > > > the > > > standard is written to allow it. > > > > > > >-----Original Message----- > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > >Dave Sheehy > > > >Sent: Friday, November 02, 2001 4:19 PM > > > >To: IETF IP SAN Reflector > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > >> > > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > > on > > > >> page 25 of draft 8, which says: > > > >> "Except for the commands marked for immediate delivery the > > > iSCSI > > > >> target layer MUST eliver the commands for execution in the > > > order > > > >> specified by CmdSN." > > > >> > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > on > > > >> each command shipped except for commands marked for immediate > > > >> delivery." > > > >> but the meaning of the term "shipped" is vague, and does not > > > >> necessarily > > > >> require that the PDUs arrive on the other end of a TCP > > > connection > > > >> in the same order that the CmdSN values were assigned to these > > > PDUs. > > > >> > > > >> Some initiators have been designed to send commands out of CmdSN > > > >> order on one connection. Consider the situation where there is > > > only > > > >> one connection and a high-level dispatcher creates a PDU for a > > > SCSI > > > >> command that involves writing immediate data to the target. > > > This PDU > > > >> is enqueued to a lower-level layer which has to setup, start, > > > and > > > >> wait-for a DMA operation to move the immediate data into an > > > onboard > > > >> buffer before the PDU can be put onto the wire. While this is > > > >> happening, the dispatcher creates another unrelated PDU for a > > > SCSI > > > >> read command (for example), and when this PDU is passed to the > > > >> lower-level layer it can be sent immediately, ahead of the > > > previous > > > >> write PDU and therefore out of order on this connection. > > > >> > > > >> The standard clearly allows this to happen if the two PDUs were > > > sent > > > >> on different connections, and seems to imply that this can also > > > happen > > > >> when the two PDUs are sent on the same connection. > > > >> > > > >> The suggestion is to put in the standard an explicit statement > > > that > > > >> this is allowed or not allowed, as appropriate. > > > >> > > > >> If this is allowed, such a statement would avoid the erroneous > > > >> assumption being made by some target implementers that within a > > > single > > > >> connection, commands will arrive in order. > > > >> > > > >> If this is not allowed, such a statement would avoid the > > > erroneous > > > >> assumption being made by some initiator implementers that within > > > a > > > >> single connection, commands can be put on the wire out of order. > > > >> > > > >> +++ > > > >> > > > >> will add an explicit statement saying that this behaviour is > > > forbidden. > > > >> 2.2.2.1 will contain: > > > >> > > > >> On any given connection, the iSCSI initiator MUST send the > > > >commands in the > > > >> order specified by CmdSN. > > > >> > > > >> +++ > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > have to > > > >order commands across the session. I don't see why it's a problem to > > > extend > > > >that to the connection as well. I, for one, believe we should take > > > >a liberal > > > >stance on this. > > > > > > > >Dave Sheehy > > > > > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## From owner-ips@ece.cmu.edu Wed Nov 7 17:50:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA25405 for ; Wed, 7 Nov 2001 17:50:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7LiC607864 for ips-outgoing; Wed, 7 Nov 2001 16:44:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from philotas.hosting.pacbell.net (philotas.hosting.pacbell.net [216.100.99.24]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7Li5l07857 for ; Wed, 7 Nov 2001 16:44:10 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by philotas.hosting.pacbell.net id QAA18974; Wed, 7 Nov 2001 16:40:14 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: Cc: Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 13:37:26 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: <200111072128.NAA09152@core.rose.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mallikarjun, Again, a simple target does not bear this burden. Only targets that do accept striping of commands have to deal with this. It is also a performance inhibitor on the target side unless we also add SHOULD to out of delivery. You may have shipped the command faster, but it then just sits somewhere till the holes are filled. (and of course the processing and data structure requirements that it implies). Sort of like speeding to the next red light. Somesh > -----Original Message----- > From: cbm@core.rose.hp.com [mailto:cbm@core.rose.hp.com]On Behalf Of > Mallikarjun C. > Sent: Wednesday, November 07, 2001 1:39 PM > To: somesh_gupta@silverbacksystems.com > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI: Out of order commands > > > Somesh, > > I see a SHOULD encouraging efficient implementations in > this case, and I think it's desirable to do that. As > you yourself argued in your last email, this allows > implementations to optimize for the most-likely case. > > IMHO, a MUST here would be going too far (connection recovery > scenario pointed out by Santosh) unless we start special > casing error recovery cases/multi-connection cases. In > effect, that again comes down to a SHOULD for a general > n-connection session case. > > Regards. > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > Somesh Gupta wrote: > > > > I think we should either have it as a MUST or not require > > it (at both ends to get the real benefit). SHOULD is one > > of those things that leads to implementation > > burden and confusion, without perhaps the feature being > > used. There are implementation as well as protocol > > considerations mixed in here. > > > > If we are to remove the restriction, we should (SHOULD) > > get the maximum benefit from it, rather than to > > accomodate an implementation choice. Out of sequence > > commands, combined with the possibility of digest errors, > > will add substantial complexity on the target side, > > without corrosponding benefit in performance. If we change > > this to SHOULD, we should also relax the requirement > > to present commands on the target side to a SHOULD. > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Wednesday, November 07, 2001 10:00 AM > > > To: ips@ece.cmu.edu > > > Subject: Re: iSCSI: Out of order commands > > > > > > > > > Mallikarjun, > > > > > > I did not see a SINGLE performance improvement that results from OOO > > > shipping. > > > I would be bad engineering to give away the "no-deadlock" mechanism we > > > have now for nothing. > > > I have also the impression that the point about deadlock that I keep > > > repeating is ignored or not understood. > > > As we stand today commands can be shipped with Immediate data > or without > > > and an implementer determined > > > to squeeze maximum bandwidth and overlap command start with > delivery will > > > choose not to work with immediate data > > > (as you have pointed out) while a low performance software > implementation > > > will use immediate data to minimize CPU cycles consumed. However both > > > will be guaranteed to work without deadlock as source and sink use the > > > same ordering. > > > Recovery is still a low probability event and should be handled with a > > > different set of considerations in mind. > > > As for the strictness of the recommendation - yes we could settle on > > > SHOULD. > > > > > > Julo > > > > > > > > > > > > > > > "Mallikarjun C." > > > Sent by: owner-ips@ece.cmu.edu > > > 07-11-01 19:41 > > > Please respond to cbm > > > > > > > > > To: Santosh Rao , ips@ece.cmu.edu > > > cc: > > > Subject: Re: iSCSI: Out of order commands > > > > > > > > > > > > Santosh, > > > > > > I have only one comment on your responses. > > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > > reason being that it can see out-of-order arrival of commands due to > > > > commands being dropped on digest errors. In such a case, it > must block > > > > further command processing until holes are filled. > > > > > > I made two convenient assumptions if you noticed, :-), one of which > > > is that target forces session recovery on *any* error that it sees > > > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > > > error. With that assumption, a target can afford not to implement > > > a scoreboard. > > > > > > As I said in a private note, I guess what primarily bothers me about > > > OOO commands on a connection is that it requires the receiver to > > > undo this "optimization" on its end - most notably on a single > > > connection. TCP experts may comment on how/if they dealt with a > > > similar issue. > > > > > > OTOH, you had some valid comments on exceptions to ordering during > > > connection recovery. Perhaps we can move on by making Julian's > > > proposed stipulation a SHOULD.... > > > -- > > > Mallikarjun > > > > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > > > > Santosh Rao wrote: > > > > > > > > Mallikarjun, > > > > > > > > Some comments below. > > > > > > > > Regards, > > > > Santosh > > > > > > > > "Mallikarjun C." wrote: > > > > > > > > > > Rod and Julian, > > > > > > > > > > This has been an interesting thread of discussion. Some > > > > > comments - > > > > > > > > > > 1.My first reaction was - allowing out-of-order command > > > > > transmission on the same connection deprives targets of > > > > > an implementation choice. Targets which support only > > > > > single-connection sessions and only support session > > > > > recovery (reasonable assumptions in my mind) can no > > > > > longer afford *not to* implement a command scoreboard. > > > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > > reason being that it can see out-of-order arrival of commands due to > > > > commands being dropped on digest errors. In such a case, it > must block > > > > further command processing until holes are filled. > > > > > > > > Thus, there is no getting away from implementing a sequencer at the > > > > target. Given this, I think it is unreasonable to restrict initiator > > > > implementation flexibility by imposing a strict ordering requirement > > > > within the connection. > > > > > > > > > 2.Any end-node efficiency that is sought to be achieved > > > > > by transmitting CmdSNs out-of-order from the initiator > > > > > would be lost on the other end-node, since the target > > > > > now must wait for re-ordering the commands. > > > > > > > > It has to handle this situation anyway to deal with holes caused by > > > > digest errors. This scenario occurs even with initiators that issue > > > > commands in order. > > > > > > > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > > > link badwidth (albeit at the expense of end-node efficiency), > > > > > compared to idling the link waiting for outbound DMA. > > > > > We have to determine if this is a reasonable trade-off. > > > > > > > > > > 4.I can see Rod's point that prefetching all immediate > > > > > data can be a burden on the NIC resources. But, two > > > > > questions - > > > > > - could the NIC not use unsolicited separate data > > > > > PDUs in these cases? [ I realize that InitialR2T > > > > > has to be "no" to let it happen... ] > > > > > - could the NIC have a memory architecture that > > > > > allows data prefetching for the next command (so > > > > > this is a non-issue from the protocol perspective)? > > > > > This scheme incurs one DMA delay for every new > > > > > burst of commands. > > > > > > > > > > 5.Another (perhaps radical at this point) option is to do > > > > > away with immediate unsolicited data, to stick only with > > > > > separate unsolicited data. I would personally be okay > > > > > with the choice, particularly if this feature (that > > > > > helps software implementations) starts making hardware > > > > > design complicated/expensive. > > > > > > > > > > So, to summarize - > > > > > > > > > > option immediate allow > > > > > data in spec? out-of-order? > > > > > > > > > > (A) (5) above no no > > > > > (B) No real reason to do this. no yes > > > > > (C) (4) above yes no > > > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > > > > > >From the arguments I heard so far, I am leaning towards > > > > > option A, and option C in that order. > > > > > > > > > > Comments? > > > > > -- > > > > > Mallikarjun > > > > > > > > > > Mallikarjun Chadalapaka > > > > > Networked Storage Architecture > > > > > Network Storage Solutions Organization > > > > > MS 5668 Hewlett-Packard, Roseville. > > > > > cbm@rose.hp.com > > > > > > > > > > Rod Harrison wrote: > > > > > > > > > > > > Julian, > > > > > > > > > > > > I don't understand what you are proposing here, > what do you > > > mean by > > > > > > "multiplexed" DMA? > > > > > > > > > > > > The problem is that the DMAs take some time, > the more there > > > are > > > > > > queued the longer the last DMAs queued take to complete. Some > > > commands > > > > > > require DMAs to complete before they can be sent, i.e. > Writes with > > > > > > immediate data, some commands do not, i.e. Reads and > writes with no > > > > > > immediate data. The iSCSI HBA wants to be able to send > commands as > > > > > > soon a possible, which for a read after a write can be > before the > > > > > > write's DMA has completed. Maintaining an ordered queue > for commands > > > > > > to be sent on the HBA is expensive and redundant since > the target > > > > > > already knows how to queue commands before committing > them to its > > > SCSI > > > > > > layer. > > > > > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > > > change the > > > > > > order of commands from the OS, but the DMAs those > commands need are > > > > > > unlikely to complete in the same order, and as I mentioned some > > > > > > commands need no DMA. If the HBA can't send commands > out of CmdSN > > > > > > order it has to maintain an ordered queue of commands > waiting to be > > > > > > sent, and potentially buffer a lot of data. For an HBA > this makes > > > > > > immediate data almost impossible to support. > > > > > > > > > > > > I don't see the problem with allowing out of > order commands > > > given > > > > > > that the target already has to deal with very similar > problems. I > > > > > > think we are getting in to the area of implementation > choices here, > > > > > > which is inappropriate for a specification. > > > > > > > > > > > > - Rod > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Julian Satran > > > > > > Sent: Monday, November 05, 2001 10:06 PM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: Re: iSCSI: Out of order commands, was current > UNH Plugfest > > > > > > > > > > > > Rod, > > > > > > > > > > > > I don't see any reason why DMA operations cant be > "multiplexed" with > > > > > > commands. > > > > > > If you have scheduled a long outbound DMA you are > doomed regardless > > > of > > > > > > the > > > > > > command ordering. > > > > > > And if you have scheduled DMA operations piecemeal then you can > > > insert > > > > > > your commands in correct order. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Rod Harrison" > > > > > > 05-11-01 20:48 > > > > > > Please respond to "Rod Harrison" > > > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > > > cc: > > > > > > Subject: iSCSI: Out of order commands, > was current > > > UNH > > > > > > Plugfest > > > > > > > > > > > > [ Subject changed ] > > > > > > > > > > > > Julian, > > > > > > > > > > > > The ordering difference is introduced > between the > > > > > > host > > > > > > side driver > > > > > > and the iSCSI HBA. The host side driver must present > SCSI commands > > > to > > > > > > the HBA in the order they are received from the OS to > prevent read > > > > > > after write dependency failures. The HBA might reorder > the commands > > > > > > depending on when DMA completes. The reordering can't > be done ahead > > > of > > > > > > time in the host driver since it doesn't know how long each DMA > > > might > > > > > > take. As long as the HBA assigns CmdSN in the order it receives > > > > > > commands the desired host ordering is preserved. > > > > > > > > > > > > - Rod > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Julian Satran > > > > > > Sent: Monday, November 05, 2001 12:35 AM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > > > Rod, > > > > > > > > > > > > I all examples give the point I find hard to understand > is why is > > > the > > > > > > ordering on the wire different from the presentation > order to the > > > > > > initiator. You can get as many overlaps as you want by > presenting > > > the > > > > > > commands to the initiator in the desired order. > > > > > > What we are considering here is the case in which you > want to ship > > > in > > > > > > an > > > > > > order different than the one you present the commands. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Rod Harrison" > > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > 04-11-01 04:42 > > > > > > Please respond to "Rod Harrison" > > > > > > > > > > > > To: "Barry Reinhold" , "Dave > > > > > > Sheehy" > > > > > > , "IETF IP SAN Reflector" > > > > > > > > > > > > cc: > > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > > > Barry, > > > > > > > > > > > > In general I agree but I don't think this is as > > > much > > > > > > of a > > > > > > corner case > > > > > > as it at first appears. Targets will have code very > similar to that > > > > > > needed to handle out of order commands to deal with > digest errors. > > > > > > Targets also need to queue commands whilst waiting for both > > > solicited > > > > > > and unsolicited data to arrive. Queuing out of order > commands seems > > > > > > little extra work. > > > > > > > > > > > > From an initiators point of view there are > > > > > > efficiency, > > > > > > and probably > > > > > > performance gains to be had from sending commands out > of order. Bob > > > > > > Russell gave the example of a read being sent whilst > write data DMA > > > is > > > > > > happening, and a similar situation can arise with DMA for writes > > > > > > overtaking that of earlier writes if the initiator has > multiple DMA > > > > > > engines. In this case the initiator might be forced to > let the wire > > > go > > > > > > idle if it can't send the data from completed DMAs as soon as > > > > > > possible. > > > > > > > > > > > > We already have a command queue at the > target to > > > > > > enforce > > > > > > correct > > > > > > serialisation of commands, doing the same thing at the > initiator is > > > > > > redundant. > > > > > > > > > > > > Finally, I don't believe we should be writing a > > > > > > standard > > > > > > to work > > > > > > around poor coding and test coverage, especially at the cost of > > > > > > potential efficiency gains. > > > > > > > > > > > > I agree with Dave and Santosh that > commands being > > > > > > sent > > > > > > out of order > > > > > > on a single session should be allowed by the standard. > > > > > > > > > > > > - Rod > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Barry Reinhold > > > > > > Sent: Friday, November 02, 2001 5:24 PM > > > > > > To: Dave Sheehy; IETF IP SAN Reflector > > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > > > Using features such as out of order command delivery on > a connection > > > > > > tend to > > > > > > be the sort of things that lead to interoperability > problems. It is > > > > > > unexpected and probably going to hit poorly tested code > paths even > > > if > > > > > > the > > > > > > standard is written to allow it. > > > > > > > > > > > > >-----Original Message----- > > > > > > >From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > > > Of > > > > > > >Dave Sheehy > > > > > > >Sent: Friday, November 02, 2001 4:19 PM > > > > > > >To: IETF IP SAN Reflector > > > > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > > > > >> > > > > > > >> The behavior of targets is clearly specified in Section > > > 2.2.2.3 > > > > > > on > > > > > > >> page 25 of draft 8, which says: > > > > > > >> "Except for the commands marked for immediate > delivery the > > > > > > iSCSI > > > > > > >> target layer MUST eliver the commands for > execution in the > > > > > > order > > > > > > >> specified by CmdSN." > > > > > > >> > > > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > > > > >> "- CmdSN - the current command Sequence Number > advanced by 1 > > > > > > on > > > > > > >> each command shipped except for commands marked for > > > immediate > > > > > > >> delivery." > > > > > > >> but the meaning of the term "shipped" is vague, > and does not > > > > > > >> necessarily > > > > > > >> require that the PDUs arrive on the other end of a TCP > > > > > > connection > > > > > > >> in the same order that the CmdSN values were > assigned to these > > > > > > PDUs. > > > > > > >> > > > > > > >> Some initiators have been designed to send commands out of > > > CmdSN > > > > > > >> order on one connection. Consider the situation > where there > > > is > > > > > > only > > > > > > >> one connection and a high-level dispatcher > creates a PDU for a > > > > > > SCSI > > > > > > >> command that involves writing immediate data to > the target. > > > > > > This PDU > > > > > > >> is enqueued to a lower-level layer which has to > setup, start, > > > > > > and > > > > > > >> wait-for a DMA operation to move the immediate > data into an > > > > > > onboard > > > > > > >> buffer before the PDU can be put onto the wire. > While this is > > > > > > >> happening, the dispatcher creates another > unrelated PDU for a > > > > > > SCSI > > > > > > >> read command (for example), and when this PDU is > passed to the > > > > > > >> lower-level layer it can be sent immediately, ahead of the > > > > > > previous > > > > > > >> write PDU and therefore out of order on this connection. > > > > > > >> > > > > > > >> The standard clearly allows this to happen if the two PDUs > > > were > > > > > > sent > > > > > > >> on different connections, and seems to imply that this can > > > also > > > > > > happen > > > > > > >> when the two PDUs are sent on the same connection. > > > > > > >> > > > > > > >> The suggestion is to put in the standard an > explicit statement > > > > > > that > > > > > > >> this is allowed or not allowed, as appropriate. > > > > > > >> > > > > > > >> If this is allowed, such a statement would avoid > the erroneous > > > > > > >> assumption being made by some target implementers > that within > > > a > > > > > > single > > > > > > >> connection, commands will arrive in order. > > > > > > >> > > > > > > >> If this is not allowed, such a statement would avoid the > > > > > > erroneous > > > > > > >> assumption being made by some initiator implementers that > > > within > > > > > > a > > > > > > >> single connection, commands can be put on the wire out of > > > order. > > > > > > >> > > > > > > >> +++ > > > > > > >> > > > > > > >> will add an explicit statement saying that this behaviour is > > > > > > forbidden. > > > > > > >> 2.2.2.1 will contain: > > > > > > >> > > > > > > >> On any given connection, the iSCSI initiator MUST send the > > > > > > >commands in the > > > > > > >> order specified by CmdSN. > > > > > > >> > > > > > > >> +++ > > > > > > > > > > > > > >Why do you feel this behavior should be forbidden? > Targets already > > > > > > have to > > > > > > >order commands across the session. I don't see why > it's a problem > > > to > > > > > > extend > > > > > > >that to the connection as well. I, for one, believe we > should take > > > > > > >a liberal > > > > > > >stance on this. > > > > > > > > > > > > > >Dave Sheehy > > > > > > > > > > > > > > > -- > > > > ################################## > > > > Santosh Rao > > > > Software Design Engineer, > > > > HP-UX iSCSI Driver Team, > > > > Hewlett Packard, Cupertino. > > > > email : santoshr@cup.hp.com > > > > Phone : 408-447-3751 > > > > ################################## > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Wed Nov 7 19:04:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26880 for ; Wed, 7 Nov 2001 19:04:42 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7Nlxp17118 for ips-outgoing; Wed, 7 Nov 2001 18:47:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from goliath.xo.com (goliath.xo.com [207.155.252.78]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7Nlrl17111 for ; Wed, 7 Nov 2001 18:47:56 -0500 (EST) Received: from rooster ([66.89.96.162]) by goliath.xo.com id SAA21886; Wed, 7 Nov 2001 18:47:50 -0500 (EST) [ConcentricHost SMTP Relay 1.14] From: "Dave Francheski" To: Cc: Subject: iSCSI initiator availability for Windows ? Date: Wed, 7 Nov 2001 15:39:05 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi all, This may not be the proper forum for this question, but I have exhausted almost all other options at this point. Does anybody know of a Windows based (e.g., Windows 2000) version of an iSCSI initiator implemented to Revision 6 (or greater) of the iSCSI standard? I have Cisco's version of the iSCSI Initiator for Windows but it is implemented to the first original draft of iSCSI, and doesn't interoperate with other more recent iSCSI target implementations. Thankyou for any and all support you can provide. regards, -Dave David Francheski Seven Systems Technologies 575 Menlo Drive Suite 2 Rocklin CA 916-577-1281 davef@seven-systems.com From owner-ips@ece.cmu.edu Wed Nov 7 19:09:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26913 for ; Wed, 7 Nov 2001 19:09:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7NCWY14669 for ips-outgoing; Wed, 7 Nov 2001 18:12:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls16.mediaone.net (chmls16.mediaone.net [24.147.1.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7NBjl14611 for ; Wed, 7 Nov 2001 18:11:49 -0500 (EST) Received: from breinhold ([140.186.40.85]) by chmls16.mediaone.net (8.11.1/8.11.1) with SMTP id fA7NAOT26397; Wed, 7 Nov 2001 18:10:25 -0500 (EST) From: "Barry Reinhold" To: "Robert D. Russell" , "Somesh Gupta" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 18:09:36 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Bob, I can't speak for targets, but OOO commands on a session with a single connection sure increases the complexity of the code path we take. To me the issue is still that these types of situations tend to be poorly tested and lead to interoperability issues. If we do go down this path, the spec. should make it very clear that this is expected behavior. Some statement with a shall in it should be written (for the receiver), so that there is a conformance test items to pass. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Robert D. Russell >Sent: Wednesday, November 07, 2001 4:13 PM >To: Somesh Gupta >Cc: Julian Satran; ips@ece.cmu.edu >Subject: RE: iSCSI: Out of order commands > > >Somesh, Julian: > >You state that dealing with OOO commands on the target >will add substantial complexity on the target. >Do you have any basis for that claim? My impression from the >plugfest is that most targets are already dealing with >it. Perhaps we need to hear from someone who is actually >building a target for which this would be a real problem. > >If anything, what we are hearing from people who really >are building initiators is that dealing with the requirement >to send commands in order will introduce substantial complexity >on the initiator. > >So why should we be saving complexity on (hypothetically) simple >targets yet requiring complexity on real initiators? > >As far as the deadlock issue is concerned, if the only way >that deadlock can occur with OOO commands on the same >connection is during the use of immediate data (which is I >think what Julian was saying), then shouldn't we change >the standard to just say that -- if the initiator sends >commands out of order on a single connection, then immediate >data MUST NOT be used on that connection in order to avoid deadlock. > >This gives everybody what they want, since initiators who find >it beneficial to deliver commands OOO will just negotiate >immediate data off. Those who really want to use immediate data >will have to ensure that commands are sent in order. >The tradeoff then becomes an implementation issue, not a >standards issue, which is where it belongs. > > >Bob Russell >InterOperability Lab >University of New Hampshire >rdr@iol.unh.edu >603-862-3774 > > >On Wed, 7 Nov 2001, Somesh Gupta wrote: > >> I think we should either have it as a MUST or not require >> it (at both ends to get the real benefit). SHOULD is one >> of those things that leads to implementation >> burden and confusion, without perhaps the feature being >> used. There are implementation as well as protocol >> considerations mixed in here. >> >> If we are to remove the restriction, we should (SHOULD) >> get the maximum benefit from it, rather than to >> accomodate an implementation choice. Out of sequence >> commands, combined with the possibility of digest errors, >> will add substantial complexity on the target side, >> without corrosponding benefit in performance. If we change >> this to SHOULD, we should also relax the requirement >> to present commands on the target side to a SHOULD. >> >> >> >> > -----Original Message----- >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >> > Julian Satran >> > Sent: Wednesday, November 07, 2001 10:00 AM >> > To: ips@ece.cmu.edu >> > Subject: Re: iSCSI: Out of order commands >> > >> > >> > Mallikarjun, >> > >> > I did not see a SINGLE performance improvement that results from OOO >> > shipping. >> > I would be bad engineering to give away the "no-deadlock" mechanism we >> > have now for nothing. >> > I have also the impression that the point about deadlock that I keep >> > repeating is ignored or not understood. >> > As we stand today commands can be shipped with Immediate data >or without >> > and an implementer determined >> > to squeeze maximum bandwidth and overlap command start with >delivery will >> > choose not to work with immediate data >> > (as you have pointed out) while a low performance software >implementation >> > will use immediate data to minimize CPU cycles consumed. However both >> > will be guaranteed to work without deadlock as source and sink use the >> > same ordering. >> > Recovery is still a low probability event and should be handled with a >> > different set of considerations in mind. >> > As for the strictness of the recommendation - yes we could settle on >> > SHOULD. >> > >> > Julo >> > >> > >> > >> > >> > "Mallikarjun C." >> > Sent by: owner-ips@ece.cmu.edu >> > 07-11-01 19:41 >> > Please respond to cbm >> > >> > >> > To: Santosh Rao , ips@ece.cmu.edu >> > cc: >> > Subject: Re: iSCSI: Out of order commands >> > >> > >> > >> > Santosh, >> > >> > I have only one comment on your responses. >> > >> > > Even a single connection target *MUST* implement a scoreboard. The >> > > reason being that it can see out-of-order arrival of commands due to >> > > commands being dropped on digest errors. In such a case, it >must block >> > > further command processing until holes are filled. >> > >> > I made two convenient assumptions if you noticed, :-), one of which >> > is that target forces session recovery on *any* error that it sees >> > (ErrorRecoveryLevel=0) - including a dropped command due to a digest >> > error. With that assumption, a target can afford not to implement >> > a scoreboard. >> > >> > As I said in a private note, I guess what primarily bothers me about >> > OOO commands on a connection is that it requires the receiver to >> > undo this "optimization" on its end - most notably on a single >> > connection. TCP experts may comment on how/if they dealt with a >> > similar issue. >> > >> > OTOH, you had some valid comments on exceptions to ordering during >> > connection recovery. Perhaps we can move on by making Julian's >> > proposed stipulation a SHOULD.... >> > -- >> > Mallikarjun >> > >> > >> > Mallikarjun Chadalapaka >> > Networked Storage Architecture >> > Network Storage Solutions Organization >> > MS 5668 Hewlett-Packard, Roseville. >> > cbm@rose.hp.com >> > >> > >> > Santosh Rao wrote: >> > > >> > > Mallikarjun, >> > > >> > > Some comments below. >> > > >> > > Regards, >> > > Santosh >> > > >> > > "Mallikarjun C." wrote: >> > > > >> > > > Rod and Julian, >> > > > >> > > > This has been an interesting thread of discussion. Some >> > > > comments - >> > > > >> > > > 1.My first reaction was - allowing out-of-order command >> > > > transmission on the same connection deprives targets of >> > > > an implementation choice. Targets which support only >> > > > single-connection sessions and only support session >> > > > recovery (reasonable assumptions in my mind) can no >> > > > longer afford *not to* implement a command scoreboard. >> > > >> > > Even a single connection target *MUST* implement a scoreboard. The >> > > reason being that it can see out-of-order arrival of commands due to >> > > commands being dropped on digest errors. In such a case, it >must block >> > > further command processing until holes are filled. >> > > >> > > Thus, there is no getting away from implementing a sequencer at the >> > > target. Given this, I think it is unreasonable to restrict initiator >> > > implementation flexibility by imposing a strict ordering requirement >> > > within the connection. >> > > >> > > > 2.Any end-node efficiency that is sought to be achieved >> > > > by transmitting CmdSNs out-of-order from the initiator >> > > > would be lost on the other end-node, since the target >> > > > now must wait for re-ordering the commands. >> > > >> > > It has to handle this situation anyway to deal with holes caused by >> > > digest errors. This scenario occurs even with initiators that issue >> > > commands in order. >> > > >> > > > >> > > > 3.The flipside is that out-of-order transmission saves >> > > > link badwidth (albeit at the expense of end-node efficiency), >> > > > compared to idling the link waiting for outbound DMA. >> > > > We have to determine if this is a reasonable trade-off. >> > > > >> > > > 4.I can see Rod's point that prefetching all immediate >> > > > data can be a burden on the NIC resources. But, two >> > > > questions - >> > > > - could the NIC not use unsolicited separate data >> > > > PDUs in these cases? [ I realize that InitialR2T >> > > > has to be "no" to let it happen... ] >> > > > - could the NIC have a memory architecture that >> > > > allows data prefetching for the next command (so >> > > > this is a non-issue from the protocol perspective)? >> > > > This scheme incurs one DMA delay for every new >> > > > burst of commands. >> > > > >> > > > 5.Another (perhaps radical at this point) option is to do >> > > > away with immediate unsolicited data, to stick only with >> > > > separate unsolicited data. I would personally be okay >> > > > with the choice, particularly if this feature (that >> > > > helps software implementations) starts making hardware >> > > > design complicated/expensive. >> > > > >> > > > So, to summarize - >> > > > >> > > > option immediate allow >> > > > data in spec? out-of-order? >> > > > >> > > > (A) (5) above no no >> > > > (B) No real reason to do this. no yes >> > > > (C) (4) above yes no >> > > > (D) pros & cons (1), (2) & (3) yes yes >> > > > >> > > > >From the arguments I heard so far, I am leaning towards >> > > > option A, and option C in that order. >> > > > >> > > > Comments? >> > > > -- >> > > > Mallikarjun >> > > > >> > > > Mallikarjun Chadalapaka >> > > > Networked Storage Architecture >> > > > Network Storage Solutions Organization >> > > > MS 5668 Hewlett-Packard, Roseville. >> > > > cbm@rose.hp.com >> > > > >> > > > Rod Harrison wrote: >> > > > > >> > > > > Julian, >> > > > > >> > > > > I don't understand what you are proposing here, >what do you >> > mean by >> > > > > "multiplexed" DMA? >> > > > > >> > > > > The problem is that the DMAs take some time, the >more there >> > are >> > > > > queued the longer the last DMAs queued take to complete. Some >> > commands >> > > > > require DMAs to complete before they can be sent, i.e. >Writes with >> > > > > immediate data, some commands do not, i.e. Reads and >writes with no >> > > > > immediate data. The iSCSI HBA wants to be able to send >commands as >> > > > > soon a possible, which for a read after a write can be before the >> > > > > write's DMA has completed. Maintaining an ordered queue >for commands >> > > > > to be sent on the HBA is expensive and redundant since the target >> > > > > already knows how to queue commands before committing them to its >> > SCSI >> > > > > layer. >> > > > > >> > > > > The iSCSI HBA and its host driver are not at liberty to >> > change the >> > > > > order of commands from the OS, but the DMAs those >commands need are >> > > > > unlikely to complete in the same order, and as I mentioned some >> > > > > commands need no DMA. If the HBA can't send commands out of CmdSN >> > > > > order it has to maintain an ordered queue of commands >waiting to be >> > > > > sent, and potentially buffer a lot of data. For an HBA this makes >> > > > > immediate data almost impossible to support. >> > > > > >> > > > > I don't see the problem with allowing out of >order commands >> > given >> > > > > that the target already has to deal with very similar problems. I >> > > > > think we are getting in to the area of implementation >choices here, >> > > > > which is inappropriate for a specification. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Julian Satran >> > > > > Sent: Monday, November 05, 2001 10:06 PM >> > > > > To: ips@ece.cmu.edu >> > > > > Subject: Re: iSCSI: Out of order commands, was current >UNH Plugfest >> > > > > >> > > > > Rod, >> > > > > >> > > > > I don't see any reason why DMA operations cant be >"multiplexed" with >> > > > > commands. >> > > > > If you have scheduled a long outbound DMA you are doomed >regardless >> > of >> > > > > the >> > > > > command ordering. >> > > > > And if you have scheduled DMA operations piecemeal then you can >> > insert >> > > > > your commands in correct order. >> > > > > >> > > > > Julo >> > > > > >> > > > > "Rod Harrison" >> > > > > 05-11-01 20:48 >> > > > > Please respond to "Rod Harrison" >> > > > > >> > > > > To: Julian Satran/Haifa/IBM@IBMIL, >> > > > > cc: >> > > > > Subject: iSCSI: Out of order commands, was current >> > UNH >> > > > > Plugfest >> > > > > >> > > > > [ Subject changed ] >> > > > > >> > > > > Julian, >> > > > > >> > > > > The ordering difference is introduced >between the >> > > > > host >> > > > > side driver >> > > > > and the iSCSI HBA. The host side driver must present >SCSI commands >> > to >> > > > > the HBA in the order they are received from the OS to >prevent read >> > > > > after write dependency failures. The HBA might reorder >the commands >> > > > > depending on when DMA completes. The reordering can't be >done ahead >> > of >> > > > > time in the host driver since it doesn't know how long each DMA >> > might >> > > > > take. As long as the HBA assigns CmdSN in the order it receives >> > > > > commands the desired host ordering is preserved. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Julian Satran >> > > > > Sent: Monday, November 05, 2001 12:35 AM >> > > > > To: ips@ece.cmu.edu >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Rod, >> > > > > >> > > > > I all examples give the point I find hard to understand is why is >> > the >> > > > > ordering on the wire different from the presentation order to the >> > > > > initiator. You can get as many overlaps as you want by >presenting >> > the >> > > > > commands to the initiator in the desired order. >> > > > > What we are considering here is the case in which you >want to ship >> > in >> > > > > an >> > > > > order different than the one you present the commands. >> > > > > >> > > > > Julo >> > > > > >> > > > > "Rod Harrison" >> > > > > Sent by: owner-ips@ece.cmu.edu >> > > > > 04-11-01 04:42 >> > > > > Please respond to "Rod Harrison" >> > > > > >> > > > > To: "Barry Reinhold" , "Dave >> > > > > Sheehy" >> > > > > , "IETF IP SAN Reflector" >> > > > > >> > > > > cc: >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Barry, >> > > > > >> > > > > In general I agree but I don't think this is as >> > much >> > > > > of a >> > > > > corner case >> > > > > as it at first appears. Targets will have code very >similar to that >> > > > > needed to handle out of order commands to deal with >digest errors. >> > > > > Targets also need to queue commands whilst waiting for both >> > solicited >> > > > > and unsolicited data to arrive. Queuing out of order >commands seems >> > > > > little extra work. >> > > > > >> > > > > From an initiators point of view there are >> > > > > efficiency, >> > > > > and probably >> > > > > performance gains to be had from sending commands out of >order. Bob >> > > > > Russell gave the example of a read being sent whilst >write data DMA >> > is >> > > > > happening, and a similar situation can arise with DMA for writes >> > > > > overtaking that of earlier writes if the initiator has >multiple DMA >> > > > > engines. In this case the initiator might be forced to >let the wire >> > go >> > > > > idle if it can't send the data from completed DMAs as soon as >> > > > > possible. >> > > > > >> > > > > We already have a command queue at the target to >> > > > > enforce >> > > > > correct >> > > > > serialisation of commands, doing the same thing at the >initiator is >> > > > > redundant. >> > > > > >> > > > > Finally, I don't believe we should be writing a >> > > > > standard >> > > > > to work >> > > > > around poor coding and test coverage, especially at the cost of >> > > > > potential efficiency gains. >> > > > > >> > > > > I agree with Dave and Santosh that >commands being >> > > > > sent >> > > > > out of order >> > > > > on a single session should be allowed by the standard. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Barry Reinhold >> > > > > Sent: Friday, November 02, 2001 5:24 PM >> > > > > To: Dave Sheehy; IETF IP SAN Reflector >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Using features such as out of order command delivery on >a connection >> > > > > tend to >> > > > > be the sort of things that lead to interoperability >problems. It is >> > > > > unexpected and probably going to hit poorly tested code >paths even >> > if >> > > > > the >> > > > > standard is written to allow it. >> > > > > >> > > > > >-----Original Message----- >> > > > > >From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > > > > Of >> > > > > >Dave Sheehy >> > > > > >Sent: Friday, November 02, 2001 4:19 PM >> > > > > >To: IETF IP SAN Reflector >> > > > > >Subject: Re: iSCSI: current UNH Plugfest >> > > > > > >> > > > > > >> > > > > > >> > > > > >> 3. Can commands be sent out of order on the same connection? >> > > > > >> >> > > > > >> The behavior of targets is clearly specified in Section >> > 2.2.2.3 >> > > > > on >> > > > > >> page 25 of draft 8, which says: >> > > > > >> "Except for the commands marked for immediate >delivery the >> > > > > iSCSI >> > > > > >> target layer MUST eliver the commands for >execution in the >> > > > > order >> > > > > >> specified by CmdSN." >> > > > > >> >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: >> > > > > >> "- CmdSN - the current command Sequence Number >advanced by 1 >> > > > > on >> > > > > >> each command shipped except for commands marked for >> > immediate >> > > > > >> delivery." >> > > > > >> but the meaning of the term "shipped" is vague, >and does not >> > > > > >> necessarily >> > > > > >> require that the PDUs arrive on the other end of a TCP >> > > > > connection >> > > > > >> in the same order that the CmdSN values were >assigned to these >> > > > > PDUs. >> > > > > >> >> > > > > >> Some initiators have been designed to send commands out of >> > CmdSN >> > > > > >> order on one connection. Consider the situation >where there >> > is >> > > > > only >> > > > > >> one connection and a high-level dispatcher creates >a PDU for a >> > > > > SCSI >> > > > > >> command that involves writing immediate data to the target. >> > > > > This PDU >> > > > > >> is enqueued to a lower-level layer which has to >setup, start, >> > > > > and >> > > > > >> wait-for a DMA operation to move the immediate data into an >> > > > > onboard >> > > > > >> buffer before the PDU can be put onto the wire. >While this is >> > > > > >> happening, the dispatcher creates another >unrelated PDU for a >> > > > > SCSI >> > > > > >> read command (for example), and when this PDU is >passed to the >> > > > > >> lower-level layer it can be sent immediately, ahead of the >> > > > > previous >> > > > > >> write PDU and therefore out of order on this connection. >> > > > > >> >> > > > > >> The standard clearly allows this to happen if the two PDUs >> > were >> > > > > sent >> > > > > >> on different connections, and seems to imply that this can >> > also >> > > > > happen >> > > > > >> when the two PDUs are sent on the same connection. >> > > > > >> >> > > > > >> The suggestion is to put in the standard an >explicit statement >> > > > > that >> > > > > >> this is allowed or not allowed, as appropriate. >> > > > > >> >> > > > > >> If this is allowed, such a statement would avoid >the erroneous >> > > > > >> assumption being made by some target implementers >that within >> > a >> > > > > single >> > > > > >> connection, commands will arrive in order. >> > > > > >> >> > > > > >> If this is not allowed, such a statement would avoid the >> > > > > erroneous >> > > > > >> assumption being made by some initiator implementers that >> > within >> > > > > a >> > > > > >> single connection, commands can be put on the wire out of >> > order. >> > > > > >> >> > > > > >> +++ >> > > > > >> >> > > > > >> will add an explicit statement saying that this behaviour is >> > > > > forbidden. >> > > > > >> 2.2.2.1 will contain: >> > > > > >> >> > > > > >> On any given connection, the iSCSI initiator MUST send the >> > > > > >commands in the >> > > > > >> order specified by CmdSN. >> > > > > >> >> > > > > >> +++ >> > > > > > >> > > > > >Why do you feel this behavior should be forbidden? >Targets already >> > > > > have to >> > > > > >order commands across the session. I don't see why it's >a problem >> > to >> > > > > extend >> > > > > >that to the connection as well. I, for one, believe we >should take >> > > > > >a liberal >> > > > > >stance on this. >> > > > > > >> > > > > >Dave Sheehy >> > > > > > >> > > >> > > -- >> > > ################################## >> > > Santosh Rao >> > > Software Design Engineer, >> > > HP-UX iSCSI Driver Team, >> > > Hewlett Packard, Cupertino. >> > > email : santoshr@cup.hp.com >> > > Phone : 408-447-3751 >> > > ################################## >> > >> > >> > >> > >> > From owner-ips@ece.cmu.edu Wed Nov 7 19:13:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26955 for ; Wed, 7 Nov 2001 19:13:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA7NIbH15100 for ips-outgoing; Wed, 7 Nov 2001 18:18:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA7NIZl15093 for ; Wed, 7 Nov 2001 18:18:35 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA7NITu11814; Wed, 7 Nov 2001 15:18:29 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA7NIO605392; Wed, 7 Nov 2001 15:18:24 -0800 From: "Bill Strahm" To: "Keith McCloghrie" , Subject: RE: FC Management MIB - issues Date: Wed, 7 Nov 2001 15:18:18 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: <200111072103.NAA04316@cisco.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit An interesting issue Keith... Currently there are a couple of tables in the FCIP MIB that AUGMENT tables in this MIB, and the FCIP MIB also imports a few of the indexes from the FCMGMT MIB. Will you be able to co-ordinate changes with the FCIP MIB editors (I am assuming they will not be able to adjust their MIB by next week) however can updated versions of these MIBs come out in the future ? Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Keith McCloghrie Sent: Wednesday, November 07, 2001 1:03 PM To: ips@ece.cmu.edu Cc: Keith McCloghrie Subject: FC Management MIB - issues Hi, As indicated by Elizabeth's message (below), I've agreed to be the interim editor for the FC Management MIB. So, first, I'd like to list a set of issues that have been raised concerning the most recent draft: draft-ietf-ipfc-fcmgmt-int-mib-07.txt. They are: 1. The use of SMIv2 is mandatory. 2. This MIB seems to have been defined with the notion that it will be the only MIB that a Fibre Channel product will require. The notion of an agent implementing just a single MIB was abandoned by the IETF in 1992 as being non-scaleable. Rather, this is another MIB in the continuing series of MIBs defined by the IETF, and thus, it needs to be consistent with its predecessors. In other words, there are existing MIBs which all SNMP agents must support, even if the support of Fibre Channel interfaces is the only functionality that they have. Thus, as the Fibre Channel Integration MIB, it is essential that this MIB contain only objects for information which is specific to Fibre Channel. All objects which apply in non-Fibre Channel environments need to be removed. (This applies even it were to require the definition of other new MIBs to contain information which is not already defined in other existing MIBs, but needed by Fibre Channel products.) Note that this issue applies to a large fraction of the objects defined in this MIB. 3. The text needs to include an explanation of the relationship between this MIB and RFC 2837. Is it a replacement or is it complementary ? If complementary, which agents implement which MIB; do some agents implement both ? 4. Every SNMP agent implements the ifTable. The ifTable counters are undoubtedly the MIB objects most well-used by administrators in SNMP management. SNMP agents need to implement a row in the ifTable for each of their network interfaces, including their Fibre Channel interfaces. The IF-MIB requires a media-specific MIB to specify how that type of interface uses the ifTable (see section 4 in RFC 2863). RFC 2837 doesn't do that, and nor (currently) does this MIB. That needs to be fixed. This will likely result in many tables in this MIB being indexed by ifIndex. 5. There are a number of objects related to the "Simple Name Service", and the definitions refer to Fibre Channel's GS-3 spec. However, GS-3 defines two Name Services: a Zoned Name Service and a Unzoned Name Service, but GS-3 does NOT use the term "Simple" for either of them !! This ambiguity needs to be resolved. 6. It is essential that the Counter32 or Counter64 (not OCTET STRINGs) syntax be used for counters. Thanks, Keith. -------------- Forwarded message: > From: "Elizabeth Rodriguez" > To: , > "IPS Mailing List (E-mail)" > Subject: FC Management MIB -- Transfer from IPFC To IPS WG > Cc: , , , > , , , > , > "Elizabeth Rodriguez" , > > > All, > > The IPFC working group has only one item left on its charter. > It is that of the FC Management MIB. It has been identified > that this MIB does not follow the IETF guidelines for MIBs > in its current format, in its lack of focus, and in its overlap > with existing MIBs. Rework of this MIB will take some time. > The content of this MIB will fit in well with the IPS WG, > especially because the subject matter experts participate in > this WG. > > For these reasons, the working group chairs, with the INT and TSV area > directors, have determined that this effort should be moved from the > IPFC working group to the IPS working group. Upon transferring of this > work, the IPFC working group will have completed the items in its > charter and the IPFC WG will be closed. > > The IPS working group has a technical advisor for MIB work -- Keith > McCloghrie. Since it has been determined that the current MIB has > issues with format, Keith McCloghrie has agreed to become the interim > editor of this MIB. As part of the re-architecture, the MIB will be > evaluated with respect to fit with other IPS WG MIBs, and may take the > form of a single new MIB or multiple MIBs, as appropriate. The IPS > working group will also be seeking Fibre Channel expertise to help > formulate the new MIB, including an editor with FC and MIB experience. > > The IPFC working group would also like to thank Steven Blumenau for all > his work on the original MIB. > > Elizabeth Rodriguez & David Black, IPS Working Group Chairs > Murali Rajagopal, IPFC Working Group Chair > From owner-ips@ece.cmu.edu Wed Nov 7 21:34:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA29421 for ; Wed, 7 Nov 2001 21:34:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA81nXa24701 for ips-outgoing; Wed, 7 Nov 2001 20:49:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from philotas.hosting.pacbell.net (philotas.hosting.pacbell.net [216.100.99.24]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA81nVl24695 for ; Wed, 7 Nov 2001 20:49:31 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by philotas.hosting.pacbell.net id UAA26719; Wed, 7 Nov 2001 20:49:00 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "Rod Harrison" , "Barry Reinhold" , "Robert D. Russell" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 17:46:13 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Rod, Sorry to butt in, but targets are not doing serialized processing. They will process many many commands simultaneously. We had prior discussions on this and unfortunately we agreed on ordered delivery at the iSCSI level. The real boost in performance is to have ordering semantics at a higher layer, so that the user code can determine whether it needs ordering or not. As an example, in the general/normal case of a computer talking to a disk, the ordering rule could be relaxed completely. Somesh > -----Original Message----- > From: Rod Harrison [mailto:rod.harrison@windriver.com] > Sent: Wednesday, November 07, 2001 5:34 PM > To: Barry Reinhold; Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Barry, > > I'm curious, where do you see the extra complexity between the > following? > > Assuming all the following commands are writes ... > > c1, c3, c2, c4 and > > c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. > > In both cases the target has to queue commands 2, 3, and 4 whilst > waiting for its R2T on c1 to be satisfied. > > Even if the target doesn't support any unsolicited data it still has > to queue commands 2, 3, and 4. I can't see how the target can avoid > queuing commands, in which case the order of arrival seems to make > little difference. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Barry Reinhold > Sent: Wednesday, November 07, 2001 3:10 PM > To: Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Bob, > I can't speak for targets, but OOO commands on a session with a > single > connection sure increases the complexity of the code path we take. > To me the issue is still that these types of situations tend to be > poorly > tested and lead to interoperability issues. If we do go down this > path, the > spec. should make it very clear that this is expected behavior. Some > statement with a shall in it should be written (for the receiver), so > that > there is a conformance test items to pass. > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > >Robert D. Russell > >Sent: Wednesday, November 07, 2001 4:13 PM > >To: Somesh Gupta > >Cc: Julian Satran; ips@ece.cmu.edu > >Subject: RE: iSCSI: Out of order commands > > > > > >Somesh, Julian: > > > >You state that dealing with OOO commands on the target > >will add substantial complexity on the target. > >Do you have any basis for that claim? My impression from the > >plugfest is that most targets are already dealing with > >it. Perhaps we need to hear from someone who is actually > >building a target for which this would be a real problem. > > > >If anything, what we are hearing from people who really > >are building initiators is that dealing with the requirement > >to send commands in order will introduce substantial complexity > >on the initiator. > > > >So why should we be saving complexity on (hypothetically) simple > >targets yet requiring complexity on real initiators? > > > >As far as the deadlock issue is concerned, if the only way > >that deadlock can occur with OOO commands on the same > >connection is during the use of immediate data (which is I > >think what Julian was saying), then shouldn't we change > >the standard to just say that -- if the initiator sends > >commands out of order on a single connection, then immediate > >data MUST NOT be used on that connection in order to avoid deadlock. > > > >This gives everybody what they want, since initiators who find > >it beneficial to deliver commands OOO will just negotiate > >immediate data off. Those who really want to use immediate data > >will have to ensure that commands are sent in order. > >The tradeoff then becomes an implementation issue, not a > >standards issue, which is where it belongs. > > > > > >Bob Russell > >InterOperability Lab > >University of New Hampshire > >rdr@iol.unh.edu > >603-862-3774 > > > > > >On Wed, 7 Nov 2001, Somesh Gupta wrote: > > > >> I think we should either have it as a MUST or not require > >> it (at both ends to get the real benefit). SHOULD is one > >> of those things that leads to implementation > >> burden and confusion, without perhaps the feature being > >> used. There are implementation as well as protocol > >> considerations mixed in here. > >> > >> If we are to remove the restriction, we should (SHOULD) > >> get the maximum benefit from it, rather than to > >> accomodate an implementation choice. Out of sequence > >> commands, combined with the possibility of digest errors, > >> will add substantial complexity on the target side, > >> without corrosponding benefit in performance. If we change > >> this to SHOULD, we should also relax the requirement > >> to present commands on the target side to a SHOULD. > >> > >> > >> > >> > -----Original Message----- > >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of > >> > Julian Satran > >> > Sent: Wednesday, November 07, 2001 10:00 AM > >> > To: ips@ece.cmu.edu > >> > Subject: Re: iSCSI: Out of order commands > >> > > >> > > >> > Mallikarjun, > >> > > >> > I did not see a SINGLE performance improvement that results from > OOO > >> > shipping. > >> > I would be bad engineering to give away the "no-deadlock" > mechanism we > >> > have now for nothing. > >> > I have also the impression that the point about deadlock that I > keep > >> > repeating is ignored or not understood. > >> > As we stand today commands can be shipped with Immediate data > >or without > >> > and an implementer determined > >> > to squeeze maximum bandwidth and overlap command start with > >delivery will > >> > choose not to work with immediate data > >> > (as you have pointed out) while a low performance software > >implementation > >> > will use immediate data to minimize CPU cycles consumed. However > both > >> > will be guaranteed to work without deadlock as source and sink > use the > >> > same ordering. > >> > Recovery is still a low probability event and should be handled > with a > >> > different set of considerations in mind. > >> > As for the strictness of the recommendation - yes we could settle > on > >> > SHOULD. > >> > > >> > Julo > >> > > >> > > >> > > >> > > >> > "Mallikarjun C." > >> > Sent by: owner-ips@ece.cmu.edu > >> > 07-11-01 19:41 > >> > Please respond to cbm > >> > > >> > > >> > To: Santosh Rao , > ips@ece.cmu.edu > >> > cc: > >> > Subject: Re: iSCSI: Out of order commands > >> > > >> > > >> > > >> > Santosh, > >> > > >> > I have only one comment on your responses. > >> > > >> > > Even a single connection target *MUST* implement a scoreboard. > The > >> > > reason being that it can see out-of-order arrival of commands > due to > >> > > commands being dropped on digest errors. In such a case, it > >must block > >> > > further command processing until holes are filled. > >> > > >> > I made two convenient assumptions if you noticed, :-), one of > which > >> > is that target forces session recovery on *any* error that it > sees > >> > (ErrorRecoveryLevel=0) - including a dropped command due to a > digest > >> > error. With that assumption, a target can afford not to > implement > >> > a scoreboard. > >> > > >> > As I said in a private note, I guess what primarily bothers me > about > >> > OOO commands on a connection is that it requires the receiver to > >> > undo this "optimization" on its end - most notably on a single > >> > connection. TCP experts may comment on how/if they dealt with a > >> > similar issue. > >> > > >> > OTOH, you had some valid comments on exceptions to ordering > during > >> > connection recovery. Perhaps we can move on by making Julian's > >> > proposed stipulation a SHOULD.... > >> > -- > >> > Mallikarjun > >> > > >> > > >> > Mallikarjun Chadalapaka > >> > Networked Storage Architecture > >> > Network Storage Solutions Organization > >> > MS 5668 Hewlett-Packard, Roseville. > >> > cbm@rose.hp.com > >> > > >> > > >> > Santosh Rao wrote: > >> > > > >> > > Mallikarjun, > >> > > > >> > > Some comments below. > >> > > > >> > > Regards, > >> > > Santosh > >> > > > >> > > "Mallikarjun C." wrote: > >> > > > > >> > > > Rod and Julian, > >> > > > > >> > > > This has been an interesting thread of discussion. Some > >> > > > comments - > >> > > > > >> > > > 1.My first reaction was - allowing out-of-order command > >> > > > transmission on the same connection deprives targets of > >> > > > an implementation choice. Targets which support only > >> > > > single-connection sessions and only support session > >> > > > recovery (reasonable assumptions in my mind) can no > >> > > > longer afford *not to* implement a command scoreboard. > >> > > > >> > > Even a single connection target *MUST* implement a scoreboard. > The > >> > > reason being that it can see out-of-order arrival of commands > due to > >> > > commands being dropped on digest errors. In such a case, it > >must block > >> > > further command processing until holes are filled. > >> > > > >> > > Thus, there is no getting away from implementing a sequencer at > the > >> > > target. Given this, I think it is unreasonable to restrict > initiator > >> > > implementation flexibility by imposing a strict ordering > requirement > >> > > within the connection. > >> > > > >> > > > 2.Any end-node efficiency that is sought to be achieved > >> > > > by transmitting CmdSNs out-of-order from the initiator > >> > > > would be lost on the other end-node, since the target > >> > > > now must wait for re-ordering the commands. > >> > > > >> > > It has to handle this situation anyway to deal with holes > caused by > >> > > digest errors. This scenario occurs even with initiators that > issue > >> > > commands in order. > >> > > > >> > > > > >> > > > 3.The flipside is that out-of-order transmission saves > >> > > > link badwidth (albeit at the expense of end-node > efficiency), > >> > > > compared to idling the link waiting for outbound DMA. > >> > > > We have to determine if this is a reasonable trade-off. > >> > > > > >> > > > 4.I can see Rod's point that prefetching all immediate > >> > > > data can be a burden on the NIC resources. But, two > >> > > > questions - > >> > > > - could the NIC not use unsolicited separate data > >> > > > PDUs in these cases? [ I realize that InitialR2T > >> > > > has to be "no" to let it happen... ] > >> > > > - could the NIC have a memory architecture that > >> > > > allows data prefetching for the next command (so > >> > > > this is a non-issue from the protocol perspective)? > >> > > > This scheme incurs one DMA delay for every new > >> > > > burst of commands. > >> > > > > >> > > > 5.Another (perhaps radical at this point) option is to do > >> > > > away with immediate unsolicited data, to stick only with > >> > > > separate unsolicited data. I would personally be okay > >> > > > with the choice, particularly if this feature (that > >> > > > helps software implementations) starts making hardware > >> > > > design complicated/expensive. > >> > > > > >> > > > So, to summarize - > >> > > > > >> > > > option immediate allow > >> > > > data in spec? > out-of-order? > >> > > > > >> > > > (A) (5) above no no > >> > > > (B) No real reason to do this. no yes > >> > > > (C) (4) above yes no > >> > > > (D) pros & cons (1), (2) & (3) yes yes > >> > > > > >> > > > >From the arguments I heard so far, I am leaning towards > >> > > > option A, and option C in that order. > >> > > > > >> > > > Comments? > >> > > > -- > >> > > > Mallikarjun > >> > > > > >> > > > Mallikarjun Chadalapaka > >> > > > Networked Storage Architecture > >> > > > Network Storage Solutions Organization > >> > > > MS 5668 Hewlett-Packard, Roseville. > >> > > > cbm@rose.hp.com > >> > > > > >> > > > Rod Harrison wrote: > >> > > > > > >> > > > > Julian, > >> > > > > > >> > > > > I don't understand what you are proposing here, > >what do you > >> > mean by > >> > > > > "multiplexed" DMA? > >> > > > > > >> > > > > The problem is that the DMAs take some time, the > >more there > >> > are > >> > > > > queued the longer the last DMAs queued take to complete. > Some > >> > commands > >> > > > > require DMAs to complete before they can be sent, i.e. > >Writes with > >> > > > > immediate data, some commands do not, i.e. Reads and > >writes with no > >> > > > > immediate data. The iSCSI HBA wants to be able to send > >commands as > >> > > > > soon a possible, which for a read after a write can be > before the > >> > > > > write's DMA has completed. Maintaining an ordered queue > >for commands > >> > > > > to be sent on the HBA is expensive and redundant since the > target > >> > > > > already knows how to queue commands before committing them > to its > >> > SCSI > >> > > > > layer. > >> > > > > > >> > > > > The iSCSI HBA and its host driver are not at > liberty to > >> > change the > >> > > > > order of commands from the OS, but the DMAs those > >commands need are > >> > > > > unlikely to complete in the same order, and as I mentioned > some > >> > > > > commands need no DMA. If the HBA can't send commands out of > CmdSN > >> > > > > order it has to maintain an ordered queue of commands > >waiting to be > >> > > > > sent, and potentially buffer a lot of data. For an HBA this > makes > >> > > > > immediate data almost impossible to support. > >> > > > > > >> > > > > I don't see the problem with allowing out of > >order commands > >> > given > >> > > > > that the target already has to deal with very similar > problems. I > >> > > > > think we are getting in to the area of implementation > >choices here, > >> > > > > which is inappropriate for a specification. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Julian Satran > >> > > > > Sent: Monday, November 05, 2001 10:06 PM > >> > > > > To: ips@ece.cmu.edu > >> > > > > Subject: Re: iSCSI: Out of order commands, was current > >UNH Plugfest > >> > > > > > >> > > > > Rod, > >> > > > > > >> > > > > I don't see any reason why DMA operations cant be > >"multiplexed" with > >> > > > > commands. > >> > > > > If you have scheduled a long outbound DMA you are doomed > >regardless > >> > of > >> > > > > the > >> > > > > command ordering. > >> > > > > And if you have scheduled DMA operations piecemeal then you > can > >> > insert > >> > > > > your commands in correct order. > >> > > > > > >> > > > > Julo > >> > > > > > >> > > > > "Rod Harrison" > >> > > > > 05-11-01 20:48 > >> > > > > Please respond to "Rod Harrison" > >> > > > > > >> > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > >> > > > > cc: > >> > > > > Subject: iSCSI: Out of order commands, was > current > >> > UNH > >> > > > > Plugfest > >> > > > > > >> > > > > [ Subject changed ] > >> > > > > > >> > > > > Julian, > >> > > > > > >> > > > > The ordering difference is introduced > >between the > >> > > > > host > >> > > > > side driver > >> > > > > and the iSCSI HBA. The host side driver must present > >SCSI commands > >> > to > >> > > > > the HBA in the order they are received from the OS to > >prevent read > >> > > > > after write dependency failures. The HBA might reorder > >the commands > >> > > > > depending on when DMA completes. The reordering can't be > >done ahead > >> > of > >> > > > > time in the host driver since it doesn't know how long each > DMA > >> > might > >> > > > > take. As long as the HBA assigns CmdSN in the order it > receives > >> > > > > commands the desired host ordering is preserved. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Julian Satran > >> > > > > Sent: Monday, November 05, 2001 12:35 AM > >> > > > > To: ips@ece.cmu.edu > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Rod, > >> > > > > > >> > > > > I all examples give the point I find hard to understand is > why is > >> > the > >> > > > > ordering on the wire different from the presentation order > to the > >> > > > > initiator. You can get as many overlaps as you want by > >presenting > >> > the > >> > > > > commands to the initiator in the desired order. > >> > > > > What we are considering here is the case in which you > >want to ship > >> > in > >> > > > > an > >> > > > > order different than the one you present the commands. > >> > > > > > >> > > > > Julo > >> > > > > > >> > > > > "Rod Harrison" > >> > > > > Sent by: owner-ips@ece.cmu.edu > >> > > > > 04-11-01 04:42 > >> > > > > Please respond to "Rod Harrison" > >> > > > > > >> > > > > To: "Barry Reinhold" , > "Dave > >> > > > > Sheehy" > >> > > > > , "IETF IP SAN Reflector" > >> > > > > > >> > > > > cc: > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Barry, > >> > > > > > >> > > > > In general I agree but I don't think this > is as > >> > much > >> > > > > of a > >> > > > > corner case > >> > > > > as it at first appears. Targets will have code very > >similar to that > >> > > > > needed to handle out of order commands to deal with > >digest errors. > >> > > > > Targets also need to queue commands whilst waiting for both > >> > solicited > >> > > > > and unsolicited data to arrive. Queuing out of order > >commands seems > >> > > > > little extra work. > >> > > > > > >> > > > > From an initiators point of view there are > >> > > > > efficiency, > >> > > > > and probably > >> > > > > performance gains to be had from sending commands out of > >order. Bob > >> > > > > Russell gave the example of a read being sent whilst > >write data DMA > >> > is > >> > > > > happening, and a similar situation can arise with DMA for > writes > >> > > > > overtaking that of earlier writes if the initiator has > >multiple DMA > >> > > > > engines. In this case the initiator might be forced to > >let the wire > >> > go > >> > > > > idle if it can't send the data from completed DMAs as soon > as > >> > > > > possible. > >> > > > > > >> > > > > We already have a command queue at the > target to > >> > > > > enforce > >> > > > > correct > >> > > > > serialisation of commands, doing the same thing at the > >initiator is > >> > > > > redundant. > >> > > > > > >> > > > > Finally, I don't believe we should be > writing a > >> > > > > standard > >> > > > > to work > >> > > > > around poor coding and test coverage, especially at the > cost of > >> > > > > potential efficiency gains. > >> > > > > > >> > > > > I agree with Dave and Santosh that > >commands being > >> > > > > sent > >> > > > > out of order > >> > > > > on a single session should be allowed by the standard. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Barry Reinhold > >> > > > > Sent: Friday, November 02, 2001 5:24 PM > >> > > > > To: Dave Sheehy; IETF IP SAN Reflector > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Using features such as out of order command delivery on > >a connection > >> > > > > tend to > >> > > > > be the sort of things that lead to interoperability > >problems. It is > >> > > > > unexpected and probably going to hit poorly tested code > >paths even > >> > if > >> > > > > the > >> > > > > standard is written to allow it. > >> > > > > > >> > > > > >-----Original Message----- > >> > > > > >From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > > > > Of > >> > > > > >Dave Sheehy > >> > > > > >Sent: Friday, November 02, 2001 4:19 PM > >> > > > > >To: IETF IP SAN Reflector > >> > > > > >Subject: Re: iSCSI: current UNH Plugfest > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > >> 3. Can commands be sent out of order on the same > connection? > >> > > > > >> > >> > > > > >> The behavior of targets is clearly specified in > Section > >> > 2.2.2.3 > >> > > > > on > >> > > > > >> page 25 of draft 8, which says: > >> > > > > >> "Except for the commands marked for immediate > >delivery the > >> > > > > iSCSI > >> > > > > >> target layer MUST eliver the commands for > >execution in the > >> > > > > order > >> > > > > >> specified by CmdSN." > >> > > > > >> > >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >> > > > > >> "- CmdSN - the current command Sequence Number > >advanced by 1 > >> > > > > on > >> > > > > >> each command shipped except for commands marked for > >> > immediate > >> > > > > >> delivery." > >> > > > > >> but the meaning of the term "shipped" is vague, > >and does not > >> > > > > >> necessarily > >> > > > > >> require that the PDUs arrive on the other end of a > TCP > >> > > > > connection > >> > > > > >> in the same order that the CmdSN values were > >assigned to these > >> > > > > PDUs. > >> > > > > >> > >> > > > > >> Some initiators have been designed to send commands > out of > >> > CmdSN > >> > > > > >> order on one connection. Consider the situation > >where there > >> > is > >> > > > > only > >> > > > > >> one connection and a high-level dispatcher creates > >a PDU for a > >> > > > > SCSI > >> > > > > >> command that involves writing immediate data to the > target. > >> > > > > This PDU > >> > > > > >> is enqueued to a lower-level layer which has to > >setup, start, > >> > > > > and > >> > > > > >> wait-for a DMA operation to move the immediate data > into an > >> > > > > onboard > >> > > > > >> buffer before the PDU can be put onto the wire. > >While this is > >> > > > > >> happening, the dispatcher creates another > >unrelated PDU for a > >> > > > > SCSI > >> > > > > >> read command (for example), and when this PDU is > >passed to the > >> > > > > >> lower-level layer it can be sent immediately, ahead > of the > >> > > > > previous > >> > > > > >> write PDU and therefore out of order on this > connection. > >> > > > > >> > >> > > > > >> The standard clearly allows this to happen if the two > PDUs > >> > were > >> > > > > sent > >> > > > > >> on different connections, and seems to imply that > this can > >> > also > >> > > > > happen > >> > > > > >> when the two PDUs are sent on the same connection. > >> > > > > >> > >> > > > > >> The suggestion is to put in the standard an > >explicit statement > >> > > > > that > >> > > > > >> this is allowed or not allowed, as appropriate. > >> > > > > >> > >> > > > > >> If this is allowed, such a statement would avoid > >the erroneous > >> > > > > >> assumption being made by some target implementers > >that within > >> > a > >> > > > > single > >> > > > > >> connection, commands will arrive in order. > >> > > > > >> > >> > > > > >> If this is not allowed, such a statement would avoid > the > >> > > > > erroneous > >> > > > > >> assumption being made by some initiator implementers > that > >> > within > >> > > > > a > >> > > > > >> single connection, commands can be put on the wire > out of > >> > order. > >> > > > > >> > >> > > > > >> +++ > >> > > > > >> > >> > > > > >> will add an explicit statement saying that this > behaviour is > >> > > > > forbidden. > >> > > > > >> 2.2.2.1 will contain: > >> > > > > >> > >> > > > > >> On any given connection, the iSCSI initiator MUST send > the > >> > > > > >commands in the > >> > > > > >> order specified by CmdSN. > >> > > > > >> > >> > > > > >> +++ > >> > > > > > > >> > > > > >Why do you feel this behavior should be forbidden? > >Targets already > >> > > > > have to > >> > > > > >order commands across the session. I don't see why it's > >a problem > >> > to > >> > > > > extend > >> > > > > >that to the connection as well. I, for one, believe we > >should take > >> > > > > >a liberal > >> > > > > >stance on this. > >> > > > > > > >> > > > > >Dave Sheehy > >> > > > > > > >> > > > >> > > -- > >> > > ################################## > >> > > Santosh Rao > >> > > Software Design Engineer, > >> > > HP-UX iSCSI Driver Team, > >> > > Hewlett Packard, Cupertino. > >> > > email : santoshr@cup.hp.com > >> > > Phone : 408-447-3751 > >> > > ################################## > >> > > >> > > >> > > >> > > >> > > > > From owner-ips@ece.cmu.edu Wed Nov 7 21:34:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA29598 for ; Wed, 7 Nov 2001 21:34:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA81Wk423623 for ips-outgoing; Wed, 7 Nov 2001 20:32:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.windriver.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA81WTl23609 for ; Wed, 7 Nov 2001 20:32:30 -0500 (EST) Received: from london ([192.168.1.62]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id RAA16795; Wed, 7 Nov 2001 17:31:04 -0800 (PST) From: "Rod Harrison" To: "Barry Reinhold" , "Robert D. Russell" , "Somesh Gupta" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 17:34:14 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Barry, I'm curious, where do you see the extra complexity between the following? Assuming all the following commands are writes ... c1, c3, c2, c4 and c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. In both cases the target has to queue commands 2, 3, and 4 whilst waiting for its R2T on c1 to be satisfied. Even if the target doesn't support any unsolicited data it still has to queue commands 2, 3, and 4. I can't see how the target can avoid queuing commands, in which case the order of arrival seems to make little difference. - Rod -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Barry Reinhold Sent: Wednesday, November 07, 2001 3:10 PM To: Robert D. Russell; Somesh Gupta Cc: Julian Satran; ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Bob, I can't speak for targets, but OOO commands on a session with a single connection sure increases the complexity of the code path we take. To me the issue is still that these types of situations tend to be poorly tested and lead to interoperability issues. If we do go down this path, the spec. should make it very clear that this is expected behavior. Some statement with a shall in it should be written (for the receiver), so that there is a conformance test items to pass. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Robert D. Russell >Sent: Wednesday, November 07, 2001 4:13 PM >To: Somesh Gupta >Cc: Julian Satran; ips@ece.cmu.edu >Subject: RE: iSCSI: Out of order commands > > >Somesh, Julian: > >You state that dealing with OOO commands on the target >will add substantial complexity on the target. >Do you have any basis for that claim? My impression from the >plugfest is that most targets are already dealing with >it. Perhaps we need to hear from someone who is actually >building a target for which this would be a real problem. > >If anything, what we are hearing from people who really >are building initiators is that dealing with the requirement >to send commands in order will introduce substantial complexity >on the initiator. > >So why should we be saving complexity on (hypothetically) simple >targets yet requiring complexity on real initiators? > >As far as the deadlock issue is concerned, if the only way >that deadlock can occur with OOO commands on the same >connection is during the use of immediate data (which is I >think what Julian was saying), then shouldn't we change >the standard to just say that -- if the initiator sends >commands out of order on a single connection, then immediate >data MUST NOT be used on that connection in order to avoid deadlock. > >This gives everybody what they want, since initiators who find >it beneficial to deliver commands OOO will just negotiate >immediate data off. Those who really want to use immediate data >will have to ensure that commands are sent in order. >The tradeoff then becomes an implementation issue, not a >standards issue, which is where it belongs. > > >Bob Russell >InterOperability Lab >University of New Hampshire >rdr@iol.unh.edu >603-862-3774 > > >On Wed, 7 Nov 2001, Somesh Gupta wrote: > >> I think we should either have it as a MUST or not require >> it (at both ends to get the real benefit). SHOULD is one >> of those things that leads to implementation >> burden and confusion, without perhaps the feature being >> used. There are implementation as well as protocol >> considerations mixed in here. >> >> If we are to remove the restriction, we should (SHOULD) >> get the maximum benefit from it, rather than to >> accomodate an implementation choice. Out of sequence >> commands, combined with the possibility of digest errors, >> will add substantial complexity on the target side, >> without corrosponding benefit in performance. If we change >> this to SHOULD, we should also relax the requirement >> to present commands on the target side to a SHOULD. >> >> >> >> > -----Original Message----- >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >> > Julian Satran >> > Sent: Wednesday, November 07, 2001 10:00 AM >> > To: ips@ece.cmu.edu >> > Subject: Re: iSCSI: Out of order commands >> > >> > >> > Mallikarjun, >> > >> > I did not see a SINGLE performance improvement that results from OOO >> > shipping. >> > I would be bad engineering to give away the "no-deadlock" mechanism we >> > have now for nothing. >> > I have also the impression that the point about deadlock that I keep >> > repeating is ignored or not understood. >> > As we stand today commands can be shipped with Immediate data >or without >> > and an implementer determined >> > to squeeze maximum bandwidth and overlap command start with >delivery will >> > choose not to work with immediate data >> > (as you have pointed out) while a low performance software >implementation >> > will use immediate data to minimize CPU cycles consumed. However both >> > will be guaranteed to work without deadlock as source and sink use the >> > same ordering. >> > Recovery is still a low probability event and should be handled with a >> > different set of considerations in mind. >> > As for the strictness of the recommendation - yes we could settle on >> > SHOULD. >> > >> > Julo >> > >> > >> > >> > >> > "Mallikarjun C." >> > Sent by: owner-ips@ece.cmu.edu >> > 07-11-01 19:41 >> > Please respond to cbm >> > >> > >> > To: Santosh Rao , ips@ece.cmu.edu >> > cc: >> > Subject: Re: iSCSI: Out of order commands >> > >> > >> > >> > Santosh, >> > >> > I have only one comment on your responses. >> > >> > > Even a single connection target *MUST* implement a scoreboard. The >> > > reason being that it can see out-of-order arrival of commands due to >> > > commands being dropped on digest errors. In such a case, it >must block >> > > further command processing until holes are filled. >> > >> > I made two convenient assumptions if you noticed, :-), one of which >> > is that target forces session recovery on *any* error that it sees >> > (ErrorRecoveryLevel=0) - including a dropped command due to a digest >> > error. With that assumption, a target can afford not to implement >> > a scoreboard. >> > >> > As I said in a private note, I guess what primarily bothers me about >> > OOO commands on a connection is that it requires the receiver to >> > undo this "optimization" on its end - most notably on a single >> > connection. TCP experts may comment on how/if they dealt with a >> > similar issue. >> > >> > OTOH, you had some valid comments on exceptions to ordering during >> > connection recovery. Perhaps we can move on by making Julian's >> > proposed stipulation a SHOULD.... >> > -- >> > Mallikarjun >> > >> > >> > Mallikarjun Chadalapaka >> > Networked Storage Architecture >> > Network Storage Solutions Organization >> > MS 5668 Hewlett-Packard, Roseville. >> > cbm@rose.hp.com >> > >> > >> > Santosh Rao wrote: >> > > >> > > Mallikarjun, >> > > >> > > Some comments below. >> > > >> > > Regards, >> > > Santosh >> > > >> > > "Mallikarjun C." wrote: >> > > > >> > > > Rod and Julian, >> > > > >> > > > This has been an interesting thread of discussion. Some >> > > > comments - >> > > > >> > > > 1.My first reaction was - allowing out-of-order command >> > > > transmission on the same connection deprives targets of >> > > > an implementation choice. Targets which support only >> > > > single-connection sessions and only support session >> > > > recovery (reasonable assumptions in my mind) can no >> > > > longer afford *not to* implement a command scoreboard. >> > > >> > > Even a single connection target *MUST* implement a scoreboard. The >> > > reason being that it can see out-of-order arrival of commands due to >> > > commands being dropped on digest errors. In such a case, it >must block >> > > further command processing until holes are filled. >> > > >> > > Thus, there is no getting away from implementing a sequencer at the >> > > target. Given this, I think it is unreasonable to restrict initiator >> > > implementation flexibility by imposing a strict ordering requirement >> > > within the connection. >> > > >> > > > 2.Any end-node efficiency that is sought to be achieved >> > > > by transmitting CmdSNs out-of-order from the initiator >> > > > would be lost on the other end-node, since the target >> > > > now must wait for re-ordering the commands. >> > > >> > > It has to handle this situation anyway to deal with holes caused by >> > > digest errors. This scenario occurs even with initiators that issue >> > > commands in order. >> > > >> > > > >> > > > 3.The flipside is that out-of-order transmission saves >> > > > link badwidth (albeit at the expense of end-node efficiency), >> > > > compared to idling the link waiting for outbound DMA. >> > > > We have to determine if this is a reasonable trade-off. >> > > > >> > > > 4.I can see Rod's point that prefetching all immediate >> > > > data can be a burden on the NIC resources. But, two >> > > > questions - >> > > > - could the NIC not use unsolicited separate data >> > > > PDUs in these cases? [ I realize that InitialR2T >> > > > has to be "no" to let it happen... ] >> > > > - could the NIC have a memory architecture that >> > > > allows data prefetching for the next command (so >> > > > this is a non-issue from the protocol perspective)? >> > > > This scheme incurs one DMA delay for every new >> > > > burst of commands. >> > > > >> > > > 5.Another (perhaps radical at this point) option is to do >> > > > away with immediate unsolicited data, to stick only with >> > > > separate unsolicited data. I would personally be okay >> > > > with the choice, particularly if this feature (that >> > > > helps software implementations) starts making hardware >> > > > design complicated/expensive. >> > > > >> > > > So, to summarize - >> > > > >> > > > option immediate allow >> > > > data in spec? out-of-order? >> > > > >> > > > (A) (5) above no no >> > > > (B) No real reason to do this. no yes >> > > > (C) (4) above yes no >> > > > (D) pros & cons (1), (2) & (3) yes yes >> > > > >> > > > >From the arguments I heard so far, I am leaning towards >> > > > option A, and option C in that order. >> > > > >> > > > Comments? >> > > > -- >> > > > Mallikarjun >> > > > >> > > > Mallikarjun Chadalapaka >> > > > Networked Storage Architecture >> > > > Network Storage Solutions Organization >> > > > MS 5668 Hewlett-Packard, Roseville. >> > > > cbm@rose.hp.com >> > > > >> > > > Rod Harrison wrote: >> > > > > >> > > > > Julian, >> > > > > >> > > > > I don't understand what you are proposing here, >what do you >> > mean by >> > > > > "multiplexed" DMA? >> > > > > >> > > > > The problem is that the DMAs take some time, the >more there >> > are >> > > > > queued the longer the last DMAs queued take to complete. Some >> > commands >> > > > > require DMAs to complete before they can be sent, i.e. >Writes with >> > > > > immediate data, some commands do not, i.e. Reads and >writes with no >> > > > > immediate data. The iSCSI HBA wants to be able to send >commands as >> > > > > soon a possible, which for a read after a write can be before the >> > > > > write's DMA has completed. Maintaining an ordered queue >for commands >> > > > > to be sent on the HBA is expensive and redundant since the target >> > > > > already knows how to queue commands before committing them to its >> > SCSI >> > > > > layer. >> > > > > >> > > > > The iSCSI HBA and its host driver are not at liberty to >> > change the >> > > > > order of commands from the OS, but the DMAs those >commands need are >> > > > > unlikely to complete in the same order, and as I mentioned some >> > > > > commands need no DMA. If the HBA can't send commands out of CmdSN >> > > > > order it has to maintain an ordered queue of commands >waiting to be >> > > > > sent, and potentially buffer a lot of data. For an HBA this makes >> > > > > immediate data almost impossible to support. >> > > > > >> > > > > I don't see the problem with allowing out of >order commands >> > given >> > > > > that the target already has to deal with very similar problems. I >> > > > > think we are getting in to the area of implementation >choices here, >> > > > > which is inappropriate for a specification. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Julian Satran >> > > > > Sent: Monday, November 05, 2001 10:06 PM >> > > > > To: ips@ece.cmu.edu >> > > > > Subject: Re: iSCSI: Out of order commands, was current >UNH Plugfest >> > > > > >> > > > > Rod, >> > > > > >> > > > > I don't see any reason why DMA operations cant be >"multiplexed" with >> > > > > commands. >> > > > > If you have scheduled a long outbound DMA you are doomed >regardless >> > of >> > > > > the >> > > > > command ordering. >> > > > > And if you have scheduled DMA operations piecemeal then you can >> > insert >> > > > > your commands in correct order. >> > > > > >> > > > > Julo >> > > > > >> > > > > "Rod Harrison" >> > > > > 05-11-01 20:48 >> > > > > Please respond to "Rod Harrison" >> > > > > >> > > > > To: Julian Satran/Haifa/IBM@IBMIL, >> > > > > cc: >> > > > > Subject: iSCSI: Out of order commands, was current >> > UNH >> > > > > Plugfest >> > > > > >> > > > > [ Subject changed ] >> > > > > >> > > > > Julian, >> > > > > >> > > > > The ordering difference is introduced >between the >> > > > > host >> > > > > side driver >> > > > > and the iSCSI HBA. The host side driver must present >SCSI commands >> > to >> > > > > the HBA in the order they are received from the OS to >prevent read >> > > > > after write dependency failures. The HBA might reorder >the commands >> > > > > depending on when DMA completes. The reordering can't be >done ahead >> > of >> > > > > time in the host driver since it doesn't know how long each DMA >> > might >> > > > > take. As long as the HBA assigns CmdSN in the order it receives >> > > > > commands the desired host ordering is preserved. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Julian Satran >> > > > > Sent: Monday, November 05, 2001 12:35 AM >> > > > > To: ips@ece.cmu.edu >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Rod, >> > > > > >> > > > > I all examples give the point I find hard to understand is why is >> > the >> > > > > ordering on the wire different from the presentation order to the >> > > > > initiator. You can get as many overlaps as you want by >presenting >> > the >> > > > > commands to the initiator in the desired order. >> > > > > What we are considering here is the case in which you >want to ship >> > in >> > > > > an >> > > > > order different than the one you present the commands. >> > > > > >> > > > > Julo >> > > > > >> > > > > "Rod Harrison" >> > > > > Sent by: owner-ips@ece.cmu.edu >> > > > > 04-11-01 04:42 >> > > > > Please respond to "Rod Harrison" >> > > > > >> > > > > To: "Barry Reinhold" , "Dave >> > > > > Sheehy" >> > > > > , "IETF IP SAN Reflector" >> > > > > >> > > > > cc: >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Barry, >> > > > > >> > > > > In general I agree but I don't think this is as >> > much >> > > > > of a >> > > > > corner case >> > > > > as it at first appears. Targets will have code very >similar to that >> > > > > needed to handle out of order commands to deal with >digest errors. >> > > > > Targets also need to queue commands whilst waiting for both >> > solicited >> > > > > and unsolicited data to arrive. Queuing out of order >commands seems >> > > > > little extra work. >> > > > > >> > > > > From an initiators point of view there are >> > > > > efficiency, >> > > > > and probably >> > > > > performance gains to be had from sending commands out of >order. Bob >> > > > > Russell gave the example of a read being sent whilst >write data DMA >> > is >> > > > > happening, and a similar situation can arise with DMA for writes >> > > > > overtaking that of earlier writes if the initiator has >multiple DMA >> > > > > engines. In this case the initiator might be forced to >let the wire >> > go >> > > > > idle if it can't send the data from completed DMAs as soon as >> > > > > possible. >> > > > > >> > > > > We already have a command queue at the target to >> > > > > enforce >> > > > > correct >> > > > > serialisation of commands, doing the same thing at the >initiator is >> > > > > redundant. >> > > > > >> > > > > Finally, I don't believe we should be writing a >> > > > > standard >> > > > > to work >> > > > > around poor coding and test coverage, especially at the cost of >> > > > > potential efficiency gains. >> > > > > >> > > > > I agree with Dave and Santosh that >commands being >> > > > > sent >> > > > > out of order >> > > > > on a single session should be allowed by the standard. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Barry Reinhold >> > > > > Sent: Friday, November 02, 2001 5:24 PM >> > > > > To: Dave Sheehy; IETF IP SAN Reflector >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Using features such as out of order command delivery on >a connection >> > > > > tend to >> > > > > be the sort of things that lead to interoperability >problems. It is >> > > > > unexpected and probably going to hit poorly tested code >paths even >> > if >> > > > > the >> > > > > standard is written to allow it. >> > > > > >> > > > > >-----Original Message----- >> > > > > >From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > > > > Of >> > > > > >Dave Sheehy >> > > > > >Sent: Friday, November 02, 2001 4:19 PM >> > > > > >To: IETF IP SAN Reflector >> > > > > >Subject: Re: iSCSI: current UNH Plugfest >> > > > > > >> > > > > > >> > > > > > >> > > > > >> 3. Can commands be sent out of order on the same connection? >> > > > > >> >> > > > > >> The behavior of targets is clearly specified in Section >> > 2.2.2.3 >> > > > > on >> > > > > >> page 25 of draft 8, which says: >> > > > > >> "Except for the commands marked for immediate >delivery the >> > > > > iSCSI >> > > > > >> target layer MUST eliver the commands for >execution in the >> > > > > order >> > > > > >> specified by CmdSN." >> > > > > >> >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: >> > > > > >> "- CmdSN - the current command Sequence Number >advanced by 1 >> > > > > on >> > > > > >> each command shipped except for commands marked for >> > immediate >> > > > > >> delivery." >> > > > > >> but the meaning of the term "shipped" is vague, >and does not >> > > > > >> necessarily >> > > > > >> require that the PDUs arrive on the other end of a TCP >> > > > > connection >> > > > > >> in the same order that the CmdSN values were >assigned to these >> > > > > PDUs. >> > > > > >> >> > > > > >> Some initiators have been designed to send commands out of >> > CmdSN >> > > > > >> order on one connection. Consider the situation >where there >> > is >> > > > > only >> > > > > >> one connection and a high-level dispatcher creates >a PDU for a >> > > > > SCSI >> > > > > >> command that involves writing immediate data to the target. >> > > > > This PDU >> > > > > >> is enqueued to a lower-level layer which has to >setup, start, >> > > > > and >> > > > > >> wait-for a DMA operation to move the immediate data into an >> > > > > onboard >> > > > > >> buffer before the PDU can be put onto the wire. >While this is >> > > > > >> happening, the dispatcher creates another >unrelated PDU for a >> > > > > SCSI >> > > > > >> read command (for example), and when this PDU is >passed to the >> > > > > >> lower-level layer it can be sent immediately, ahead of the >> > > > > previous >> > > > > >> write PDU and therefore out of order on this connection. >> > > > > >> >> > > > > >> The standard clearly allows this to happen if the two PDUs >> > were >> > > > > sent >> > > > > >> on different connections, and seems to imply that this can >> > also >> > > > > happen >> > > > > >> when the two PDUs are sent on the same connection. >> > > > > >> >> > > > > >> The suggestion is to put in the standard an >explicit statement >> > > > > that >> > > > > >> this is allowed or not allowed, as appropriate. >> > > > > >> >> > > > > >> If this is allowed, such a statement would avoid >the erroneous >> > > > > >> assumption being made by some target implementers >that within >> > a >> > > > > single >> > > > > >> connection, commands will arrive in order. >> > > > > >> >> > > > > >> If this is not allowed, such a statement would avoid the >> > > > > erroneous >> > > > > >> assumption being made by some initiator implementers that >> > within >> > > > > a >> > > > > >> single connection, commands can be put on the wire out of >> > order. >> > > > > >> >> > > > > >> +++ >> > > > > >> >> > > > > >> will add an explicit statement saying that this behaviour is >> > > > > forbidden. >> > > > > >> 2.2.2.1 will contain: >> > > > > >> >> > > > > >> On any given connection, the iSCSI initiator MUST send the >> > > > > >commands in the >> > > > > >> order specified by CmdSN. >> > > > > >> >> > > > > >> +++ >> > > > > > >> > > > > >Why do you feel this behavior should be forbidden? >Targets already >> > > > > have to >> > > > > >order commands across the session. I don't see why it's >a problem >> > to >> > > > > extend >> > > > > >that to the connection as well. I, for one, believe we >should take >> > > > > >a liberal >> > > > > >stance on this. >> > > > > > >> > > > > >Dave Sheehy >> > > > > > >> > > >> > > -- >> > > ################################## >> > > Santosh Rao >> > > Software Design Engineer, >> > > HP-UX iSCSI Driver Team, >> > > Hewlett Packard, Cupertino. >> > > email : santoshr@cup.hp.com >> > > Phone : 408-447-3751 >> > > ################################## >> > >> > >> > >> > >> > From owner-ips@ece.cmu.edu Wed Nov 7 22:25:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA29468 for ; Wed, 7 Nov 2001 21:34:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA81vPg25315 for ips-outgoing; Wed, 7 Nov 2001 20:57:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.wrs.com (unknown-1-11.windriver.com [147.11.1.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA81v9l25297 for ; Wed, 7 Nov 2001 20:57:09 -0500 (EST) Received: from london ([192.168.1.62]) by mail.wrs.com (8.9.3/8.9.1) with SMTP id RAA02123; Wed, 7 Nov 2001 17:54:46 -0800 (PST) From: "Rod Harrison" To: , "Barry Reinhold" , "Robert D. Russell" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Wed, 7 Nov 2001 17:57:56 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh, The spec prohibits processing commands simultaneously, if by processing you mean handing off to the SCSI layer. If by processing simultaneously you mean preparing data for committing to SCSI when the CmdSN allows then surely reception of commands out of order is a no brainer. By the way, I am in support of allowing the target to make the decision on when to commit the SCSI layer rather than having to use the CmdSN. That decision can be made on the basis of device type, LUN, and queue tag. Forcing CmdSN order of committal to SCSI has always seemed overly restrictive to me. - Rod -----Original Message----- From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] Sent: Wednesday, November 07, 2001 5:46 PM To: Rod Harrison; Barry Reinhold; Robert D. Russell Cc: Julian Satran; ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Rod, Sorry to butt in, but targets are not doing serialized processing. They will process many many commands simultaneously. We had prior discussions on this and unfortunately we agreed on ordered delivery at the iSCSI level. The real boost in performance is to have ordering semantics at a higher layer, so that the user code can determine whether it needs ordering or not. As an example, in the general/normal case of a computer talking to a disk, the ordering rule could be relaxed completely. Somesh > -----Original Message----- > From: Rod Harrison [mailto:rod.harrison@windriver.com] > Sent: Wednesday, November 07, 2001 5:34 PM > To: Barry Reinhold; Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Barry, > > I'm curious, where do you see the extra complexity between the > following? > > Assuming all the following commands are writes ... > > c1, c3, c2, c4 and > > c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. > > In both cases the target has to queue commands 2, 3, and 4 whilst > waiting for its R2T on c1 to be satisfied. > > Even if the target doesn't support any unsolicited data it still has > to queue commands 2, 3, and 4. I can't see how the target can avoid > queuing commands, in which case the order of arrival seems to make > little difference. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Barry Reinhold > Sent: Wednesday, November 07, 2001 3:10 PM > To: Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Bob, > I can't speak for targets, but OOO commands on a session with a > single > connection sure increases the complexity of the code path we take. > To me the issue is still that these types of situations tend to be > poorly > tested and lead to interoperability issues. If we do go down this > path, the > spec. should make it very clear that this is expected behavior. Some > statement with a shall in it should be written (for the receiver), so > that > there is a conformance test items to pass. > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > >Robert D. Russell > >Sent: Wednesday, November 07, 2001 4:13 PM > >To: Somesh Gupta > >Cc: Julian Satran; ips@ece.cmu.edu > >Subject: RE: iSCSI: Out of order commands > > > > > >Somesh, Julian: > > > >You state that dealing with OOO commands on the target > >will add substantial complexity on the target. > >Do you have any basis for that claim? My impression from the > >plugfest is that most targets are already dealing with > >it. Perhaps we need to hear from someone who is actually > >building a target for which this would be a real problem. > > > >If anything, what we are hearing from people who really > >are building initiators is that dealing with the requirement > >to send commands in order will introduce substantial complexity > >on the initiator. > > > >So why should we be saving complexity on (hypothetically) simple > >targets yet requiring complexity on real initiators? > > > >As far as the deadlock issue is concerned, if the only way > >that deadlock can occur with OOO commands on the same > >connection is during the use of immediate data (which is I > >think what Julian was saying), then shouldn't we change > >the standard to just say that -- if the initiator sends > >commands out of order on a single connection, then immediate > >data MUST NOT be used on that connection in order to avoid deadlock. > > > >This gives everybody what they want, since initiators who find > >it beneficial to deliver commands OOO will just negotiate > >immediate data off. Those who really want to use immediate data > >will have to ensure that commands are sent in order. > >The tradeoff then becomes an implementation issue, not a > >standards issue, which is where it belongs. > > > > > >Bob Russell > >InterOperability Lab > >University of New Hampshire > >rdr@iol.unh.edu > >603-862-3774 > > > > > >On Wed, 7 Nov 2001, Somesh Gupta wrote: > > > >> I think we should either have it as a MUST or not require > >> it (at both ends to get the real benefit). SHOULD is one > >> of those things that leads to implementation > >> burden and confusion, without perhaps the feature being > >> used. There are implementation as well as protocol > >> considerations mixed in here. > >> > >> If we are to remove the restriction, we should (SHOULD) > >> get the maximum benefit from it, rather than to > >> accomodate an implementation choice. Out of sequence > >> commands, combined with the possibility of digest errors, > >> will add substantial complexity on the target side, > >> without corrosponding benefit in performance. If we change > >> this to SHOULD, we should also relax the requirement > >> to present commands on the target side to a SHOULD. > >> > >> > >> > >> > -----Original Message----- > >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of > >> > Julian Satran > >> > Sent: Wednesday, November 07, 2001 10:00 AM > >> > To: ips@ece.cmu.edu > >> > Subject: Re: iSCSI: Out of order commands > >> > > >> > > >> > Mallikarjun, > >> > > >> > I did not see a SINGLE performance improvement that results from > OOO > >> > shipping. > >> > I would be bad engineering to give away the "no-deadlock" > mechanism we > >> > have now for nothing. > >> > I have also the impression that the point about deadlock that I > keep > >> > repeating is ignored or not understood. > >> > As we stand today commands can be shipped with Immediate data > >or without > >> > and an implementer determined > >> > to squeeze maximum bandwidth and overlap command start with > >delivery will > >> > choose not to work with immediate data > >> > (as you have pointed out) while a low performance software > >implementation > >> > will use immediate data to minimize CPU cycles consumed. However > both > >> > will be guaranteed to work without deadlock as source and sink > use the > >> > same ordering. > >> > Recovery is still a low probability event and should be handled > with a > >> > different set of considerations in mind. > >> > As for the strictness of the recommendation - yes we could settle > on > >> > SHOULD. > >> > > >> > Julo > >> > > >> > > >> > > >> > > >> > "Mallikarjun C." > >> > Sent by: owner-ips@ece.cmu.edu > >> > 07-11-01 19:41 > >> > Please respond to cbm > >> > > >> > > >> > To: Santosh Rao , > ips@ece.cmu.edu > >> > cc: > >> > Subject: Re: iSCSI: Out of order commands > >> > > >> > > >> > > >> > Santosh, > >> > > >> > I have only one comment on your responses. > >> > > >> > > Even a single connection target *MUST* implement a scoreboard. > The > >> > > reason being that it can see out-of-order arrival of commands > due to > >> > > commands being dropped on digest errors. In such a case, it > >must block > >> > > further command processing until holes are filled. > >> > > >> > I made two convenient assumptions if you noticed, :-), one of > which > >> > is that target forces session recovery on *any* error that it > sees > >> > (ErrorRecoveryLevel=0) - including a dropped command due to a > digest > >> > error. With that assumption, a target can afford not to > implement > >> > a scoreboard. > >> > > >> > As I said in a private note, I guess what primarily bothers me > about > >> > OOO commands on a connection is that it requires the receiver to > >> > undo this "optimization" on its end - most notably on a single > >> > connection. TCP experts may comment on how/if they dealt with a > >> > similar issue. > >> > > >> > OTOH, you had some valid comments on exceptions to ordering > during > >> > connection recovery. Perhaps we can move on by making Julian's > >> > proposed stipulation a SHOULD.... > >> > -- > >> > Mallikarjun > >> > > >> > > >> > Mallikarjun Chadalapaka > >> > Networked Storage Architecture > >> > Network Storage Solutions Organization > >> > MS 5668 Hewlett-Packard, Roseville. > >> > cbm@rose.hp.com > >> > > >> > > >> > Santosh Rao wrote: > >> > > > >> > > Mallikarjun, > >> > > > >> > > Some comments below. > >> > > > >> > > Regards, > >> > > Santosh > >> > > > >> > > "Mallikarjun C." wrote: > >> > > > > >> > > > Rod and Julian, > >> > > > > >> > > > This has been an interesting thread of discussion. Some > >> > > > comments - > >> > > > > >> > > > 1.My first reaction was - allowing out-of-order command > >> > > > transmission on the same connection deprives targets of > >> > > > an implementation choice. Targets which support only > >> > > > single-connection sessions and only support session > >> > > > recovery (reasonable assumptions in my mind) can no > >> > > > longer afford *not to* implement a command scoreboard. > >> > > > >> > > Even a single connection target *MUST* implement a scoreboard. > The > >> > > reason being that it can see out-of-order arrival of commands > due to > >> > > commands being dropped on digest errors. In such a case, it > >must block > >> > > further command processing until holes are filled. > >> > > > >> > > Thus, there is no getting away from implementing a sequencer at > the > >> > > target. Given this, I think it is unreasonable to restrict > initiator > >> > > implementation flexibility by imposing a strict ordering > requirement > >> > > within the connection. > >> > > > >> > > > 2.Any end-node efficiency that is sought to be achieved > >> > > > by transmitting CmdSNs out-of-order from the initiator > >> > > > would be lost on the other end-node, since the target > >> > > > now must wait for re-ordering the commands. > >> > > > >> > > It has to handle this situation anyway to deal with holes > caused by > >> > > digest errors. This scenario occurs even with initiators that > issue > >> > > commands in order. > >> > > > >> > > > > >> > > > 3.The flipside is that out-of-order transmission saves > >> > > > link badwidth (albeit at the expense of end-node > efficiency), > >> > > > compared to idling the link waiting for outbound DMA. > >> > > > We have to determine if this is a reasonable trade-off. > >> > > > > >> > > > 4.I can see Rod's point that prefetching all immediate > >> > > > data can be a burden on the NIC resources. But, two > >> > > > questions - > >> > > > - could the NIC not use unsolicited separate data > >> > > > PDUs in these cases? [ I realize that InitialR2T > >> > > > has to be "no" to let it happen... ] > >> > > > - could the NIC have a memory architecture that > >> > > > allows data prefetching for the next command (so > >> > > > this is a non-issue from the protocol perspective)? > >> > > > This scheme incurs one DMA delay for every new > >> > > > burst of commands. > >> > > > > >> > > > 5.Another (perhaps radical at this point) option is to do > >> > > > away with immediate unsolicited data, to stick only with > >> > > > separate unsolicited data. I would personally be okay > >> > > > with the choice, particularly if this feature (that > >> > > > helps software implementations) starts making hardware > >> > > > design complicated/expensive. > >> > > > > >> > > > So, to summarize - > >> > > > > >> > > > option immediate allow > >> > > > data in spec? > out-of-order? > >> > > > > >> > > > (A) (5) above no no > >> > > > (B) No real reason to do this. no yes > >> > > > (C) (4) above yes no > >> > > > (D) pros & cons (1), (2) & (3) yes yes > >> > > > > >> > > > >From the arguments I heard so far, I am leaning towards > >> > > > option A, and option C in that order. > >> > > > > >> > > > Comments? > >> > > > -- > >> > > > Mallikarjun > >> > > > > >> > > > Mallikarjun Chadalapaka > >> > > > Networked Storage Architecture > >> > > > Network Storage Solutions Organization > >> > > > MS 5668 Hewlett-Packard, Roseville. > >> > > > cbm@rose.hp.com > >> > > > > >> > > > Rod Harrison wrote: > >> > > > > > >> > > > > Julian, > >> > > > > > >> > > > > I don't understand what you are proposing here, > >what do you > >> > mean by > >> > > > > "multiplexed" DMA? > >> > > > > > >> > > > > The problem is that the DMAs take some time, the > >more there > >> > are > >> > > > > queued the longer the last DMAs queued take to complete. > Some > >> > commands > >> > > > > require DMAs to complete before they can be sent, i.e. > >Writes with > >> > > > > immediate data, some commands do not, i.e. Reads and > >writes with no > >> > > > > immediate data. The iSCSI HBA wants to be able to send > >commands as > >> > > > > soon a possible, which for a read after a write can be > before the > >> > > > > write's DMA has completed. Maintaining an ordered queue > >for commands > >> > > > > to be sent on the HBA is expensive and redundant since the > target > >> > > > > already knows how to queue commands before committing them > to its > >> > SCSI > >> > > > > layer. > >> > > > > > >> > > > > The iSCSI HBA and its host driver are not at > liberty to > >> > change the > >> > > > > order of commands from the OS, but the DMAs those > >commands need are > >> > > > > unlikely to complete in the same order, and as I mentioned > some > >> > > > > commands need no DMA. If the HBA can't send commands out of > CmdSN > >> > > > > order it has to maintain an ordered queue of commands > >waiting to be > >> > > > > sent, and potentially buffer a lot of data. For an HBA this > makes > >> > > > > immediate data almost impossible to support. > >> > > > > > >> > > > > I don't see the problem with allowing out of > >order commands > >> > given > >> > > > > that the target already has to deal with very similar > problems. I > >> > > > > think we are getting in to the area of implementation > >choices here, > >> > > > > which is inappropriate for a specification. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Julian Satran > >> > > > > Sent: Monday, November 05, 2001 10:06 PM > >> > > > > To: ips@ece.cmu.edu > >> > > > > Subject: Re: iSCSI: Out of order commands, was current > >UNH Plugfest > >> > > > > > >> > > > > Rod, > >> > > > > > >> > > > > I don't see any reason why DMA operations cant be > >"multiplexed" with > >> > > > > commands. > >> > > > > If you have scheduled a long outbound DMA you are doomed > >regardless > >> > of > >> > > > > the > >> > > > > command ordering. > >> > > > > And if you have scheduled DMA operations piecemeal then you > can > >> > insert > >> > > > > your commands in correct order. > >> > > > > > >> > > > > Julo > >> > > > > > >> > > > > "Rod Harrison" > >> > > > > 05-11-01 20:48 > >> > > > > Please respond to "Rod Harrison" > >> > > > > > >> > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > >> > > > > cc: > >> > > > > Subject: iSCSI: Out of order commands, was > current > >> > UNH > >> > > > > Plugfest > >> > > > > > >> > > > > [ Subject changed ] > >> > > > > > >> > > > > Julian, > >> > > > > > >> > > > > The ordering difference is introduced > >between the > >> > > > > host > >> > > > > side driver > >> > > > > and the iSCSI HBA. The host side driver must present > >SCSI commands > >> > to > >> > > > > the HBA in the order they are received from the OS to > >prevent read > >> > > > > after write dependency failures. The HBA might reorder > >the commands > >> > > > > depending on when DMA completes. The reordering can't be > >done ahead > >> > of > >> > > > > time in the host driver since it doesn't know how long each > DMA > >> > might > >> > > > > take. As long as the HBA assigns CmdSN in the order it > receives > >> > > > > commands the desired host ordering is preserved. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Julian Satran > >> > > > > Sent: Monday, November 05, 2001 12:35 AM > >> > > > > To: ips@ece.cmu.edu > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Rod, > >> > > > > > >> > > > > I all examples give the point I find hard to understand is > why is > >> > the > >> > > > > ordering on the wire different from the presentation order > to the > >> > > > > initiator. You can get as many overlaps as you want by > >presenting > >> > the > >> > > > > commands to the initiator in the desired order. > >> > > > > What we are considering here is the case in which you > >want to ship > >> > in > >> > > > > an > >> > > > > order different than the one you present the commands. > >> > > > > > >> > > > > Julo > >> > > > > > >> > > > > "Rod Harrison" > >> > > > > Sent by: owner-ips@ece.cmu.edu > >> > > > > 04-11-01 04:42 > >> > > > > Please respond to "Rod Harrison" > >> > > > > > >> > > > > To: "Barry Reinhold" , > "Dave > >> > > > > Sheehy" > >> > > > > , "IETF IP SAN Reflector" > >> > > > > > >> > > > > cc: > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Barry, > >> > > > > > >> > > > > In general I agree but I don't think this > is as > >> > much > >> > > > > of a > >> > > > > corner case > >> > > > > as it at first appears. Targets will have code very > >similar to that > >> > > > > needed to handle out of order commands to deal with > >digest errors. > >> > > > > Targets also need to queue commands whilst waiting for both > >> > solicited > >> > > > > and unsolicited data to arrive. Queuing out of order > >commands seems > >> > > > > little extra work. > >> > > > > > >> > > > > From an initiators point of view there are > >> > > > > efficiency, > >> > > > > and probably > >> > > > > performance gains to be had from sending commands out of > >order. Bob > >> > > > > Russell gave the example of a read being sent whilst > >write data DMA > >> > is > >> > > > > happening, and a similar situation can arise with DMA for > writes > >> > > > > overtaking that of earlier writes if the initiator has > >multiple DMA > >> > > > > engines. In this case the initiator might be forced to > >let the wire > >> > go > >> > > > > idle if it can't send the data from completed DMAs as soon > as > >> > > > > possible. > >> > > > > > >> > > > > We already have a command queue at the > target to > >> > > > > enforce > >> > > > > correct > >> > > > > serialisation of commands, doing the same thing at the > >initiator is > >> > > > > redundant. > >> > > > > > >> > > > > Finally, I don't believe we should be > writing a > >> > > > > standard > >> > > > > to work > >> > > > > around poor coding and test coverage, especially at the > cost of > >> > > > > potential efficiency gains. > >> > > > > > >> > > > > I agree with Dave and Santosh that > >commands being > >> > > > > sent > >> > > > > out of order > >> > > > > on a single session should be allowed by the standard. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Barry Reinhold > >> > > > > Sent: Friday, November 02, 2001 5:24 PM > >> > > > > To: Dave Sheehy; IETF IP SAN Reflector > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Using features such as out of order command delivery on > >a connection > >> > > > > tend to > >> > > > > be the sort of things that lead to interoperability > >problems. It is > >> > > > > unexpected and probably going to hit poorly tested code > >paths even > >> > if > >> > > > > the > >> > > > > standard is written to allow it. > >> > > > > > >> > > > > >-----Original Message----- > >> > > > > >From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > > > > Of > >> > > > > >Dave Sheehy > >> > > > > >Sent: Friday, November 02, 2001 4:19 PM > >> > > > > >To: IETF IP SAN Reflector > >> > > > > >Subject: Re: iSCSI: current UNH Plugfest > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > >> 3. Can commands be sent out of order on the same > connection? > >> > > > > >> > >> > > > > >> The behavior of targets is clearly specified in > Section > >> > 2.2.2.3 > >> > > > > on > >> > > > > >> page 25 of draft 8, which says: > >> > > > > >> "Except for the commands marked for immediate > >delivery the > >> > > > > iSCSI > >> > > > > >> target layer MUST eliver the commands for > >execution in the > >> > > > > order > >> > > > > >> specified by CmdSN." > >> > > > > >> > >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >> > > > > >> "- CmdSN - the current command Sequence Number > >advanced by 1 > >> > > > > on > >> > > > > >> each command shipped except for commands marked for > >> > immediate > >> > > > > >> delivery." > >> > > > > >> but the meaning of the term "shipped" is vague, > >and does not > >> > > > > >> necessarily > >> > > > > >> require that the PDUs arrive on the other end of a > TCP > >> > > > > connection > >> > > > > >> in the same order that the CmdSN values were > >assigned to these > >> > > > > PDUs. > >> > > > > >> > >> > > > > >> Some initiators have been designed to send commands > out of > >> > CmdSN > >> > > > > >> order on one connection. Consider the situation > >where there > >> > is > >> > > > > only > >> > > > > >> one connection and a high-level dispatcher creates > >a PDU for a > >> > > > > SCSI > >> > > > > >> command that involves writing immediate data to the > target. > >> > > > > This PDU > >> > > > > >> is enqueued to a lower-level layer which has to > >setup, start, > >> > > > > and > >> > > > > >> wait-for a DMA operation to move the immediate data > into an > >> > > > > onboard > >> > > > > >> buffer before the PDU can be put onto the wire. > >While this is > >> > > > > >> happening, the dispatcher creates another > >unrelated PDU for a > >> > > > > SCSI > >> > > > > >> read command (for example), and when this PDU is > >passed to the > >> > > > > >> lower-level layer it can be sent immediately, ahead > of the > >> > > > > previous > >> > > > > >> write PDU and therefore out of order on this > connection. > >> > > > > >> > >> > > > > >> The standard clearly allows this to happen if the two > PDUs > >> > were > >> > > > > sent > >> > > > > >> on different connections, and seems to imply that > this can > >> > also > >> > > > > happen > >> > > > > >> when the two PDUs are sent on the same connection. > >> > > > > >> > >> > > > > >> The suggestion is to put in the standard an > >explicit statement > >> > > > > that > >> > > > > >> this is allowed or not allowed, as appropriate. > >> > > > > >> > >> > > > > >> If this is allowed, such a statement would avoid > >the erroneous > >> > > > > >> assumption being made by some target implementers > >that within > >> > a > >> > > > > single > >> > > > > >> connection, commands will arrive in order. > >> > > > > >> > >> > > > > >> If this is not allowed, such a statement would avoid > the > >> > > > > erroneous > >> > > > > >> assumption being made by some initiator implementers > that > >> > within > >> > > > > a > >> > > > > >> single connection, commands can be put on the wire > out of > >> > order. > >> > > > > >> > >> > > > > >> +++ > >> > > > > >> > >> > > > > >> will add an explicit statement saying that this > behaviour is > >> > > > > forbidden. > >> > > > > >> 2.2.2.1 will contain: > >> > > > > >> > >> > > > > >> On any given connection, the iSCSI initiator MUST send > the > >> > > > > >commands in the > >> > > > > >> order specified by CmdSN. > >> > > > > >> > >> > > > > >> +++ > >> > > > > > > >> > > > > >Why do you feel this behavior should be forbidden? > >Targets already > >> > > > > have to > >> > > > > >order commands across the session. I don't see why it's > >a problem > >> > to > >> > > > > extend > >> > > > > >that to the connection as well. I, for one, believe we > >should take > >> > > > > >a liberal > >> > > > > >stance on this. > >> > > > > > > >> > > > > >Dave Sheehy > >> > > > > > > >> > > > >> > > -- > >> > > ################################## > >> > > Santosh Rao > >> > > Software Design Engineer, > >> > > HP-UX iSCSI Driver Team, > >> > > Hewlett Packard, Cupertino. > >> > > email : santoshr@cup.hp.com > >> > > Phone : 408-447-3751 > >> > > ################################## > >> > > >> > > >> > > >> > > >> > > > > From owner-ips@ece.cmu.edu Wed Nov 7 22:46:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA01853 for ; Wed, 7 Nov 2001 22:46:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA82XNk27683 for ips-outgoing; Wed, 7 Nov 2001 21:33:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel11.hp.com (palrel11.hp.com [156.153.255.246]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA82XKl27676 for ; Wed, 7 Nov 2001 21:33:20 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel11.hp.com (Postfix) with ESMTP id 04F541F5AC for ; Wed, 7 Nov 2001 18:33:15 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id SAA28480; Wed, 7 Nov 2001 18:33:09 -0800 (PST) Message-ID: <3BE9F0EE.4A9A5C23@cup.hp.com> Date: Wed, 07 Nov 2001 18:41:50 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: cbm@rose.hp.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands (holes due to protocol error) References: <200111070106.RAA07404@core.rose.hp.com> <3BE8995C.892512E8@cup.hp.com> <200111071730.JAA18255@core.rose.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mallikarjun , Julian & All, A target *MUST* implement command ordering even if it is only supporting single connection sessions and is only using session recovery. Such targets will still see holes in CmdSN sequence on a protocol error and need to stall the processing of subsequently received commands. There is no getting away from implementing command ordering at the target unless iscsi makes command ordering itself optional to implement and negotiate this at login. Given the above, iscsi MUST NOT attempt to enforce upon initiators a requirement to transmit commands in order within a given connection. There is no tangible requirement for this and implementation flexibility should be allowed as long as a scheme exists to maintain end-to-end ordering. Regards, Santosh "Mallikarjun C." wrote: > > Santosh, > > I have only one comment on your responses. > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > I made two convenient assumptions if you noticed, :-), one of which > is that target forces session recovery on *any* error that it sees > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > error. With that assumption, a target can afford not to implement > a scoreboard. > > As I said in a private note, I guess what primarily bothers me about > OOO commands on a connection is that it requires the receiver to > undo this "optimization" on its end - most notably on a single > connection. TCP experts may comment on how/if they dealt with a > similar issue. > > OTOH, you had some valid comments on exceptions to ordering during > connection recovery. Perhaps we can move on by making Julian's > proposed stipulation a SHOULD.... > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > Santosh Rao wrote: > > > > Mallikarjun, > > > > Some comments below. > > > > Regards, > > Santosh > > > > "Mallikarjun C." wrote: > > > > > > Rod and Julian, > > > > > > This has been an interesting thread of discussion. Some > > > comments - > > > > > > 1.My first reaction was - allowing out-of-order command > > > transmission on the same connection deprives targets of > > > an implementation choice. Targets which support only > > > single-connection sessions and only support session > > > recovery (reasonable assumptions in my mind) can no > > > longer afford *not to* implement a command scoreboard. > > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > > > Thus, there is no getting away from implementing a sequencer at the > > target. Given this, I think it is unreasonable to restrict initiator > > implementation flexibility by imposing a strict ordering requirement > > within the connection. > > > > > 2.Any end-node efficiency that is sought to be achieved > > > by transmitting CmdSNs out-of-order from the initiator > > > would be lost on the other end-node, since the target > > > now must wait for re-ordering the commands. > > > > It has to handle this situation anyway to deal with holes caused by > > digest errors. This scenario occurs even with initiators that issue > > commands in order. > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > link badwidth (albeit at the expense of end-node efficiency), > > > compared to idling the link waiting for outbound DMA. > > > We have to determine if this is a reasonable trade-off. > > > > > > 4.I can see Rod's point that prefetching all immediate > > > data can be a burden on the NIC resources. But, two > > > questions - > > > - could the NIC not use unsolicited separate data > > > PDUs in these cases? [ I realize that InitialR2T > > > has to be "no" to let it happen... ] > > > - could the NIC have a memory architecture that > > > allows data prefetching for the next command (so > > > this is a non-issue from the protocol perspective)? > > > This scheme incurs one DMA delay for every new > > > burst of commands. > > > > > > 5.Another (perhaps radical at this point) option is to do > > > away with immediate unsolicited data, to stick only with > > > separate unsolicited data. I would personally be okay > > > with the choice, particularly if this feature (that > > > helps software implementations) starts making hardware > > > design complicated/expensive. > > > > > > So, to summarize - > > > > > > option immediate allow > > > data in spec? out-of-order? > > > > > > (A) (5) above no no > > > (B) No real reason to do this. no yes > > > (C) (4) above yes no > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > >From the arguments I heard so far, I am leaning towards > > > option A, and option C in that order. > > > > > > Comments? > > > -- > > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > Rod Harrison wrote: > > > > > > > > Julian, > > > > > > > > I don't understand what you are proposing here, what do you mean by > > > > "multiplexed" DMA? > > > > > > > > The problem is that the DMAs take some time, the more there are > > > > queued the longer the last DMAs queued take to complete. Some commands > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > soon a possible, which for a read after a write can be before the > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > to be sent on the HBA is expensive and redundant since the target > > > > already knows how to queue commands before committing them to its SCSI > > > > layer. > > > > > > > > The iSCSI HBA and its host driver are not at liberty to change the > > > > order of commands from the OS, but the DMAs those commands need are > > > > unlikely to complete in the same order, and as I mentioned some > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > order it has to maintain an ordered queue of commands waiting to be > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > immediate data almost impossible to support. > > > > > > > > I don't see the problem with allowing out of order commands given > > > > that the target already has to deal with very similar problems. I > > > > think we are getting in to the area of implementation choices here, > > > > which is inappropriate for a specification. > > > > > > > > - Rod > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > Julian Satran > > > > Sent: Monday, November 05, 2001 10:06 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > > > Rod, > > > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > > commands. > > > > If you have scheduled a long outbound DMA you are doomed regardless of > > > > the > > > > command ordering. > > > > And if you have scheduled DMA operations piecemeal then you can insert > > > > your commands in correct order. > > > > > > > > Julo > > > > > > > > "Rod Harrison" > > > > 05-11-01 20:48 > > > > Please respond to "Rod Harrison" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > cc: > > > > Subject: iSCSI: Out of order commands, was current UNH > > > > Plugfest > > > > > > > > [ Subject changed ] > > > > > > > > Julian, > > > > > > > > The ordering difference is introduced between the > > > > host > > > > side driver > > > > and the iSCSI HBA. The host side driver must present SCSI commands to > > > > the HBA in the order they are received from the OS to prevent read > > > > after write dependency failures. The HBA might reorder the commands > > > > depending on when DMA completes. The reordering can't be done ahead of > > > > time in the host driver since it doesn't know how long each DMA might > > > > take. As long as the HBA assigns CmdSN in the order it receives > > > > commands the desired host ordering is preserved. > > > > > > > > - Rod > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > Julian Satran > > > > Sent: Monday, November 05, 2001 12:35 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > Rod, > > > > > > > > I all examples give the point I find hard to understand is why is the > > > > ordering on the wire different from the presentation order to the > > > > initiator. You can get as many overlaps as you want by presenting the > > > > commands to the initiator in the desired order. > > > > What we are considering here is the case in which you want to ship in > > > > an > > > > order different than the one you present the commands. > > > > > > > > Julo > > > > > > > > "Rod Harrison" > > > > Sent by: owner-ips@ece.cmu.edu > > > > 04-11-01 04:42 > > > > Please respond to "Rod Harrison" > > > > > > > > To: "Barry Reinhold" , "Dave > > > > Sheehy" > > > > , "IETF IP SAN Reflector" > > > > > > > > cc: > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > Barry, > > > > > > > > In general I agree but I don't think this is as much > > > > of a > > > > corner case > > > > as it at first appears. Targets will have code very similar to that > > > > needed to handle out of order commands to deal with digest errors. > > > > Targets also need to queue commands whilst waiting for both solicited > > > > and unsolicited data to arrive. Queuing out of order commands seems > > > > little extra work. > > > > > > > > From an initiators point of view there are > > > > efficiency, > > > > and probably > > > > performance gains to be had from sending commands out of order. Bob > > > > Russell gave the example of a read being sent whilst write data DMA is > > > > happening, and a similar situation can arise with DMA for writes > > > > overtaking that of earlier writes if the initiator has multiple DMA > > > > engines. In this case the initiator might be forced to let the wire go > > > > idle if it can't send the data from completed DMAs as soon as > > > > possible. > > > > > > > > We already have a command queue at the target to > > > > enforce > > > > correct > > > > serialisation of commands, doing the same thing at the initiator is > > > > redundant. > > > > > > > > Finally, I don't believe we should be writing a > > > > standard > > > > to work > > > > around poor coding and test coverage, especially at the cost of > > > > potential efficiency gains. > > > > > > > > I agree with Dave and Santosh that commands being > > > > sent > > > > out of order > > > > on a single session should be allowed by the standard. > > > > > > > > - Rod > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > Barry Reinhold > > > > Sent: Friday, November 02, 2001 5:24 PM > > > > To: Dave Sheehy; IETF IP SAN Reflector > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > Using features such as out of order command delivery on a connection > > > > tend to > > > > be the sort of things that lead to interoperability problems. It is > > > > unexpected and probably going to hit poorly tested code paths even if > > > > the > > > > standard is written to allow it. > > > > > > > > >-----Original Message----- > > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > Of > > > > >Dave Sheehy > > > > >Sent: Friday, November 02, 2001 4:19 PM > > > > >To: IETF IP SAN Reflector > > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > > >> > > > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > > > on > > > > >> page 25 of draft 8, which says: > > > > >> "Except for the commands marked for immediate delivery the > > > > iSCSI > > > > >> target layer MUST eliver the commands for execution in the > > > > order > > > > >> specified by CmdSN." > > > > >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > > on > > > > >> each command shipped except for commands marked for immediate > > > > >> delivery." > > > > >> but the meaning of the term "shipped" is vague, and does not > > > > >> necessarily > > > > >> require that the PDUs arrive on the other end of a TCP > > > > connection > > > > >> in the same order that the CmdSN values were assigned to these > > > > PDUs. > > > > >> > > > > >> Some initiators have been designed to send commands out of CmdSN > > > > >> order on one connection. Consider the situation where there is > > > > only > > > > >> one connection and a high-level dispatcher creates a PDU for a > > > > SCSI > > > > >> command that involves writing immediate data to the target. > > > > This PDU > > > > >> is enqueued to a lower-level layer which has to setup, start, > > > > and > > > > >> wait-for a DMA operation to move the immediate data into an > > > > onboard > > > > >> buffer before the PDU can be put onto the wire. While this is > > > > >> happening, the dispatcher creates another unrelated PDU for a > > > > SCSI > > > > >> read command (for example), and when this PDU is passed to the > > > > >> lower-level layer it can be sent immediately, ahead of the > > > > previous > > > > >> write PDU and therefore out of order on this connection. > > > > >> > > > > >> The standard clearly allows this to happen if the two PDUs were > > > > sent > > > > >> on different connections, and seems to imply that this can also > > > > happen > > > > >> when the two PDUs are sent on the same connection. > > > > >> > > > > >> The suggestion is to put in the standard an explicit statement > > > > that > > > > >> this is allowed or not allowed, as appropriate. > > > > >> > > > > >> If this is allowed, such a statement would avoid the erroneous > > > > >> assumption being made by some target implementers that within a > > > > single > > > > >> connection, commands will arrive in order. > > > > >> > > > > >> If this is not allowed, such a statement would avoid the > > > > erroneous > > > > >> assumption being made by some initiator implementers that within > > > > a > > > > >> single connection, commands can be put on the wire out of order. > > > > >> > > > > >> +++ > > > > >> > > > > >> will add an explicit statement saying that this behaviour is > > > > forbidden. > > > > >> 2.2.2.1 will contain: > > > > >> > > > > >> On any given connection, the iSCSI initiator MUST send the > > > > >commands in the > > > > >> order specified by CmdSN. > > > > >> > > > > >> +++ > > > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > > have to > > > > >order commands across the session. I don't see why it's a problem to > > > > extend > > > > >that to the connection as well. I, for one, believe we should take > > > > >a liberal > > > > >stance on this. > > > > > > > > > >Dave Sheehy > > > > > > > > > -- > > ################################## > > Santosh Rao > > Software Design Engineer, > > HP-UX iSCSI Driver Team, > > Hewlett Packard, Cupertino. > > email : santoshr@cup.hp.com > > Phone : 408-447-3751 > > ################################## -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Wed Nov 7 22:50:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA01943 for ; Wed, 7 Nov 2001 22:50:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA83CZg00148 for ips-outgoing; Wed, 7 Nov 2001 22:12:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from harbor.sj.gadzoox.com (harbormail.gadzoox.com [216.52.31.23]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA80Tql19656 for ; Wed, 7 Nov 2001 19:29:52 -0500 (EST) Received: by harbor.gadzoox.com with Internet Mail Service (5.5.2653.19) id ; Wed, 7 Nov 2001 16:29:50 -0800 Message-ID: <24552A8180A1AE4BA4424F6C616EDB5E2EE909@harbor.gadzoox.com> From: John Nguyen To: "'Keith McCloghrie'" Cc: ips@ece.cmu.edu Subject: RE: FC Management MIB - issues Date: Wed, 7 Nov 2001 16:29:49 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Keith, I'm John Nguyen, software engineer from Gadzoox Networks. My company is also member of SNIA, DMTF, and thus many other organizations, who are deploying Fibre Channel into SAN and working with other joint vendors to propose many FC related standards. As we all are moving further in SAN and SAN market, it may be a concern to think about bringing ZONE into FC Management MIB. DMTF and SNIA are those who may already have approved specifications on zoning that can be incorporated. I'm not sure if there are any ideas or discussions around the table about this! Thanks for taking mine as input to utilize zoning in the new FC mgmt MIB. Regards, John Nguyen Software Engineer 408.361.6142 Gadzoox Networks -----Original Message----- From: Keith McCloghrie [mailto:kzm@cisco.com] Sent: Wednesday, November 07, 2001 1:03 PM To: ips@ece.cmu.edu Cc: kzm@cisco.com Subject: FC Management MIB - issues Hi, As indicated by Elizabeth's message (below), I've agreed to be the interim editor for the FC Management MIB. So, first, I'd like to list a set of issues that have been raised concerning the most recent draft: draft-ietf-ipfc-fcmgmt-int-mib-07.txt. They are: 1. The use of SMIv2 is mandatory. 2. This MIB seems to have been defined with the notion that it will be the only MIB that a Fibre Channel product will require. The notion of an agent implementing just a single MIB was abandoned by the IETF in 1992 as being non-scaleable. Rather, this is another MIB in the continuing series of MIBs defined by the IETF, and thus, it needs to be consistent with its predecessors. In other words, there are existing MIBs which all SNMP agents must support, even if the support of Fibre Channel interfaces is the only functionality that they have. Thus, as the Fibre Channel Integration MIB, it is essential that this MIB contain only objects for information which is specific to Fibre Channel. All objects which apply in non-Fibre Channel environments need to be removed. (This applies even it were to require the definition of other new MIBs to contain information which is not already defined in other existing MIBs, but needed by Fibre Channel products.) Note that this issue applies to a large fraction of the objects defined in this MIB. 3. The text needs to include an explanation of the relationship between this MIB and RFC 2837. Is it a replacement or is it complementary ? If complementary, which agents implement which MIB; do some agents implement both ? 4. Every SNMP agent implements the ifTable. The ifTable counters are undoubtedly the MIB objects most well-used by administrators in SNMP management. SNMP agents need to implement a row in the ifTable for each of their network interfaces, including their Fibre Channel interfaces. The IF-MIB requires a media-specific MIB to specify how that type of interface uses the ifTable (see section 4 in RFC 2863). RFC 2837 doesn't do that, and nor (currently) does this MIB. That needs to be fixed. This will likely result in many tables in this MIB being indexed by ifIndex. 5. There are a number of objects related to the "Simple Name Service", and the definitions refer to Fibre Channel's GS-3 spec. However, GS-3 defines two Name Services: a Zoned Name Service and a Unzoned Name Service, but GS-3 does NOT use the term "Simple" for either of them !! This ambiguity needs to be resolved. 6. It is essential that the Counter32 or Counter64 (not OCTET STRINGs) syntax be used for counters. Thanks, Keith. -------------- Forwarded message: > From: "Elizabeth Rodriguez" > To: , > "IPS Mailing List (E-mail)" > Subject: FC Management MIB -- Transfer from IPFC To IPS WG > Cc: , , , > , , , > , > "Elizabeth Rodriguez" , > > > All, > > The IPFC working group has only one item left on its charter. > It is that of the FC Management MIB. It has been identified > that this MIB does not follow the IETF guidelines for MIBs > in its current format, in its lack of focus, and in its overlap > with existing MIBs. Rework of this MIB will take some time. > The content of this MIB will fit in well with the IPS WG, > especially because the subject matter experts participate in > this WG. > > For these reasons, the working group chairs, with the INT and TSV area > directors, have determined that this effort should be moved from the > IPFC working group to the IPS working group. Upon transferring of this > work, the IPFC working group will have completed the items in its > charter and the IPFC WG will be closed. > > The IPS working group has a technical advisor for MIB work -- Keith > McCloghrie. Since it has been determined that the current MIB has > issues with format, Keith McCloghrie has agreed to become the interim > editor of this MIB. As part of the re-architecture, the MIB will be > evaluated with respect to fit with other IPS WG MIBs, and may take the > form of a single new MIB or multiple MIBs, as appropriate. The IPS > working group will also be seeking Fibre Channel expertise to help > formulate the new MIB, including an editor with FC and MIB experience. > > The IPFC working group would also like to thank Steven Blumenau for all > his work on the original MIB. > > Elizabeth Rodriguez & David Black, IPS Working Group Chairs > Murali Rajagopal, IPFC Working Group Chair > From owner-ips@ece.cmu.edu Thu Nov 8 02:08:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA18430 for ; Thu, 8 Nov 2001 02:08:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8627t10723 for ips-outgoing; Thu, 8 Nov 2001 01:02:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8625l10715 for ; Thu, 8 Nov 2001 01:02:05 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA40414 for ; Thu, 8 Nov 2001 07:01:58 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA861vw77958 for ; Thu, 8 Nov 2001 07:01:57 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 8 Nov 2001 08:01:54 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 08/11/2001 08:01:57, Serialize complete at 08/11/2001 08:01:57 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Robert, I am not saying that handling OOO commands will create more complexity (targets already do that over several connections and it does not matter for them). However allowing initiators to ship them out of order creates a potential deadlock that does not appear otherwise. If a target is missing a command in a queue (and there are no errors) the this command is bound to be first on some connection under the current set of rules. If we allow OOO shipping then the missing command can be somewhere "inside" the window on some connection and if the target has just filled his queue and has room in the staging buffer only for the command it is waiting for and that command happens to be the first to pass to SCSI you have a deadlock. Julo "Robert D. Russell" 07-11-01 23:13 Please respond to "Robert D. Russell" To: Somesh Gupta cc: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Somesh, Julian: You state that dealing with OOO commands on the target will add substantial complexity on the target. Do you have any basis for that claim? My impression from the plugfest is that most targets are already dealing with it. Perhaps we need to hear from someone who is actually building a target for which this would be a real problem. If anything, what we are hearing from people who really are building initiators is that dealing with the requirement to send commands in order will introduce substantial complexity on the initiator. So why should we be saving complexity on (hypothetically) simple targets yet requiring complexity on real initiators? As far as the deadlock issue is concerned, if the only way that deadlock can occur with OOO commands on the same connection is during the use of immediate data (which is I think what Julian was saying), then shouldn't we change the standard to just say that -- if the initiator sends commands out of order on a single connection, then immediate data MUST NOT be used on that connection in order to avoid deadlock. This gives everybody what they want, since initiators who find it beneficial to deliver commands OOO will just negotiate immediate data off. Those who really want to use immediate data will have to ensure that commands are sent in order. The tradeoff then becomes an implementation issue, not a standards issue, which is where it belongs. Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Wed, 7 Nov 2001, Somesh Gupta wrote: > I think we should either have it as a MUST or not require > it (at both ends to get the real benefit). SHOULD is one > of those things that leads to implementation > burden and confusion, without perhaps the feature being > used. There are implementation as well as protocol > considerations mixed in here. > > If we are to remove the restriction, we should (SHOULD) > get the maximum benefit from it, rather than to > accomodate an implementation choice. Out of sequence > commands, combined with the possibility of digest errors, > will add substantial complexity on the target side, > without corrosponding benefit in performance. If we change > this to SHOULD, we should also relax the requirement > to present commands on the target side to a SHOULD. > > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Wednesday, November 07, 2001 10:00 AM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Out of order commands > > > > > > Mallikarjun, > > > > I did not see a SINGLE performance improvement that results from OOO > > shipping. > > I would be bad engineering to give away the "no-deadlock" mechanism we > > have now for nothing. > > I have also the impression that the point about deadlock that I keep > > repeating is ignored or not understood. > > As we stand today commands can be shipped with Immediate data or without > > and an implementer determined > > to squeeze maximum bandwidth and overlap command start with delivery will > > choose not to work with immediate data > > (as you have pointed out) while a low performance software implementation > > will use immediate data to minimize CPU cycles consumed. However both > > will be guaranteed to work without deadlock as source and sink use the > > same ordering. > > Recovery is still a low probability event and should be handled with a > > different set of considerations in mind. > > As for the strictness of the recommendation - yes we could settle on > > SHOULD. > > > > Julo > > > > > > > > > > "Mallikarjun C." > > Sent by: owner-ips@ece.cmu.edu > > 07-11-01 19:41 > > Please respond to cbm > > > > > > To: Santosh Rao , ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: Out of order commands > > > > > > > > Santosh, > > > > I have only one comment on your responses. > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > I made two convenient assumptions if you noticed, :-), one of which > > is that target forces session recovery on *any* error that it sees > > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > > error. With that assumption, a target can afford not to implement > > a scoreboard. > > > > As I said in a private note, I guess what primarily bothers me about > > OOO commands on a connection is that it requires the receiver to > > undo this "optimization" on its end - most notably on a single > > connection. TCP experts may comment on how/if they dealt with a > > similar issue. > > > > OTOH, you had some valid comments on exceptions to ordering during > > connection recovery. Perhaps we can move on by making Julian's > > proposed stipulation a SHOULD.... > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > Santosh Rao wrote: > > > > > > Mallikarjun, > > > > > > Some comments below. > > > > > > Regards, > > > Santosh > > > > > > "Mallikarjun C." wrote: > > > > > > > > Rod and Julian, > > > > > > > > This has been an interesting thread of discussion. Some > > > > comments - > > > > > > > > 1.My first reaction was - allowing out-of-order command > > > > transmission on the same connection deprives targets of > > > > an implementation choice. Targets which support only > > > > single-connection sessions and only support session > > > > recovery (reasonable assumptions in my mind) can no > > > > longer afford *not to* implement a command scoreboard. > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > > > Thus, there is no getting away from implementing a sequencer at the > > > target. Given this, I think it is unreasonable to restrict initiator > > > implementation flexibility by imposing a strict ordering requirement > > > within the connection. > > > > > > > 2.Any end-node efficiency that is sought to be achieved > > > > by transmitting CmdSNs out-of-order from the initiator > > > > would be lost on the other end-node, since the target > > > > now must wait for re-ordering the commands. > > > > > > It has to handle this situation anyway to deal with holes caused by > > > digest errors. This scenario occurs even with initiators that issue > > > commands in order. > > > > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > > link badwidth (albeit at the expense of end-node efficiency), > > > > compared to idling the link waiting for outbound DMA. > > > > We have to determine if this is a reasonable trade-off. > > > > > > > > 4.I can see Rod's point that prefetching all immediate > > > > data can be a burden on the NIC resources. But, two > > > > questions - > > > > - could the NIC not use unsolicited separate data > > > > PDUs in these cases? [ I realize that InitialR2T > > > > has to be "no" to let it happen... ] > > > > - could the NIC have a memory architecture that > > > > allows data prefetching for the next command (so > > > > this is a non-issue from the protocol perspective)? > > > > This scheme incurs one DMA delay for every new > > > > burst of commands. > > > > > > > > 5.Another (perhaps radical at this point) option is to do > > > > away with immediate unsolicited data, to stick only with > > > > separate unsolicited data. I would personally be okay > > > > with the choice, particularly if this feature (that > > > > helps software implementations) starts making hardware > > > > design complicated/expensive. > > > > > > > > So, to summarize - > > > > > > > > option immediate allow > > > > data in spec? out-of-order? > > > > > > > > (A) (5) above no no > > > > (B) No real reason to do this. no yes > > > > (C) (4) above yes no > > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > > > >From the arguments I heard so far, I am leaning towards > > > > option A, and option C in that order. > > > > > > > > Comments? > > > > -- > > > > Mallikarjun > > > > > > > > Mallikarjun Chadalapaka > > > > Networked Storage Architecture > > > > Network Storage Solutions Organization > > > > MS 5668 Hewlett-Packard, Roseville. > > > > cbm@rose.hp.com > > > > > > > > Rod Harrison wrote: > > > > > > > > > > Julian, > > > > > > > > > > I don't understand what you are proposing here, what do you > > mean by > > > > > "multiplexed" DMA? > > > > > > > > > > The problem is that the DMAs take some time, the more there > > are > > > > > queued the longer the last DMAs queued take to complete. Some > > commands > > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > > soon a possible, which for a read after a write can be before the > > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > > to be sent on the HBA is expensive and redundant since the target > > > > > already knows how to queue commands before committing them to its > > SCSI > > > > > layer. > > > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > > change the > > > > > order of commands from the OS, but the DMAs those commands need are > > > > > unlikely to complete in the same order, and as I mentioned some > > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > > order it has to maintain an ordered queue of commands waiting to be > > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > > immediate data almost impossible to support. > > > > > > > > > > I don't see the problem with allowing out of order commands > > given > > > > > that the target already has to deal with very similar problems. I > > > > > think we are getting in to the area of implementation choices here, > > > > > which is inappropriate for a specification. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Julian Satran > > > > > Sent: Monday, November 05, 2001 10:06 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > > > > > Rod, > > > > > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > > > commands. > > > > > If you have scheduled a long outbound DMA you are doomed regardless > > of > > > > > the > > > > > command ordering. > > > > > And if you have scheduled DMA operations piecemeal then you can > > insert > > > > > your commands in correct order. > > > > > > > > > > Julo > > > > > > > > > > "Rod Harrison" > > > > > 05-11-01 20:48 > > > > > Please respond to "Rod Harrison" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > > cc: > > > > > Subject: iSCSI: Out of order commands, was current > > UNH > > > > > Plugfest > > > > > > > > > > [ Subject changed ] > > > > > > > > > > Julian, > > > > > > > > > > The ordering difference is introduced between the > > > > > host > > > > > side driver > > > > > and the iSCSI HBA. The host side driver must present SCSI commands > > to > > > > > the HBA in the order they are received from the OS to prevent read > > > > > after write dependency failures. The HBA might reorder the commands > > > > > depending on when DMA completes. The reordering can't be done ahead > > of > > > > > time in the host driver since it doesn't know how long each DMA > > might > > > > > take. As long as the HBA assigns CmdSN in the order it receives > > > > > commands the desired host ordering is preserved. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Julian Satran > > > > > Sent: Monday, November 05, 2001 12:35 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Rod, > > > > > > > > > > I all examples give the point I find hard to understand is why is > > the > > > > > ordering on the wire different from the presentation order to the > > > > > initiator. You can get as many overlaps as you want by presenting > > the > > > > > commands to the initiator in the desired order. > > > > > What we are considering here is the case in which you want to ship > > in > > > > > an > > > > > order different than the one you present the commands. > > > > > > > > > > Julo > > > > > > > > > > "Rod Harrison" > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > 04-11-01 04:42 > > > > > Please respond to "Rod Harrison" > > > > > > > > > > To: "Barry Reinhold" , "Dave > > > > > Sheehy" > > > > > , "IETF IP SAN Reflector" > > > > > > > > > > cc: > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Barry, > > > > > > > > > > In general I agree but I don't think this is as > > much > > > > > of a > > > > > corner case > > > > > as it at first appears. Targets will have code very similar to that > > > > > needed to handle out of order commands to deal with digest errors. > > > > > Targets also need to queue commands whilst waiting for both > > solicited > > > > > and unsolicited data to arrive. Queuing out of order commands seems > > > > > little extra work. > > > > > > > > > > From an initiators point of view there are > > > > > efficiency, > > > > > and probably > > > > > performance gains to be had from sending commands out of order. Bob > > > > > Russell gave the example of a read being sent whilst write data DMA > > is > > > > > happening, and a similar situation can arise with DMA for writes > > > > > overtaking that of earlier writes if the initiator has multiple DMA > > > > > engines. In this case the initiator might be forced to let the wire > > go > > > > > idle if it can't send the data from completed DMAs as soon as > > > > > possible. > > > > > > > > > > We already have a command queue at the target to > > > > > enforce > > > > > correct > > > > > serialisation of commands, doing the same thing at the initiator is > > > > > redundant. > > > > > > > > > > Finally, I don't believe we should be writing a > > > > > standard > > > > > to work > > > > > around poor coding and test coverage, especially at the cost of > > > > > potential efficiency gains. > > > > > > > > > > I agree with Dave and Santosh that commands being > > > > > sent > > > > > out of order > > > > > on a single session should be allowed by the standard. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Barry Reinhold > > > > > Sent: Friday, November 02, 2001 5:24 PM > > > > > To: Dave Sheehy; IETF IP SAN Reflector > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Using features such as out of order command delivery on a connection > > > > > tend to > > > > > be the sort of things that lead to interoperability problems. It is > > > > > unexpected and probably going to hit poorly tested code paths even > > if > > > > > the > > > > > standard is written to allow it. > > > > > > > > > > >-----Original Message----- > > > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > > Of > > > > > >Dave Sheehy > > > > > >Sent: Friday, November 02, 2001 4:19 PM > > > > > >To: IETF IP SAN Reflector > > > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > > > >> > > > > > >> The behavior of targets is clearly specified in Section > > 2.2.2.3 > > > > > on > > > > > >> page 25 of draft 8, which says: > > > > > >> "Except for the commands marked for immediate delivery the > > > > > iSCSI > > > > > >> target layer MUST eliver the commands for execution in the > > > > > order > > > > > >> specified by CmdSN." > > > > > >> > > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > > > on > > > > > >> each command shipped except for commands marked for > > immediate > > > > > >> delivery." > > > > > >> but the meaning of the term "shipped" is vague, and does not > > > > > >> necessarily > > > > > >> require that the PDUs arrive on the other end of a TCP > > > > > connection > > > > > >> in the same order that the CmdSN values were assigned to these > > > > > PDUs. > > > > > >> > > > > > >> Some initiators have been designed to send commands out of > > CmdSN > > > > > >> order on one connection. Consider the situation where there > > is > > > > > only > > > > > >> one connection and a high-level dispatcher creates a PDU for a > > > > > SCSI > > > > > >> command that involves writing immediate data to the target. > > > > > This PDU > > > > > >> is enqueued to a lower-level layer which has to setup, start, > > > > > and > > > > > >> wait-for a DMA operation to move the immediate data into an > > > > > onboard > > > > > >> buffer before the PDU can be put onto the wire. While this is > > > > > >> happening, the dispatcher creates another unrelated PDU for a > > > > > SCSI > > > > > >> read command (for example), and when this PDU is passed to the > > > > > >> lower-level layer it can be sent immediately, ahead of the > > > > > previous > > > > > >> write PDU and therefore out of order on this connection. > > > > > >> > > > > > >> The standard clearly allows this to happen if the two PDUs > > were > > > > > sent > > > > > >> on different connections, and seems to imply that this can > > also > > > > > happen > > > > > >> when the two PDUs are sent on the same connection. > > > > > >> > > > > > >> The suggestion is to put in the standard an explicit statement > > > > > that > > > > > >> this is allowed or not allowed, as appropriate. > > > > > >> > > > > > >> If this is allowed, such a statement would avoid the erroneous > > > > > >> assumption being made by some target implementers that within > > a > > > > > single > > > > > >> connection, commands will arrive in order. > > > > > >> > > > > > >> If this is not allowed, such a statement would avoid the > > > > > erroneous > > > > > >> assumption being made by some initiator implementers that > > within > > > > > a > > > > > >> single connection, commands can be put on the wire out of > > order. > > > > > >> > > > > > >> +++ > > > > > >> > > > > > >> will add an explicit statement saying that this behaviour is > > > > > forbidden. > > > > > >> 2.2.2.1 will contain: > > > > > >> > > > > > >> On any given connection, the iSCSI initiator MUST send the > > > > > >commands in the > > > > > >> order specified by CmdSN. > > > > > >> > > > > > >> +++ > > > > > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > > > have to > > > > > >order commands across the session. I don't see why it's a problem > > to > > > > > extend > > > > > >that to the connection as well. I, for one, believe we should take > > > > > >a liberal > > > > > >stance on this. > > > > > > > > > > > >Dave Sheehy > > > > > > > > > > > > -- > > > ################################## > > > Santosh Rao > > > Software Design Engineer, > > > HP-UX iSCSI Driver Team, > > > Hewlett Packard, Cupertino. > > > email : santoshr@cup.hp.com > > > Phone : 408-447-3751 > > > ################################## > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 09:19:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA26719 for ; Thu, 8 Nov 2001 09:19:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8D7cW16494 for ips-outgoing; Thu, 8 Nov 2001 08:07:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls05.mediaone.net (chmls05.mediaone.net [24.147.1.143]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8D7Ml16472 for ; Thu, 8 Nov 2001 08:07:22 -0500 (EST) Received: from breinhold ([140.186.40.85]) by chmls05.mediaone.net (8.11.1/8.11.1) with SMTP id fA8D6uk26188; Thu, 8 Nov 2001 08:06:56 -0500 (EST) From: "Barry Reinhold" To: "Robert D. Russell" , "Somesh Gupta" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 08:06:14 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Bob, I can't speak for targets, but OOO commands on a session with a single connection sure increases the complexity of the code path we take. To me the issue is still that these types of situations tend to be poorly tested and lead to interoperability issues. If we do go down this path, the spec. should make it very clear that this is expected behavior. Some statement with a shall in it should be written (for the receiver), so that there is a conformance test items to pass. >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Robert D. Russell >Sent: Wednesday, November 07, 2001 4:13 PM >To: Somesh Gupta >Cc: Julian Satran; ips@ece.cmu.edu >Subject: RE: iSCSI: Out of order commands > > >Somesh, Julian: > >You state that dealing with OOO commands on the target >will add substantial complexity on the target. >Do you have any basis for that claim? My impression from the >plugfest is that most targets are already dealing with >it. Perhaps we need to hear from someone who is actually >building a target for which this would be a real problem. > >If anything, what we are hearing from people who really >are building initiators is that dealing with the requirement >to send commands in order will introduce substantial complexity >on the initiator. > >So why should we be saving complexity on (hypothetically) simple >targets yet requiring complexity on real initiators? > >As far as the deadlock issue is concerned, if the only way >that deadlock can occur with OOO commands on the same >connection is during the use of immediate data (which is I >think what Julian was saying), then shouldn't we change >the standard to just say that -- if the initiator sends >commands out of order on a single connection, then immediate >data MUST NOT be used on that connection in order to avoid deadlock. > >This gives everybody what they want, since initiators who find >it beneficial to deliver commands OOO will just negotiate >immediate data off. Those who really want to use immediate data >will have to ensure that commands are sent in order. >The tradeoff then becomes an implementation issue, not a >standards issue, which is where it belongs. > > >Bob Russell >InterOperability Lab >University of New Hampshire >rdr@iol.unh.edu >603-862-3774 > > >On Wed, 7 Nov 2001, Somesh Gupta wrote: > >> I think we should either have it as a MUST or not require >> it (at both ends to get the real benefit). SHOULD is one >> of those things that leads to implementation >> burden and confusion, without perhaps the feature being >> used. There are implementation as well as protocol >> considerations mixed in here. >> >> If we are to remove the restriction, we should (SHOULD) >> get the maximum benefit from it, rather than to >> accomodate an implementation choice. Out of sequence >> commands, combined with the possibility of digest errors, >> will add substantial complexity on the target side, >> without corrosponding benefit in performance. If we change >> this to SHOULD, we should also relax the requirement >> to present commands on the target side to a SHOULD. >> >> >> >> > -----Original Message----- >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >> > Julian Satran >> > Sent: Wednesday, November 07, 2001 10:00 AM >> > To: ips@ece.cmu.edu >> > Subject: Re: iSCSI: Out of order commands >> > >> > >> > Mallikarjun, >> > >> > I did not see a SINGLE performance improvement that results from OOO >> > shipping. >> > I would be bad engineering to give away the "no-deadlock" mechanism we >> > have now for nothing. >> > I have also the impression that the point about deadlock that I keep >> > repeating is ignored or not understood. >> > As we stand today commands can be shipped with Immediate data >or without >> > and an implementer determined >> > to squeeze maximum bandwidth and overlap command start with >delivery will >> > choose not to work with immediate data >> > (as you have pointed out) while a low performance software >implementation >> > will use immediate data to minimize CPU cycles consumed. However both >> > will be guaranteed to work without deadlock as source and sink use the >> > same ordering. >> > Recovery is still a low probability event and should be handled with a >> > different set of considerations in mind. >> > As for the strictness of the recommendation - yes we could settle on >> > SHOULD. >> > >> > Julo >> > >> > >> > >> > >> > "Mallikarjun C." >> > Sent by: owner-ips@ece.cmu.edu >> > 07-11-01 19:41 >> > Please respond to cbm >> > >> > >> > To: Santosh Rao , ips@ece.cmu.edu >> > cc: >> > Subject: Re: iSCSI: Out of order commands >> > >> > >> > >> > Santosh, >> > >> > I have only one comment on your responses. >> > >> > > Even a single connection target *MUST* implement a scoreboard. The >> > > reason being that it can see out-of-order arrival of commands due to >> > > commands being dropped on digest errors. In such a case, it >must block >> > > further command processing until holes are filled. >> > >> > I made two convenient assumptions if you noticed, :-), one of which >> > is that target forces session recovery on *any* error that it sees >> > (ErrorRecoveryLevel=0) - including a dropped command due to a digest >> > error. With that assumption, a target can afford not to implement >> > a scoreboard. >> > >> > As I said in a private note, I guess what primarily bothers me about >> > OOO commands on a connection is that it requires the receiver to >> > undo this "optimization" on its end - most notably on a single >> > connection. TCP experts may comment on how/if they dealt with a >> > similar issue. >> > >> > OTOH, you had some valid comments on exceptions to ordering during >> > connection recovery. Perhaps we can move on by making Julian's >> > proposed stipulation a SHOULD.... >> > -- >> > Mallikarjun >> > >> > >> > Mallikarjun Chadalapaka >> > Networked Storage Architecture >> > Network Storage Solutions Organization >> > MS 5668 Hewlett-Packard, Roseville. >> > cbm@rose.hp.com >> > >> > >> > Santosh Rao wrote: >> > > >> > > Mallikarjun, >> > > >> > > Some comments below. >> > > >> > > Regards, >> > > Santosh >> > > >> > > "Mallikarjun C." wrote: >> > > > >> > > > Rod and Julian, >> > > > >> > > > This has been an interesting thread of discussion. Some >> > > > comments - >> > > > >> > > > 1.My first reaction was - allowing out-of-order command >> > > > transmission on the same connection deprives targets of >> > > > an implementation choice. Targets which support only >> > > > single-connection sessions and only support session >> > > > recovery (reasonable assumptions in my mind) can no >> > > > longer afford *not to* implement a command scoreboard. >> > > >> > > Even a single connection target *MUST* implement a scoreboard. The >> > > reason being that it can see out-of-order arrival of commands due to >> > > commands being dropped on digest errors. In such a case, it >must block >> > > further command processing until holes are filled. >> > > >> > > Thus, there is no getting away from implementing a sequencer at the >> > > target. Given this, I think it is unreasonable to restrict initiator >> > > implementation flexibility by imposing a strict ordering requirement >> > > within the connection. >> > > >> > > > 2.Any end-node efficiency that is sought to be achieved >> > > > by transmitting CmdSNs out-of-order from the initiator >> > > > would be lost on the other end-node, since the target >> > > > now must wait for re-ordering the commands. >> > > >> > > It has to handle this situation anyway to deal with holes caused by >> > > digest errors. This scenario occurs even with initiators that issue >> > > commands in order. >> > > >> > > > >> > > > 3.The flipside is that out-of-order transmission saves >> > > > link badwidth (albeit at the expense of end-node efficiency), >> > > > compared to idling the link waiting for outbound DMA. >> > > > We have to determine if this is a reasonable trade-off. >> > > > >> > > > 4.I can see Rod's point that prefetching all immediate >> > > > data can be a burden on the NIC resources. But, two >> > > > questions - >> > > > - could the NIC not use unsolicited separate data >> > > > PDUs in these cases? [ I realize that InitialR2T >> > > > has to be "no" to let it happen... ] >> > > > - could the NIC have a memory architecture that >> > > > allows data prefetching for the next command (so >> > > > this is a non-issue from the protocol perspective)? >> > > > This scheme incurs one DMA delay for every new >> > > > burst of commands. >> > > > >> > > > 5.Another (perhaps radical at this point) option is to do >> > > > away with immediate unsolicited data, to stick only with >> > > > separate unsolicited data. I would personally be okay >> > > > with the choice, particularly if this feature (that >> > > > helps software implementations) starts making hardware >> > > > design complicated/expensive. >> > > > >> > > > So, to summarize - >> > > > >> > > > option immediate allow >> > > > data in spec? out-of-order? >> > > > >> > > > (A) (5) above no no >> > > > (B) No real reason to do this. no yes >> > > > (C) (4) above yes no >> > > > (D) pros & cons (1), (2) & (3) yes yes >> > > > >> > > > >From the arguments I heard so far, I am leaning towards >> > > > option A, and option C in that order. >> > > > >> > > > Comments? >> > > > -- >> > > > Mallikarjun >> > > > >> > > > Mallikarjun Chadalapaka >> > > > Networked Storage Architecture >> > > > Network Storage Solutions Organization >> > > > MS 5668 Hewlett-Packard, Roseville. >> > > > cbm@rose.hp.com >> > > > >> > > > Rod Harrison wrote: >> > > > > >> > > > > Julian, >> > > > > >> > > > > I don't understand what you are proposing here, >what do you >> > mean by >> > > > > "multiplexed" DMA? >> > > > > >> > > > > The problem is that the DMAs take some time, the >more there >> > are >> > > > > queued the longer the last DMAs queued take to complete. Some >> > commands >> > > > > require DMAs to complete before they can be sent, i.e. >Writes with >> > > > > immediate data, some commands do not, i.e. Reads and >writes with no >> > > > > immediate data. The iSCSI HBA wants to be able to send >commands as >> > > > > soon a possible, which for a read after a write can be before the >> > > > > write's DMA has completed. Maintaining an ordered queue >for commands >> > > > > to be sent on the HBA is expensive and redundant since the target >> > > > > already knows how to queue commands before committing them to its >> > SCSI >> > > > > layer. >> > > > > >> > > > > The iSCSI HBA and its host driver are not at liberty to >> > change the >> > > > > order of commands from the OS, but the DMAs those >commands need are >> > > > > unlikely to complete in the same order, and as I mentioned some >> > > > > commands need no DMA. If the HBA can't send commands out of CmdSN >> > > > > order it has to maintain an ordered queue of commands >waiting to be >> > > > > sent, and potentially buffer a lot of data. For an HBA this makes >> > > > > immediate data almost impossible to support. >> > > > > >> > > > > I don't see the problem with allowing out of >order commands >> > given >> > > > > that the target already has to deal with very similar problems. I >> > > > > think we are getting in to the area of implementation >choices here, >> > > > > which is inappropriate for a specification. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Julian Satran >> > > > > Sent: Monday, November 05, 2001 10:06 PM >> > > > > To: ips@ece.cmu.edu >> > > > > Subject: Re: iSCSI: Out of order commands, was current >UNH Plugfest >> > > > > >> > > > > Rod, >> > > > > >> > > > > I don't see any reason why DMA operations cant be >"multiplexed" with >> > > > > commands. >> > > > > If you have scheduled a long outbound DMA you are doomed >regardless >> > of >> > > > > the >> > > > > command ordering. >> > > > > And if you have scheduled DMA operations piecemeal then you can >> > insert >> > > > > your commands in correct order. >> > > > > >> > > > > Julo >> > > > > >> > > > > "Rod Harrison" >> > > > > 05-11-01 20:48 >> > > > > Please respond to "Rod Harrison" >> > > > > >> > > > > To: Julian Satran/Haifa/IBM@IBMIL, >> > > > > cc: >> > > > > Subject: iSCSI: Out of order commands, was current >> > UNH >> > > > > Plugfest >> > > > > >> > > > > [ Subject changed ] >> > > > > >> > > > > Julian, >> > > > > >> > > > > The ordering difference is introduced >between the >> > > > > host >> > > > > side driver >> > > > > and the iSCSI HBA. The host side driver must present >SCSI commands >> > to >> > > > > the HBA in the order they are received from the OS to >prevent read >> > > > > after write dependency failures. The HBA might reorder >the commands >> > > > > depending on when DMA completes. The reordering can't be >done ahead >> > of >> > > > > time in the host driver since it doesn't know how long each DMA >> > might >> > > > > take. As long as the HBA assigns CmdSN in the order it receives >> > > > > commands the desired host ordering is preserved. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Julian Satran >> > > > > Sent: Monday, November 05, 2001 12:35 AM >> > > > > To: ips@ece.cmu.edu >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Rod, >> > > > > >> > > > > I all examples give the point I find hard to understand is why is >> > the >> > > > > ordering on the wire different from the presentation order to the >> > > > > initiator. You can get as many overlaps as you want by >presenting >> > the >> > > > > commands to the initiator in the desired order. >> > > > > What we are considering here is the case in which you >want to ship >> > in >> > > > > an >> > > > > order different than the one you present the commands. >> > > > > >> > > > > Julo >> > > > > >> > > > > "Rod Harrison" >> > > > > Sent by: owner-ips@ece.cmu.edu >> > > > > 04-11-01 04:42 >> > > > > Please respond to "Rod Harrison" >> > > > > >> > > > > To: "Barry Reinhold" , "Dave >> > > > > Sheehy" >> > > > > , "IETF IP SAN Reflector" >> > > > > >> > > > > cc: >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Barry, >> > > > > >> > > > > In general I agree but I don't think this is as >> > much >> > > > > of a >> > > > > corner case >> > > > > as it at first appears. Targets will have code very >similar to that >> > > > > needed to handle out of order commands to deal with >digest errors. >> > > > > Targets also need to queue commands whilst waiting for both >> > solicited >> > > > > and unsolicited data to arrive. Queuing out of order >commands seems >> > > > > little extra work. >> > > > > >> > > > > From an initiators point of view there are >> > > > > efficiency, >> > > > > and probably >> > > > > performance gains to be had from sending commands out of >order. Bob >> > > > > Russell gave the example of a read being sent whilst >write data DMA >> > is >> > > > > happening, and a similar situation can arise with DMA for writes >> > > > > overtaking that of earlier writes if the initiator has >multiple DMA >> > > > > engines. In this case the initiator might be forced to >let the wire >> > go >> > > > > idle if it can't send the data from completed DMAs as soon as >> > > > > possible. >> > > > > >> > > > > We already have a command queue at the target to >> > > > > enforce >> > > > > correct >> > > > > serialisation of commands, doing the same thing at the >initiator is >> > > > > redundant. >> > > > > >> > > > > Finally, I don't believe we should be writing a >> > > > > standard >> > > > > to work >> > > > > around poor coding and test coverage, especially at the cost of >> > > > > potential efficiency gains. >> > > > > >> > > > > I agree with Dave and Santosh that >commands being >> > > > > sent >> > > > > out of order >> > > > > on a single session should be allowed by the standard. >> > > > > >> > > > > - Rod >> > > > > >> > > > > -----Original Message----- >> > > > > From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > Of >> > > > > Barry Reinhold >> > > > > Sent: Friday, November 02, 2001 5:24 PM >> > > > > To: Dave Sheehy; IETF IP SAN Reflector >> > > > > Subject: RE: iSCSI: current UNH Plugfest >> > > > > >> > > > > Using features such as out of order command delivery on >a connection >> > > > > tend to >> > > > > be the sort of things that lead to interoperability >problems. It is >> > > > > unexpected and probably going to hit poorly tested code >paths even >> > if >> > > > > the >> > > > > standard is written to allow it. >> > > > > >> > > > > >-----Original Message----- >> > > > > >From: owner-ips@ece.cmu.edu >[mailto:owner-ips@ece.cmu.edu]On Behalf >> > > > > Of >> > > > > >Dave Sheehy >> > > > > >Sent: Friday, November 02, 2001 4:19 PM >> > > > > >To: IETF IP SAN Reflector >> > > > > >Subject: Re: iSCSI: current UNH Plugfest >> > > > > > >> > > > > > >> > > > > > >> > > > > >> 3. Can commands be sent out of order on the same connection? >> > > > > >> >> > > > > >> The behavior of targets is clearly specified in Section >> > 2.2.2.3 >> > > > > on >> > > > > >> page 25 of draft 8, which says: >> > > > > >> "Except for the commands marked for immediate >delivery the >> > > > > iSCSI >> > > > > >> target layer MUST eliver the commands for >execution in the >> > > > > order >> > > > > >> specified by CmdSN." >> > > > > >> >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: >> > > > > >> "- CmdSN - the current command Sequence Number >advanced by 1 >> > > > > on >> > > > > >> each command shipped except for commands marked for >> > immediate >> > > > > >> delivery." >> > > > > >> but the meaning of the term "shipped" is vague, >and does not >> > > > > >> necessarily >> > > > > >> require that the PDUs arrive on the other end of a TCP >> > > > > connection >> > > > > >> in the same order that the CmdSN values were >assigned to these >> > > > > PDUs. >> > > > > >> >> > > > > >> Some initiators have been designed to send commands out of >> > CmdSN >> > > > > >> order on one connection. Consider the situation >where there >> > is >> > > > > only >> > > > > >> one connection and a high-level dispatcher creates >a PDU for a >> > > > > SCSI >> > > > > >> command that involves writing immediate data to the target. >> > > > > This PDU >> > > > > >> is enqueued to a lower-level layer which has to >setup, start, >> > > > > and >> > > > > >> wait-for a DMA operation to move the immediate data into an >> > > > > onboard >> > > > > >> buffer before the PDU can be put onto the wire. >While this is >> > > > > >> happening, the dispatcher creates another >unrelated PDU for a >> > > > > SCSI >> > > > > >> read command (for example), and when this PDU is >passed to the >> > > > > >> lower-level layer it can be sent immediately, ahead of the >> > > > > previous >> > > > > >> write PDU and therefore out of order on this connection. >> > > > > >> >> > > > > >> The standard clearly allows this to happen if the two PDUs >> > were >> > > > > sent >> > > > > >> on different connections, and seems to imply that this can >> > also >> > > > > happen >> > > > > >> when the two PDUs are sent on the same connection. >> > > > > >> >> > > > > >> The suggestion is to put in the standard an >explicit statement >> > > > > that >> > > > > >> this is allowed or not allowed, as appropriate. >> > > > > >> >> > > > > >> If this is allowed, such a statement would avoid >the erroneous >> > > > > >> assumption being made by some target implementers >that within >> > a >> > > > > single >> > > > > >> connection, commands will arrive in order. >> > > > > >> >> > > > > >> If this is not allowed, such a statement would avoid the >> > > > > erroneous >> > > > > >> assumption being made by some initiator implementers that >> > within >> > > > > a >> > > > > >> single connection, commands can be put on the wire out of >> > order. >> > > > > >> >> > > > > >> +++ >> > > > > >> >> > > > > >> will add an explicit statement saying that this behaviour is >> > > > > forbidden. >> > > > > >> 2.2.2.1 will contain: >> > > > > >> >> > > > > >> On any given connection, the iSCSI initiator MUST send the >> > > > > >commands in the >> > > > > >> order specified by CmdSN. >> > > > > >> >> > > > > >> +++ >> > > > > > >> > > > > >Why do you feel this behavior should be forbidden? >Targets already >> > > > > have to >> > > > > >order commands across the session. I don't see why it's >a problem >> > to >> > > > > extend >> > > > > >that to the connection as well. I, for one, believe we >should take >> > > > > >a liberal >> > > > > >stance on this. >> > > > > > >> > > > > >Dave Sheehy >> > > > > > >> > > >> > > -- >> > > ################################## >> > > Santosh Rao >> > > Software Design Engineer, >> > > HP-UX iSCSI Driver Team, >> > > Hewlett Packard, Cupertino. >> > > email : santoshr@cup.hp.com >> > > Phone : 408-447-3751 >> > > ################################## >> > >> > >> > >> > >> > From owner-ips@ece.cmu.edu Thu Nov 8 10:31:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA28460 for ; Thu, 8 Nov 2001 10:31:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8EL3e21444 for ips-outgoing; Thu, 8 Nov 2001 09:21:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls05.mediaone.net (chmls05.mediaone.net [24.147.1.143]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8EKgl21420 for ; Thu, 8 Nov 2001 09:20:47 -0500 (EST) Received: from breinhold ([140.186.40.85]) by chmls05.mediaone.net (8.11.1/8.11.1) with SMTP id fA8EK6k10617; Thu, 8 Nov 2001 09:20:06 -0500 (EST) From: "Barry Reinhold" To: "Rod Harrison" , "Robert D. Russell" , "Somesh Gupta" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 09:19:24 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Rod, I can see your point, from the perspective of a target that is a box combining an iSCSI entity with a SCSI entity. However, from the perspective of an "iSCSI only" entity, delivery does not have to be delayed until C1 completes. In this case you only need to ensure that the commands are delivered in order to the SCSI layer. The SCSI layer has to address the semantics of what the operation entails in terms of command execution. Thus the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry about the fact that C1 generates data in response to the R2T. However the command sequence of C1, C3, C2, C4 forces the iSCSI layer to buffer C3. This is significant to a hardware accelerated iSCSI layer. >-----Original Message----- >From: Rod Harrison [mailto:rod.harrison@windriver.com] >Sent: Wednesday, November 07, 2001 8:34 PM >To: Barry Reinhold; Robert D. Russell; Somesh Gupta >Cc: Julian Satran; ips@ece.cmu.edu >Subject: RE: iSCSI: Out of order commands > > >Barry, > > I'm curious, where do you see the extra complexity between the >following? > > Assuming all the following commands are writes ... > >c1, c3, c2, c4 and > >c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. > > In both cases the target has to queue commands 2, 3, and 4 whilst >waiting for its R2T on c1 to be satisfied. > > Even if the target doesn't support any unsolicited data it still has >to queue commands 2, 3, and 4. I can't see how the target can avoid >queuing commands, in which case the order of arrival seems to make >little difference. > > - Rod > >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Barry Reinhold >Sent: Wednesday, November 07, 2001 3:10 PM >To: Robert D. Russell; Somesh Gupta >Cc: Julian Satran; ips@ece.cmu.edu >Subject: RE: iSCSI: Out of order commands > > >Bob, > I can't speak for targets, but OOO commands on a session with a >single >connection sure increases the complexity of the code path we take. > To me the issue is still that these types of situations tend to be >poorly >tested and lead to interoperability issues. If we do go down this >path, the >spec. should make it very clear that this is expected behavior. Some >statement with a shall in it should be written (for the receiver), so >that >there is a conformance test items to pass. > >>-----Original Message----- >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf >Of >>Robert D. Russell >>Sent: Wednesday, November 07, 2001 4:13 PM >>To: Somesh Gupta >>Cc: Julian Satran; ips@ece.cmu.edu >>Subject: RE: iSCSI: Out of order commands >> >> >>Somesh, Julian: >> >>You state that dealing with OOO commands on the target >>will add substantial complexity on the target. >>Do you have any basis for that claim? My impression from the >>plugfest is that most targets are already dealing with >>it. Perhaps we need to hear from someone who is actually >>building a target for which this would be a real problem. >> >>If anything, what we are hearing from people who really >>are building initiators is that dealing with the requirement >>to send commands in order will introduce substantial complexity >>on the initiator. >> >>So why should we be saving complexity on (hypothetically) simple >>targets yet requiring complexity on real initiators? >> >>As far as the deadlock issue is concerned, if the only way >>that deadlock can occur with OOO commands on the same >>connection is during the use of immediate data (which is I >>think what Julian was saying), then shouldn't we change >>the standard to just say that -- if the initiator sends >>commands out of order on a single connection, then immediate >>data MUST NOT be used on that connection in order to avoid deadlock. >> >>This gives everybody what they want, since initiators who find >>it beneficial to deliver commands OOO will just negotiate >>immediate data off. Those who really want to use immediate data >>will have to ensure that commands are sent in order. >>The tradeoff then becomes an implementation issue, not a >>standards issue, which is where it belongs. >> >> >>Bob Russell >>InterOperability Lab >>University of New Hampshire >>rdr@iol.unh.edu >>603-862-3774 >> >> >>On Wed, 7 Nov 2001, Somesh Gupta wrote: >> >>> I think we should either have it as a MUST or not require >>> it (at both ends to get the real benefit). SHOULD is one >>> of those things that leads to implementation >>> burden and confusion, without perhaps the feature being >>> used. There are implementation as well as protocol >>> considerations mixed in here. >>> >>> If we are to remove the restriction, we should (SHOULD) >>> get the maximum benefit from it, rather than to >>> accomodate an implementation choice. Out of sequence >>> commands, combined with the possibility of digest errors, >>> will add substantial complexity on the target side, >>> without corrosponding benefit in performance. If we change >>> this to SHOULD, we should also relax the requirement >>> to present commands on the target side to a SHOULD. >>> >>> >>> >>> > -----Original Message----- >>> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On >Behalf Of >>> > Julian Satran >>> > Sent: Wednesday, November 07, 2001 10:00 AM >>> > To: ips@ece.cmu.edu >>> > Subject: Re: iSCSI: Out of order commands >>> > >>> > >>> > Mallikarjun, >>> > >>> > I did not see a SINGLE performance improvement that results from >OOO >>> > shipping. >>> > I would be bad engineering to give away the "no-deadlock" >mechanism we >>> > have now for nothing. >>> > I have also the impression that the point about deadlock that I >keep >>> > repeating is ignored or not understood. >>> > As we stand today commands can be shipped with Immediate data >>or without >>> > and an implementer determined >>> > to squeeze maximum bandwidth and overlap command start with >>delivery will >>> > choose not to work with immediate data >>> > (as you have pointed out) while a low performance software >>implementation >>> > will use immediate data to minimize CPU cycles consumed. However >both >>> > will be guaranteed to work without deadlock as source and sink >use the >>> > same ordering. >>> > Recovery is still a low probability event and should be handled >with a >>> > different set of considerations in mind. >>> > As for the strictness of the recommendation - yes we could settle >on >>> > SHOULD. >>> > >>> > Julo >>> > >>> > >>> > >>> > >>> > "Mallikarjun C." >>> > Sent by: owner-ips@ece.cmu.edu >>> > 07-11-01 19:41 >>> > Please respond to cbm >>> > >>> > >>> > To: Santosh Rao , >ips@ece.cmu.edu >>> > cc: >>> > Subject: Re: iSCSI: Out of order commands >>> > >>> > >>> > >>> > Santosh, >>> > >>> > I have only one comment on your responses. >>> > >>> > > Even a single connection target *MUST* implement a scoreboard. >The >>> > > reason being that it can see out-of-order arrival of commands >due to >>> > > commands being dropped on digest errors. In such a case, it >>must block >>> > > further command processing until holes are filled. >>> > >>> > I made two convenient assumptions if you noticed, :-), one of >which >>> > is that target forces session recovery on *any* error that it >sees >>> > (ErrorRecoveryLevel=0) - including a dropped command due to a >digest >>> > error. With that assumption, a target can afford not to >implement >>> > a scoreboard. >>> > >>> > As I said in a private note, I guess what primarily bothers me >about >>> > OOO commands on a connection is that it requires the receiver to >>> > undo this "optimization" on its end - most notably on a single >>> > connection. TCP experts may comment on how/if they dealt with a >>> > similar issue. >>> > >>> > OTOH, you had some valid comments on exceptions to ordering >during >>> > connection recovery. Perhaps we can move on by making Julian's >>> > proposed stipulation a SHOULD.... >>> > -- >>> > Mallikarjun >>> > >>> > >>> > Mallikarjun Chadalapaka >>> > Networked Storage Architecture >>> > Network Storage Solutions Organization >>> > MS 5668 Hewlett-Packard, Roseville. >>> > cbm@rose.hp.com >>> > >>> > >>> > Santosh Rao wrote: >>> > > >>> > > Mallikarjun, >>> > > >>> > > Some comments below. >>> > > >>> > > Regards, >>> > > Santosh >>> > > >>> > > "Mallikarjun C." wrote: >>> > > > >>> > > > Rod and Julian, >>> > > > >>> > > > This has been an interesting thread of discussion. Some >>> > > > comments - >>> > > > >>> > > > 1.My first reaction was - allowing out-of-order command >>> > > > transmission on the same connection deprives targets of >>> > > > an implementation choice. Targets which support only >>> > > > single-connection sessions and only support session >>> > > > recovery (reasonable assumptions in my mind) can no >>> > > > longer afford *not to* implement a command scoreboard. >>> > > >>> > > Even a single connection target *MUST* implement a scoreboard. >The >>> > > reason being that it can see out-of-order arrival of commands >due to >>> > > commands being dropped on digest errors. In such a case, it >>must block >>> > > further command processing until holes are filled. >>> > > >>> > > Thus, there is no getting away from implementing a sequencer at >the >>> > > target. Given this, I think it is unreasonable to restrict >initiator >>> > > implementation flexibility by imposing a strict ordering >requirement >>> > > within the connection. >>> > > >>> > > > 2.Any end-node efficiency that is sought to be achieved >>> > > > by transmitting CmdSNs out-of-order from the initiator >>> > > > would be lost on the other end-node, since the target >>> > > > now must wait for re-ordering the commands. >>> > > >>> > > It has to handle this situation anyway to deal with holes >caused by >>> > > digest errors. This scenario occurs even with initiators that >issue >>> > > commands in order. >>> > > >>> > > > >>> > > > 3.The flipside is that out-of-order transmission saves >>> > > > link badwidth (albeit at the expense of end-node >efficiency), >>> > > > compared to idling the link waiting for outbound DMA. >>> > > > We have to determine if this is a reasonable trade-off. >>> > > > >>> > > > 4.I can see Rod's point that prefetching all immediate >>> > > > data can be a burden on the NIC resources. But, two >>> > > > questions - >>> > > > - could the NIC not use unsolicited separate data >>> > > > PDUs in these cases? [ I realize that InitialR2T >>> > > > has to be "no" to let it happen... ] >>> > > > - could the NIC have a memory architecture that >>> > > > allows data prefetching for the next command (so >>> > > > this is a non-issue from the protocol perspective)? >>> > > > This scheme incurs one DMA delay for every new >>> > > > burst of commands. >>> > > > >>> > > > 5.Another (perhaps radical at this point) option is to do >>> > > > away with immediate unsolicited data, to stick only with >>> > > > separate unsolicited data. I would personally be okay >>> > > > with the choice, particularly if this feature (that >>> > > > helps software implementations) starts making hardware >>> > > > design complicated/expensive. >>> > > > >>> > > > So, to summarize - >>> > > > >>> > > > option immediate allow >>> > > > data in spec? >out-of-order? >>> > > > >>> > > > (A) (5) above no no >>> > > > (B) No real reason to do this. no yes >>> > > > (C) (4) above yes no >>> > > > (D) pros & cons (1), (2) & (3) yes yes >>> > > > >>> > > > >From the arguments I heard so far, I am leaning towards >>> > > > option A, and option C in that order. >>> > > > >>> > > > Comments? >>> > > > -- >>> > > > Mallikarjun >>> > > > >>> > > > Mallikarjun Chadalapaka >>> > > > Networked Storage Architecture >>> > > > Network Storage Solutions Organization >>> > > > MS 5668 Hewlett-Packard, Roseville. >>> > > > cbm@rose.hp.com >>> > > > >>> > > > Rod Harrison wrote: >>> > > > > >>> > > > > Julian, >>> > > > > >>> > > > > I don't understand what you are proposing here, >>what do you >>> > mean by >>> > > > > "multiplexed" DMA? >>> > > > > >>> > > > > The problem is that the DMAs take some time, the >>more there >>> > are >>> > > > > queued the longer the last DMAs queued take to complete. >Some >>> > commands >>> > > > > require DMAs to complete before they can be sent, i.e. >>Writes with >>> > > > > immediate data, some commands do not, i.e. Reads and >>writes with no >>> > > > > immediate data. The iSCSI HBA wants to be able to send >>commands as >>> > > > > soon a possible, which for a read after a write can be >before the >>> > > > > write's DMA has completed. Maintaining an ordered queue >>for commands >>> > > > > to be sent on the HBA is expensive and redundant since the >target >>> > > > > already knows how to queue commands before committing them >to its >>> > SCSI >>> > > > > layer. >>> > > > > >>> > > > > The iSCSI HBA and its host driver are not at >liberty to >>> > change the >>> > > > > order of commands from the OS, but the DMAs those >>commands need are >>> > > > > unlikely to complete in the same order, and as I mentioned >some >>> > > > > commands need no DMA. If the HBA can't send commands out of >CmdSN >>> > > > > order it has to maintain an ordered queue of commands >>waiting to be >>> > > > > sent, and potentially buffer a lot of data. For an HBA this >makes >>> > > > > immediate data almost impossible to support. >>> > > > > >>> > > > > I don't see the problem with allowing out of >>order commands >>> > given >>> > > > > that the target already has to deal with very similar >problems. I >>> > > > > think we are getting in to the area of implementation >>choices here, >>> > > > > which is inappropriate for a specification. >>> > > > > >>> > > > > - Rod >>> > > > > >>> > > > > -----Original Message----- >>> > > > > From: owner-ips@ece.cmu.edu >>[mailto:owner-ips@ece.cmu.edu]On Behalf >>> > Of >>> > > > > Julian Satran >>> > > > > Sent: Monday, November 05, 2001 10:06 PM >>> > > > > To: ips@ece.cmu.edu >>> > > > > Subject: Re: iSCSI: Out of order commands, was current >>UNH Plugfest >>> > > > > >>> > > > > Rod, >>> > > > > >>> > > > > I don't see any reason why DMA operations cant be >>"multiplexed" with >>> > > > > commands. >>> > > > > If you have scheduled a long outbound DMA you are doomed >>regardless >>> > of >>> > > > > the >>> > > > > command ordering. >>> > > > > And if you have scheduled DMA operations piecemeal then you >can >>> > insert >>> > > > > your commands in correct order. >>> > > > > >>> > > > > Julo >>> > > > > >>> > > > > "Rod Harrison" >>> > > > > 05-11-01 20:48 >>> > > > > Please respond to "Rod Harrison" >>> > > > > >>> > > > > To: Julian Satran/Haifa/IBM@IBMIL, > >>> > > > > cc: >>> > > > > Subject: iSCSI: Out of order commands, was >current >>> > UNH >>> > > > > Plugfest >>> > > > > >>> > > > > [ Subject changed ] >>> > > > > >>> > > > > Julian, >>> > > > > >>> > > > > The ordering difference is introduced >>between the >>> > > > > host >>> > > > > side driver >>> > > > > and the iSCSI HBA. The host side driver must present >>SCSI commands >>> > to >>> > > > > the HBA in the order they are received from the OS to >>prevent read >>> > > > > after write dependency failures. The HBA might reorder >>the commands >>> > > > > depending on when DMA completes. The reordering can't be >>done ahead >>> > of >>> > > > > time in the host driver since it doesn't know how long each >DMA >>> > might >>> > > > > take. As long as the HBA assigns CmdSN in the order it >receives >>> > > > > commands the desired host ordering is preserved. >>> > > > > >>> > > > > - Rod >>> > > > > >>> > > > > -----Original Message----- >>> > > > > From: owner-ips@ece.cmu.edu >>[mailto:owner-ips@ece.cmu.edu]On Behalf >>> > Of >>> > > > > Julian Satran >>> > > > > Sent: Monday, November 05, 2001 12:35 AM >>> > > > > To: ips@ece.cmu.edu >>> > > > > Subject: RE: iSCSI: current UNH Plugfest >>> > > > > >>> > > > > Rod, >>> > > > > >>> > > > > I all examples give the point I find hard to understand is >why is >>> > the >>> > > > > ordering on the wire different from the presentation order >to the >>> > > > > initiator. You can get as many overlaps as you want by >>presenting >>> > the >>> > > > > commands to the initiator in the desired order. >>> > > > > What we are considering here is the case in which you >>want to ship >>> > in >>> > > > > an >>> > > > > order different than the one you present the commands. >>> > > > > >>> > > > > Julo >>> > > > > >>> > > > > "Rod Harrison" >>> > > > > Sent by: owner-ips@ece.cmu.edu >>> > > > > 04-11-01 04:42 >>> > > > > Please respond to "Rod Harrison" >>> > > > > >>> > > > > To: "Barry Reinhold" , >"Dave >>> > > > > Sheehy" >>> > > > > , "IETF IP SAN Reflector" >>> > > > > >>> > > > > cc: >>> > > > > Subject: RE: iSCSI: current UNH Plugfest >>> > > > > >>> > > > > Barry, >>> > > > > >>> > > > > In general I agree but I don't think this >is as >>> > much >>> > > > > of a >>> > > > > corner case >>> > > > > as it at first appears. Targets will have code very >>similar to that >>> > > > > needed to handle out of order commands to deal with >>digest errors. >>> > > > > Targets also need to queue commands whilst waiting for both >>> > solicited >>> > > > > and unsolicited data to arrive. Queuing out of order >>commands seems >>> > > > > little extra work. >>> > > > > >>> > > > > From an initiators point of view there are >>> > > > > efficiency, >>> > > > > and probably >>> > > > > performance gains to be had from sending commands out of >>order. Bob >>> > > > > Russell gave the example of a read being sent whilst >>write data DMA >>> > is >>> > > > > happening, and a similar situation can arise with DMA for >writes >>> > > > > overtaking that of earlier writes if the initiator has >>multiple DMA >>> > > > > engines. In this case the initiator might be forced to >>let the wire >>> > go >>> > > > > idle if it can't send the data from completed DMAs as soon >as >>> > > > > possible. >>> > > > > >>> > > > > We already have a command queue at the >target to >>> > > > > enforce >>> > > > > correct >>> > > > > serialisation of commands, doing the same thing at the >>initiator is >>> > > > > redundant. >>> > > > > >>> > > > > Finally, I don't believe we should be >writing a >>> > > > > standard >>> > > > > to work >>> > > > > around poor coding and test coverage, especially at the >cost of >>> > > > > potential efficiency gains. >>> > > > > >>> > > > > I agree with Dave and Santosh that >>commands being >>> > > > > sent >>> > > > > out of order >>> > > > > on a single session should be allowed by the standard. >>> > > > > >>> > > > > - Rod >>> > > > > >>> > > > > -----Original Message----- >>> > > > > From: owner-ips@ece.cmu.edu >>[mailto:owner-ips@ece.cmu.edu]On Behalf >>> > Of >>> > > > > Barry Reinhold >>> > > > > Sent: Friday, November 02, 2001 5:24 PM >>> > > > > To: Dave Sheehy; IETF IP SAN Reflector >>> > > > > Subject: RE: iSCSI: current UNH Plugfest >>> > > > > >>> > > > > Using features such as out of order command delivery on >>a connection >>> > > > > tend to >>> > > > > be the sort of things that lead to interoperability >>problems. It is >>> > > > > unexpected and probably going to hit poorly tested code >>paths even >>> > if >>> > > > > the >>> > > > > standard is written to allow it. >>> > > > > >>> > > > > >-----Original Message----- >>> > > > > >From: owner-ips@ece.cmu.edu >>[mailto:owner-ips@ece.cmu.edu]On Behalf >>> > > > > Of >>> > > > > >Dave Sheehy >>> > > > > >Sent: Friday, November 02, 2001 4:19 PM >>> > > > > >To: IETF IP SAN Reflector >>> > > > > >Subject: Re: iSCSI: current UNH Plugfest >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > >> 3. Can commands be sent out of order on the same >connection? >>> > > > > >> >>> > > > > >> The behavior of targets is clearly specified in >Section >>> > 2.2.2.3 >>> > > > > on >>> > > > > >> page 25 of draft 8, which says: >>> > > > > >> "Except for the commands marked for immediate >>delivery the >>> > > > > iSCSI >>> > > > > >> target layer MUST eliver the commands for >>execution in the >>> > > > > order >>> > > > > >> specified by CmdSN." >>> > > > > >> >>> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: >>> > > > > >> "- CmdSN - the current command Sequence Number >>advanced by 1 >>> > > > > on >>> > > > > >> each command shipped except for commands marked for >>> > immediate >>> > > > > >> delivery." >>> > > > > >> but the meaning of the term "shipped" is vague, >>and does not >>> > > > > >> necessarily >>> > > > > >> require that the PDUs arrive on the other end of a >TCP >>> > > > > connection >>> > > > > >> in the same order that the CmdSN values were >>assigned to these >>> > > > > PDUs. >>> > > > > >> >>> > > > > >> Some initiators have been designed to send commands >out of >>> > CmdSN >>> > > > > >> order on one connection. Consider the situation >>where there >>> > is >>> > > > > only >>> > > > > >> one connection and a high-level dispatcher creates >>a PDU for a >>> > > > > SCSI >>> > > > > >> command that involves writing immediate data to the >target. >>> > > > > This PDU >>> > > > > >> is enqueued to a lower-level layer which has to >>setup, start, >>> > > > > and >>> > > > > >> wait-for a DMA operation to move the immediate data >into an >>> > > > > onboard >>> > > > > >> buffer before the PDU can be put onto the wire. >>While this is >>> > > > > >> happening, the dispatcher creates another >>unrelated PDU for a >>> > > > > SCSI >>> > > > > >> read command (for example), and when this PDU is >>passed to the >>> > > > > >> lower-level layer it can be sent immediately, ahead >of the >>> > > > > previous >>> > > > > >> write PDU and therefore out of order on this >connection. >>> > > > > >> >>> > > > > >> The standard clearly allows this to happen if the two >PDUs >>> > were >>> > > > > sent >>> > > > > >> on different connections, and seems to imply that >this can >>> > also >>> > > > > happen >>> > > > > >> when the two PDUs are sent on the same connection. >>> > > > > >> >>> > > > > >> The suggestion is to put in the standard an >>explicit statement >>> > > > > that >>> > > > > >> this is allowed or not allowed, as appropriate. >>> > > > > >> >>> > > > > >> If this is allowed, such a statement would avoid >>the erroneous >>> > > > > >> assumption being made by some target implementers >>that within >>> > a >>> > > > > single >>> > > > > >> connection, commands will arrive in order. >>> > > > > >> >>> > > > > >> If this is not allowed, such a statement would avoid >the >>> > > > > erroneous >>> > > > > >> assumption being made by some initiator implementers >that >>> > within >>> > > > > a >>> > > > > >> single connection, commands can be put on the wire >out of >>> > order. >>> > > > > >> >>> > > > > >> +++ >>> > > > > >> >>> > > > > >> will add an explicit statement saying that this >behaviour is >>> > > > > forbidden. >>> > > > > >> 2.2.2.1 will contain: >>> > > > > >> >>> > > > > >> On any given connection, the iSCSI initiator MUST send >the >>> > > > > >commands in the >>> > > > > >> order specified by CmdSN. >>> > > > > >> >>> > > > > >> +++ >>> > > > > > >>> > > > > >Why do you feel this behavior should be forbidden? >>Targets already >>> > > > > have to >>> > > > > >order commands across the session. I don't see why it's >>a problem >>> > to >>> > > > > extend >>> > > > > >that to the connection as well. I, for one, believe we >>should take >>> > > > > >a liberal >>> > > > > >stance on this. >>> > > > > > >>> > > > > >Dave Sheehy >>> > > > > > >>> > > >>> > > -- >>> > > ################################## >>> > > Santosh Rao >>> > > Software Design Engineer, >>> > > HP-UX iSCSI Driver Team, >>> > > Hewlett Packard, Cupertino. >>> > > email : santoshr@cup.hp.com >>> > > Phone : 408-447-3751 >>> > > ################################## >>> > >>> > >>> > >>> > >>> >> > From owner-ips@ece.cmu.edu Thu Nov 8 12:10:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00946 for ; Thu, 8 Nov 2001 12:10:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8Fowq28425 for ips-outgoing; Thu, 8 Nov 2001 10:50:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8Foul28420 for ; Thu, 8 Nov 2001 10:50:56 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id KAA12983; Thu, 8 Nov 2001 10:50:50 -0500 (EST) Date: Thu, 8 Nov 2001 10:50:49 -0500 From: "Robert D. Russell" To: Julian Satran cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: > However allowing initiators to ship them out of order creates a > potential deadlock that does not appear otherwise. > > If a target is missing a command in a queue (and there are no errors) then > this command is bound to be first on some connection under the current set > of rules. > > If we allow OOO shipping then the missing command can be somewhere > "inside" the window on some connection and if the target has just filled > his queue and has room in the staging buffer only for the command it is > waiting for and that command happens to be the first to pass to SCSI you > have a deadlock. I must be understanding something. Your example is certainly correct if the target has no control over the number of commands sent to it by the initiator. But the target does control this number through the MaxCmdSN field. For the scenario you are describing to occur, wouldn't it be necessary for the target to advertize a MaxCmdSN value bigger than it actually has resources to handle? It seems to me that if a target can only handle x new commands, then its queueing capacity is x, and in the SCSI Response PDU it should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the formula in section 2.2.2.1. This in turn controls the number of commands the initiator can send, and thus prevents the incoming commands from overfilling the target's queue. So isn't the deadlock caused by a broken target? Isn't the target advertizing a queueing capacity greater than it actually has and isn't that the cause of the deadlock? An alternative explanation of the deadlock might be that the target is advertizing the correct MaxCmdSN, but the initiator is sending commands beyond what it is allowed to send. However, in this case the target should silently ignore any non-immediate command outside the allowed range. For the deadlock to occur, wouldn't the target have to be queuing commands outside the allowed range instead of ignoring them? So in this case too, the target is broken and that is what causes the deadlock. Where am I going wrong in this reasoning? Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Thu Nov 8 13:00:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA02750 for ; Thu, 8 Nov 2001 13:00:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8H3tT04064 for ips-outgoing; Thu, 8 Nov 2001 12:03:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8H3rl04059 for ; Thu, 8 Nov 2001 12:03:53 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id BCF421A8B; Thu, 8 Nov 2001 10:03:52 -0700 (MST) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 24739615; Thu, 8 Nov 2001 10:03:52 -0700 (MST) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 08 Nov 2001 10:03:51 -0700 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Nov 2001 10:03:51 -0700 Message-ID: <9F8400020EC5D311AF62009027C39616048A4230@axcs09.cos.agilent.com> From: "FRYMAN,ROBERT (A-Roseville,ex1)" To: "'Barry Reinhold'" , Rod Harrison , "Robert D. Russell" , Somesh Gupta Cc: Julian Satran , ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 10:03:50 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Thus the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry > about the fact that C1 generates data in response to the R2T. > However the command sequence of C1, C3, C2, C4 forces the iSCSI > layer to buffer C3. This is significant to a hardware accelerated > iSCSI layer. This is significant to a target hardware accelerated iSCSI layer. By forcing the initiator to order the iSCSI commands, you move the significants from the target to the initiator, but it is still a complexity added to a hardware accelerator. The problem I have is that we are trying to add a requirement that only effects a specific area, but adds complexity to a larger area. If we say that initiators must order commands on a single connection, then doesn't this mean that we have just added complexity and slowdown to the enterprise storage area? In the Fibre channel realm, a standard base configuration was to have a target (JBOD, array, etc.) with 2 ports, and an initiator with two ports. These would be connected on two seperate physical connections, so if a problem happens on one, the other will take over. We allow the same thing in iSCSI, plus specified how the customer can use both connections to increase bandwidth as long as both connections work. It seems to me that any customer that wants a fault tolerant solution, and gets increased bandwidth for free due to the ability to have multiple connections, would take advantage of that. This being the case, these targets have to have reordering capability. If we force the initiators to send the comands in order, we just added complexity and slowdown that adds no benifit. As I see it (and I could be totally off base here), we have a choice of complexity and slowdown in the enterprise market, or adding a little complexity to simple targets. Am I flawed in this thinking? Scott Fryman Agilent Technologies > -----Original Message----- > From: Barry Reinhold [mailto:bbrtrebia@mediaone.net] > Sent: Thursday, November 08, 2001 6:19 AM > To: Rod Harrison; Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Rod, > I can see your point, from the perspective of a target > that is a box > combining an iSCSI entity with a SCSI entity. However, from > the perspective > of an "iSCSI only" entity, delivery does not have to be > delayed until C1 > completes. In this case you only need to ensure that the commands are > delivered in order to the SCSI layer. The SCSI layer has to > address the > semantics of what the operation entails in terms of command > execution. Thus > the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry > about the fact > that C1 generates data in response to the R2T. However the > command sequence > of C1, C3, C2, C4 forces the iSCSI layer to buffer C3. This > is significant > to a hardware accelerated iSCSI layer. > > >-----Original Message----- > >From: Rod Harrison [mailto:rod.harrison@windriver.com] > >Sent: Wednesday, November 07, 2001 8:34 PM > >To: Barry Reinhold; Robert D. Russell; Somesh Gupta > >Cc: Julian Satran; ips@ece.cmu.edu > >Subject: RE: iSCSI: Out of order commands > > > > > >Barry, > > > > I'm curious, where do you see the extra complexity between the > >following? > > > > Assuming all the following commands are writes ... > > > >c1, c3, c2, c4 and > > > >c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. > > > > In both cases the target has to queue commands 2, 3, > and 4 whilst > >waiting for its R2T on c1 to be satisfied. > > > > Even if the target doesn't support any unsolicited data > it still has > >to queue commands 2, 3, and 4. I can't see how the target can avoid > >queuing commands, in which case the order of arrival seems to make > >little difference. > > > > - Rod > > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of > >Barry Reinhold > >Sent: Wednesday, November 07, 2001 3:10 PM > >To: Robert D. Russell; Somesh Gupta > >Cc: Julian Satran; ips@ece.cmu.edu > >Subject: RE: iSCSI: Out of order commands > > > > > >Bob, > > I can't speak for targets, but OOO commands on a session with a > >single > >connection sure increases the complexity of the code path we take. > > To me the issue is still that these types of situations > tend to be > >poorly > >tested and lead to interoperability issues. If we do go down this > >path, the > >spec. should make it very clear that this is expected behavior. Some > >statement with a shall in it should be written (for the receiver), so > >that > >there is a conformance test items to pass. > > > >>-----Original Message----- > >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > >Of > >>Robert D. Russell > >>Sent: Wednesday, November 07, 2001 4:13 PM > >>To: Somesh Gupta > >>Cc: Julian Satran; ips@ece.cmu.edu > >>Subject: RE: iSCSI: Out of order commands > >> > >> > >>Somesh, Julian: > >> > >>You state that dealing with OOO commands on the target > >>will add substantial complexity on the target. > >>Do you have any basis for that claim? My impression from the > >>plugfest is that most targets are already dealing with > >>it. Perhaps we need to hear from someone who is actually > >>building a target for which this would be a real problem. > >> > >>If anything, what we are hearing from people who really > >>are building initiators is that dealing with the requirement > >>to send commands in order will introduce substantial complexity > >>on the initiator. > >> > >>So why should we be saving complexity on (hypothetically) simple > >>targets yet requiring complexity on real initiators? > >> > >>As far as the deadlock issue is concerned, if the only way > >>that deadlock can occur with OOO commands on the same > >>connection is during the use of immediate data (which is I > >>think what Julian was saying), then shouldn't we change > >>the standard to just say that -- if the initiator sends > >>commands out of order on a single connection, then immediate > >>data MUST NOT be used on that connection in order to avoid deadlock. > >> > >>This gives everybody what they want, since initiators who find > >>it beneficial to deliver commands OOO will just negotiate > >>immediate data off. Those who really want to use immediate data > >>will have to ensure that commands are sent in order. > >>The tradeoff then becomes an implementation issue, not a > >>standards issue, which is where it belongs. > >> > >> > >>Bob Russell > >>InterOperability Lab > >>University of New Hampshire > >>rdr@iol.unh.edu > >>603-862-3774 > >> > >> > >>On Wed, 7 Nov 2001, Somesh Gupta wrote: > >> > >>> I think we should either have it as a MUST or not require > >>> it (at both ends to get the real benefit). SHOULD is one > >>> of those things that leads to implementation > >>> burden and confusion, without perhaps the feature being > >>> used. There are implementation as well as protocol > >>> considerations mixed in here. > >>> > >>> If we are to remove the restriction, we should (SHOULD) > >>> get the maximum benefit from it, rather than to > >>> accomodate an implementation choice. Out of sequence > >>> commands, combined with the possibility of digest errors, > >>> will add substantial complexity on the target side, > >>> without corrosponding benefit in performance. If we change > >>> this to SHOULD, we should also relax the requirement > >>> to present commands on the target side to a SHOULD. > >>> > >>> > >>> > >>> > -----Original Message----- > >>> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > >Behalf Of > >>> > Julian Satran > >>> > Sent: Wednesday, November 07, 2001 10:00 AM > >>> > To: ips@ece.cmu.edu > >>> > Subject: Re: iSCSI: Out of order commands > >>> > > >>> > > >>> > Mallikarjun, > >>> > > >>> > I did not see a SINGLE performance improvement that results from > >OOO > >>> > shipping. > >>> > I would be bad engineering to give away the "no-deadlock" > >mechanism we > >>> > have now for nothing. > >>> > I have also the impression that the point about deadlock that I > >keep > >>> > repeating is ignored or not understood. > >>> > As we stand today commands can be shipped with Immediate data > >>or without > >>> > and an implementer determined > >>> > to squeeze maximum bandwidth and overlap command start with > >>delivery will > >>> > choose not to work with immediate data > >>> > (as you have pointed out) while a low performance software > >>implementation > >>> > will use immediate data to minimize CPU cycles > consumed. However > >both > >>> > will be guaranteed to work without deadlock as source and sink > >use the > >>> > same ordering. > >>> > Recovery is still a low probability event and should be handled > >with a > >>> > different set of considerations in mind. > >>> > As for the strictness of the recommendation - yes we > could settle > >on > >>> > SHOULD. > >>> > > >>> > Julo > >>> > > >>> > > >>> > > >>> > > >>> > "Mallikarjun C." > >>> > Sent by: owner-ips@ece.cmu.edu > >>> > 07-11-01 19:41 > >>> > Please respond to cbm > >>> > > >>> > > >>> > To: Santosh Rao , > >ips@ece.cmu.edu > >>> > cc: > >>> > Subject: Re: iSCSI: Out of order commands > >>> > > >>> > > >>> > > >>> > Santosh, > >>> > > >>> > I have only one comment on your responses. > >>> > > >>> > > Even a single connection target *MUST* implement a scoreboard. > >The > >>> > > reason being that it can see out-of-order arrival of commands > >due to > >>> > > commands being dropped on digest errors. In such a case, it > >>must block > >>> > > further command processing until holes are filled. > >>> > > >>> > I made two convenient assumptions if you noticed, :-), one of > >which > >>> > is that target forces session recovery on *any* error that it > >sees > >>> > (ErrorRecoveryLevel=0) - including a dropped command due to a > >digest > >>> > error. With that assumption, a target can afford not to > >implement > >>> > a scoreboard. > >>> > > >>> > As I said in a private note, I guess what primarily bothers me > >about > >>> > OOO commands on a connection is that it requires the receiver to > >>> > undo this "optimization" on its end - most notably on a single > >>> > connection. TCP experts may comment on how/if they dealt with a > >>> > similar issue. > >>> > > >>> > OTOH, you had some valid comments on exceptions to ordering > >during > >>> > connection recovery. Perhaps we can move on by making Julian's > >>> > proposed stipulation a SHOULD.... > >>> > -- > >>> > Mallikarjun > >>> > > >>> > > >>> > Mallikarjun Chadalapaka > >>> > Networked Storage Architecture > >>> > Network Storage Solutions Organization > >>> > MS 5668 Hewlett-Packard, Roseville. > >>> > cbm@rose.hp.com > >>> > > >>> > > >>> > Santosh Rao wrote: > >>> > > > >>> > > Mallikarjun, > >>> > > > >>> > > Some comments below. > >>> > > > >>> > > Regards, > >>> > > Santosh > >>> > > > >>> > > "Mallikarjun C." wrote: > >>> > > > > >>> > > > Rod and Julian, > >>> > > > > >>> > > > This has been an interesting thread of discussion. Some > >>> > > > comments - > >>> > > > > >>> > > > 1.My first reaction was - allowing out-of-order command > >>> > > > transmission on the same connection deprives targets of > >>> > > > an implementation choice. Targets which support only > >>> > > > single-connection sessions and only support session > >>> > > > recovery (reasonable assumptions in my mind) can no > >>> > > > longer afford *not to* implement a command scoreboard. > >>> > > > >>> > > Even a single connection target *MUST* implement a scoreboard. > >The > >>> > > reason being that it can see out-of-order arrival of commands > >due to > >>> > > commands being dropped on digest errors. In such a case, it > >>must block > >>> > > further command processing until holes are filled. > >>> > > > >>> > > Thus, there is no getting away from implementing a > sequencer at > >the > >>> > > target. Given this, I think it is unreasonable to restrict > >initiator > >>> > > implementation flexibility by imposing a strict ordering > >requirement > >>> > > within the connection. > >>> > > > >>> > > > 2.Any end-node efficiency that is sought to be achieved > >>> > > > by transmitting CmdSNs out-of-order from the initiator > >>> > > > would be lost on the other end-node, since the target > >>> > > > now must wait for re-ordering the commands. > >>> > > > >>> > > It has to handle this situation anyway to deal with holes > >caused by > >>> > > digest errors. This scenario occurs even with initiators that > >issue > >>> > > commands in order. > >>> > > > >>> > > > > >>> > > > 3.The flipside is that out-of-order transmission saves > >>> > > > link badwidth (albeit at the expense of end-node > >efficiency), > >>> > > > compared to idling the link waiting for outbound DMA. > >>> > > > We have to determine if this is a reasonable trade-off. > >>> > > > > >>> > > > 4.I can see Rod's point that prefetching all immediate > >>> > > > data can be a burden on the NIC resources. But, two > >>> > > > questions - > >>> > > > - could the NIC not use unsolicited separate data > >>> > > > PDUs in these cases? [ I realize that InitialR2T > >>> > > > has to be "no" to let it happen... ] > >>> > > > - could the NIC have a memory architecture that > >>> > > > allows data prefetching for the next command (so > >>> > > > this is a non-issue from the protocol > perspective)? > >>> > > > This scheme incurs one DMA delay for every new > >>> > > > burst of commands. > >>> > > > > >>> > > > 5.Another (perhaps radical at this point) option is to do > >>> > > > away with immediate unsolicited data, to stick only with > >>> > > > separate unsolicited data. I would personally be okay > >>> > > > with the choice, particularly if this feature (that > >>> > > > helps software implementations) starts making hardware > >>> > > > design complicated/expensive. > >>> > > > > >>> > > > So, to summarize - > >>> > > > > >>> > > > option immediate allow > >>> > > > data in spec? > >out-of-order? > >>> > > > > >>> > > > (A) (5) above no no > >>> > > > (B) No real reason to do this. no yes > >>> > > > (C) (4) above yes no > >>> > > > (D) pros & cons (1), (2) & (3) yes yes > >>> > > > > >>> > > > >From the arguments I heard so far, I am leaning towards > >>> > > > option A, and option C in that order. > >>> > > > > >>> > > > Comments? > >>> > > > -- > >>> > > > Mallikarjun > >>> > > > > >>> > > > Mallikarjun Chadalapaka > >>> > > > Networked Storage Architecture > >>> > > > Network Storage Solutions Organization > >>> > > > MS 5668 Hewlett-Packard, Roseville. > >>> > > > cbm@rose.hp.com > >>> > > > > >>> > > > Rod Harrison wrote: > >>> > > > > > >>> > > > > Julian, > >>> > > > > > >>> > > > > I don't understand what you are proposing here, > >>what do you > >>> > mean by > >>> > > > > "multiplexed" DMA? > >>> > > > > > >>> > > > > The problem is that the DMAs take some time, the > >>more there > >>> > are > >>> > > > > queued the longer the last DMAs queued take to complete. > >Some > >>> > commands > >>> > > > > require DMAs to complete before they can be sent, i.e. > >>Writes with > >>> > > > > immediate data, some commands do not, i.e. Reads and > >>writes with no > >>> > > > > immediate data. The iSCSI HBA wants to be able to send > >>commands as > >>> > > > > soon a possible, which for a read after a write can be > >before the > >>> > > > > write's DMA has completed. Maintaining an ordered queue > >>for commands > >>> > > > > to be sent on the HBA is expensive and redundant since the > >target > >>> > > > > already knows how to queue commands before committing them > >to its > >>> > SCSI > >>> > > > > layer. > >>> > > > > > >>> > > > > The iSCSI HBA and its host driver are not at > >liberty to > >>> > change the > >>> > > > > order of commands from the OS, but the DMAs those > >>commands need are > >>> > > > > unlikely to complete in the same order, and as I mentioned > >some > >>> > > > > commands need no DMA. If the HBA can't send > commands out of > >CmdSN > >>> > > > > order it has to maintain an ordered queue of commands > >>waiting to be > >>> > > > > sent, and potentially buffer a lot of data. For > an HBA this > >makes > >>> > > > > immediate data almost impossible to support. > >>> > > > > > >>> > > > > I don't see the problem with allowing out of > >>order commands > >>> > given > >>> > > > > that the target already has to deal with very similar > >problems. I > >>> > > > > think we are getting in to the area of implementation > >>choices here, > >>> > > > > which is inappropriate for a specification. > >>> > > > > > >>> > > > > - Rod > >>> > > > > > >>> > > > > -----Original Message----- > >>> > > > > From: owner-ips@ece.cmu.edu > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > >>> > Of > >>> > > > > Julian Satran > >>> > > > > Sent: Monday, November 05, 2001 10:06 PM > >>> > > > > To: ips@ece.cmu.edu > >>> > > > > Subject: Re: iSCSI: Out of order commands, was current > >>UNH Plugfest > >>> > > > > > >>> > > > > Rod, > >>> > > > > > >>> > > > > I don't see any reason why DMA operations cant be > >>"multiplexed" with > >>> > > > > commands. > >>> > > > > If you have scheduled a long outbound DMA you are doomed > >>regardless > >>> > of > >>> > > > > the > >>> > > > > command ordering. > >>> > > > > And if you have scheduled DMA operations > piecemeal then you > >can > >>> > insert > >>> > > > > your commands in correct order. > >>> > > > > > >>> > > > > Julo > >>> > > > > > >>> > > > > "Rod Harrison" > >>> > > > > 05-11-01 20:48 > >>> > > > > Please respond to "Rod Harrison" > >>> > > > > > >>> > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > >>> > > > > cc: > >>> > > > > Subject: iSCSI: Out of order commands, was > >current > >>> > UNH > >>> > > > > Plugfest > >>> > > > > > >>> > > > > [ Subject changed ] > >>> > > > > > >>> > > > > Julian, > >>> > > > > > >>> > > > > The ordering difference is introduced > >>between the > >>> > > > > host > >>> > > > > side driver > >>> > > > > and the iSCSI HBA. The host side driver must present > >>SCSI commands > >>> > to > >>> > > > > the HBA in the order they are received from the OS to > >>prevent read > >>> > > > > after write dependency failures. The HBA might reorder > >>the commands > >>> > > > > depending on when DMA completes. The reordering can't be > >>done ahead > >>> > of > >>> > > > > time in the host driver since it doesn't know how > long each > >DMA > >>> > might > >>> > > > > take. As long as the HBA assigns CmdSN in the order it > >receives > >>> > > > > commands the desired host ordering is preserved. > >>> > > > > > >>> > > > > - Rod > >>> > > > > > >>> > > > > -----Original Message----- > >>> > > > > From: owner-ips@ece.cmu.edu > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > >>> > Of > >>> > > > > Julian Satran > >>> > > > > Sent: Monday, November 05, 2001 12:35 AM > >>> > > > > To: ips@ece.cmu.edu > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > >>> > > > > > >>> > > > > Rod, > >>> > > > > > >>> > > > > I all examples give the point I find hard to understand is > >why is > >>> > the > >>> > > > > ordering on the wire different from the presentation order > >to the > >>> > > > > initiator. You can get as many overlaps as you want by > >>presenting > >>> > the > >>> > > > > commands to the initiator in the desired order. > >>> > > > > What we are considering here is the case in which you > >>want to ship > >>> > in > >>> > > > > an > >>> > > > > order different than the one you present the commands. > >>> > > > > > >>> > > > > Julo > >>> > > > > > >>> > > > > "Rod Harrison" > >>> > > > > Sent by: owner-ips@ece.cmu.edu > >>> > > > > 04-11-01 04:42 > >>> > > > > Please respond to "Rod Harrison" > >>> > > > > > >>> > > > > To: "Barry Reinhold" , > >"Dave > >>> > > > > Sheehy" > >>> > > > > , "IETF IP SAN Reflector" > >>> > > > > > >>> > > > > cc: > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > >>> > > > > > >>> > > > > Barry, > >>> > > > > > >>> > > > > In general I agree but I don't think this > >is as > >>> > much > >>> > > > > of a > >>> > > > > corner case > >>> > > > > as it at first appears. Targets will have code very > >>similar to that > >>> > > > > needed to handle out of order commands to deal with > >>digest errors. > >>> > > > > Targets also need to queue commands whilst > waiting for both > >>> > solicited > >>> > > > > and unsolicited data to arrive. Queuing out of order > >>commands seems > >>> > > > > little extra work. > >>> > > > > > >>> > > > > From an initiators point of view > there are > >>> > > > > efficiency, > >>> > > > > and probably > >>> > > > > performance gains to be had from sending commands out of > >>order. Bob > >>> > > > > Russell gave the example of a read being sent whilst > >>write data DMA > >>> > is > >>> > > > > happening, and a similar situation can arise with DMA for > >writes > >>> > > > > overtaking that of earlier writes if the initiator has > >>multiple DMA > >>> > > > > engines. In this case the initiator might be forced to > >>let the wire > >>> > go > >>> > > > > idle if it can't send the data from completed DMAs as soon > >as > >>> > > > > possible. > >>> > > > > > >>> > > > > We already have a command queue at the > >target to > >>> > > > > enforce > >>> > > > > correct > >>> > > > > serialisation of commands, doing the same thing at the > >>initiator is > >>> > > > > redundant. > >>> > > > > > >>> > > > > Finally, I don't believe we should be > >writing a > >>> > > > > standard > >>> > > > > to work > >>> > > > > around poor coding and test coverage, especially at the > >cost of > >>> > > > > potential efficiency gains. > >>> > > > > > >>> > > > > I agree with Dave and Santosh that > >>commands being > >>> > > > > sent > >>> > > > > out of order > >>> > > > > on a single session should be allowed by the standard. > >>> > > > > > >>> > > > > - Rod > >>> > > > > > >>> > > > > -----Original Message----- > >>> > > > > From: owner-ips@ece.cmu.edu > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > >>> > Of > >>> > > > > Barry Reinhold > >>> > > > > Sent: Friday, November 02, 2001 5:24 PM > >>> > > > > To: Dave Sheehy; IETF IP SAN Reflector > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > >>> > > > > > >>> > > > > Using features such as out of order command delivery on > >>a connection > >>> > > > > tend to > >>> > > > > be the sort of things that lead to interoperability > >>problems. It is > >>> > > > > unexpected and probably going to hit poorly tested code > >>paths even > >>> > if > >>> > > > > the > >>> > > > > standard is written to allow it. > >>> > > > > > >>> > > > > >-----Original Message----- > >>> > > > > >From: owner-ips@ece.cmu.edu > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > >>> > > > > Of > >>> > > > > >Dave Sheehy > >>> > > > > >Sent: Friday, November 02, 2001 4:19 PM > >>> > > > > >To: IETF IP SAN Reflector > >>> > > > > >Subject: Re: iSCSI: current UNH Plugfest > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > >> 3. Can commands be sent out of order on the same > >connection? > >>> > > > > >> > >>> > > > > >> The behavior of targets is clearly specified in > >Section > >>> > 2.2.2.3 > >>> > > > > on > >>> > > > > >> page 25 of draft 8, which says: > >>> > > > > >> "Except for the commands marked for immediate > >>delivery the > >>> > > > > iSCSI > >>> > > > > >> target layer MUST eliver the commands for > >>execution in the > >>> > > > > order > >>> > > > > >> specified by CmdSN." > >>> > > > > >> > >>> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >>> > > > > >> "- CmdSN - the current command Sequence Number > >>advanced by 1 > >>> > > > > on > >>> > > > > >> each command shipped except for commands > marked for > >>> > immediate > >>> > > > > >> delivery." > >>> > > > > >> but the meaning of the term "shipped" is vague, > >>and does not > >>> > > > > >> necessarily > >>> > > > > >> require that the PDUs arrive on the other end of a > >TCP > >>> > > > > connection > >>> > > > > >> in the same order that the CmdSN values were > >>assigned to these > >>> > > > > PDUs. > >>> > > > > >> > >>> > > > > >> Some initiators have been designed to send commands > >out of > >>> > CmdSN > >>> > > > > >> order on one connection. Consider the situation > >>where there > >>> > is > >>> > > > > only > >>> > > > > >> one connection and a high-level dispatcher creates > >>a PDU for a > >>> > > > > SCSI > >>> > > > > >> command that involves writing immediate data to the > >target. > >>> > > > > This PDU > >>> > > > > >> is enqueued to a lower-level layer which has to > >>setup, start, > >>> > > > > and > >>> > > > > >> wait-for a DMA operation to move the immediate data > >into an > >>> > > > > onboard > >>> > > > > >> buffer before the PDU can be put onto the wire. > >>While this is > >>> > > > > >> happening, the dispatcher creates another > >>unrelated PDU for a > >>> > > > > SCSI > >>> > > > > >> read command (for example), and when this PDU is > >>passed to the > >>> > > > > >> lower-level layer it can be sent immediately, ahead > >of the > >>> > > > > previous > >>> > > > > >> write PDU and therefore out of order on this > >connection. > >>> > > > > >> > >>> > > > > >> The standard clearly allows this to happen > if the two > >PDUs > >>> > were > >>> > > > > sent > >>> > > > > >> on different connections, and seems to imply that > >this can > >>> > also > >>> > > > > happen > >>> > > > > >> when the two PDUs are sent on the same connection. > >>> > > > > >> > >>> > > > > >> The suggestion is to put in the standard an > >>explicit statement > >>> > > > > that > >>> > > > > >> this is allowed or not allowed, as appropriate. > >>> > > > > >> > >>> > > > > >> If this is allowed, such a statement would avoid > >>the erroneous > >>> > > > > >> assumption being made by some target implementers > >>that within > >>> > a > >>> > > > > single > >>> > > > > >> connection, commands will arrive in order. > >>> > > > > >> > >>> > > > > >> If this is not allowed, such a statement would avoid > >the > >>> > > > > erroneous > >>> > > > > >> assumption being made by some initiator implementers > >that > >>> > within > >>> > > > > a > >>> > > > > >> single connection, commands can be put on the wire > >out of > >>> > order. > >>> > > > > >> > >>> > > > > >> +++ > >>> > > > > >> > >>> > > > > >> will add an explicit statement saying that this > >behaviour is > >>> > > > > forbidden. > >>> > > > > >> 2.2.2.1 will contain: > >>> > > > > >> > >>> > > > > >> On any given connection, the iSCSI initiator MUST send > >the > >>> > > > > >commands in the > >>> > > > > >> order specified by CmdSN. > >>> > > > > >> > >>> > > > > >> +++ > >>> > > > > > > >>> > > > > >Why do you feel this behavior should be forbidden? > >>Targets already > >>> > > > > have to > >>> > > > > >order commands across the session. I don't see why it's > >>a problem > >>> > to > >>> > > > > extend > >>> > > > > >that to the connection as well. I, for one, believe we > >>should take > >>> > > > > >a liberal > >>> > > > > >stance on this. > >>> > > > > > > >>> > > > > >Dave Sheehy > >>> > > > > > > >>> > > > >>> > > -- > >>> > > ################################## > >>> > > Santosh Rao > >>> > > Software Design Engineer, > >>> > > HP-UX iSCSI Driver Team, > >>> > > Hewlett Packard, Cupertino. > >>> > > email : santoshr@cup.hp.com > >>> > > Phone : 408-447-3751 > >>> > > ################################## > >>> > > >>> > > >>> > > >>> > > >>> > >> > > > From owner-ips@ece.cmu.edu Thu Nov 8 13:02:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA02797 for ; Thu, 8 Nov 2001 13:02:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8HYqo06373 for ips-outgoing; Thu, 8 Nov 2001 12:34:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8HYol06365 for ; Thu, 8 Nov 2001 12:34:50 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA69646 for ; Thu, 8 Nov 2001 18:34:43 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA8HYhw52034 for ; Thu, 8 Nov 2001 18:34:43 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 8 Nov 2001 19:34:41 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 08/11/2001 19:34:42, Serialize complete at 08/11/2001 19:34:42 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Robert, We are pursuing a very impractical track. Show me please one SINGLE advantage for OOO shipping. Julo "Robert D. Russell" Sent by: owner-ips@ece.cmu.edu 08-11-01 17:50 Please respond to "Robert D. Russell" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Julian: > However allowing initiators to ship them out of order creates a > potential deadlock that does not appear otherwise. > > If a target is missing a command in a queue (and there are no errors) then > this command is bound to be first on some connection under the current set > of rules. > > If we allow OOO shipping then the missing command can be somewhere > "inside" the window on some connection and if the target has just filled > his queue and has room in the staging buffer only for the command it is > waiting for and that command happens to be the first to pass to SCSI you > have a deadlock. I must be understanding something. Your example is certainly correct if the target has no control over the number of commands sent to it by the initiator. But the target does control this number through the MaxCmdSN field. For the scenario you are describing to occur, wouldn't it be necessary for the target to advertize a MaxCmdSN value bigger than it actually has resources to handle? It seems to me that if a target can only handle x new commands, then its queueing capacity is x, and in the SCSI Response PDU it should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the formula in section 2.2.2.1. This in turn controls the number of commands the initiator can send, and thus prevents the incoming commands from overfilling the target's queue. So isn't the deadlock caused by a broken target? Isn't the target advertizing a queueing capacity greater than it actually has and isn't that the cause of the deadlock? An alternative explanation of the deadlock might be that the target is advertizing the correct MaxCmdSN, but the initiator is sending commands beyond what it is allowed to send. However, in this case the target should silently ignore any non-immediate command outside the allowed range. For the deadlock to occur, wouldn't the target have to be queuing commands outside the allowed range instead of ignoring them? So in this case too, the target is broken and that is what causes the deadlock. Where am I going wrong in this reasoning? Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Thu Nov 8 13:09:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA03045 for ; Thu, 8 Nov 2001 13:09:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8GvoN03654 for ips-outgoing; Thu, 8 Nov 2001 11:57:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8Gvml03649 for ; Thu, 8 Nov 2001 11:57:48 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id IAA27008; Thu, 8 Nov 2001 08:57:36 -0800 (PST) From: Keith McCloghrie Message-Id: <200111081657.IAA27008@cisco.com> Subject: Re: FC Management MIB - issues To: bill@sanera.net (Bill Strahm) Date: Thu, 8 Nov 2001 08:57:36 -0800 (PST) Cc: kzm@cisco.com (Keith McCloghrie), ips@ece.cmu.edu In-Reply-To: from "Bill Strahm" at Nov 07, 2001 03:18:18 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Bill, You raise a good point. One of the reasons to move the FCMGMT MIB to this WG was to ensure coordination with this WG's MIBs. You're right that the FCIP MIB is dependent on the FCMGMT MIB, since every table in the FCIP MIB is INDEXed by fcConnUnitId, which it IMPORTS from the FCMGMT MIB. As and when a new structure is adopted for the FCMGMT MIB, the FCIP MIB will be affected, and will need to be updated accordingly. However, I too don't think there's time to achieve the required coordination before the Internet-Draft deadline. Keith. > An interesting issue Keith... > Currently there are a couple of tables in the FCIP MIB that AUGMENT tables > in this MIB, and the FCIP MIB also imports a few of the indexes from the > FCMGMT MIB. > > Will you be able to co-ordinate changes with the FCIP MIB editors (I am > assuming they will not be able to adjust their MIB by next week) however can > updated versions of these MIBs come out in the future ? > > Bill > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Keith McCloghrie > Sent: Wednesday, November 07, 2001 1:03 PM > To: ips@ece.cmu.edu > Cc: Keith McCloghrie > Subject: FC Management MIB - issues > > > Hi, > > As indicated by Elizabeth's message (below), I've agreed to be the > interim editor for the FC Management MIB. So, first, I'd like to > list a set of issues that have been raised concerning the most > recent draft: draft-ietf-ipfc-fcmgmt-int-mib-07.txt. They are: > > 1. The use of SMIv2 is mandatory. > > 2. This MIB seems to have been defined with the notion that it will be > the only MIB that a Fibre Channel product will require. The notion of > an agent implementing just a single MIB was abandoned by the IETF in > 1992 as being non-scaleable. Rather, this is another MIB in the > continuing series of MIBs defined by the IETF, and thus, it needs to be > consistent with its predecessors. In other words, there are existing > MIBs which all SNMP agents must support, even if the support of Fibre > Channel interfaces is the only functionality that they have. Thus, as > the Fibre Channel Integration MIB, it is essential that this MIB > contain only objects for information which is specific to Fibre > Channel. All objects which apply in non-Fibre Channel environments > need to be removed. (This applies even it were to require the > definition of other new MIBs to contain information which is not > already defined in other existing MIBs, but needed by Fibre Channel > products.) Note that this issue applies to a large fraction of the > objects defined in this MIB. > > 3. The text needs to include an explanation of the relationship between > this MIB and RFC 2837. Is it a replacement or is it complementary ? > If complementary, which agents implement which MIB; do some agents > implement both ? > > 4. Every SNMP agent implements the ifTable. The ifTable counters are > undoubtedly the MIB objects most well-used by administrators in SNMP > management. SNMP agents need to implement a row in the ifTable for > each of their network interfaces, including their Fibre Channel > interfaces. The IF-MIB requires a media-specific MIB to specify how > that type of interface uses the ifTable (see section 4 in RFC 2863). > RFC 2837 doesn't do that, and nor (currently) does this MIB. That > needs to be fixed. This will likely result in many tables in this > MIB being indexed by ifIndex. > > 5. There are a number of objects related to the "Simple Name Service", > and the definitions refer to Fibre Channel's GS-3 spec. However, GS-3 > defines two Name Services: a Zoned Name Service and a Unzoned Name > Service, but GS-3 does NOT use the term "Simple" for either of them !! > This ambiguity needs to be resolved. > > 6. It is essential that the Counter32 or Counter64 (not OCTET STRINGs) > syntax be used for counters. > > Thanks, > Keith. > -------------- > Forwarded message: > > From: "Elizabeth Rodriguez" > > To: , > > "IPS Mailing List (E-mail)" > > Subject: FC Management MIB -- Transfer from IPFC To IPS WG > > Cc: , , , > > , , , > > , > > "Elizabeth Rodriguez" > , > > > > > > All, > > > > The IPFC working group has only one item left on its charter. > > It is that of the FC Management MIB. It has been identified > > that this MIB does not follow the IETF guidelines for MIBs > > in its current format, in its lack of focus, and in its overlap > > with existing MIBs. Rework of this MIB will take some time. > > The content of this MIB will fit in well with the IPS WG, > > especially because the subject matter experts participate in > > this WG. > > > > For these reasons, the working group chairs, with the INT and TSV area > > directors, have determined that this effort should be moved from the > > IPFC working group to the IPS working group. Upon transferring of this > > work, the IPFC working group will have completed the items in its > > charter and the IPFC WG will be closed. > > > > The IPS working group has a technical advisor for MIB work -- Keith > > McCloghrie. Since it has been determined that the current MIB has > > issues with format, Keith McCloghrie has agreed to become the interim > > editor of this MIB. As part of the re-architecture, the MIB will be > > evaluated with respect to fit with other IPS WG MIBs, and may take the > > form of a single new MIB or multiple MIBs, as appropriate. The IPS > > working group will also be seeking Fibre Channel expertise to help > > formulate the new MIB, including an editor with FC and MIB experience. > > > > The IPFC working group would also like to thank Steven Blumenau for all > > his work on the original MIB. > > > > Elizabeth Rodriguez & David Black, IPS Working Group Chairs > > Murali Rajagopal, IPFC Working Group Chair > > > > From owner-ips@ece.cmu.edu Thu Nov 8 13:50:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04030 for ; Thu, 8 Nov 2001 13:50:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8Ho3907456 for ips-outgoing; Thu, 8 Nov 2001 12:50:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8Ho0l07452 for ; Thu, 8 Nov 2001 12:50:00 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA07582 for ; Thu, 8 Nov 2001 18:49:50 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA8Hnnw19386 for ; Thu, 8 Nov 2001 18:49:49 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 8 Nov 2001 19:49:47 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 08/11/2001 19:49:48, Serialize complete at 08/11/2001 19:49:48 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ron, Targets will most likely advertise a total (command and data) window larger than they can accommodate on any long haul link. With the current ordering rules nothing bad will happen. And both initiator and target will gain. I did not see a SINGLE reason (performance, memory etc.) to remove this ordering requirement. And the ordering requirement is not global it is per connection. It says that if you send on c1,c2,c3,c4.c5 with c1 & c3 going on connection 1 you should not send c3 before c1 on connection 1 but you can ship c3 before c2 if c2 goes on connection 2. Many other schemes like some of the recovery, task abort connection cleanup are all based on ordering being preserved within the connection. The whole discussion thread seems also related to some perceived gain from relaxing this restriction - but no one was able to show a single scenario showing a real gain. This type of source and sink ordering is a common requirement in most distributed systems. Julo Ron Grinfeld 08-11-01 11:28 Please respond to Ron Grinfeld To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: iSCSI: Out of order commands Julian, Can you clarify the deadlock scenario a little bit more (taking into account that a target will not advertise a command window larger than the number of commands it can support) ? Rong -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Thursday, November 08, 2001 8:02 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Robert, I am not saying that handling OOO commands will create more complexity (targets already do that over several connections and it does not matter for them). However allowing initiators to ship them out of order creates a potential deadlock that does not appear otherwise. If a target is missing a command in a queue (and there are no errors) the this command is bound to be first on some connection under the current set of rules. If we allow OOO shipping then the missing command can be somewhere "inside" the window on some connection and if the target has just filled his queue and has room in the staging buffer only for the command it is waiting for and that command happens to be the first to pass to SCSI you have a deadlock. Julo "Robert D. Russell" 07-11-01 23:13 Please respond to "Robert D. Russell" To: Somesh Gupta cc: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Somesh, Julian: You state that dealing with OOO commands on the target will add substantial complexity on the target. Do you have any basis for that claim? My impression from the plugfest is that most targets are already dealing with it. Perhaps we need to hear from someone who is actually building a target for which this would be a real problem. If anything, what we are hearing from people who really are building initiators is that dealing with the requirement to send commands in order will introduce substantial complexity on the initiator. So why should we be saving complexity on (hypothetically) simple targets yet requiring complexity on real initiators? As far as the deadlock issue is concerned, if the only way that deadlock can occur with OOO commands on the same connection is during the use of immediate data (which is I think what Julian was saying), then shouldn't we change the standard to just say that -- if the initiator sends commands out of order on a single connection, then immediate data MUST NOT be used on that connection in order to avoid deadlock. This gives everybody what they want, since initiators who find it beneficial to deliver commands OOO will just negotiate immediate data off. Those who really want to use immediate data will have to ensure that commands are sent in order. The tradeoff then becomes an implementation issue, not a standards issue, which is where it belongs. Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Wed, 7 Nov 2001, Somesh Gupta wrote: > I think we should either have it as a MUST or not require > it (at both ends to get the real benefit). SHOULD is one > of those things that leads to implementation > burden and confusion, without perhaps the feature being > used. There are implementation as well as protocol > considerations mixed in here. > > If we are to remove the restriction, we should (SHOULD) > get the maximum benefit from it, rather than to > accomodate an implementation choice. Out of sequence > commands, combined with the possibility of digest errors, > will add substantial complexity on the target side, > without corrosponding benefit in performance. If we change > this to SHOULD, we should also relax the requirement > to present commands on the target side to a SHOULD. > > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Wednesday, November 07, 2001 10:00 AM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI: Out of order commands > > > > > > Mallikarjun, > > > > I did not see a SINGLE performance improvement that results from OOO > > shipping. > > I would be bad engineering to give away the "no-deadlock" mechanism we > > have now for nothing. > > I have also the impression that the point about deadlock that I keep > > repeating is ignored or not understood. > > As we stand today commands can be shipped with Immediate data or without > > and an implementer determined > > to squeeze maximum bandwidth and overlap command start with delivery will > > choose not to work with immediate data > > (as you have pointed out) while a low performance software implementation > > will use immediate data to minimize CPU cycles consumed. However both > > will be guaranteed to work without deadlock as source and sink use the > > same ordering. > > Recovery is still a low probability event and should be handled with a > > different set of considerations in mind. > > As for the strictness of the recommendation - yes we could settle on > > SHOULD. > > > > Julo > > > > > > > > > > "Mallikarjun C." > > Sent by: owner-ips@ece.cmu.edu > > 07-11-01 19:41 > > Please respond to cbm > > > > > > To: Santosh Rao , ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: Out of order commands > > > > > > > > Santosh, > > > > I have only one comment on your responses. > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > I made two convenient assumptions if you noticed, :-), one of which > > is that target forces session recovery on *any* error that it sees > > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > > error. With that assumption, a target can afford not to implement > > a scoreboard. > > > > As I said in a private note, I guess what primarily bothers me about > > OOO commands on a connection is that it requires the receiver to > > undo this "optimization" on its end - most notably on a single > > connection. TCP experts may comment on how/if they dealt with a > > similar issue. > > > > OTOH, you had some valid comments on exceptions to ordering during > > connection recovery. Perhaps we can move on by making Julian's > > proposed stipulation a SHOULD.... > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > Santosh Rao wrote: > > > > > > Mallikarjun, > > > > > > Some comments below. > > > > > > Regards, > > > Santosh > > > > > > "Mallikarjun C." wrote: > > > > > > > > Rod and Julian, > > > > > > > > This has been an interesting thread of discussion. Some > > > > comments - > > > > > > > > 1.My first reaction was - allowing out-of-order command > > > > transmission on the same connection deprives targets of > > > > an implementation choice. Targets which support only > > > > single-connection sessions and only support session > > > > recovery (reasonable assumptions in my mind) can no > > > > longer afford *not to* implement a command scoreboard. > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > > > Thus, there is no getting away from implementing a sequencer at the > > > target. Given this, I think it is unreasonable to restrict initiator > > > implementation flexibility by imposing a strict ordering requirement > > > within the connection. > > > > > > > 2.Any end-node efficiency that is sought to be achieved > > > > by transmitting CmdSNs out-of-order from the initiator > > > > would be lost on the other end-node, since the target > > > > now must wait for re-ordering the commands. > > > > > > It has to handle this situation anyway to deal with holes caused by > > > digest errors. This scenario occurs even with initiators that issue > > > commands in order. > > > > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > > link badwidth (albeit at the expense of end-node efficiency), > > > > compared to idling the link waiting for outbound DMA. > > > > We have to determine if this is a reasonable trade-off. > > > > > > > > 4.I can see Rod's point that prefetching all immediate > > > > data can be a burden on the NIC resources. But, two > > > > questions - > > > > - could the NIC not use unsolicited separate data > > > > PDUs in these cases? [ I realize that InitialR2T > > > > has to be "no" to let it happen... ] > > > > - could the NIC have a memory architecture that > > > > allows data prefetching for the next command (so > > > > this is a non-issue from the protocol perspective)? > > > > This scheme incurs one DMA delay for every new > > > > burst of commands. > > > > > > > > 5.Another (perhaps radical at this point) option is to do > > > > away with immediate unsolicited data, to stick only with > > > > separate unsolicited data. I would personally be okay > > > > with the choice, particularly if this feature (that > > > > helps software implementations) starts making hardware > > > > design complicated/expensive. > > > > > > > > So, to summarize - > > > > > > > > option immediate allow > > > > data in spec? out-of-order? > > > > > > > > (A) (5) above no no > > > > (B) No real reason to do this. no yes > > > > (C) (4) above yes no > > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > > > >From the arguments I heard so far, I am leaning towards > > > > option A, and option C in that order. > > > > > > > > Comments? > > > > -- > > > > Mallikarjun > > > > > > > > Mallikarjun Chadalapaka > > > > Networked Storage Architecture > > > > Network Storage Solutions Organization > > > > MS 5668 Hewlett-Packard, Roseville. > > > > cbm@rose.hp.com > > > > > > > > Rod Harrison wrote: > > > > > > > > > > Julian, > > > > > > > > > > I don't understand what you are proposing here, what do you > > mean by > > > > > "multiplexed" DMA? > > > > > > > > > > The problem is that the DMAs take some time, the more there > > are > > > > > queued the longer the last DMAs queued take to complete. Some > > commands > > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > > soon a possible, which for a read after a write can be before the > > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > > to be sent on the HBA is expensive and redundant since the target > > > > > already knows how to queue commands before committing them to its > > SCSI > > > > > layer. > > > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > > change the > > > > > order of commands from the OS, but the DMAs those commands need are > > > > > unlikely to complete in the same order, and as I mentioned some > > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > > order it has to maintain an ordered queue of commands waiting to be > > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > > immediate data almost impossible to support. > > > > > > > > > > I don't see the problem with allowing out of order commands > > given > > > > > that the target already has to deal with very similar problems. I > > > > > think we are getting in to the area of implementation choices here, > > > > > which is inappropriate for a specification. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Julian Satran > > > > > Sent: Monday, November 05, 2001 10:06 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > > > > > Rod, > > > > > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > > > commands. > > > > > If you have scheduled a long outbound DMA you are doomed regardless > > of > > > > > the > > > > > command ordering. > > > > > And if you have scheduled DMA operations piecemeal then you can > > insert > > > > > your commands in correct order. > > > > > > > > > > Julo > > > > > > > > > > "Rod Harrison" > > > > > 05-11-01 20:48 > > > > > Please respond to "Rod Harrison" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > > cc: > > > > > Subject: iSCSI: Out of order commands, was current > > UNH > > > > > Plugfest > > > > > > > > > > [ Subject changed ] > > > > > > > > > > Julian, > > > > > > > > > > The ordering difference is introduced between the > > > > > host > > > > > side driver > > > > > and the iSCSI HBA. The host side driver must present SCSI commands > > to > > > > > the HBA in the order they are received from the OS to prevent read > > > > > after write dependency failures. The HBA might reorder the commands > > > > > depending on when DMA completes. The reordering can't be done ahead > > of > > > > > time in the host driver since it doesn't know how long each DMA > > might > > > > > take. As long as the HBA assigns CmdSN in the order it receives > > > > > commands the desired host ordering is preserved. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Julian Satran > > > > > Sent: Monday, November 05, 2001 12:35 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Rod, > > > > > > > > > > I all examples give the point I find hard to understand is why is > > the > > > > > ordering on the wire different from the presentation order to the > > > > > initiator. You can get as many overlaps as you want by presenting > > the > > > > > commands to the initiator in the desired order. > > > > > What we are considering here is the case in which you want to ship > > in > > > > > an > > > > > order different than the one you present the commands. > > > > > > > > > > Julo > > > > > > > > > > "Rod Harrison" > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > 04-11-01 04:42 > > > > > Please respond to "Rod Harrison" > > > > > > > > > > To: "Barry Reinhold" , "Dave > > > > > Sheehy" > > > > > , "IETF IP SAN Reflector" > > > > > > > > > > cc: > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Barry, > > > > > > > > > > In general I agree but I don't think this is as > > much > > > > > of a > > > > > corner case > > > > > as it at first appears. Targets will have code very similar to that > > > > > needed to handle out of order commands to deal with digest errors. > > > > > Targets also need to queue commands whilst waiting for both > > solicited > > > > > and unsolicited data to arrive. Queuing out of order commands seems > > > > > little extra work. > > > > > > > > > > From an initiators point of view there are > > > > > efficiency, > > > > > and probably > > > > > performance gains to be had from sending commands out of order. Bob > > > > > Russell gave the example of a read being sent whilst write data DMA > > is > > > > > happening, and a similar situation can arise with DMA for writes > > > > > overtaking that of earlier writes if the initiator has multiple DMA > > > > > engines. In this case the initiator might be forced to let the wire > > go > > > > > idle if it can't send the data from completed DMAs as soon as > > > > > possible. > > > > > > > > > > We already have a command queue at the target to > > > > > enforce > > > > > correct > > > > > serialisation of commands, doing the same thing at the initiator is > > > > > redundant. > > > > > > > > > > Finally, I don't believe we should be writing a > > > > > standard > > > > > to work > > > > > around poor coding and test coverage, especially at the cost of > > > > > potential efficiency gains. > > > > > > > > > > I agree with Dave and Santosh that commands being > > > > > sent > > > > > out of order > > > > > on a single session should be allowed by the standard. > > > > > > > > > > - Rod > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Barry Reinhold > > > > > Sent: Friday, November 02, 2001 5:24 PM > > > > > To: Dave Sheehy; IETF IP SAN Reflector > > > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > > > > > Using features such as out of order command delivery on a connection > > > > > tend to > > > > > be the sort of things that lead to interoperability problems. It is > > > > > unexpected and probably going to hit poorly tested code paths even > > if > > > > > the > > > > > standard is written to allow it. > > > > > > > > > > >-----Original Message----- > > > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > > Of > > > > > >Dave Sheehy > > > > > >Sent: Friday, November 02, 2001 4:19 PM > > > > > >To: IETF IP SAN Reflector > > > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > > > >> > > > > > >> The behavior of targets is clearly specified in Section > > 2.2.2.3 > > > > > on > > > > > >> page 25 of draft 8, which says: > > > > > >> "Except for the commands marked for immediate delivery the > > > > > iSCSI > > > > > >> target layer MUST eliver the commands for execution in the > > > > > order > > > > > >> specified by CmdSN." > > > > > >> > > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > > > on > > > > > >> each command shipped except for commands marked for > > immediate > > > > > >> delivery." > > > > > >> but the meaning of the term "shipped" is vague, and does not > > > > > >> necessarily > > > > > >> require that the PDUs arrive on the other end of a TCP > > > > > connection > > > > > >> in the same order that the CmdSN values were assigned to these > > > > > PDUs. > > > > > >> > > > > > >> Some initiators have been designed to send commands out of > > CmdSN > > > > > >> order on one connection. Consider the situation where there > > is > > > > > only > > > > > >> one connection and a high-level dispatcher creates a PDU for a > > > > > SCSI > > > > > >> command that involves writing immediate data to the target. > > > > > This PDU > > > > > >> is enqueued to a lower-level layer which has to setup, start, > > > > > and > > > > > >> wait-for a DMA operation to move the immediate data into an > > > > > onboard > > > > > >> buffer before the PDU can be put onto the wire. While this is > > > > > >> happening, the dispatcher creates another unrelated PDU for a > > > > > SCSI > > > > > >> read command (for example), and when this PDU is passed to the > > > > > >> lower-level layer it can be sent immediately, ahead of the > > > > > previous > > > > > >> write PDU and therefore out of order on this connection. > > > > > >> > > > > > >> The standard clearly allows this to happen if the two PDUs > > were > > > > > sent > > > > > >> on different connections, and seems to imply that this can > > also > > > > > happen > > > > > >> when the two PDUs are sent on the same connection. > > > > > >> > > > > > >> The suggestion is to put in the standard an explicit statement > > > > > that > > > > > >> this is allowed or not allowed, as appropriate. > > > > > >> > > > > > >> If this is allowed, such a statement would avoid the erroneous > > > > > >> assumption being made by some target implementers that within > > a > > > > > single > > > > > >> connection, commands will arrive in order. > > > > > >> > > > > > >> If this is not allowed, such a statement would avoid the > > > > > erroneous > > > > > >> assumption being made by some initiator implementers that > > within > > > > > a > > > > > >> single connection, commands can be put on the wire out of > > order. > > > > > >> > > > > > >> +++ > > > > > >> > > > > > >> will add an explicit statement saying that this behaviour is > > > > > forbidden. > > > > > >> 2.2.2.1 will contain: > > > > > >> > > > > > >> On any given connection, the iSCSI initiator MUST send the > > > > > >commands in the > > > > > >> order specified by CmdSN. > > > > > >> > > > > > >> +++ > > > > > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > > > have to > > > > > >order commands across the session. I don't see why it's a problem > > to > > > > > extend > > > > > >that to the connection as well. I, for one, believe we should take > > > > > >a liberal > > > > > >stance on this. > > > > > > > > > > > >Dave Sheehy > > > > > > > > > > > > -- > > > ################################## > > > Santosh Rao > > > Software Design Engineer, > > > HP-UX iSCSI Driver Team, > > > Hewlett Packard, Cupertino. > > > email : santoshr@cup.hp.com > > > Phone : 408-447-3751 > > > ################################## > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 13:57:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04141 for ; Thu, 8 Nov 2001 13:57:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8I5Qf08655 for ips-outgoing; Thu, 8 Nov 2001 13:05:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail1.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8I5Nl08644 for ; Thu, 8 Nov 2001 13:05:23 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by hoemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fA8I5Ge13508 for ; Thu, 8 Nov 2001 13:05:16 -0500 (EST) Content-Class: urn:content-classes:message Subject: RE: FCencap: List ALL SOF/EOF codes MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C1687F.E91178DB" Date: Thu, 8 Nov 2001 13:05:12 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Message-ID: <9410A0F67DFE7F4D998D4538236498360408F1@nc8220exchange.ral.lucent.com> Thread-Topic: FCencap: List ALL SOF/EOF codes Thread-Index: AcFnoTsgh4Vd+Je1TY2tZE+qkiTg1wA2XkEw From: "Elizabeth Rodriguez" To: "Franco Travostino" , , "IPS Reflector" Cc: "Murali Rajagopal" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------_=_NextPart_001_01C1687F.E91178DB Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable (Participant mode) =20 I disagree with this motion.=20 We had this discussion back in January, and basically came to the conclusion that Class 1 and Class 4 should not be included, for the reasons that class 1 really cannot be supported across the IP network and class 4 is not really not defined yets, so the codes are not guaranteed to remain constant. =20 Even if we do decide to accept Ralph's arguments to include class 1 and class 4 SOF/EOF codes, we cannot take ALL the codes from FC-BB and incorporate them into the FC Encapsulation draft. Recall, we started out by including all the SOF/EOF codes from FC-BB-2. We reevaluated in January 2001, when we analyzed the codes themselves. Several that we excluded I think were valid (e.g. for class 1 and class 4), but others were completely bogus and undefined anywhere other than in FC-BB. We cannot just blindly accept those codes. =20 If this motion is considered, we need to reopen that evaluation made in the January interim meeting and make a determination as to what codes need to be included in FC Common Encapsulation, and make sure not to include invalid codes. =20 Thanks, =20 Elizabeth -----Original Message----- From: Franco Travostino [mailto:travos@nortelnetworks.com] Sent: Wednesday, November 07, 2001 9:32 AM To: ENDL_TX@computer.org; IPS Reflector Cc: Murali Rajagopal; Elizabeth Rodriguez Subject: Re: FCencap: List ALL SOF/EOF codes I agree with this motion.=20 You wrote under another heading: "Vague may be vague but it also is not unnecessarily constraining." What a troubling statement was this. I'm glad we now appear to be moving towards a constructive end after all. thanks -franco At 08:15 AM 11/7/2001, Ralph Weber wrote: Upon further reflection, I think the right thing to do is to list all the SOF/EOF codes defined in FC-BB in the FC Encapsulation draft. FIRST There is nothing in the FC Encapsulation draft other than to omission of Class 1 SOF/EOF codes that prevents encapsulating FC Class 1 frames for TCP transport. Sure, a TCP ULP that is smarter than anything anybody has thought about will be required to do it. BUT there is (or should be) nothing the the FC Encapsulation draft that prevents such a protocol from being invented. AND the FC Encapsulation draft specifically says that you need the wisdom of some other protocol document in order to get any use out of the FC Encapsulation draft. Why force the mad man that devises a way to transport Class 1 over TCP/IP to revise the FC Encapsulation SOF/EOF tables? SECOND It is conceivable that a future version of iFCP (or maybe even FCIP) might want to support Class 4. Again, there is nothing in the FC Encapsulation draft that prevents this, except the omission of the SOF/EOF codes. FINALLY I believe that the elimination of all SOF/EOF codes other than Class 2, Class 3, AND CLASS F is a hold over from the early FCIP work, before the FC Encapsulation was split into a separate draft. I believe that decision was right for FCIP but wrong for an FC Encapsulation intended to be used by ALL FC protocols running over TCP/IP. Thanks for your consideration. Ralph... ------_=_NextPart_001_01C1687F.E91178DB Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
(Participant mode)
 
I=20 disagree with this motion. 
We=20 had this discussion back in January, and basically came to the = conclusion=20 that Class 1 and Class 4 should not be included, for the reasons = that class=20 1 really cannot be supported across the IP network and class 4 is not = really not=20 defined yets, so the codes are not guaranteed to remain=20 constant.  
Even=20 if we do decide to accept Ralph's arguments to include class 1 and class = 4=20 SOF/EOF codes, we cannot take ALL the codes from FC-BB and incorporate = them into=20 the FC Encapsulation draft.
Recall, we started out by including all=20 the SOF/EOF codes from FC-BB-2.  We reevaluated in January = 2001, when=20 we analyzed the codes themselves.
Several that we excluded I think were valid = (e.g. for=20 class 1 and class 4), but others were completely bogus and undefined = anywhere=20 other than in FC-BB.
We=20 cannot just blindly accept those codes.
 
If=20 this motion is considered, we need to reopen that evaluation made in the = January=20 interim meeting and make a determination as to what codes need to be = included in=20 FC Common Encapsulation, and make sure not to include invalid=20 codes.
 
Thanks,
 
Elizabeth
-----Original Message-----
From: Franco Travostino=20 [mailto:travos@nortelnetworks.com]
Sent: Wednesday, November = 07,=20 2001 9:32 AM
To: ENDL_TX@computer.org; IPS = Reflector
Cc:=20 Murali Rajagopal; Elizabeth Rodriguez
Subject: Re: FCencap: = List ALL=20 SOF/EOF codes


I agree with this = motion.=20

You wrote under another heading: "Vague may be vague but it = also is=20 not unnecessarily constraining." What a troubling statement was this. = I'm glad=20 we now appear to be moving towards a constructive end after=20 all.

thanks
-franco

At 08:15 AM 11/7/2001, Ralph = Weber=20 wrote:
Upon further reflection, I = think the=20 right thing to do
is to list all the SOF/EOF codes defined in = FC-BB=20 in
the FC Encapsulation draft.

FIRST

There is = nothing in=20 the FC Encapsulation draft other
than to omission of Class 1 = SOF/EOF=20 codes that prevents
encapsulating FC Class 1 frames for TCP=20 transport.
Sure, a TCP ULP that is smarter than anything = anybody
has=20 thought about will be required to do it.  BUT
there is (or = should=20 be) nothing the the FC Encapsulation
draft that prevents such a = protocol=20 from being invented.
AND the FC Encapsulation draft specifically = says=20 that
you need the wisdom of some other protocol document = in
order to=20 get any use out of the FC Encapsulation draft.
Why force the mad = man that=20 devises a way to transport
Class 1 over TCP/IP to revise the FC=20 Encapsulation
SOF/EOF tables?

SECOND

It is = conceivable that=20 a future version of iFCP
(or maybe even FCIP) might want to = support Class=20 4.
Again, there is nothing in the FC Encapsulation
draft that = prevents=20 this, except the omission
of the SOF/EOF = codes.

FINALLY

I=20 believe that the elimination of all SOF/EOF
codes other than = Class 2,=20 Class 3, AND CLASS F
is a hold over from the early FCIP work,=20 before
the FC Encapsulation was split into a separate
draft. I = believe=20 that decision was right for
FCIP but wrong for an FC = Encapsulation=20 intended
to be used by ALL FC protocols running=20 over
TCP/IP.

Thanks for your=20 = consideration.

Ralph...
------_=_NextPart_001_01C1687F.E91178DB-- From owner-ips@ece.cmu.edu Thu Nov 8 13:57:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04153 for ; Thu, 8 Nov 2001 13:57:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8IWVt10892 for ips-outgoing; Thu, 8 Nov 2001 13:32:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8IWTl10887 for ; Thu, 8 Nov 2001 13:32:29 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id NAA11184; Thu, 8 Nov 2001 13:32:05 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "Robert D. Russell" , "Julian Satran" Cc: Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 10:29:18 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Bob, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Robert D. Russell > Sent: Thursday, November 08, 2001 7:51 AM > To: Julian Satran > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Julian: > > > However allowing initiators to ship them out of order creates a > > potential deadlock that does not appear otherwise. > > > > If a target is missing a command in a queue (and there are no > errors) then > > this command is bound to be first on some connection under the > current set > > of rules. > > > > If we allow OOO shipping then the missing command can be somewhere > > "inside" the window on some connection and if the target has > just filled > > his queue and has room in the staging buffer only for the command it is > > waiting for and that command happens to be the first to pass to > SCSI you > > have a deadlock. > > > I must be understanding something. > > Your example is certainly correct if the target has no control > over the number of commands sent to it by the initiator. > But the target does control this number through the MaxCmdSN field. > For the scenario you are describing to occur, wouldn't it be > necessary for the target to advertize a MaxCmdSN value bigger than > it actually has resources to handle? > > It seems to me that if a target can only handle x new commands, > then its queueing capacity is x, and in the SCSI Response PDU it > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > formula in section 2.2.2.1. This in turn controls the number of > commands the initiator can send, and thus prevents the incoming > commands from overfilling the target's queue. This is actually a weakness of the multiple connections over multiple adapters model (it is not related to ordering). The target advertises a command window on a session wide basis. At the same time, it has to provide the resource to the adapter to be able to DMA those commands to. Since there is no restriction on which connection the commands may be received, either the target has to allocate the max resources needed to each adapter (thus committing n times the resources actually required), or has to "lie" (it would not be a complete lie) which could result in running out of resources. One way to fix that would be to have the credit on a per connection basis. > > So isn't the deadlock caused by a broken target? > Isn't the target advertizing a queueing capacity greater than it > actually has and isn't that the cause of the deadlock? > > An alternative explanation of the deadlock might be that the > target is advertizing the correct MaxCmdSN, but the initiator is > sending commands beyond what it is allowed to send. > > However, in this case the target should silently ignore any > non-immediate command outside the allowed range. > For the deadlock to occur, wouldn't the target have to be queuing > commands outside the allowed range instead of ignoring them? > > So in this case too, the target is broken and that is what causes > the deadlock. > > Where am I going wrong in this reasoning? > > Thanks, > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > From owner-ips@ece.cmu.edu Thu Nov 8 14:08:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA04332 for ; Thu, 8 Nov 2001 14:08:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8IOrG10202 for ips-outgoing; Thu, 8 Nov 2001 13:24:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8IOol10195 for ; Thu, 8 Nov 2001 13:24:50 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id NAA25874; Thu, 8 Nov 2001 13:23:50 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "FRYMAN,ROBERT \(A-Roseville,ex1\)" , "'Barry Reinhold'" , "Rod Harrison" , "Robert D. Russell" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 10:21:01 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <9F8400020EC5D311AF62009027C39616048A4230@axcs09.cos.agilent.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Robert, Comments below. You make a point a lot of us agree with but it requires also relaxing delivery rules. Somesh > -----Original Message----- > From: FRYMAN,ROBERT (A-Roseville,ex1) [mailto:robert_fryman@agilent.com] > Sent: Thursday, November 08, 2001 9:04 AM > To: 'Barry Reinhold'; Rod Harrison; Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > Thus the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry > > about the fact that C1 generates data in response to the R2T. > > However the command sequence of C1, C3, C2, C4 forces the iSCSI > > layer to buffer C3. This is significant to a hardware accelerated > > iSCSI layer. > > This is significant to a target hardware accelerated iSCSI layer. By > forcing the initiator to order the iSCSI commands, you move the > significants > from the target to the initiator, but it is still a complexity added to a > hardware accelerator. > > The problem I have is that we are trying to add a requirement that only > effects a specific area, but adds complexity to a larger area. If we say > that initiators must order commands on a single connection, then doesn't > this mean that we have just added complexity and slowdown to the > enterprise > storage area? > > In the Fibre channel realm, a standard base configuration was to have a > target (JBOD, array, etc.) with 2 ports, and an initiator with two ports. > These would be connected on two seperate physical connections, so if a > problem happens on one, the other will take over. We allow the same thing > in iSCSI, plus specified how the customer can use both connections to > increase bandwidth as long as both connections work. It seems to me that > any customer that wants a fault tolerant solution, and gets increased > bandwidth for free due to the ability to have multiple connections, would > take advantage of that. The assumption that the ordered delivery on a session spread across multiple controller boards will perform well is something I don't think is a valid assumption. However, if the application did not require ordered delivery, then of course, then of course all the synchronization bottlenecks are removed and data flows freely. In this case we could have a connection wide sequence # for recovering from digest errors and connection errors, and perhaps a session wide sequence number for applications that required ordering at a higher level. In this case, the complexity of ordering would be borne only by applications that did want to have multiple connections, and that did require ordering. > > This being the case, these targets have to have reordering capability. If > we force the initiators to send the comands in order, we just added > complexity and slowdown that adds no benifit. > > As I see it (and I could be totally off base here), we have a choice of > complexity and slowdown in the enterprise market, or adding a little > complexity to simple targets. Am I flawed in this thinking? > > Scott Fryman > Agilent Technologies > > > > -----Original Message----- > > From: Barry Reinhold [mailto:bbrtrebia@mediaone.net] > > Sent: Thursday, November 08, 2001 6:19 AM > > To: Rod Harrison; Robert D. Russell; Somesh Gupta > > Cc: Julian Satran; ips@ece.cmu.edu > > Subject: RE: iSCSI: Out of order commands > > > > > > Rod, > > I can see your point, from the perspective of a target > > that is a box > > combining an iSCSI entity with a SCSI entity. However, from > > the perspective > > of an "iSCSI only" entity, delivery does not have to be > > delayed until C1 > > completes. In this case you only need to ensure that the commands are > > delivered in order to the SCSI layer. The SCSI layer has to > > address the > > semantics of what the operation entails in terms of command > > execution. Thus > > the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry > > about the fact > > that C1 generates data in response to the R2T. However the > > command sequence > > of C1, C3, C2, C4 forces the iSCSI layer to buffer C3. This > > is significant > > to a hardware accelerated iSCSI layer. > > > > >-----Original Message----- > > >From: Rod Harrison [mailto:rod.harrison@windriver.com] > > >Sent: Wednesday, November 07, 2001 8:34 PM > > >To: Barry Reinhold; Robert D. Russell; Somesh Gupta > > >Cc: Julian Satran; ips@ece.cmu.edu > > >Subject: RE: iSCSI: Out of order commands > > > > > > > > >Barry, > > > > > > I'm curious, where do you see the extra complexity between the > > >following? > > > > > > Assuming all the following commands are writes ... > > > > > >c1, c3, c2, c4 and > > > > > >c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. > > > > > > In both cases the target has to queue commands 2, 3, > > and 4 whilst > > >waiting for its R2T on c1 to be satisfied. > > > > > > Even if the target doesn't support any unsolicited data > > it still has > > >to queue commands 2, 3, and 4. I can't see how the target can avoid > > >queuing commands, in which case the order of arrival seems to make > > >little difference. > > > > > > - Rod > > > > > >-----Original Message----- > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > > Behalf Of > > >Barry Reinhold > > >Sent: Wednesday, November 07, 2001 3:10 PM > > >To: Robert D. Russell; Somesh Gupta > > >Cc: Julian Satran; ips@ece.cmu.edu > > >Subject: RE: iSCSI: Out of order commands > > > > > > > > >Bob, > > > I can't speak for targets, but OOO commands on a session with a > > >single > > >connection sure increases the complexity of the code path we take. > > > To me the issue is still that these types of situations > > tend to be > > >poorly > > >tested and lead to interoperability issues. If we do go down this > > >path, the > > >spec. should make it very clear that this is expected behavior. Some > > >statement with a shall in it should be written (for the receiver), so > > >that > > >there is a conformance test items to pass. > > > > > >>-----Original Message----- > > >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > >Of > > >>Robert D. Russell > > >>Sent: Wednesday, November 07, 2001 4:13 PM > > >>To: Somesh Gupta > > >>Cc: Julian Satran; ips@ece.cmu.edu > > >>Subject: RE: iSCSI: Out of order commands > > >> > > >> > > >>Somesh, Julian: > > >> > > >>You state that dealing with OOO commands on the target > > >>will add substantial complexity on the target. > > >>Do you have any basis for that claim? My impression from the > > >>plugfest is that most targets are already dealing with > > >>it. Perhaps we need to hear from someone who is actually > > >>building a target for which this would be a real problem. > > >> > > >>If anything, what we are hearing from people who really > > >>are building initiators is that dealing with the requirement > > >>to send commands in order will introduce substantial complexity > > >>on the initiator. > > >> > > >>So why should we be saving complexity on (hypothetically) simple > > >>targets yet requiring complexity on real initiators? > > >> > > >>As far as the deadlock issue is concerned, if the only way > > >>that deadlock can occur with OOO commands on the same > > >>connection is during the use of immediate data (which is I > > >>think what Julian was saying), then shouldn't we change > > >>the standard to just say that -- if the initiator sends > > >>commands out of order on a single connection, then immediate > > >>data MUST NOT be used on that connection in order to avoid deadlock. > > >> > > >>This gives everybody what they want, since initiators who find > > >>it beneficial to deliver commands OOO will just negotiate > > >>immediate data off. Those who really want to use immediate data > > >>will have to ensure that commands are sent in order. > > >>The tradeoff then becomes an implementation issue, not a > > >>standards issue, which is where it belongs. > > >> > > >> > > >>Bob Russell > > >>InterOperability Lab > > >>University of New Hampshire > > >>rdr@iol.unh.edu > > >>603-862-3774 > > >> > > >> > > >>On Wed, 7 Nov 2001, Somesh Gupta wrote: > > >> > > >>> I think we should either have it as a MUST or not require > > >>> it (at both ends to get the real benefit). SHOULD is one > > >>> of those things that leads to implementation > > >>> burden and confusion, without perhaps the feature being > > >>> used. There are implementation as well as protocol > > >>> considerations mixed in here. > > >>> > > >>> If we are to remove the restriction, we should (SHOULD) > > >>> get the maximum benefit from it, rather than to > > >>> accomodate an implementation choice. Out of sequence > > >>> commands, combined with the possibility of digest errors, > > >>> will add substantial complexity on the target side, > > >>> without corrosponding benefit in performance. If we change > > >>> this to SHOULD, we should also relax the requirement > > >>> to present commands on the target side to a SHOULD. > > >>> > > >>> > > >>> > > >>> > -----Original Message----- > > >>> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > > >Behalf Of > > >>> > Julian Satran > > >>> > Sent: Wednesday, November 07, 2001 10:00 AM > > >>> > To: ips@ece.cmu.edu > > >>> > Subject: Re: iSCSI: Out of order commands > > >>> > > > >>> > > > >>> > Mallikarjun, > > >>> > > > >>> > I did not see a SINGLE performance improvement that results from > > >OOO > > >>> > shipping. > > >>> > I would be bad engineering to give away the "no-deadlock" > > >mechanism we > > >>> > have now for nothing. > > >>> > I have also the impression that the point about deadlock that I > > >keep > > >>> > repeating is ignored or not understood. > > >>> > As we stand today commands can be shipped with Immediate data > > >>or without > > >>> > and an implementer determined > > >>> > to squeeze maximum bandwidth and overlap command start with > > >>delivery will > > >>> > choose not to work with immediate data > > >>> > (as you have pointed out) while a low performance software > > >>implementation > > >>> > will use immediate data to minimize CPU cycles > > consumed. However > > >both > > >>> > will be guaranteed to work without deadlock as source and sink > > >use the > > >>> > same ordering. > > >>> > Recovery is still a low probability event and should be handled > > >with a > > >>> > different set of considerations in mind. > > >>> > As for the strictness of the recommendation - yes we > > could settle > > >on > > >>> > SHOULD. > > >>> > > > >>> > Julo > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > "Mallikarjun C." > > >>> > Sent by: owner-ips@ece.cmu.edu > > >>> > 07-11-01 19:41 > > >>> > Please respond to cbm > > >>> > > > >>> > > > >>> > To: Santosh Rao , > > >ips@ece.cmu.edu > > >>> > cc: > > >>> > Subject: Re: iSCSI: Out of order commands > > >>> > > > >>> > > > >>> > > > >>> > Santosh, > > >>> > > > >>> > I have only one comment on your responses. > > >>> > > > >>> > > Even a single connection target *MUST* implement a scoreboard. > > >The > > >>> > > reason being that it can see out-of-order arrival of commands > > >due to > > >>> > > commands being dropped on digest errors. In such a case, it > > >>must block > > >>> > > further command processing until holes are filled. > > >>> > > > >>> > I made two convenient assumptions if you noticed, :-), one of > > >which > > >>> > is that target forces session recovery on *any* error that it > > >sees > > >>> > (ErrorRecoveryLevel=0) - including a dropped command due to a > > >digest > > >>> > error. With that assumption, a target can afford not to > > >implement > > >>> > a scoreboard. > > >>> > > > >>> > As I said in a private note, I guess what primarily bothers me > > >about > > >>> > OOO commands on a connection is that it requires the receiver to > > >>> > undo this "optimization" on its end - most notably on a single > > >>> > connection. TCP experts may comment on how/if they dealt with a > > >>> > similar issue. > > >>> > > > >>> > OTOH, you had some valid comments on exceptions to ordering > > >during > > >>> > connection recovery. Perhaps we can move on by making Julian's > > >>> > proposed stipulation a SHOULD.... > > >>> > -- > > >>> > Mallikarjun > > >>> > > > >>> > > > >>> > Mallikarjun Chadalapaka > > >>> > Networked Storage Architecture > > >>> > Network Storage Solutions Organization > > >>> > MS 5668 Hewlett-Packard, Roseville. > > >>> > cbm@rose.hp.com > > >>> > > > >>> > > > >>> > Santosh Rao wrote: > > >>> > > > > >>> > > Mallikarjun, > > >>> > > > > >>> > > Some comments below. > > >>> > > > > >>> > > Regards, > > >>> > > Santosh > > >>> > > > > >>> > > "Mallikarjun C." wrote: > > >>> > > > > > >>> > > > Rod and Julian, > > >>> > > > > > >>> > > > This has been an interesting thread of discussion. Some > > >>> > > > comments - > > >>> > > > > > >>> > > > 1.My first reaction was - allowing out-of-order command > > >>> > > > transmission on the same connection deprives targets of > > >>> > > > an implementation choice. Targets which support only > > >>> > > > single-connection sessions and only support session > > >>> > > > recovery (reasonable assumptions in my mind) can no > > >>> > > > longer afford *not to* implement a command scoreboard. > > >>> > > > > >>> > > Even a single connection target *MUST* implement a scoreboard. > > >The > > >>> > > reason being that it can see out-of-order arrival of commands > > >due to > > >>> > > commands being dropped on digest errors. In such a case, it > > >>must block > > >>> > > further command processing until holes are filled. > > >>> > > > > >>> > > Thus, there is no getting away from implementing a > > sequencer at > > >the > > >>> > > target. Given this, I think it is unreasonable to restrict > > >initiator > > >>> > > implementation flexibility by imposing a strict ordering > > >requirement > > >>> > > within the connection. > > >>> > > > > >>> > > > 2.Any end-node efficiency that is sought to be achieved > > >>> > > > by transmitting CmdSNs out-of-order from the initiator > > >>> > > > would be lost on the other end-node, since the target > > >>> > > > now must wait for re-ordering the commands. > > >>> > > > > >>> > > It has to handle this situation anyway to deal with holes > > >caused by > > >>> > > digest errors. This scenario occurs even with initiators that > > >issue > > >>> > > commands in order. > > >>> > > > > >>> > > > > > >>> > > > 3.The flipside is that out-of-order transmission saves > > >>> > > > link badwidth (albeit at the expense of end-node > > >efficiency), > > >>> > > > compared to idling the link waiting for outbound DMA. > > >>> > > > We have to determine if this is a reasonable trade-off. > > >>> > > > > > >>> > > > 4.I can see Rod's point that prefetching all immediate > > >>> > > > data can be a burden on the NIC resources. But, two > > >>> > > > questions - > > >>> > > > - could the NIC not use unsolicited separate data > > >>> > > > PDUs in these cases? [ I realize that InitialR2T > > >>> > > > has to be "no" to let it happen... ] > > >>> > > > - could the NIC have a memory architecture that > > >>> > > > allows data prefetching for the next command (so > > >>> > > > this is a non-issue from the protocol > > perspective)? > > >>> > > > This scheme incurs one DMA delay for every new > > >>> > > > burst of commands. > > >>> > > > > > >>> > > > 5.Another (perhaps radical at this point) option is to do > > >>> > > > away with immediate unsolicited data, to stick only with > > >>> > > > separate unsolicited data. I would personally be okay > > >>> > > > with the choice, particularly if this feature (that > > >>> > > > helps software implementations) starts making hardware > > >>> > > > design complicated/expensive. > > >>> > > > > > >>> > > > So, to summarize - > > >>> > > > > > >>> > > > option immediate allow > > >>> > > > data in spec? > > >out-of-order? > > >>> > > > > > >>> > > > (A) (5) above no no > > >>> > > > (B) No real reason to do this. no yes > > >>> > > > (C) (4) above yes no > > >>> > > > (D) pros & cons (1), (2) & (3) yes yes > > >>> > > > > > >>> > > > >From the arguments I heard so far, I am leaning towards > > >>> > > > option A, and option C in that order. > > >>> > > > > > >>> > > > Comments? > > >>> > > > -- > > >>> > > > Mallikarjun > > >>> > > > > > >>> > > > Mallikarjun Chadalapaka > > >>> > > > Networked Storage Architecture > > >>> > > > Network Storage Solutions Organization > > >>> > > > MS 5668 Hewlett-Packard, Roseville. > > >>> > > > cbm@rose.hp.com > > >>> > > > > > >>> > > > Rod Harrison wrote: > > >>> > > > > > > >>> > > > > Julian, > > >>> > > > > > > >>> > > > > I don't understand what you are proposing here, > > >>what do you > > >>> > mean by > > >>> > > > > "multiplexed" DMA? > > >>> > > > > > > >>> > > > > The problem is that the DMAs take some time, the > > >>more there > > >>> > are > > >>> > > > > queued the longer the last DMAs queued take to complete. > > >Some > > >>> > commands > > >>> > > > > require DMAs to complete before they can be sent, i.e. > > >>Writes with > > >>> > > > > immediate data, some commands do not, i.e. Reads and > > >>writes with no > > >>> > > > > immediate data. The iSCSI HBA wants to be able to send > > >>commands as > > >>> > > > > soon a possible, which for a read after a write can be > > >before the > > >>> > > > > write's DMA has completed. Maintaining an ordered queue > > >>for commands > > >>> > > > > to be sent on the HBA is expensive and redundant since the > > >target > > >>> > > > > already knows how to queue commands before committing them > > >to its > > >>> > SCSI > > >>> > > > > layer. > > >>> > > > > > > >>> > > > > The iSCSI HBA and its host driver are not at > > >liberty to > > >>> > change the > > >>> > > > > order of commands from the OS, but the DMAs those > > >>commands need are > > >>> > > > > unlikely to complete in the same order, and as I mentioned > > >some > > >>> > > > > commands need no DMA. If the HBA can't send > > commands out of > > >CmdSN > > >>> > > > > order it has to maintain an ordered queue of commands > > >>waiting to be > > >>> > > > > sent, and potentially buffer a lot of data. For > > an HBA this > > >makes > > >>> > > > > immediate data almost impossible to support. > > >>> > > > > > > >>> > > > > I don't see the problem with allowing out of > > >>order commands > > >>> > given > > >>> > > > > that the target already has to deal with very similar > > >problems. I > > >>> > > > > think we are getting in to the area of implementation > > >>choices here, > > >>> > > > > which is inappropriate for a specification. > > >>> > > > > > > >>> > > > > - Rod > > >>> > > > > > > >>> > > > > -----Original Message----- > > >>> > > > > From: owner-ips@ece.cmu.edu > > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > > >>> > Of > > >>> > > > > Julian Satran > > >>> > > > > Sent: Monday, November 05, 2001 10:06 PM > > >>> > > > > To: ips@ece.cmu.edu > > >>> > > > > Subject: Re: iSCSI: Out of order commands, was current > > >>UNH Plugfest > > >>> > > > > > > >>> > > > > Rod, > > >>> > > > > > > >>> > > > > I don't see any reason why DMA operations cant be > > >>"multiplexed" with > > >>> > > > > commands. > > >>> > > > > If you have scheduled a long outbound DMA you are doomed > > >>regardless > > >>> > of > > >>> > > > > the > > >>> > > > > command ordering. > > >>> > > > > And if you have scheduled DMA operations > > piecemeal then you > > >can > > >>> > insert > > >>> > > > > your commands in correct order. > > >>> > > > > > > >>> > > > > Julo > > >>> > > > > > > >>> > > > > "Rod Harrison" > > >>> > > > > 05-11-01 20:48 > > >>> > > > > Please respond to "Rod Harrison" > > >>> > > > > > > >>> > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > > > >>> > > > > cc: > > >>> > > > > Subject: iSCSI: Out of order commands, was > > >current > > >>> > UNH > > >>> > > > > Plugfest > > >>> > > > > > > >>> > > > > [ Subject changed ] > > >>> > > > > > > >>> > > > > Julian, > > >>> > > > > > > >>> > > > > The ordering difference is introduced > > >>between the > > >>> > > > > host > > >>> > > > > side driver > > >>> > > > > and the iSCSI HBA. The host side driver must present > > >>SCSI commands > > >>> > to > > >>> > > > > the HBA in the order they are received from the OS to > > >>prevent read > > >>> > > > > after write dependency failures. The HBA might reorder > > >>the commands > > >>> > > > > depending on when DMA completes. The reordering can't be > > >>done ahead > > >>> > of > > >>> > > > > time in the host driver since it doesn't know how > > long each > > >DMA > > >>> > might > > >>> > > > > take. As long as the HBA assigns CmdSN in the order it > > >receives > > >>> > > > > commands the desired host ordering is preserved. > > >>> > > > > > > >>> > > > > - Rod > > >>> > > > > > > >>> > > > > -----Original Message----- > > >>> > > > > From: owner-ips@ece.cmu.edu > > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > > >>> > Of > > >>> > > > > Julian Satran > > >>> > > > > Sent: Monday, November 05, 2001 12:35 AM > > >>> > > > > To: ips@ece.cmu.edu > > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > > >>> > > > > > > >>> > > > > Rod, > > >>> > > > > > > >>> > > > > I all examples give the point I find hard to understand is > > >why is > > >>> > the > > >>> > > > > ordering on the wire different from the presentation order > > >to the > > >>> > > > > initiator. You can get as many overlaps as you want by > > >>presenting > > >>> > the > > >>> > > > > commands to the initiator in the desired order. > > >>> > > > > What we are considering here is the case in which you > > >>want to ship > > >>> > in > > >>> > > > > an > > >>> > > > > order different than the one you present the commands. > > >>> > > > > > > >>> > > > > Julo > > >>> > > > > > > >>> > > > > "Rod Harrison" > > >>> > > > > Sent by: owner-ips@ece.cmu.edu > > >>> > > > > 04-11-01 04:42 > > >>> > > > > Please respond to "Rod Harrison" > > >>> > > > > > > >>> > > > > To: "Barry Reinhold" , > > >"Dave > > >>> > > > > Sheehy" > > >>> > > > > , "IETF IP SAN Reflector" > > >>> > > > > > > >>> > > > > cc: > > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > > >>> > > > > > > >>> > > > > Barry, > > >>> > > > > > > >>> > > > > In general I agree but I don't think this > > >is as > > >>> > much > > >>> > > > > of a > > >>> > > > > corner case > > >>> > > > > as it at first appears. Targets will have code very > > >>similar to that > > >>> > > > > needed to handle out of order commands to deal with > > >>digest errors. > > >>> > > > > Targets also need to queue commands whilst > > waiting for both > > >>> > solicited > > >>> > > > > and unsolicited data to arrive. Queuing out of order > > >>commands seems > > >>> > > > > little extra work. > > >>> > > > > > > >>> > > > > From an initiators point of view > > there are > > >>> > > > > efficiency, > > >>> > > > > and probably > > >>> > > > > performance gains to be had from sending commands out of > > >>order. Bob > > >>> > > > > Russell gave the example of a read being sent whilst > > >>write data DMA > > >>> > is > > >>> > > > > happening, and a similar situation can arise with DMA for > > >writes > > >>> > > > > overtaking that of earlier writes if the initiator has > > >>multiple DMA > > >>> > > > > engines. In this case the initiator might be forced to > > >>let the wire > > >>> > go > > >>> > > > > idle if it can't send the data from completed DMAs as soon > > >as > > >>> > > > > possible. > > >>> > > > > > > >>> > > > > We already have a command queue at the > > >target to > > >>> > > > > enforce > > >>> > > > > correct > > >>> > > > > serialisation of commands, doing the same thing at the > > >>initiator is > > >>> > > > > redundant. > > >>> > > > > > > >>> > > > > Finally, I don't believe we should be > > >writing a > > >>> > > > > standard > > >>> > > > > to work > > >>> > > > > around poor coding and test coverage, especially at the > > >cost of > > >>> > > > > potential efficiency gains. > > >>> > > > > > > >>> > > > > I agree with Dave and Santosh that > > >>commands being > > >>> > > > > sent > > >>> > > > > out of order > > >>> > > > > on a single session should be allowed by the standard. > > >>> > > > > > > >>> > > > > - Rod > > >>> > > > > > > >>> > > > > -----Original Message----- > > >>> > > > > From: owner-ips@ece.cmu.edu > > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > > >>> > Of > > >>> > > > > Barry Reinhold > > >>> > > > > Sent: Friday, November 02, 2001 5:24 PM > > >>> > > > > To: Dave Sheehy; IETF IP SAN Reflector > > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > > >>> > > > > > > >>> > > > > Using features such as out of order command delivery on > > >>a connection > > >>> > > > > tend to > > >>> > > > > be the sort of things that lead to interoperability > > >>problems. It is > > >>> > > > > unexpected and probably going to hit poorly tested code > > >>paths even > > >>> > if > > >>> > > > > the > > >>> > > > > standard is written to allow it. > > >>> > > > > > > >>> > > > > >-----Original Message----- > > >>> > > > > >From: owner-ips@ece.cmu.edu > > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > > >>> > > > > Of > > >>> > > > > >Dave Sheehy > > >>> > > > > >Sent: Friday, November 02, 2001 4:19 PM > > >>> > > > > >To: IETF IP SAN Reflector > > >>> > > > > >Subject: Re: iSCSI: current UNH Plugfest > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > >> 3. Can commands be sent out of order on the same > > >connection? > > >>> > > > > >> > > >>> > > > > >> The behavior of targets is clearly specified in > > >Section > > >>> > 2.2.2.3 > > >>> > > > > on > > >>> > > > > >> page 25 of draft 8, which says: > > >>> > > > > >> "Except for the commands marked for immediate > > >>delivery the > > >>> > > > > iSCSI > > >>> > > > > >> target layer MUST eliver the commands for > > >>execution in the > > >>> > > > > order > > >>> > > > > >> specified by CmdSN." > > >>> > > > > >> > > >>> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > >>> > > > > >> "- CmdSN - the current command Sequence Number > > >>advanced by 1 > > >>> > > > > on > > >>> > > > > >> each command shipped except for commands > > marked for > > >>> > immediate > > >>> > > > > >> delivery." > > >>> > > > > >> but the meaning of the term "shipped" is vague, > > >>and does not > > >>> > > > > >> necessarily > > >>> > > > > >> require that the PDUs arrive on the other end of a > > >TCP > > >>> > > > > connection > > >>> > > > > >> in the same order that the CmdSN values were > > >>assigned to these > > >>> > > > > PDUs. > > >>> > > > > >> > > >>> > > > > >> Some initiators have been designed to send commands > > >out of > > >>> > CmdSN > > >>> > > > > >> order on one connection. Consider the situation > > >>where there > > >>> > is > > >>> > > > > only > > >>> > > > > >> one connection and a high-level dispatcher creates > > >>a PDU for a > > >>> > > > > SCSI > > >>> > > > > >> command that involves writing immediate data to the > > >target. > > >>> > > > > This PDU > > >>> > > > > >> is enqueued to a lower-level layer which has to > > >>setup, start, > > >>> > > > > and > > >>> > > > > >> wait-for a DMA operation to move the immediate data > > >into an > > >>> > > > > onboard > > >>> > > > > >> buffer before the PDU can be put onto the wire. > > >>While this is > > >>> > > > > >> happening, the dispatcher creates another > > >>unrelated PDU for a > > >>> > > > > SCSI > > >>> > > > > >> read command (for example), and when this PDU is > > >>passed to the > > >>> > > > > >> lower-level layer it can be sent immediately, ahead > > >of the > > >>> > > > > previous > > >>> > > > > >> write PDU and therefore out of order on this > > >connection. > > >>> > > > > >> > > >>> > > > > >> The standard clearly allows this to happen > > if the two > > >PDUs > > >>> > were > > >>> > > > > sent > > >>> > > > > >> on different connections, and seems to imply that > > >this can > > >>> > also > > >>> > > > > happen > > >>> > > > > >> when the two PDUs are sent on the same connection. > > >>> > > > > >> > > >>> > > > > >> The suggestion is to put in the standard an > > >>explicit statement > > >>> > > > > that > > >>> > > > > >> this is allowed or not allowed, as appropriate. > > >>> > > > > >> > > >>> > > > > >> If this is allowed, such a statement would avoid > > >>the erroneous > > >>> > > > > >> assumption being made by some target implementers > > >>that within > > >>> > a > > >>> > > > > single > > >>> > > > > >> connection, commands will arrive in order. > > >>> > > > > >> > > >>> > > > > >> If this is not allowed, such a statement would avoid > > >the > > >>> > > > > erroneous > > >>> > > > > >> assumption being made by some initiator implementers > > >that > > >>> > within > > >>> > > > > a > > >>> > > > > >> single connection, commands can be put on the wire > > >out of > > >>> > order. > > >>> > > > > >> > > >>> > > > > >> +++ > > >>> > > > > >> > > >>> > > > > >> will add an explicit statement saying that this > > >behaviour is > > >>> > > > > forbidden. > > >>> > > > > >> 2.2.2.1 will contain: > > >>> > > > > >> > > >>> > > > > >> On any given connection, the iSCSI initiator MUST send > > >the > > >>> > > > > >commands in the > > >>> > > > > >> order specified by CmdSN. > > >>> > > > > >> > > >>> > > > > >> +++ > > >>> > > > > > > > >>> > > > > >Why do you feel this behavior should be forbidden? > > >>Targets already > > >>> > > > > have to > > >>> > > > > >order commands across the session. I don't see why it's > > >>a problem > > >>> > to > > >>> > > > > extend > > >>> > > > > >that to the connection as well. I, for one, believe we > > >>should take > > >>> > > > > >a liberal > > >>> > > > > >stance on this. > > >>> > > > > > > > >>> > > > > >Dave Sheehy > > >>> > > > > > > > >>> > > > > >>> > > -- > > >>> > > ################################## > > >>> > > Santosh Rao > > >>> > > Software Design Engineer, > > >>> > > HP-UX iSCSI Driver Team, > > >>> > > Hewlett Packard, Cupertino. > > >>> > > email : santoshr@cup.hp.com > > >>> > > Phone : 408-447-3751 > > >>> > > ################################## > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > >> > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 14:43:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA04863 for ; Thu, 8 Nov 2001 14:43:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8IgiS11599 for ips-outgoing; Thu, 8 Nov 2001 13:42:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8Iggl11593 for ; Thu, 8 Nov 2001 13:42:42 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id KAA24981; Thu, 8 Nov 2001 10:42:30 -0800 (PST) From: Keith McCloghrie Message-Id: <200111081842.KAA24981@cisco.com> Subject: FC Management MIB - proposed changes To: ips@ece.cmu.edu Date: Thu, 8 Nov 2001 10:42:30 -0800 (PST) Cc: kzm@cisco.com (Keith McCloghrie) X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Based on the issues that I listed in my previous message (yesterday), I have a set of changes that I'd like to make to the FC-Mgmt MIB, providing there are no objections from the WG. These changes (the first six correspond to the issues list) are: 1. MIB will be written using SMIv2. 2. All objects which apply in non-Fibre Channel environments will be removed. (Note that this applies to a large fraction of the objects defined in this MIB.) 2a. For those objects which will be removed, if there are any which are: a) not already covered in other MIBs, and b) considered essential to the management of Fibre Channel-based products, then it will be necessary to consider whether other MIB(s) need to be modified/created as a home for them, and if so which WG(s) should undertake such work. 3. With 20x20 hindsight, the following observations can be made with respect to RFC 2837: - the reason that we now have the issue of how it relates to the FC-Mgmt MIB is because it was written as a Fabric Element MIB; this issue would not have arisen if it had been written as a Fibre Channel interface MIB. - it should have extended the ifTable, rather than overlapping with it, - it should not have defined the fcFeModuleTable, which overlaps with the Entity MIB. - and it should have specified (at least) its octet counters as Counter64. Given these problems, I propose that the new FC-Mgmt MIB be specified as a replacement for RFC 2837. 4. This MIB will include a media-specific MIB to specify how Fibre Channel interfaces use the ifTable (see section 4 in RFC 2863). This will result in many tables in this MIB being indexed by ifIndex. 5. Regarding the the MIB objects for the "Simple Name Service", I see two possible solutions: i. retain the MIB objects but focus them on GS-3's Unzoned Name Service. ii. remove the MIB objects for the "Simple Name Service" from this MIB. If there is WG consensus that a MIB is needed for one of the GS-3 Name Services, and for which one, then the appropriate set of MIB objects can be defined in a new MIB. Of these two, I propose to investigate solution i), and if it proves feasible, then to adopt it; if not, to fall back to solution ii). 6. The Counter32 or Counter64 syntax will be used for counters. 7. In addition to the above, it has been suggested that MIB objects to support Class 1 are no longer needed. If I don't hear any objections, I will assume that I can make this change also. 8. Yesterday, John Nguyen (jnguyen@gadzoox.com) asked in an email about "bringing ZONE into FC Management MIB". His suggestion would seem to be in addition to, rather than as a replacement for, anything which will be in the new FC-Mgmt MIB. Thus, at this point in time, I'd like to suggest that the changes listed above represent a large enough amount of work for us to chew on. Therefore, I propose that our answer to John's query is to ask him to re-raise this issue at a point in the future when the new FC-Mgmt MIB is becoming more stable. 9. Are there any other changes that anyone would like to see at this time ? Also note that with this MIB now being in a different WG, the I-D needs to have a draft-ietf-ips-xxx name. I propose to use draft-ietf-ips-fcmgmt-mib-00.txt. Keith. From owner-ips@ece.cmu.edu Thu Nov 8 14:48:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA04952 for ; Thu, 8 Nov 2001 14:48:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8J03212985 for ips-outgoing; Thu, 8 Nov 2001 14:00:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8J00l12972 for ; Thu, 8 Nov 2001 14:00:00 -0500 (EST) Received: from xparelay2.corp.hp.com (unknown [15.58.137.112]) by palrel12.hp.com (Postfix) with ESMTP id 115B41F92A; Thu, 8 Nov 2001 10:59:54 -0800 (PST) Received: from xpabh2.corp.hp.com (xpabh2.corp.hp.com [15.58.136.192]) by xparelay2.corp.hp.com (Postfix) with ESMTP id 53CA61F535; Thu, 8 Nov 2001 03:58:56 -0800 (PST) Received: by xpabh2.corp.hp.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Nov 2001 10:59:48 -0800 Message-ID: <028FE7141C79D511B65100D0B74FE875BCA5C6@xrose01.rose.hp.com> From: "SPASIC,PREDRAG (HP-Roseville,ex1)" To: "'Keith McCloghrie'" , ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Date: Thu, 8 Nov 2001 10:59:40 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Keith, Point 8: Although there are quite a few issues around zoning management, most of them revolving around perceived inadequate security of SNMP, without this information MIB will not be complete. Zoning management is extensively defined in FC-GS3, is specific to FC, and I do not see any reasons to delay this for some point in the future. Regards Predrag Spasic, Hewlett Packard Details: https://ecardfile.com/id/predrag_spasic > -----Original Message----- > From: Keith McCloghrie [mailto:kzm@cisco.com] > Sent: Thursday, November 08, 2001 10:43 AM > To: ips@ece.cmu.edu > Cc: kzm@cisco.com > Subject: FC Management MIB - proposed changes > > > > Based on the issues that I listed in my previous message > (yesterday), I > have a set of changes that I'd like to make to the FC-Mgmt MIB, > providing there are no objections from the WG. These changes (the > first six correspond to the issues list) are: > > 1. MIB will be written using SMIv2. > > 2. All objects which apply in non-Fibre Channel environments will be > removed. (Note that this applies to a large fraction of the objects > defined in this MIB.) > > 2a. For those objects which will be removed, if there are any which > are: a) not already covered in other MIBs, and b) considered essential > to the management of Fibre Channel-based products, then it will be > necessary to consider whether other MIB(s) need to be modified/created > as a home for them, and if so which WG(s) should undertake such work. > > 3. With 20x20 hindsight, the following observations can be made with > respect to RFC 2837: > > - the reason that we now have the issue of how it relates to the > FC-Mgmt MIB is because it was written as a Fabric Element MIB; > this issue would not have arisen if it had been written as a > Fibre Channel interface MIB. > - it should have extended the ifTable, rather than > overlapping with it, > - it should not have defined the fcFeModuleTable, which overlaps with > the Entity MIB. > - and it should have specified (at least) its octet counters > as Counter64. > > Given these problems, I propose that the new FC-Mgmt MIB be specified > as a replacement for RFC 2837. > > 4. This MIB will include a media-specific MIB to specify how Fibre > Channel interfaces use the ifTable (see section 4 in RFC 2863). This > will result in many tables in this MIB being indexed by ifIndex. > > 5. Regarding the the MIB objects for the "Simple Name Service", > I see two possible solutions: > > i. retain the MIB objects but focus them on GS-3's Unzoned > Name Service. > > ii. remove the MIB objects for the "Simple Name Service" from > this MIB. > If there is WG consensus that a MIB is needed for one of > the GS-3 Name > Services, and for which one, then the appropriate set of > MIB objects > can be defined in a new MIB. > > Of these two, I propose to investigate solution i), and if it proves > feasible, then to adopt it; if not, to fall back to solution ii). > > 6. The Counter32 or Counter64 syntax will be used for counters. > > 7. In addition to the above, it has been suggested that MIB objects to > support Class 1 are no longer needed. If I don't hear any objections, > I will assume that I can make this change also. > > 8. Yesterday, John Nguyen (jnguyen@gadzoox.com) asked in an > email about > "bringing ZONE into FC Management MIB". His suggestion would seem to > be in addition to, rather than as a replacement for, anything > which will > be in the new FC-Mgmt MIB. Thus, at this point in time, I'd like to > suggest that the changes listed above represent a large enough amount > of work for us to chew on. Therefore, I propose that our answer to > John's query is to ask him to re-raise this issue at a point in the > future when the new FC-Mgmt MIB is becoming more stable. > > 9. Are there any other changes that anyone would like to see at this > time ? > > Also note that with this MIB now being in a different WG, the > I-D needs > to have a draft-ietf-ips-xxx name. I propose to use > draft-ietf-ips-fcmgmt-mib-00.txt. > > Keith. > From owner-ips@ece.cmu.edu Thu Nov 8 14:50:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05024 for ; Thu, 8 Nov 2001 14:50:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8JL7f14703 for ips-outgoing; Thu, 8 Nov 2001 14:21:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from glatton.xo.com ([207.155.248.78]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8JL5l14695 for ; Thu, 8 Nov 2001 14:21:05 -0500 (EST) Received: from blekinge ([66.89.96.162]) by glatton.xo.com id OAA17024; Thu, 8 Nov 2001 14:20:59 -0500 (EST) [ConcentricHost SMTP Relay 1.14] Message-ID: <002a01c1688a$63793d30$1001010a@blekinge> From: "Peter Mellquist" To: "Julian Satran" Cc: "IPS" References: Subject: Re: iSCSI over TLS Date: Thu, 8 Nov 2001 11:20:08 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Thank you for the clarification. It makes sense moving toward 10GbE to use IPSEC. It would also really be beneficial to allow iSCSI to utilize TLS, essentially have iSCSI support either IPSEC or TLS rather than just IPSEC. This would only help to proliferate secure iSCSI as well as allow more products to incorporate strong security in a flexible manner ( there are a number of export issues with having strong security embedded in silicon around a TOE) It would not require much work in terms of the RFC effort, all we would need is another IANA port ( iSCSI and iSCSI/TLS) and a default cipher suite. It would be better have the standard support TLS rather than have proprietary port numbers and cipher suites resulting in lack of interoperation. Thanks, Peter Mellquist Seven Systems Technologies 575 Menlo Drive Suite 2 Rocklin CA 916-577-1275 peterm@seven-systems.com ----- Original Message ----- From: "Julian Satran" To: Sent: Tuesday, November 06, 2001 10:16 PM Subject: Re: iSCSI over TLS > Peter, > > A group of us seriously considered TLS. The main reason for dropping it > was that it would interfere with any mechanism we could think of doing > framing and steering and we thought that framing and steering are > essential at 10Gbps and over. > > Julo > > > > > "Peter Mellquist" > Sent by: owner-ips@ece.cmu.edu > 07-11-01 02:15 > Please respond to "Peter Mellquist" > > > To: > cc: > Subject: iSCSI over TLS > > > > I am aware that the ips group is leaning toward IPSEC as for the security > solution but I am interested if anyone is also considering using Transport > Layer Security (TLS)? > > I am concerned that the requirement for IPSEC might make TOEs more > complex > than they need to be. Can TLS be optionally used as well as defined by the > specification? This could allow TOE vendors to only be concerned with > providing normal IPv4 / ipv6 and leave the security to a higher layer. A > TLS > stack sitting above the TOE could then handle security very well. Also, I > anticipate that the first generation of TOEs will not support IPSEC. With > a > iSCSI/TLS we could enable security solutions with the first generation of > TOEs and get speed and security. > > Are any TOE vendors planning to support IPSEC? > > Can TLS or IPSEC be supported? > > -peter > > > > Peter Mellquist > Seven Systems Technologies > 575 Menlo Drive Suite 2 > Rocklin CA > 916-577-1275 > peterm@seven-systems.com > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 15:21:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA05873 for ; Thu, 8 Nov 2001 15:21:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8JwvZ17845 for ips-outgoing; Thu, 8 Nov 2001 14:58:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8Jwtl17838 for ; Thu, 8 Nov 2001 14:58:55 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA8Jwnu17357 for ; Thu, 8 Nov 2001 11:58:49 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA8Jwh624326 for ; Thu, 8 Nov 2001 11:58:43 -0800 From: "Bill Strahm" To: "Ips@Ece. Cmu. Edu" Subject: RE: FC Management MIB - proposed changes Date: Thu, 8 Nov 2001 11:58:37 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <028FE7141C79D511B65100D0B74FE875BCA5C6@xrose01.rose.hp.com> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Predrag, The problem is that I am not sure we understand zoning management well enough right now, as compared to simply managing the transport. For that reason I would hate to delay the FC MIBs for a long time to figure out how to correctly manage zoning. There is nothing stopping you or John from proposing a seperate MIB and working on getting it standardized, but I am not certain it belongs in this effort (which is to standardize the Transport management) Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of SPASIC,PREDRAG (HP-Roseville,ex1) Sent: Thursday, November 08, 2001 11:00 AM To: 'Keith McCloghrie'; ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Keith, Point 8: Although there are quite a few issues around zoning management, most of them revolving around perceived inadequate security of SNMP, without this information MIB will not be complete. Zoning management is extensively defined in FC-GS3, is specific to FC, and I do not see any reasons to delay this for some point in the future. Regards Predrag Spasic, Hewlett Packard Details: https://ecardfile.com/id/predrag_spasic > -----Original Message----- > From: Keith McCloghrie [mailto:kzm@cisco.com] > Sent: Thursday, November 08, 2001 10:43 AM > To: ips@ece.cmu.edu > Cc: kzm@cisco.com > Subject: FC Management MIB - proposed changes > > > > Based on the issues that I listed in my previous message > (yesterday), I > have a set of changes that I'd like to make to the FC-Mgmt MIB, > providing there are no objections from the WG. These changes (the > first six correspond to the issues list) are: > > 1. MIB will be written using SMIv2. > > 2. All objects which apply in non-Fibre Channel environments will be > removed. (Note that this applies to a large fraction of the objects > defined in this MIB.) > > 2a. For those objects which will be removed, if there are any which > are: a) not already covered in other MIBs, and b) considered essential > to the management of Fibre Channel-based products, then it will be > necessary to consider whether other MIB(s) need to be modified/created > as a home for them, and if so which WG(s) should undertake such work. > > 3. With 20x20 hindsight, the following observations can be made with > respect to RFC 2837: > > - the reason that we now have the issue of how it relates to the > FC-Mgmt MIB is because it was written as a Fabric Element MIB; > this issue would not have arisen if it had been written as a > Fibre Channel interface MIB. > - it should have extended the ifTable, rather than > overlapping with it, > - it should not have defined the fcFeModuleTable, which overlaps with > the Entity MIB. > - and it should have specified (at least) its octet counters > as Counter64. > > Given these problems, I propose that the new FC-Mgmt MIB be specified > as a replacement for RFC 2837. > > 4. This MIB will include a media-specific MIB to specify how Fibre > Channel interfaces use the ifTable (see section 4 in RFC 2863). This > will result in many tables in this MIB being indexed by ifIndex. > > 5. Regarding the the MIB objects for the "Simple Name Service", > I see two possible solutions: > > i. retain the MIB objects but focus them on GS-3's Unzoned > Name Service. > > ii. remove the MIB objects for the "Simple Name Service" from > this MIB. > If there is WG consensus that a MIB is needed for one of > the GS-3 Name > Services, and for which one, then the appropriate set of > MIB objects > can be defined in a new MIB. > > Of these two, I propose to investigate solution i), and if it proves > feasible, then to adopt it; if not, to fall back to solution ii). > > 6. The Counter32 or Counter64 syntax will be used for counters. > > 7. In addition to the above, it has been suggested that MIB objects to > support Class 1 are no longer needed. If I don't hear any objections, > I will assume that I can make this change also. > > 8. Yesterday, John Nguyen (jnguyen@gadzoox.com) asked in an > email about > "bringing ZONE into FC Management MIB". His suggestion would seem to > be in addition to, rather than as a replacement for, anything > which will > be in the new FC-Mgmt MIB. Thus, at this point in time, I'd like to > suggest that the changes listed above represent a large enough amount > of work for us to chew on. Therefore, I propose that our answer to > John's query is to ask him to re-raise this issue at a point in the > future when the new FC-Mgmt MIB is becoming more stable. > > 9. Are there any other changes that anyone would like to see at this > time ? > > Also note that with this MIB now being in a different WG, the > I-D needs > to have a draft-ietf-ips-xxx name. I propose to use > draft-ietf-ips-fcmgmt-mib-00.txt. > > Keith. > From owner-ips@ece.cmu.edu Thu Nov 8 15:21:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA05884 for ; Thu, 8 Nov 2001 15:21:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8Jel616290 for ips-outgoing; Thu, 8 Nov 2001 14:40:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from glatton.xo.com ([207.155.248.78]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8Jegl16283 for ; Thu, 8 Nov 2001 14:40:42 -0500 (EST) Received: from blekinge ([66.89.96.162]) by glatton.xo.com id OAA00346; Thu, 8 Nov 2001 14:40:28 -0500 (EST) [ConcentricHost SMTP Relay 1.14] Message-ID: <006f01c1688d$1c7c2520$1001010a@blekinge> From: "Peter Mellquist" To: , "IPS" Cc: "IPS" References: <020001c16721$536f4f70$1001010a@blekinge> <200111062227010305.0AE8F00C@opulentsystems.com> Subject: Re: iSCSI over TLS Date: Thu, 8 Nov 2001 11:39:40 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I am not advocating having a TOE implement TLS. I would count on the TOE to provide general purpose TCP/IP services and offload the host of these services. Security with TLS would be provided at a higher layer which could or could not use hardware acceleration. For example, my iSCSI target server uses a TLS implementation which in turn would interfaces with a TOE. The TLS implementation uses ECC ( Elliptical Curve Crypto as the default cipher suite) in which the ECC engine is provided either in software or hardware. By doing so, I get flexibility to include any type of security. I can also deal with all the export legalities better. I agree that TOEs will also exist to take over everything up to layer 4. That's fine. This will be applicable to some but not all applications. I would really like the iSCSI standard to be flexible in the area of security rather than have too much biased toward layer 4 TOEs. -peter ----- Original Message ----- From: "Sukanta Ganguly" To: "Peter Mellquist" ; "IPS" Sent: Tuesday, November 06, 2001 10:27 PM Subject: Re: iSCSI over TLS > Peter, > A very good point. I am not sure if the TOE vendors have plans of implementing IPSec and/or TLS. But allowing TLS as another mechanism is also going to increase the complexity on the TOE side. The more logic that is applied to the TOE the more expensive and difficult it is going to get. > The TOE vendors take over the packet processing at layer 4 and hence is already fairly restrictive scale-wise. Adding TLS will make it more difficult. However, a good mix of TLS on software and a synergistic TOE can make a good combination. Hence I like the idea. I am not sure if any TOE vendors have any comment of this ??? > > > SG > > *********** REPLY SEPARATOR *********** > > On 11/6/2001 at 4:15 PM Peter Mellquist wrote: > > >I am aware that the ips group is leaning toward IPSEC as for the security > >solution but I am interested if anyone is also considering using Transport > >Layer Security (TLS)? > > > >I am concerned that the requirement for IPSEC might make TOEs more complex > >than they need to be. Can TLS be optionally used as well as defined by the > >specification? This could allow TOE vendors to only be concerned with > >providing normal IPv4 / ipv6 and leave the security to a higher layer. A > >TLS > >stack sitting above the TOE could then handle security very well. Also, I > >anticipate that the first generation of TOEs will not support IPSEC. With a > >iSCSI/TLS we could enable security solutions with the first generation of > >TOEs and get speed and security. > > > >Are any TOE vendors planning to support IPSEC? > > > >Can TLS or IPSEC be supported? > > > >-peter > > > > > > > >Peter Mellquist > >Seven Systems Technologies > >575 Menlo Drive Suite 2 > >Rocklin CA > >916-577-1275 > >peterm@seven-systems.com > > > > From owner-ips@ece.cmu.edu Thu Nov 8 15:23:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA05961 for ; Thu, 8 Nov 2001 15:23:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8JuQE17636 for ips-outgoing; Thu, 8 Nov 2001 14:56:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8JuNl17631 for ; Thu, 8 Nov 2001 14:56:23 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id OAA19996; Thu, 8 Nov 2001 14:56:15 -0500 (EST) Date: Thu, 8 Nov 2001 14:56:15 -0500 From: "Robert D. Russell" To: Julian Satran cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: On Thu, 8 Nov 2001, Julian Satran wrote: > Robert, > > We are pursuing a very impractical track. Show me please one SINGLE > advantage for OOO shipping. > > Julo > In your message to Mallikarjun, you stressed the need for the "no deadlock" mechanism and claimed that it was not understood. I am trying to understand it and the example you gave seems to me to be wrong. Please explain. Thanks, Bob > > > > Mallikarjun, > > > > I did not see a SINGLE performance improvement that results from OOO > > shipping. > > I would be bad engineering to give away the "no-deadlock" mechanism we > > have now for nothing. > > I have also the impression that the point about deadlock that I keep > > repeating is ignored or not understood. > > > > "Robert D. Russell" > Sent by: owner-ips@ece.cmu.edu > 08-11-01 17:50 > Please respond to "Robert D. Russell" > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > Julian: > > > However allowing initiators to ship them out of order creates a > > potential deadlock that does not appear otherwise. > > > > If a target is missing a command in a queue (and there are no errors) > then > > this command is bound to be first on some connection under the current > set > > of rules. > > > > If we allow OOO shipping then the missing command can be somewhere > > "inside" the window on some connection and if the target has just filled > > > his queue and has room in the staging buffer only for the command it is > > waiting for and that command happens to be the first to pass to SCSI you > > > have a deadlock. > > > I must be misunderstanding something. > > Your example is certainly correct if the target has no control > over the number of commands sent to it by the initiator. > But the target does control this number through the MaxCmdSN field. > For the scenario you are describing to occur, wouldn't it be > necessary for the target to advertize a MaxCmdSN value bigger than > it actually has resources to handle? > > It seems to me that if a target can only handle x new commands, > then its queueing capacity is x, and in the SCSI Response PDU it > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > formula in section 2.2.2.1. This in turn controls the number of > commands the initiator can send, and thus prevents the incoming > commands from overfilling the target's queue. > > So isn't the deadlock caused by a broken target? > Isn't the target advertizing a queueing capacity greater than it > actually has and isn't that the cause of the deadlock? > > An alternative explanation of the deadlock might be that the > target is advertizing the correct MaxCmdSN, but the initiator is > sending commands beyond what it is allowed to send. > > However, in this case the target should silently ignore any > non-immediate command outside the allowed range. > For the deadlock to occur, wouldn't the target have to be queuing > commands outside the allowed range instead of ignoring them? > > So in this case too, the target is broken and that is what causes > the deadlock. > > Where am I going wrong in this reasoning? > > Thanks, > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > > From owner-ips@ece.cmu.edu Thu Nov 8 15:24:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA05998 for ; Thu, 8 Nov 2001 15:24:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8K1vE18158 for ips-outgoing; Thu, 8 Nov 2001 15:01:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8K1tl18154 for ; Thu, 8 Nov 2001 15:01:55 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA8K1nu17396 for ; Thu, 8 Nov 2001 12:01:49 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA8K1h624409 for ; Thu, 8 Nov 2001 12:01:43 -0800 From: "Bill Strahm" To: Subject: RE: FC Management MIB - proposed changes Date: Thu, 8 Nov 2001 12:01:37 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <200111081842.KAA24981@cisco.com> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Keith, On the topic of Simple Name Service, would it be possible to make this generic enough to work within the iSNS MIB. I would like to see the iSNS MIB be able to cover both iSCSI and FC if it is at all possible. Any reasons this can't be done ? Does the iSNS editors want to try to incorporate this into their design (how hard would it be, is it even possible ?) Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Keith McCloghrie Sent: Thursday, November 08, 2001 10:43 AM To: ips@ece.cmu.edu Cc: Keith McCloghrie Subject: FC Management MIB - proposed changes 5. Regarding the the MIB objects for the "Simple Name Service", I see two possible solutions: i. retain the MIB objects but focus them on GS-3's Unzoned Name Service. ii. remove the MIB objects for the "Simple Name Service" from this MIB. If there is WG consensus that a MIB is needed for one of the GS-3 Name Services, and for which one, then the appropriate set of MIB objects can be defined in a new MIB. Of these two, I propose to investigate solution i), and if it proves feasible, then to adopt it; if not, to fall back to solution ii). Keith. From owner-ips@ece.cmu.edu Thu Nov 8 15:24:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA06011 for ; Thu, 8 Nov 2001 15:24:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8JiKC16590 for ips-outgoing; Thu, 8 Nov 2001 14:44:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8JiIl16584 for ; Thu, 8 Nov 2001 14:44:18 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id LAA14240; Thu, 8 Nov 2001 11:44:10 -0800 (PST) From: Keith McCloghrie Message-Id: <200111081944.LAA14240@cisco.com> Subject: Re: FC Management MIB - proposed changes To: predrag_spasic@hp.com ("SPASIC,PREDRAG (HP-Roseville,ex1)") Date: Thu, 8 Nov 2001 11:44:10 -0800 (PST) Cc: kzm@cisco.com ('Keith McCloghrie'), ips@ece.cmu.edu In-Reply-To: <028FE7141C79D511B65100D0B74FE875BCA5C6@xrose01.rose.hp.com> from "SPASIC,PREDRAG (HP-Roseville,ex1)" at Nov 08, 2001 10:59:40 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Predrag, > Point 8: > Although there are quite a few issues around zoning management, > most of them revolving around perceived inadequate security of > SNMP, I assume you mean the inadequate security of SNMPv1 and SNMPv2c, i.e., nobody perceives that SNMPv3 is insecure, right ?? > without this information MIB will not be complete. Are you saying that this MIB will not be complete, or that Fibre Channel management will not be complete ? > Zoning management is extensively defined in FC-GS3, is specific > to FC, and I do not see any reasons to delay this for some point > in the future. I see several options: 1. have the first version of the new FC-Mgmt MIB include Zoning, 2. start work on another MIB to address Zoning in parallel with working on the first version of the new FC-Mgmt MIB. 3. complete the first version (Internet-Draft) of the new FC-Mgmt MIB, and defer the addition of Zoning until a second or subsequent Internet-Draft of this MIB, but before its WG Last Call. 4. defer the start of work on another MIB to address Zoning until after completing the first version (Internet-Draft) of the new FC-Mgmt MIB. 5. finish the new FC-Mgmt MIB, including having it published as an RFC, and only then work on extending it to address Zoning. I'm not proposing #5 (as you may have feared). Rather, I'm proposing that we not do #1 - specifically, that producing the first version of the new FC-Mgmt MIB is a large enough task, that we should do it first before tackling Zoning. I also think #2 would require coodination between something new and something which is changing, and therefore would be more work/more difficult than is warranted/necessary. So, I'm proposing either #3 or #4; at this point, I don't have a feel for which would be better. What do you think ? Keith. > Regards > > Predrag Spasic, > Hewlett Packard > Details: https://ecardfile.com/id/predrag_spasic > > > > -----Original Message----- > > From: Keith McCloghrie [mailto:kzm@cisco.com] > > Sent: Thursday, November 08, 2001 10:43 AM > > To: ips@ece.cmu.edu > > Cc: kzm@cisco.com > > Subject: FC Management MIB - proposed changes > > > > > > > > Based on the issues that I listed in my previous message > > (yesterday), I > > have a set of changes that I'd like to make to the FC-Mgmt MIB, > > providing there are no objections from the WG. These changes (the > > first six correspond to the issues list) are: > > > > 1. MIB will be written using SMIv2. > > > > 2. All objects which apply in non-Fibre Channel environments will be > > removed. (Note that this applies to a large fraction of the objects > > defined in this MIB.) > > > > 2a. For those objects which will be removed, if there are any which > > are: a) not already covered in other MIBs, and b) considered essential > > to the management of Fibre Channel-based products, then it will be > > necessary to consider whether other MIB(s) need to be modified/created > > as a home for them, and if so which WG(s) should undertake such work. > > > > 3. With 20x20 hindsight, the following observations can be made with > > respect to RFC 2837: > > > > - the reason that we now have the issue of how it relates to the > > FC-Mgmt MIB is because it was written as a Fabric Element MIB; > > this issue would not have arisen if it had been written as a > > Fibre Channel interface MIB. > > - it should have extended the ifTable, rather than > > overlapping with it, > > - it should not have defined the fcFeModuleTable, which overlaps with > > the Entity MIB. > > - and it should have specified (at least) its octet counters > > as Counter64. > > > > Given these problems, I propose that the new FC-Mgmt MIB be specified > > as a replacement for RFC 2837. > > > > 4. This MIB will include a media-specific MIB to specify how Fibre > > Channel interfaces use the ifTable (see section 4 in RFC 2863). This > > will result in many tables in this MIB being indexed by ifIndex. > > > > 5. Regarding the the MIB objects for the "Simple Name Service", > > I see two possible solutions: > > > > i. retain the MIB objects but focus them on GS-3's Unzoned > > Name Service. > > > > ii. remove the MIB objects for the "Simple Name Service" from > > this MIB. > > If there is WG consensus that a MIB is needed for one of > > the GS-3 Name > > Services, and for which one, then the appropriate set of > > MIB objects > > can be defined in a new MIB. > > > > Of these two, I propose to investigate solution i), and if it proves > > feasible, then to adopt it; if not, to fall back to solution ii). > > > > 6. The Counter32 or Counter64 syntax will be used for counters. > > > > 7. In addition to the above, it has been suggested that MIB objects to > > support Class 1 are no longer needed. If I don't hear any objections, > > I will assume that I can make this change also. > > > > 8. Yesterday, John Nguyen (jnguyen@gadzoox.com) asked in an > > email about > > "bringing ZONE into FC Management MIB". His suggestion would seem to > > be in addition to, rather than as a replacement for, anything > > which will > > be in the new FC-Mgmt MIB. Thus, at this point in time, I'd like to > > suggest that the changes listed above represent a large enough amount > > of work for us to chew on. Therefore, I propose that our answer to > > John's query is to ask him to re-raise this issue at a point in the > > future when the new FC-Mgmt MIB is becoming more stable. > > > > 9. Are there any other changes that anyone would like to see at this > > time ? > > > > Also note that with this MIB now being in a different WG, the > > I-D needs > > to have a draft-ietf-ips-xxx name. I propose to use > > draft-ietf-ips-fcmgmt-mib-00.txt. > > > > Keith. > > > From owner-ips@ece.cmu.edu Thu Nov 8 15:25:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA06042 for ; Thu, 8 Nov 2001 15:25:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8Jker16785 for ips-outgoing; Thu, 8 Nov 2001 14:46:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (mars.iol.unh.edu [132.177.121.222]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8Jkdl16779 for ; Thu, 8 Nov 2001 14:46:39 -0500 (EST) Received: from localhost (rdr@localhost) by iol.unh.edu (8.9.3/8.9.3) with ESMTP id OAA19752; Thu, 8 Nov 2001 14:46:21 -0500 (EST) Date: Thu, 8 Nov 2001 14:46:20 -0500 From: "Robert D. Russell" To: Somesh Gupta cc: Julian Satran , ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh: > > It seems to me that if a target can only handle x new commands, > > then its queueing capacity is x, and in the SCSI Response PDU it > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > formula in section 2.2.2.1. This in turn controls the number of > > commands the initiator can send, and thus prevents the incoming > > commands from overfilling the target's queue. > > This is actually a weakness of the multiple connections over > multiple adapters model (it is not related to ordering). > The target advertises a command window on a session wide basis. > At the same time, it has to provide the resource to the adapter > to be able to DMA those commands to. Since there is no restriction > on which connection the commands may be received, either the > target has to allocate the max resources needed to each adapter > (thus committing n times the resources actually required), or > has to "lie" (it would not be a complete lie) which could > result in running out of resources. One way to fix that would be > to have the credit on a per connection basis. If I understand you correctly, you are saying that deadlock can occur even if we enforce ordered delivery by initiators and even if the advertisements are correct and even if both parties obey the advertisements. In other words, doesn't this mean that the standard can lead to a trap in implementing targets and that, in fact, the advertisements really should be on a per connection basis? However, what would be the consequences for initiators if the advertisements were on a per-connection basis? Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Thu Nov 8 16:00:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA07540 for ; Thu, 8 Nov 2001 16:00:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8KKlR19625 for ips-outgoing; Thu, 8 Nov 2001 15:20:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8KKjl19620 for ; Thu, 8 Nov 2001 15:20:45 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id MAA24302; Thu, 8 Nov 2001 12:20:33 -0800 (PST) From: Keith McCloghrie Message-Id: <200111082020.MAA24302@cisco.com> Subject: Re: FC Management MIB - proposed changes To: bill@sanera.net (Bill Strahm) Date: Thu, 8 Nov 2001 12:20:33 -0800 (PST) Cc: ips@ece.cmu.edu In-Reply-To: from "Bill Strahm" at Nov 08, 2001 12:01:37 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit If the iSNS editors think they can do this, then I think it's a good idea. Keith. > Keith, > > On the topic of Simple Name Service, would it be possible to make this > generic enough to work within the iSNS MIB. I would like to see the iSNS > MIB be able to cover both iSCSI and FC if it is at all possible. Any > reasons this can't be done ? Does the iSNS editors want to try to > incorporate this into their design (how hard would it be, is it even > possible ?) > > Bill > +========+=========+=========+=========+=========+=========+=========+ > Bill Strahm Software Development is a race between Programmers > Member of the trying to build bigger and better idiot proof software > Technical Staff and the Universe trying to produce bigger and better > bill@sanera.net idiots. > (503) 601-0263 So far the Universe is winning --- Rich Cook > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Keith McCloghrie > Sent: Thursday, November 08, 2001 10:43 AM > To: ips@ece.cmu.edu > Cc: Keith McCloghrie > Subject: FC Management MIB - proposed changes > > > > 5. Regarding the the MIB objects for the "Simple Name Service", > I see two possible solutions: > > i. retain the MIB objects but focus them on GS-3's Unzoned Name Service. > > ii. remove the MIB objects for the "Simple Name Service" from this MIB. > If there is WG consensus that a MIB is needed for one of the GS-3 Name > Services, and for which one, then the appropriate set of MIB objects > can be defined in a new MIB. > > Of these two, I propose to investigate solution i), and if it proves > feasible, then to adopt it; if not, to fall back to solution ii). > > > > Keith. > > From owner-ips@ece.cmu.edu Thu Nov 8 16:09:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA07858 for ; Thu, 8 Nov 2001 16:09:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8KhG821413 for ips-outgoing; Thu, 8 Nov 2001 15:43:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas2.cos.agilent.com (msgbas2x.cos.agilent.com [192.25.240.37]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8KhDl21409 for ; Thu, 8 Nov 2001 15:43:14 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas2.cos.agilent.com (Postfix) with ESMTP id 8863D332 for ; Thu, 8 Nov 2001 13:43:12 -0700 (MST) Received: from acropora.rose.agilent.com (acropora.rose.agilent.com [130.30.179.5]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 0A0B7120 for ; Thu, 8 Nov 2001 13:43:12 -0700 (MST) Received: (from dbs@localhost) by acropora.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) id MAA25522 for ips@ece.cmu.edu; Thu, 8 Nov 2001 12:42:06 -0800 (PST) From: Dave Sheehy Message-Id: <200111082042.MAA25522@acropora.rose.agilent.com> Subject: Re: iSCSI over TLS To: ips@ece.cmu.edu (IETF IP SAN Reflector) Date: Thu, 08 Nov 2001 12:42:04 PST X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, > A group of us seriously considered TLS. The main reason for dropping it > was that it would interfere with any mechanism we could think of doing > framing and steering and we thought that framing and steering are > essential at 10Gbps and over. If framing and steering "are essential" (your words) then why is framing not a MUST in the spec? And why are so many implementers stating (or hinting) that they are NOT going to implement it? I think there is a major disconnect here. iSCSI w/o framing or markers is dead in the water IMHO. Dave Sheehy From owner-ips@ece.cmu.edu Thu Nov 8 16:10:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA07903 for ; Thu, 8 Nov 2001 16:10:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8K1Ms18114 for ips-outgoing; Thu, 8 Nov 2001 15:01:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.bld.us.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8K1Il18109 for ; Thu, 8 Nov 2001 15:01:19 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e34.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA78012 for ; Thu, 8 Nov 2001 14:58:41 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA8K006129170 for ; Thu, 8 Nov 2001 13:00:00 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: resend of diagram from 2.5.1 To: "Andre Asselin" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 8 Nov 2001 11:59:06 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/08/2001 01:00:00 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Andre, I think it would be more correct if the Session block was included within the SCSI Port Block. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Andre Asselin 11/07/2001 08:27 AM To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: Re: iSCSI: resend of diagram from 2.5.1 (Document link: John Hufferd) John, How's this look? ----------------------------IP Network--------------------- | | | +--------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +-----------|-----------------+ | | | | | Portal Group 1 | | Portal Group 2 | | | | | | | | | | | | | | SCSI Target Port | | SCSI Target Port | | | | | | (iSCSI Target Name + | | (iSCSI Target Name + | | | | | | 't' + 1) | | 't' + 2) | | | | | +--------------------------+ +-----------------------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session | | iSCSI Session | | | | | | | | | | | | | | (iSCSI Initiator Name + | | (iSCSI Initiator Name + | | | | | | ISID + iSCSI Target Name + | | ISID + iSCSI Target Name + | | | | | | Portal Group ID) | | Portal Group ID) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-----------------------------------------------------------------+ | | | | Network Entity | +---------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC John Hufferd To: Andre Asselin/Raleigh/IBM 11/06/2001 cc: ips@ece.cmu.edu 03:33 PM From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: iSCSI: resend of diagram from 2.5.1(Document link: Andre Asselin) Andre, Now that I can see the picture that you sent. It probably now needs to have the Names of the Target Side SCSI Ports clearly called out. The picture might imply to some folks that the SCSI Port Name is associated with "iSCSI Name + TSID" where as the SCSI Port Name should be "iSCSI Name + PortalGroupTag". The picture is, I think, technically correct, the Target End of the Session can be associated with iSCSI Name+TSID, but that may cause others to go through the same set of discussions that we have held recently, and they might think that the SCSI Port Name includes the TSID. I wonder if it would be clearer to Align, in the pictures, the concept of Target End of a Session with SCSI Port Name and give it the identification of "iSCSI Name + PortalGroupTag" and leave TSID to the various text in the Draft and for it be understood as just a Handle which the implementation can use in any way it wants. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Andre Asselin 11/06/2001 07:38 AM To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: iSCSI: resend of diagram from 2.5.1 It looks like my original post with an updated diagram for section 2.5.1 got wrapped badly. Here's a second attempt that hopefully will make it through. (original) ----------------------------IP Network--------------------- | | | +--------|---------------|--------------------|---------------------+ | +----|---------------|-----+ +----|---------+ | | | +---------+ +---------+ | | +---------+ | | | | | Network | | Network | | | | Network | | | | | | Portal | | Portal | | | | Portal | | | | | +--|------+ +---------+ | | +---------+ | | | | | | | | | | | | | | Portal | | | | Portal | | | | | Group 1 | | | | Group 2 | | | +--------------------------+ +--------------+ | | | | | | | +----------------------------+ +-----------------------------+ | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | +----------------------------+ +-----------------------------+ | | | | iSCSI Target Node | | (within Network Entity, not shown) | +-------------------------------------------------------------------+ (updated) ----------------------------IP Network--------------------- | | | +--------|--------------|-----------------------|---------------------+ | | | | | | +----|----+ +----|----+ +----|----+ | | | Network | | Network | | Network | | | | Portal | | Portal | | Portal | | | +---------+ +---------+ +---------+ | | | | | | | +------|--------------|-----------------------|-------------------+ | | | +----|--------------|------+ +------|-------+ | | | | | Portal | | Portal | | | | | | Group 1 | | Group 2 | | | | | +--------------------------+ +--------------+ | | | | | | | | | | +----------------------------+ +-----------------------------+ | | | | | iSCSI Session (Target side)| | iSCSI Session (Target side) | | | | | | | | | | | | | | (iSCSI Name + TSID=2) | | (iSCSI Name + TSID=1) | | | | | +----------------------------+ +-----------------------------+ | | | | | | | | iSCSI Target Node | | | +-----------------------------------------------------------------+ | | | | Network Entity | +---------------------------------------------------------------------+ Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC From owner-ips@ece.cmu.edu Thu Nov 8 16:33:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08531 for ; Thu, 8 Nov 2001 16:33:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8L5dM23122 for ips-outgoing; Thu, 8 Nov 2001 16:05:39 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8L5cl23118 for ; Thu, 8 Nov 2001 16:05:38 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Nov 2001 16:05:33 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71ECAD8FC@CORPMX14> From: Black_David@emc.com To: dbs@acropora.rose.agilent.com, ips@ece.cmu.edu Subject: iSCSI and framing Date: Thu, 8 Nov 2001 15:55:57 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > If framing and steering "are essential" (your words) then why is framing > not a MUST in the spec? And why are so many implementers stating (or hinting) > that they are NOT going to implement it? I think there is a major disconnect > are. iSCSI w/o framing or markers is dead in the water IMHO. Because WG consensus is distinctly absent on this point. We tried to get consensus on this in London and failed twice. I consider that text to be an open issue at the moment. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Nov 8 16:52:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08940 for ; Thu, 8 Nov 2001 16:52:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8LKaE24325 for ips-outgoing; Thu, 8 Nov 2001 16:20:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8LKZl24321 for ; Thu, 8 Nov 2001 16:20:35 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Nov 2001 16:22:54 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71ECAD8FE@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: IPS anti-spam filter Date: Thu, 8 Nov 2001 16:10:54 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, Since a couple of people have bumped into this in the past few weeks, I should point out that the IPS mailing list uses an anti-spam filter that requires that posts to the list come from subscribed addresses. Those that don't are bit-bucketed in order to avoid generating email in response to spam. The filter had to be put in after we started getting bombarded with spam a while back and has been very effective. If you post from an address that's not subscribed to the list and don't see your post appear, this is why. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Thu Nov 8 16:55:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08997 for ; Thu, 8 Nov 2001 16:55:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8L1Af22760 for ips-outgoing; Thu, 8 Nov 2001 16:01:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8L18l22755 for ; Thu, 8 Nov 2001 16:01:08 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Nov 2001 16:01:03 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71ECAD8FB@CORPMX14> From: Black_David@emc.com To: kzm@cisco.com, ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Date: Thu, 8 Nov 2001 15:51:23 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, For simplicity, I would propose keeping the initial revision focused on structural and format changes only - i.e., make no functional changes that aren't required to bring the MIB into line with the overall architecture of MIBs, use of SMIv2, and the like. Comments on this, keyed to Keith's issue numbers: 1. The trap table doesn't match up with RFC 2573 - I presume correcting this is part of the conversion to SMIv2. 3. Let's keep the replacement of RFC 2837 as (a) a proposal and (b) Keith's recommendation to the WG as MIB Advisor, but not formally adopt that approach until we have a version of the MIB draft that would replace RFC 2837 available for review by the WG. 5. I think there's a misunderstanding here. There is only one name service in any FC fabric, period. The term "Simple Name Service" is historical and refers to the current approach to Fibre Channel fabric name service by contrast to the originally-proposed X.500 directory-based approach (see the original FC-GS) - the meaning of "Simple" should now be obvious, as it's by comparison to X.500 ;-). Both the Zoned and Unzoned name services are simple name services. The same objects can be used to represent both, as only one is active at any time, and the communication interfaces/protocols are identical. Client access to the name service is the same in both cases; zoned vs. unzoned only affects server behavior in terms of what is returned to a query. In addition the name service objects are described to represent only entities "known to this unit". So, in a zoned fabric, I would expect the table in a switch to be completely populated, but the table in an HBA to contain only the entries in that HBA's zone because the HBA doesn't "know" about any others. The name server objects need to be retained - these are quite important for fabric management visibility. Whether the fabric is zoned or not and how the nameserver behaves can be determined from the zone objects (i.e., for a switch, the nameserver objects contain everything and the zone objects have to be consulted to figure out what a query from a particular client to the nameserver will return). 7. I would definitely take out Class 1 support, and reference FC-MI (which prohibits Class 1 service) as an authority for doing so. 8. For the initial version, I would only carry forward the zone objects existing in the current MIB and not define any new functionality - that can be done in revisions after -00. 9. I'd prefer that the -00 version contain no additional functional changes beyond those required to support the necessary structure and format changes. Once we have that -00 baseline, functional changes can be made from there. 2, 4, and 6 look fine to me, as does the new draft name. Comments? Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Keith McCloghrie [mailto:kzm@cisco.com] > Sent: Thursday, November 08, 2001 1:43 PM > To: ips@ece.cmu.edu > Cc: kzm@cisco.com > Subject: FC Management MIB - proposed changes > > > > Based on the issues that I listed in my previous message > (yesterday), I > have a set of changes that I'd like to make to the FC-Mgmt MIB, > providing there are no objections from the WG. These changes (the > first six correspond to the issues list) are: > > 1. MIB will be written using SMIv2. > > 2. All objects which apply in non-Fibre Channel environments will be > removed. (Note that this applies to a large fraction of the objects > defined in this MIB.) > > 2a. For those objects which will be removed, if there are any which > are: a) not already covered in other MIBs, and b) considered essential > to the management of Fibre Channel-based products, then it will be > necessary to consider whether other MIB(s) need to be modified/created > as a home for them, and if so which WG(s) should undertake such work. > > 3. With 20x20 hindsight, the following observations can be made with > respect to RFC 2837: > > - the reason that we now have the issue of how it relates to the > FC-Mgmt MIB is because it was written as a Fabric Element MIB; > this issue would not have arisen if it had been written as a > Fibre Channel interface MIB. > - it should have extended the ifTable, rather than > overlapping with it, > - it should not have defined the fcFeModuleTable, which overlaps with > the Entity MIB. > - and it should have specified (at least) its octet counters > as Counter64. > > Given these problems, I propose that the new FC-Mgmt MIB be specified > as a replacement for RFC 2837. > > 4. This MIB will include a media-specific MIB to specify how Fibre > Channel interfaces use the ifTable (see section 4 in RFC 2863). This > will result in many tables in this MIB being indexed by ifIndex. > > 5. Regarding the the MIB objects for the "Simple Name Service", > I see two possible solutions: > > i. retain the MIB objects but focus them on GS-3's Unzoned > Name Service. > > ii. remove the MIB objects for the "Simple Name Service" from > this MIB. > If there is WG consensus that a MIB is needed for one of > the GS-3 Name > Services, and for which one, then the appropriate set of > MIB objects > can be defined in a new MIB. > > Of these two, I propose to investigate solution i), and if it proves > feasible, then to adopt it; if not, to fall back to solution ii). > > 6. The Counter32 or Counter64 syntax will be used for counters. > > 7. In addition to the above, it has been suggested that MIB objects to > support Class 1 are no longer needed. If I don't hear any objections, > I will assume that I can make this change also. > > 8. Yesterday, John Nguyen (jnguyen@gadzoox.com) asked in an > email about > "bringing ZONE into FC Management MIB". His suggestion would seem to > be in addition to, rather than as a replacement for, anything > which will > be in the new FC-Mgmt MIB. Thus, at this point in time, I'd like to > suggest that the changes listed above represent a large enough amount > of work for us to chew on. Therefore, I propose that our answer to > John's query is to ask him to re-raise this issue at a point in the > future when the new FC-Mgmt MIB is becoming more stable. > > 9. Are there any other changes that anyone would like to see at this > time ? > > Also note that with this MIB now being in a different WG, the > I-D needs > to have a draft-ietf-ips-xxx name. I propose to use > draft-ietf-ips-fcmgmt-mib-00.txt. > > Keith. > From owner-ips@ece.cmu.edu Thu Nov 8 17:52:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA10806 for ; Thu, 8 Nov 2001 17:52:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8MPtb29672 for ips-outgoing; Thu, 8 Nov 2001 17:25:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8MPrl29667 for ; Thu, 8 Nov 2001 17:25:53 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id RAA26198; Thu, 8 Nov 2001 17:25:19 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "Robert D. Russell" Cc: "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 14:22:30 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Comments below > -----Original Message----- > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > Sent: Thursday, November 08, 2001 11:46 AM > To: Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > Somesh: > > > > It seems to me that if a target can only handle x new commands, > > > then its queueing capacity is x, and in the SCSI Response PDU it > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > > formula in section 2.2.2.1. This in turn controls the number of > > > commands the initiator can send, and thus prevents the incoming > > > commands from overfilling the target's queue. > > > > This is actually a weakness of the multiple connections over > > multiple adapters model (it is not related to ordering). > > The target advertises a command window on a session wide basis. > > At the same time, it has to provide the resource to the adapter > > to be able to DMA those commands to. Since there is no restriction > > on which connection the commands may be received, either the > > target has to allocate the max resources needed to each adapter > > (thus committing n times the resources actually required), or > > has to "lie" (it would not be a complete lie) which could > > result in running out of resources. One way to fix that would be > > to have the credit on a per connection basis. > > If I understand you correctly, you are saying that deadlock can > occur even if we enforce ordered delivery by initiators and even > if the advertisements are correct and even if both parties obey > the advertisements. I would let Julian enlighten us on that, or whether it just leads to wholesale command drops and retries, or TASK SET FULL is returned (is TASK SET FULL a valid response in this case for a LINKED task when it is not the first command?) > > In other words, doesn't this mean that the standard can lead to a > trap in implementing targets and that, in fact, the advertisements > really should be on a per connection basis? > However, what would be the consequences for initiators if > the advertisements were on a per-connection basis? There is really no issue with adveritisement on a per connection basis. It leads to a protocol change. However, it also allows a much more performant flow of commands. > > > Thanks, > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > From owner-ips@ece.cmu.edu Thu Nov 8 17:58:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA10877 for ; Thu, 8 Nov 2001 17:58:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8LffZ26114 for ips-outgoing; Thu, 8 Nov 2001 16:41:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8Lfal26104 for ; Thu, 8 Nov 2001 16:41:36 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA199046; Thu, 8 Nov 2001 16:38:47 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA8Lf86100952; Thu, 8 Nov 2001 14:41:22 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: Out of order commands To: cbm@rose.hp.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 8 Nov 2001 13:40:14 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/08/2001 02:41:21 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, Could you comment on the concept of OOO on the ErrorRecoveryLevel>0. I had thought that "in order delivery" was part of the detection of missing PDUs and needed for timely Recovery. I was wondering if this changes the way we would use the ExpCmdSN, etc. I think your opinions on this part of the OOO discussion would be valuable. For example, how would you contrast the differences in detecting a problem and recovering from that problem etc., today vrs the OOO approach (if any). . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Mallikarjun C." @ece.cmu.edu on 11/07/2001 09:41:05 AM Please respond to cbm@rose.hp.com Sent by: owner-ips@ece.cmu.edu To: Santosh Rao , ips@ece.cmu.edu cc: Subject: Re: iSCSI: Out of order commands Santosh, I have only one comment on your responses. > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. I made two convenient assumptions if you noticed, :-), one of which is that target forces session recovery on *any* error that it sees (ErrorRecoveryLevel=0) - including a dropped command due to a digest error. With that assumption, a target can afford not to implement a scoreboard. As I said in a private note, I guess what primarily bothers me about OOO commands on a connection is that it requires the receiver to undo this "optimization" on its end - most notably on a single connection. TCP experts may comment on how/if they dealt with a similar issue. OTOH, you had some valid comments on exceptions to ordering during connection recovery. Perhaps we can move on by making Julian's proposed stipulation a SHOULD.... -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Santosh Rao wrote: > > Mallikarjun, > > Some comments below. > > Regards, > Santosh > > "Mallikarjun C." wrote: > > > > Rod and Julian, > > > > This has been an interesting thread of discussion. Some > > comments - > > > > 1.My first reaction was - allowing out-of-order command > > transmission on the same connection deprives targets of > > an implementation choice. Targets which support only > > single-connection sessions and only support session > > recovery (reasonable assumptions in my mind) can no > > longer afford *not to* implement a command scoreboard. > > Even a single connection target *MUST* implement a scoreboard. The > reason being that it can see out-of-order arrival of commands due to > commands being dropped on digest errors. In such a case, it must block > further command processing until holes are filled. > > Thus, there is no getting away from implementing a sequencer at the > target. Given this, I think it is unreasonable to restrict initiator > implementation flexibility by imposing a strict ordering requirement > within the connection. > > > 2.Any end-node efficiency that is sought to be achieved > > by transmitting CmdSNs out-of-order from the initiator > > would be lost on the other end-node, since the target > > now must wait for re-ordering the commands. > > It has to handle this situation anyway to deal with holes caused by > digest errors. This scenario occurs even with initiators that issue > commands in order. > > > > > 3.The flipside is that out-of-order transmission saves > > link badwidth (albeit at the expense of end-node efficiency), > > compared to idling the link waiting for outbound DMA. > > We have to determine if this is a reasonable trade-off. > > > > 4.I can see Rod's point that prefetching all immediate > > data can be a burden on the NIC resources. But, two > > questions - > > - could the NIC not use unsolicited separate data > > PDUs in these cases? [ I realize that InitialR2T > > has to be "no" to let it happen... ] > > - could the NIC have a memory architecture that > > allows data prefetching for the next command (so > > this is a non-issue from the protocol perspective)? > > This scheme incurs one DMA delay for every new > > burst of commands. > > > > 5.Another (perhaps radical at this point) option is to do > > away with immediate unsolicited data, to stick only with > > separate unsolicited data. I would personally be okay > > with the choice, particularly if this feature (that > > helps software implementations) starts making hardware > > design complicated/expensive. > > > > So, to summarize - > > > > option immediate allow > > data in spec? out-of-order? > > > > (A) (5) above no no > > (B) No real reason to do this. no yes > > (C) (4) above yes no > > (D) pros & cons (1), (2) & (3) yes yes > > > > >From the arguments I heard so far, I am leaning towards > > option A, and option C in that order. > > > > Comments? > > -- > > Mallikarjun > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > Rod Harrison wrote: > > > > > > Julian, > > > > > > I don't understand what you are proposing here, what do you mean by > > > "multiplexed" DMA? > > > > > > The problem is that the DMAs take some time, the more there are > > > queued the longer the last DMAs queued take to complete. Some commands > > > require DMAs to complete before they can be sent, i.e. Writes with > > > immediate data, some commands do not, i.e. Reads and writes with no > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > soon a possible, which for a read after a write can be before the > > > write's DMA has completed. Maintaining an ordered queue for commands > > > to be sent on the HBA is expensive and redundant since the target > > > already knows how to queue commands before committing them to its SCSI > > > layer. > > > > > > The iSCSI HBA and its host driver are not at liberty to change the > > > order of commands from the OS, but the DMAs those commands need are > > > unlikely to complete in the same order, and as I mentioned some > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > order it has to maintain an ordered queue of commands waiting to be > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > immediate data almost impossible to support. > > > > > > I don't see the problem with allowing out of order commands given > > > that the target already has to deal with very similar problems. I > > > think we are getting in to the area of implementation choices here, > > > which is inappropriate for a specification. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest > > > > > > Rod, > > > > > > I don't see any reason why DMA operations cant be "multiplexed" with > > > commands. > > > If you have scheduled a long outbound DMA you are doomed regardless of > > > the > > > command ordering. > > > And if you have scheduled DMA operations piecemeal then you can insert > > > your commands in correct order. > > > > > > Julo > > > > > > "Rod Harrison" > > > 05-11-01 20:48 > > > Please respond to "Rod Harrison" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > cc: > > > Subject: iSCSI: Out of order commands, was current UNH > > > Plugfest > > > > > > [ Subject changed ] > > > > > > Julian, > > > > > > The ordering difference is introduced between the > > > host > > > side driver > > > and the iSCSI HBA. The host side driver must present SCSI commands to > > > the HBA in the order they are received from the OS to prevent read > > > after write dependency failures. The HBA might reorder the commands > > > depending on when DMA completes. The reordering can't be done ahead of > > > time in the host driver since it doesn't know how long each DMA might > > > take. As long as the HBA assigns CmdSN in the order it receives > > > commands the desired host ordering is preserved. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Julian Satran > > > Sent: Monday, November 05, 2001 12:35 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Rod, > > > > > > I all examples give the point I find hard to understand is why is the > > > ordering on the wire different from the presentation order to the > > > initiator. You can get as many overlaps as you want by presenting the > > > commands to the initiator in the desired order. > > > What we are considering here is the case in which you want to ship in > > > an > > > order different than the one you present the commands. > > > > > > Julo > > > > > > "Rod Harrison" > > > Sent by: owner-ips@ece.cmu.edu > > > 04-11-01 04:42 > > > Please respond to "Rod Harrison" > > > > > > To: "Barry Reinhold" , "Dave > > > Sheehy" > > > , "IETF IP SAN Reflector" > > > > > > cc: > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Barry, > > > > > > In general I agree but I don't think this is as much > > > of a > > > corner case > > > as it at first appears. Targets will have code very similar to that > > > needed to handle out of order commands to deal with digest errors. > > > Targets also need to queue commands whilst waiting for both solicited > > > and unsolicited data to arrive. Queuing out of order commands seems > > > little extra work. > > > > > > From an initiators point of view there are > > > efficiency, > > > and probably > > > performance gains to be had from sending commands out of order. Bob > > > Russell gave the example of a read being sent whilst write data DMA is > > > happening, and a similar situation can arise with DMA for writes > > > overtaking that of earlier writes if the initiator has multiple DMA > > > engines. In this case the initiator might be forced to let the wire go > > > idle if it can't send the data from completed DMAs as soon as > > > possible. > > > > > > We already have a command queue at the target to > > > enforce > > > correct > > > serialisation of commands, doing the same thing at the initiator is > > > redundant. > > > > > > Finally, I don't believe we should be writing a > > > standard > > > to work > > > around poor coding and test coverage, especially at the cost of > > > potential efficiency gains. > > > > > > I agree with Dave and Santosh that commands being > > > sent > > > out of order > > > on a single session should be allowed by the standard. > > > > > > - Rod > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Barry Reinhold > > > Sent: Friday, November 02, 2001 5:24 PM > > > To: Dave Sheehy; IETF IP SAN Reflector > > > Subject: RE: iSCSI: current UNH Plugfest > > > > > > Using features such as out of order command delivery on a connection > > > tend to > > > be the sort of things that lead to interoperability problems. It is > > > unexpected and probably going to hit poorly tested code paths even if > > > the > > > standard is written to allow it. > > > > > > >-----Original Message----- > > > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > >Dave Sheehy > > > >Sent: Friday, November 02, 2001 4:19 PM > > > >To: IETF IP SAN Reflector > > > >Subject: Re: iSCSI: current UNH Plugfest > > > > > > > > > > > > > > > >> 3. Can commands be sent out of order on the same connection? > > > >> > > > >> The behavior of targets is clearly specified in Section 2.2.2.3 > > > on > > > >> page 25 of draft 8, which says: > > > >> "Except for the commands marked for immediate delivery the > > > iSCSI > > > >> target layer MUST eliver the commands for execution in the > > > order > > > >> specified by CmdSN." > > > >> > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > > > >> "- CmdSN - the current command Sequence Number advanced by 1 > > > on > > > >> each command shipped except for commands marked for immediate > > > >> delivery." > > > >> but the meaning of the term "shipped" is vague, and does not > > > >> necessarily > > > >> require that the PDUs arrive on the other end of a TCP > > > connection > > > >> in the same order that the CmdSN values were assigned to these > > > PDUs. > > > >> > > > >> Some initiators have been designed to send commands out of CmdSN > > > >> order on one connection. Consider the situation where there is > > > only > > > >> one connection and a high-level dispatcher creates a PDU for a > > > SCSI > > > >> command that involves writing immediate data to the target. > > > This PDU > > > >> is enqueued to a lower-level layer which has to setup, start, > > > and > > > >> wait-for a DMA operation to move the immediate data into an > > > onboard > > > >> buffer before the PDU can be put onto the wire. While this is > > > >> happening, the dispatcher creates another unrelated PDU for a > > > SCSI > > > >> read command (for example), and when this PDU is passed to the > > > >> lower-level layer it can be sent immediately, ahead of the > > > previous > > > >> write PDU and therefore out of order on this connection. > > > >> > > > >> The standard clearly allows this to happen if the two PDUs were > > > sent > > > >> on different connections, and seems to imply that this can also > > > happen > > > >> when the two PDUs are sent on the same connection. > > > >> > > > >> The suggestion is to put in the standard an explicit statement > > > that > > > >> this is allowed or not allowed, as appropriate. > > > >> > > > >> If this is allowed, such a statement would avoid the erroneous > > > >> assumption being made by some target implementers that within a > > > single > > > >> connection, commands will arrive in order. > > > >> > > > >> If this is not allowed, such a statement would avoid the > > > erroneous > > > >> assumption being made by some initiator implementers that within > > > a > > > >> single connection, commands can be put on the wire out of order. > > > >> > > > >> +++ > > > >> > > > >> will add an explicit statement saying that this behaviour is > > > forbidden. > > > >> 2.2.2.1 will contain: > > > >> > > > >> On any given connection, the iSCSI initiator MUST send the > > > >commands in the > > > >> order specified by CmdSN. > > > >> > > > >> +++ > > > > > > > >Why do you feel this behavior should be forbidden? Targets already > > > have to > > > >order commands across the session. I don't see why it's a problem to > > > extend > > > >that to the connection as well. I, for one, believe we should take > > > >a liberal > > > >stance on this. > > > > > > > >Dave Sheehy > > > > > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## From owner-ips@ece.cmu.edu Thu Nov 8 18:01:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA10963 for ; Thu, 8 Nov 2001 18:01:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8MTWN29945 for ips-outgoing; Thu, 8 Nov 2001 17:29:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from emulex.emulex.com (emulex.emulex.com [138.239.112.1]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8MTUl29938 for ; Thu, 8 Nov 2001 17:29:30 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id OAA25397 for ; Thu, 8 Nov 2001 14:29:21 -0800 (PST) Received: by xbl.ad.emulex.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Nov 2001 17:29:22 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF280A0507@xbl.ad.emulex.com> From: "Williams, Jim" To: "'ips@ece.cmu.edu'" Subject: Padding Preceding CRC Date: Thu, 8 Nov 2001 17:29:20 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The iSCSI spec indicates that the data will be padded out to a length which is a multiple of 4 before the CRC (digest) is appended. In discussing this with our hardware designers, I was told that this provides no benefit to them, but causes a fair amount of pain. As such I would like to raise the question of whether the padding should be eliminated. Computing unaligned CRCs is fairly trivial to do, and must be done anyway. This is because the padding aligns the CRC with respect to the iSCSI PDU, but one can't in general assume that the iSCSI PDU is aligned with the TCP segment on received data, and on transmitted data the source may be a general gather list of unaligned buffers. I suppose this doesn't preclude doing a lot of work to align the data before feeding it to the CRC unit, but given how easy it is to compute unaligned CRCs in hardware, there is no reason to do this. Inserting padding is extra work, however. This creates a bubble in the data flow pipe, extra information that must be passed between hardware units indicating the number of pad bytes to be inserted, and extra logic to insert the required pad data. If someone can make a case why this padding is a good thing, then certainly the hardware problems are solvable. But it looks to me like a "feature" with cost but no benefit. From owner-ips@ece.cmu.edu Thu Nov 8 18:02:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA10980 for ; Thu, 8 Nov 2001 18:02:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8M9Sf28355 for ips-outgoing; Thu, 8 Nov 2001 17:09:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8M9Ql28350 for ; Thu, 8 Nov 2001 17:09:26 -0500 (EST) Received: from xparelay2.corp.hp.com (unknown [15.58.137.112]) by palrel12.hp.com (Postfix) with ESMTP id ADB951F798; Thu, 8 Nov 2001 14:09:20 -0800 (PST) Received: from xpabh4.corp.hp.com (xpabh4.corp.hp.com [15.58.136.1]) by xparelay2.corp.hp.com (Postfix) with ESMTP id B83401F542; Thu, 8 Nov 2001 07:08:27 -0800 (PST) Received: by xpabh4.corp.hp.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Nov 2001 14:09:19 -0800 Message-ID: <028FE7141C79D511B65100D0B74FE875BCA5C7@xrose01.rose.hp.com> From: "SPASIC,PREDRAG (HP-Roseville,ex1)" To: "'Keith McCloghrie'" Cc: ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Date: Thu, 8 Nov 2001 14:09:12 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Keith, My replies are in line. One additional comment Point 6: Use Counter64 only if necessary. Counter32 would be the preferred solution. -There are no Counter64s in FC standards that we need in the MIB -Some existing tools have problems with Counter64 and Unsigned64 -No Gauge64 in SMIv2 Predrag > -----Original Message----- > From: Keith McCloghrie [mailto:kzm@cisco.com] > Sent: Thursday, November 08, 2001 11:44 AM > To: predrag_spasic@hp.com > Cc: kzm@cisco.com; ips@ece.cmu.edu > Subject: Re: FC Management MIB - proposed changes > > > Predrag, > > > Point 8: > > Although there are quite a few issues around zoning management, > > most of them revolving around perceived inadequate security of > > SNMP, > > I assume you mean the inadequate security of SNMPv1 and SNMPv2c, > i.e., nobody perceives that SNMPv3 is insecure, right ?? Yes, SNMPv3 is secure, but not that many implementations. > > > without this information MIB will not be complete. > > Are you saying that this MIB will not be complete, or that Fibre > Channel management will not be complete ? Management, of course. > > > Zoning management is extensively defined in FC-GS3, is specific > > to FC, and I do not see any reasons to delay this for some point > > in the future. > > I see several options: > > 1. have the first version of the new FC-Mgmt MIB include Zoning, > > 2. start work on another MIB to address Zoning in parallel with > working on the first version of the new FC-Mgmt MIB. > > 3. complete the first version (Internet-Draft) of the new FC-Mgmt > MIB, and defer the addition of Zoning until a second or subsequent > Internet-Draft of this MIB, but before its WG Last Call. > > 4. defer the start of work on another MIB to address Zoning until > after completing the first version (Internet-Draft) of the new > FC-Mgmt MIB. > > 5. finish the new FC-Mgmt MIB, including having it published as an > RFC, and only then work on extending it to address Zoning. Agreed. Carrying over the zoning objects defined in current MIB version is sufficient as a first step. This would not add any new functionality and allow us to focus on the MIB structure. (We might want to define trap for zoning configuration change, if not already there) This approach seems fine, but if we decide to make some of the zoning objects write- able, it may not be so easy after MIB reaches RFC status. So as long as you are not proposing 5, we are OK. > > I'm not proposing #5 (as you may have feared). Rather, I'm proposing > that we not do #1 - specifically, that producing the first version of > the new FC-Mgmt MIB is a large enough task, that we should do it first > before tackling Zoning. I also think #2 would require coodination > between something new and something which is changing, and therefore > would be more work/more difficult than is > warranted/necessary. So, I'm > proposing either #3 or #4; at this point, I don't have a feel > for which > would be better. > > What do you think ? > > Keith. > > > Regards > > > > Predrag Spasic, > > Hewlett Packard > > Details: https://ecardfile.com/id/predrag_spasic > > > > > > > -----Original Message----- > > > From: Keith McCloghrie [mailto:kzm@cisco.com] > > > Sent: Thursday, November 08, 2001 10:43 AM > > > To: ips@ece.cmu.edu > > > Cc: kzm@cisco.com > > > Subject: FC Management MIB - proposed changes > > > > > > > > > > > > Based on the issues that I listed in my previous message > > > (yesterday), I > > > have a set of changes that I'd like to make to the FC-Mgmt MIB, > > > providing there are no objections from the WG. These changes (the > > > first six correspond to the issues list) are: > > > > > > 1. MIB will be written using SMIv2. > > > > > > 2. All objects which apply in non-Fibre Channel > environments will be > > > removed. (Note that this applies to a large fraction of > the objects > > > defined in this MIB.) > > > > > > 2a. For those objects which will be removed, if there are > any which > > > are: a) not already covered in other MIBs, and b) > considered essential > > > to the management of Fibre Channel-based products, then it will be > > > necessary to consider whether other MIB(s) need to be > modified/created > > > as a home for them, and if so which WG(s) should > undertake such work. > > > > > > 3. With 20x20 hindsight, the following observations can > be made with > > > respect to RFC 2837: > > > > > > - the reason that we now have the issue of how it relates to the > > > FC-Mgmt MIB is because it was written as a Fabric Element MIB; > > > this issue would not have arisen if it had been written as a > > > Fibre Channel interface MIB. > > > - it should have extended the ifTable, rather than > > > overlapping with it, > > > - it should not have defined the fcFeModuleTable, which > overlaps with > > > the Entity MIB. > > > - and it should have specified (at least) its octet counters > > > as Counter64. > > > > > > Given these problems, I propose that the new FC-Mgmt MIB > be specified > > > as a replacement for RFC 2837. > > > > > > 4. This MIB will include a media-specific MIB to specify how Fibre > > > Channel interfaces use the ifTable (see section 4 in RFC > 2863). This > > > will result in many tables in this MIB being indexed by ifIndex. > > > > > > 5. Regarding the the MIB objects for the "Simple Name Service", > > > I see two possible solutions: > > > > > > i. retain the MIB objects but focus them on GS-3's Unzoned > > > Name Service. > > > > > > ii. remove the MIB objects for the "Simple Name Service" from > > > this MIB. > > > If there is WG consensus that a MIB is needed for one of > > > the GS-3 Name > > > Services, and for which one, then the appropriate set of > > > MIB objects > > > can be defined in a new MIB. > > > > > > Of these two, I propose to investigate solution i), and > if it proves > > > feasible, then to adopt it; if not, to fall back to solution ii). > > > > > > 6. The Counter32 or Counter64 syntax will be used for counters. > > > > > > 7. In addition to the above, it has been suggested that > MIB objects to > > > support Class 1 are no longer needed. If I don't hear > any objections, > > > I will assume that I can make this change also. > > > > > > 8. Yesterday, John Nguyen (jnguyen@gadzoox.com) asked in an > > > email about > > > "bringing ZONE into FC Management MIB". His suggestion > would seem to > > > be in addition to, rather than as a replacement for, anything > > > which will > > > be in the new FC-Mgmt MIB. Thus, at this point in time, > I'd like to > > > suggest that the changes listed above represent a large > enough amount > > > of work for us to chew on. Therefore, I propose that our > answer to > > > John's query is to ask him to re-raise this issue at a > point in the > > > future when the new FC-Mgmt MIB is becoming more stable. > > > > > > 9. Are there any other changes that anyone would like to > see at this > > > time ? > > > > > > Also note that with this MIB now being in a different WG, the > > > I-D needs > > > to have a draft-ietf-ips-xxx name. I propose to use > > > draft-ietf-ips-fcmgmt-mib-00.txt. > > > > > > Keith. > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 19:14:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA11836 for ; Thu, 8 Nov 2001 19:14:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8NEwh03686 for ips-outgoing; Thu, 8 Nov 2001 18:14:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8NEul03678 for ; Thu, 8 Nov 2001 18:14:56 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA14122; Thu, 8 Nov 2001 18:12:19 -0500 Received: from d03nm041.boulder.ibm.com (d03nm041.boulder.ibm.com [9.99.140.41]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA8NEt689166; Thu, 8 Nov 2001 16:14:55 -0700 Subject: ISCSI: Padding Preceding CRC To: "Williams, Jim" Cc: "'ips@ece.cmu.edu'" X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Prasenjit Sarkar" Date: Thu, 8 Nov 2001 15:14:20 -0800 X-MIMETrack: Serialize by Router on D03NM041/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/08/2001 03:14:55 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Isnt padding independent of CRC? "Williams, Jim" mulex.com> cc: Sent by: Subject: Padding Preceding CRC owner-ips@ece.c mu.edu 11/08/2001 02:29 PM The iSCSI spec indicates that the data will be padded out to a length which is a multiple of 4 before the CRC (digest) is appended. In discussing this with our hardware designers, I was told that this provides no benefit to them, but causes a fair amount of pain. As such I would like to raise the question of whether the padding should be eliminated. Computing unaligned CRCs is fairly trivial to do, and must be done anyway. This is because the padding aligns the CRC with respect to the iSCSI PDU, but one can't in general assume that the iSCSI PDU is aligned with the TCP segment on received data, and on transmitted data the source may be a general gather list of unaligned buffers. I suppose this doesn't preclude doing a lot of work to align the data before feeding it to the CRC unit, but given how easy it is to compute unaligned CRCs in hardware, there is no reason to do this. Inserting padding is extra work, however. This creates a bubble in the data flow pipe, extra information that must be passed between hardware units indicating the number of pad bytes to be inserted, and extra logic to insert the required pad data. If someone can make a case why this padding is a good thing, then certainly the hardware problems are solvable. But it looks to me like a "feature" with cost but no benefit. From owner-ips@ece.cmu.edu Thu Nov 8 19:15:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA11870 for ; Thu, 8 Nov 2001 19:15:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9020u07068 for ips-outgoing; Thu, 8 Nov 2001 19:02:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA901wl07063 for ; Thu, 8 Nov 2001 19:01:58 -0500 (EST) Received: by ariel.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Thu, 8 Nov 2001 16:01:52 -0800 Message-ID: From: Kevin Gibbons To: "'Keith McCloghrie'" , bill@sanera.net Cc: ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Date: Thu, 8 Nov 2001 16:01:51 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk If there is sufficient interest, it would be possible to add FC Zone information to the iSNS MIB. The iSNS MIB already provides an ability to manage iSCSI/iFCP Discovery Domains and iSCSI/iFCP Discovery Domain Sets. These are similar to FC Zones and Zone Sets. There are already situations where an iSCSI/iFCP Discovery Domain maps to an equivalent FC Zone. For example, when a FC to iSCSI/iFCP switch is used, and a Discovery Domain with native FC devices is created in the iSNS, then an equivalent FC Zone is created in the Fibre Channel Name Service to support this. If added to the iSNS MIB, the conformance section could indicate that a switch only supporting Fibre Channel only implements FC specific sections. Kevin -----Original Message----- From: Keith McCloghrie [mailto:kzm@cisco.com] Sent: Thursday, November 08, 2001 12:21 PM To: bill@sanera.net Cc: ips@ece.cmu.edu Subject: Re: FC Management MIB - proposed changes If the iSNS editors think they can do this, then I think it's a good idea. Keith. > Keith, > > On the topic of Simple Name Service, would it be possible to make this > generic enough to work within the iSNS MIB. I would like to see the iSNS > MIB be able to cover both iSCSI and FC if it is at all possible. Any > reasons this can't be done ? Does the iSNS editors want to try to > incorporate this into their design (how hard would it be, is it even > possible ?) > > Bill > +========+=========+=========+=========+=========+=========+=========+ > Bill Strahm Software Development is a race between Programmers > Member of the trying to build bigger and better idiot proof software > Technical Staff and the Universe trying to produce bigger and better > bill@sanera.net idiots. > (503) 601-0263 So far the Universe is winning --- Rich Cook > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Keith McCloghrie > Sent: Thursday, November 08, 2001 10:43 AM > To: ips@ece.cmu.edu > Cc: Keith McCloghrie > Subject: FC Management MIB - proposed changes > > > > 5. Regarding the the MIB objects for the "Simple Name Service", > I see two possible solutions: > > i. retain the MIB objects but focus them on GS-3's Unzoned Name Service. > > ii. remove the MIB objects for the "Simple Name Service" from this MIB. > If there is WG consensus that a MIB is needed for one of the GS-3 Name > Services, and for which one, then the appropriate set of MIB objects > can be defined in a new MIB. > > Of these two, I propose to investigate solution i), and if it proves > feasible, then to adopt it; if not, to fall back to solution ii). > > > > Keith. > > From owner-ips@ece.cmu.edu Thu Nov 8 19:46:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12194 for ; Thu, 8 Nov 2001 19:46:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA90LTF08391 for ips-outgoing; Thu, 8 Nov 2001 19:21:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA90LAl08379 for ; Thu, 8 Nov 2001 19:21:13 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA116012; Thu, 8 Nov 2001 19:18:33 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA90L7650084; Thu, 8 Nov 2001 17:21:07 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: Out of order commands To: Cc: "Robert D. Russell" , "Julian Satran" , X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 8 Nov 2001 16:19:54 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/08/2001 05:21:08 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Most folks I know have a buffer that applies to all the connections, not to just one buffer per connection. I fail to see the problem for most implimentations. The window should apply to the session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Somesh Gupta" @ece.cmu.edu on 11/08/2001 02:22:30 PM Please respond to Sent by: owner-ips@ece.cmu.edu To: "Robert D. Russell" cc: Julian Satran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: Out of order commands Comments below > -----Original Message----- > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > Sent: Thursday, November 08, 2001 11:46 AM > To: Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > Somesh: > > > > It seems to me that if a target can only handle x new commands, > > > then its queueing capacity is x, and in the SCSI Response PDU it > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > > formula in section 2.2.2.1. This in turn controls the number of > > > commands the initiator can send, and thus prevents the incoming > > > commands from overfilling the target's queue. > > > > This is actually a weakness of the multiple connections over > > multiple adapters model (it is not related to ordering). > > The target advertises a command window on a session wide basis. > > At the same time, it has to provide the resource to the adapter > > to be able to DMA those commands to. Since there is no restriction > > on which connection the commands may be received, either the > > target has to allocate the max resources needed to each adapter > > (thus committing n times the resources actually required), or > > has to "lie" (it would not be a complete lie) which could > > result in running out of resources. One way to fix that would be > > to have the credit on a per connection basis. > > If I understand you correctly, you are saying that deadlock can > occur even if we enforce ordered delivery by initiators and even > if the advertisements are correct and even if both parties obey > the advertisements. I would let Julian enlighten us on that, or whether it just leads to wholesale command drops and retries, or TASK SET FULL is returned (is TASK SET FULL a valid response in this case for a LINKED task when it is not the first command?) > > In other words, doesn't this mean that the standard can lead to a > trap in implementing targets and that, in fact, the advertisements > really should be on a per connection basis? > However, what would be the consequences for initiators if > the advertisements were on a per-connection basis? There is really no issue with adveritisement on a per connection basis. It leads to a protocol change. However, it also allows a much more performant flow of commands. > > > Thanks, > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > From owner-ips@ece.cmu.edu Thu Nov 8 19:48:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12229 for ; Thu, 8 Nov 2001 19:48:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA8NM5B04289 for ips-outgoing; Thu, 8 Nov 2001 18:22:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from NetworkRelay ([63.119.175.39]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA8NDSl03577 for ; Thu, 8 Nov 2001 18:13:28 -0500 (EST) Received: from quicksall.com [63.119.175.45] by NetworkRelay with ESMTP (SMTPD32-7.00) id A019268E0104; Thu, 08 Nov 2001 17:07:05 -0600 Received: from eddydesktop [24.61.64.49] by quicksall.com with ESMTP (SMTPD32-6.06) id A180B5A00F2; Thu, 08 Nov 2001 17:13:04 -0600 From: "Eddy Quicksall" To: "'Rod Harrison'" Cc: Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 18:07:04 -0500 Message-ID: <000501c168aa$f21adba0$0202a8c0@eddydesktop> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Rob, You will not know the tag type of a command that is not there yet. That is why you must only submit to the SCSI layer in CmdSN order (at least that is why I thought that was in the spec). E.g., C0 and C1 may be a simple tag, C2 may be an ordered tag and C3 may be a simple tag. C0 and C1 can execute in any order but, because C2 is ordered, C3 must execute after C2. Therefore, C3 must wait for C2 and CmdSN regiments that. Eddy -----Original Message----- From: Rod Harrison [mailto:rod.harrison@windriver.com] Sent: Wednesday, November 07, 2001 8:58 PM To: somesh_gupta@silverbacksystems.com; Barry Reinhold; Robert D. Russell Cc: Julian Satran; ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Somesh, The spec prohibits processing commands simultaneously, if by processing you mean handing off to the SCSI layer. If by processing simultaneously you mean preparing data for committing to SCSI when the CmdSN allows then surely reception of commands out of order is a no brainer. By the way, I am in support of allowing the target to make the decision on when to commit the SCSI layer rather than having to use the CmdSN. That decision can be made on the basis of device type, LUN, and queue tag. Forcing CmdSN order of committal to SCSI has always seemed overly restrictive to me. - Rod -----Original Message----- From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] Sent: Wednesday, November 07, 2001 5:46 PM To: Rod Harrison; Barry Reinhold; Robert D. Russell Cc: Julian Satran; ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Rod, Sorry to butt in, but targets are not doing serialized processing. They will process many many commands simultaneously. We had prior discussions on this and unfortunately we agreed on ordered delivery at the iSCSI level. The real boost in performance is to have ordering semantics at a higher layer, so that the user code can determine whether it needs ordering or not. As an example, in the general/normal case of a computer talking to a disk, the ordering rule could be relaxed completely. Somesh > -----Original Message----- > From: Rod Harrison [mailto:rod.harrison@windriver.com] > Sent: Wednesday, November 07, 2001 5:34 PM > To: Barry Reinhold; Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Barry, > > I'm curious, where do you see the extra complexity between the > following? > > Assuming all the following commands are writes ... > > c1, c3, c2, c4 and > > c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. > > In both cases the target has to queue commands 2, 3, and 4 whilst > waiting for its R2T on c1 to be satisfied. > > Even if the target doesn't support any unsolicited data it still has > to queue commands 2, 3, and 4. I can't see how the target can avoid > queuing commands, in which case the order of arrival seems to make > little difference. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Barry Reinhold > Sent: Wednesday, November 07, 2001 3:10 PM > To: Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Bob, > I can't speak for targets, but OOO commands on a session with a > single > connection sure increases the complexity of the code path we take. > To me the issue is still that these types of situations tend to be > poorly > tested and lead to interoperability issues. If we do go down this > path, the > spec. should make it very clear that this is expected behavior. Some > statement with a shall in it should be written (for the receiver), so > that > there is a conformance test items to pass. > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > >Robert D. Russell > >Sent: Wednesday, November 07, 2001 4:13 PM > >To: Somesh Gupta > >Cc: Julian Satran; ips@ece.cmu.edu > >Subject: RE: iSCSI: Out of order commands > > > > > >Somesh, Julian: > > > >You state that dealing with OOO commands on the target > >will add substantial complexity on the target. > >Do you have any basis for that claim? My impression from the > >plugfest is that most targets are already dealing with > >it. Perhaps we need to hear from someone who is actually > >building a target for which this would be a real problem. > > > >If anything, what we are hearing from people who really > >are building initiators is that dealing with the requirement > >to send commands in order will introduce substantial complexity > >on the initiator. > > > >So why should we be saving complexity on (hypothetically) simple > >targets yet requiring complexity on real initiators? > > > >As far as the deadlock issue is concerned, if the only way > >that deadlock can occur with OOO commands on the same > >connection is during the use of immediate data (which is I > >think what Julian was saying), then shouldn't we change > >the standard to just say that -- if the initiator sends > >commands out of order on a single connection, then immediate > >data MUST NOT be used on that connection in order to avoid deadlock. > > > >This gives everybody what they want, since initiators who find > >it beneficial to deliver commands OOO will just negotiate > >immediate data off. Those who really want to use immediate data > >will have to ensure that commands are sent in order. > >The tradeoff then becomes an implementation issue, not a > >standards issue, which is where it belongs. > > > > > >Bob Russell > >InterOperability Lab > >University of New Hampshire > >rdr@iol.unh.edu > >603-862-3774 > > > > > >On Wed, 7 Nov 2001, Somesh Gupta wrote: > > > >> I think we should either have it as a MUST or not require > >> it (at both ends to get the real benefit). SHOULD is one > >> of those things that leads to implementation > >> burden and confusion, without perhaps the feature being > >> used. There are implementation as well as protocol > >> considerations mixed in here. > >> > >> If we are to remove the restriction, we should (SHOULD) > >> get the maximum benefit from it, rather than to > >> accomodate an implementation choice. Out of sequence > >> commands, combined with the possibility of digest errors, > >> will add substantial complexity on the target side, > >> without corrosponding benefit in performance. If we change > >> this to SHOULD, we should also relax the requirement > >> to present commands on the target side to a SHOULD. > >> > >> > >> > >> > -----Original Message----- > >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of > >> > Julian Satran > >> > Sent: Wednesday, November 07, 2001 10:00 AM > >> > To: ips@ece.cmu.edu > >> > Subject: Re: iSCSI: Out of order commands > >> > > >> > > >> > Mallikarjun, > >> > > >> > I did not see a SINGLE performance improvement that results from > OOO > >> > shipping. > >> > I would be bad engineering to give away the "no-deadlock" > mechanism we > >> > have now for nothing. > >> > I have also the impression that the point about deadlock that I > keep > >> > repeating is ignored or not understood. > >> > As we stand today commands can be shipped with Immediate data > >or without > >> > and an implementer determined > >> > to squeeze maximum bandwidth and overlap command start with > >delivery will > >> > choose not to work with immediate data > >> > (as you have pointed out) while a low performance software > >implementation > >> > will use immediate data to minimize CPU cycles consumed. However > both > >> > will be guaranteed to work without deadlock as source and sink > use the > >> > same ordering. > >> > Recovery is still a low probability event and should be handled > with a > >> > different set of considerations in mind. > >> > As for the strictness of the recommendation - yes we could settle > on > >> > SHOULD. > >> > > >> > Julo > >> > > >> > > >> > > >> > > >> > "Mallikarjun C." > >> > Sent by: owner-ips@ece.cmu.edu > >> > 07-11-01 19:41 > >> > Please respond to cbm > >> > > >> > > >> > To: Santosh Rao , > ips@ece.cmu.edu > >> > cc: > >> > Subject: Re: iSCSI: Out of order commands > >> > > >> > > >> > > >> > Santosh, > >> > > >> > I have only one comment on your responses. > >> > > >> > > Even a single connection target *MUST* implement a scoreboard. > The > >> > > reason being that it can see out-of-order arrival of commands > due to > >> > > commands being dropped on digest errors. In such a case, it > >must block > >> > > further command processing until holes are filled. > >> > > >> > I made two convenient assumptions if you noticed, :-), one of > which > >> > is that target forces session recovery on *any* error that it > sees > >> > (ErrorRecoveryLevel=0) - including a dropped command due to a > digest > >> > error. With that assumption, a target can afford not to > implement > >> > a scoreboard. > >> > > >> > As I said in a private note, I guess what primarily bothers me > about > >> > OOO commands on a connection is that it requires the receiver to > >> > undo this "optimization" on its end - most notably on a single > >> > connection. TCP experts may comment on how/if they dealt with a > >> > similar issue. > >> > > >> > OTOH, you had some valid comments on exceptions to ordering > during > >> > connection recovery. Perhaps we can move on by making Julian's > >> > proposed stipulation a SHOULD.... > >> > -- > >> > Mallikarjun > >> > > >> > > >> > Mallikarjun Chadalapaka > >> > Networked Storage Architecture > >> > Network Storage Solutions Organization > >> > MS 5668 Hewlett-Packard, Roseville. > >> > cbm@rose.hp.com > >> > > >> > > >> > Santosh Rao wrote: > >> > > > >> > > Mallikarjun, > >> > > > >> > > Some comments below. > >> > > > >> > > Regards, > >> > > Santosh > >> > > > >> > > "Mallikarjun C." wrote: > >> > > > > >> > > > Rod and Julian, > >> > > > > >> > > > This has been an interesting thread of discussion. Some > >> > > > comments - > >> > > > > >> > > > 1.My first reaction was - allowing out-of-order command > >> > > > transmission on the same connection deprives targets of > >> > > > an implementation choice. Targets which support only > >> > > > single-connection sessions and only support session > >> > > > recovery (reasonable assumptions in my mind) can no > >> > > > longer afford *not to* implement a command scoreboard. > >> > > > >> > > Even a single connection target *MUST* implement a scoreboard. > The > >> > > reason being that it can see out-of-order arrival of commands > due to > >> > > commands being dropped on digest errors. In such a case, it > >must block > >> > > further command processing until holes are filled. > >> > > > >> > > Thus, there is no getting away from implementing a sequencer at > the > >> > > target. Given this, I think it is unreasonable to restrict > initiator > >> > > implementation flexibility by imposing a strict ordering > requirement > >> > > within the connection. > >> > > > >> > > > 2.Any end-node efficiency that is sought to be achieved > >> > > > by transmitting CmdSNs out-of-order from the initiator > >> > > > would be lost on the other end-node, since the target > >> > > > now must wait for re-ordering the commands. > >> > > > >> > > It has to handle this situation anyway to deal with holes > caused by > >> > > digest errors. This scenario occurs even with initiators that > issue > >> > > commands in order. > >> > > > >> > > > > >> > > > 3.The flipside is that out-of-order transmission saves > >> > > > link badwidth (albeit at the expense of end-node > efficiency), > >> > > > compared to idling the link waiting for outbound DMA. > >> > > > We have to determine if this is a reasonable trade-off. > >> > > > > >> > > > 4.I can see Rod's point that prefetching all immediate > >> > > > data can be a burden on the NIC resources. But, two > >> > > > questions - > >> > > > - could the NIC not use unsolicited separate data > >> > > > PDUs in these cases? [ I realize that InitialR2T > >> > > > has to be "no" to let it happen... ] > >> > > > - could the NIC have a memory architecture that > >> > > > allows data prefetching for the next command (so > >> > > > this is a non-issue from the protocol perspective)? > >> > > > This scheme incurs one DMA delay for every new > >> > > > burst of commands. > >> > > > > >> > > > 5.Another (perhaps radical at this point) option is to do > >> > > > away with immediate unsolicited data, to stick only with > >> > > > separate unsolicited data. I would personally be okay > >> > > > with the choice, particularly if this feature (that > >> > > > helps software implementations) starts making hardware > >> > > > design complicated/expensive. > >> > > > > >> > > > So, to summarize - > >> > > > > >> > > > option immediate allow > >> > > > data in spec? > out-of-order? > >> > > > > >> > > > (A) (5) above no no > >> > > > (B) No real reason to do this. no yes > >> > > > (C) (4) above yes no > >> > > > (D) pros & cons (1), (2) & (3) yes yes > >> > > > > >> > > > >From the arguments I heard so far, I am leaning towards > >> > > > option A, and option C in that order. > >> > > > > >> > > > Comments? > >> > > > -- > >> > > > Mallikarjun > >> > > > > >> > > > Mallikarjun Chadalapaka > >> > > > Networked Storage Architecture > >> > > > Network Storage Solutions Organization > >> > > > MS 5668 Hewlett-Packard, Roseville. > >> > > > cbm@rose.hp.com > >> > > > > >> > > > Rod Harrison wrote: > >> > > > > > >> > > > > Julian, > >> > > > > > >> > > > > I don't understand what you are proposing here, > >what do you > >> > mean by > >> > > > > "multiplexed" DMA? > >> > > > > > >> > > > > The problem is that the DMAs take some time, the > >more there > >> > are > >> > > > > queued the longer the last DMAs queued take to complete. > Some > >> > commands > >> > > > > require DMAs to complete before they can be sent, i.e. > >Writes with > >> > > > > immediate data, some commands do not, i.e. Reads and > >writes with no > >> > > > > immediate data. The iSCSI HBA wants to be able to send > >commands as > >> > > > > soon a possible, which for a read after a write can be > before the > >> > > > > write's DMA has completed. Maintaining an ordered queue > >for commands > >> > > > > to be sent on the HBA is expensive and redundant since the > target > >> > > > > already knows how to queue commands before committing them > to its > >> > SCSI > >> > > > > layer. > >> > > > > > >> > > > > The iSCSI HBA and its host driver are not at > liberty to > >> > change the > >> > > > > order of commands from the OS, but the DMAs those > >commands need are > >> > > > > unlikely to complete in the same order, and as I mentioned > some > >> > > > > commands need no DMA. If the HBA can't send commands out of > CmdSN > >> > > > > order it has to maintain an ordered queue of commands > >waiting to be > >> > > > > sent, and potentially buffer a lot of data. For an HBA this > makes > >> > > > > immediate data almost impossible to support. > >> > > > > > >> > > > > I don't see the problem with allowing out of > >order commands > >> > given > >> > > > > that the target already has to deal with very similar > problems. I > >> > > > > think we are getting in to the area of implementation > >choices here, > >> > > > > which is inappropriate for a specification. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Julian Satran > >> > > > > Sent: Monday, November 05, 2001 10:06 PM > >> > > > > To: ips@ece.cmu.edu > >> > > > > Subject: Re: iSCSI: Out of order commands, was current > >UNH Plugfest > >> > > > > > >> > > > > Rod, > >> > > > > > >> > > > > I don't see any reason why DMA operations cant be > >"multiplexed" with > >> > > > > commands. > >> > > > > If you have scheduled a long outbound DMA you are doomed > >regardless > >> > of > >> > > > > the > >> > > > > command ordering. > >> > > > > And if you have scheduled DMA operations piecemeal then you > can > >> > insert > >> > > > > your commands in correct order. > >> > > > > > >> > > > > Julo > >> > > > > > >> > > > > "Rod Harrison" > >> > > > > 05-11-01 20:48 > >> > > > > Please respond to "Rod Harrison" > >> > > > > > >> > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > >> > > > > cc: > >> > > > > Subject: iSCSI: Out of order commands, was > current > >> > UNH > >> > > > > Plugfest > >> > > > > > >> > > > > [ Subject changed ] > >> > > > > > >> > > > > Julian, > >> > > > > > >> > > > > The ordering difference is introduced > >between the > >> > > > > host > >> > > > > side driver > >> > > > > and the iSCSI HBA. The host side driver must present > >SCSI commands > >> > to > >> > > > > the HBA in the order they are received from the OS to > >prevent read > >> > > > > after write dependency failures. The HBA might reorder > >the commands > >> > > > > depending on when DMA completes. The reordering can't be > >done ahead > >> > of > >> > > > > time in the host driver since it doesn't know how long each > DMA > >> > might > >> > > > > take. As long as the HBA assigns CmdSN in the order it > receives > >> > > > > commands the desired host ordering is preserved. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Julian Satran > >> > > > > Sent: Monday, November 05, 2001 12:35 AM > >> > > > > To: ips@ece.cmu.edu > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Rod, > >> > > > > > >> > > > > I all examples give the point I find hard to understand is > why is > >> > the > >> > > > > ordering on the wire different from the presentation order > to the > >> > > > > initiator. You can get as many overlaps as you want by > >presenting > >> > the > >> > > > > commands to the initiator in the desired order. > >> > > > > What we are considering here is the case in which you > >want to ship > >> > in > >> > > > > an > >> > > > > order different than the one you present the commands. > >> > > > > > >> > > > > Julo > >> > > > > > >> > > > > "Rod Harrison" > >> > > > > Sent by: owner-ips@ece.cmu.edu > >> > > > > 04-11-01 04:42 > >> > > > > Please respond to "Rod Harrison" > >> > > > > > >> > > > > To: "Barry Reinhold" , > "Dave > >> > > > > Sheehy" > >> > > > > , "IETF IP SAN Reflector" > >> > > > > > >> > > > > cc: > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Barry, > >> > > > > > >> > > > > In general I agree but I don't think this > is as > >> > much > >> > > > > of a > >> > > > > corner case > >> > > > > as it at first appears. Targets will have code very > >similar to that > >> > > > > needed to handle out of order commands to deal with > >digest errors. > >> > > > > Targets also need to queue commands whilst waiting for both > >> > solicited > >> > > > > and unsolicited data to arrive. Queuing out of order > >commands seems > >> > > > > little extra work. > >> > > > > > >> > > > > From an initiators point of view there are > >> > > > > efficiency, > >> > > > > and probably > >> > > > > performance gains to be had from sending commands out of > >order. Bob > >> > > > > Russell gave the example of a read being sent whilst > >write data DMA > >> > is > >> > > > > happening, and a similar situation can arise with DMA for > writes > >> > > > > overtaking that of earlier writes if the initiator has > >multiple DMA > >> > > > > engines. In this case the initiator might be forced to > >let the wire > >> > go > >> > > > > idle if it can't send the data from completed DMAs as soon > as > >> > > > > possible. > >> > > > > > >> > > > > We already have a command queue at the > target to > >> > > > > enforce > >> > > > > correct > >> > > > > serialisation of commands, doing the same thing at the > >initiator is > >> > > > > redundant. > >> > > > > > >> > > > > Finally, I don't believe we should be > writing a > >> > > > > standard > >> > > > > to work > >> > > > > around poor coding and test coverage, especially at the > cost of > >> > > > > potential efficiency gains. > >> > > > > > >> > > > > I agree with Dave and Santosh that > >commands being > >> > > > > sent > >> > > > > out of order > >> > > > > on a single session should be allowed by the standard. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Barry Reinhold > >> > > > > Sent: Friday, November 02, 2001 5:24 PM > >> > > > > To: Dave Sheehy; IETF IP SAN Reflector > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Using features such as out of order command delivery on > >a connection > >> > > > > tend to > >> > > > > be the sort of things that lead to interoperability > >problems. It is > >> > > > > unexpected and probably going to hit poorly tested code > >paths even > >> > if > >> > > > > the > >> > > > > standard is written to allow it. > >> > > > > > >> > > > > >-----Original Message----- > >> > > > > >From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > > > > Of > >> > > > > >Dave Sheehy > >> > > > > >Sent: Friday, November 02, 2001 4:19 PM > >> > > > > >To: IETF IP SAN Reflector > >> > > > > >Subject: Re: iSCSI: current UNH Plugfest > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > >> 3. Can commands be sent out of order on the same > connection? > >> > > > > >> > >> > > > > >> The behavior of targets is clearly specified in > Section > >> > 2.2.2.3 > >> > > > > on > >> > > > > >> page 25 of draft 8, which says: > >> > > > > >> "Except for the commands marked for immediate > >delivery the > >> > > > > iSCSI > >> > > > > >> target layer MUST eliver the commands for > >execution in the > >> > > > > order > >> > > > > >> specified by CmdSN." > >> > > > > >> > >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >> > > > > >> "- CmdSN - the current command Sequence Number > >advanced by 1 > >> > > > > on > >> > > > > >> each command shipped except for commands marked for > >> > immediate > >> > > > > >> delivery." > >> > > > > >> but the meaning of the term "shipped" is vague, > >and does not > >> > > > > >> necessarily > >> > > > > >> require that the PDUs arrive on the other end of a > TCP > >> > > > > connection > >> > > > > >> in the same order that the CmdSN values were > >assigned to these > >> > > > > PDUs. > >> > > > > >> > >> > > > > >> Some initiators have been designed to send commands > out of > >> > CmdSN > >> > > > > >> order on one connection. Consider the situation > >where there > >> > is > >> > > > > only > >> > > > > >> one connection and a high-level dispatcher creates > >a PDU for a > >> > > > > SCSI > >> > > > > >> command that involves writing immediate data to the > target. > >> > > > > This PDU > >> > > > > >> is enqueued to a lower-level layer which has to > >setup, start, > >> > > > > and > >> > > > > >> wait-for a DMA operation to move the immediate data > into an > >> > > > > onboard > >> > > > > >> buffer before the PDU can be put onto the wire. > >While this is > >> > > > > >> happening, the dispatcher creates another > >unrelated PDU for a > >> > > > > SCSI > >> > > > > >> read command (for example), and when this PDU is > >passed to the > >> > > > > >> lower-level layer it can be sent immediately, ahead > of the > >> > > > > previous > >> > > > > >> write PDU and therefore out of order on this > connection. > >> > > > > >> > >> > > > > >> The standard clearly allows this to happen if the two > PDUs > >> > were > >> > > > > sent > >> > > > > >> on different connections, and seems to imply that > this can > >> > also > >> > > > > happen > >> > > > > >> when the two PDUs are sent on the same connection. > >> > > > > >> > >> > > > > >> The suggestion is to put in the standard an > >explicit statement > >> > > > > that > >> > > > > >> this is allowed or not allowed, as appropriate. > >> > > > > >> > >> > > > > >> If this is allowed, such a statement would avoid > >the erroneous > >> > > > > >> assumption being made by some target implementers > >that within > >> > a > >> > > > > single > >> > > > > >> connection, commands will arrive in order. > >> > > > > >> > >> > > > > >> If this is not allowed, such a statement would avoid > the > >> > > > > erroneous > >> > > > > >> assumption being made by some initiator implementers > that > >> > within > >> > > > > a > >> > > > > >> single connection, commands can be put on the wire > out of > >> > order. > >> > > > > >> > >> > > > > >> +++ > >> > > > > >> > >> > > > > >> will add an explicit statement saying that this > behaviour is > >> > > > > forbidden. > >> > > > > >> 2.2.2.1 will contain: > >> > > > > >> > >> > > > > >> On any given connection, the iSCSI initiator MUST send > the > >> > > > > >commands in the > >> > > > > >> order specified by CmdSN. > >> > > > > >> > >> > > > > >> +++ > >> > > > > > > >> > > > > >Why do you feel this behavior should be forbidden? > >Targets already > >> > > > > have to > >> > > > > >order commands across the session. I don't see why it's > >a problem > >> > to > >> > > > > extend > >> > > > > >that to the connection as well. I, for one, believe we > >should take > >> > > > > >a liberal > >> > > > > >stance on this. > >> > > > > > > >> > > > > >Dave Sheehy > >> > > > > > > >> > > > >> > > -- > >> > > ################################## > >> > > Santosh Rao > >> > > Software Design Engineer, > >> > > HP-UX iSCSI Driver Team, > >> > > Hewlett Packard, Cupertino. > >> > > email : santoshr@cup.hp.com > >> > > Phone : 408-447-3751 > >> > > ################################## > >> > > >> > > >> > > >> > > >> > > > > From owner-ips@ece.cmu.edu Thu Nov 8 20:22:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA12784 for ; Thu, 8 Nov 2001 20:22:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA90ew809755 for ips-outgoing; Thu, 8 Nov 2001 19:40:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA90eul09748 for ; Thu, 8 Nov 2001 19:40:56 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA47468; Thu, 8 Nov 2001 19:38:19 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA90es6179486; Thu, 8 Nov 2001 17:40:54 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: Out of order commands To: Cc: "Robert D. Russell" , "Julian Satran" , X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 8 Nov 2001 16:39:51 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/08/2001 05:40:54 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I guess I disagree with this design approach. The buffers on the iSCSI HBAs are for Eddy buffers. Once you have the iSCSI Header should be able to place the data directly into a Shared buffer. If you have a session that spans HBAs, that should work just fine. But the buffers match the Session Window, and the Buffers need to be shared across the HBAs, not in the HBAs. Again, the HBAs should just have the Eddy Buffers. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Somesh Gupta" on 11/08/2001 04:31:29 PM Please respond to To: John Hufferd/San Jose/IBM@IBMUS cc: "Robert D. Russell" , Julian Satran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: Out of order commands John, You are correct if all the connections are on a single adapter. In this case the buffer pool can be shared among all the connections on the adapter. However this is not true if the connections are on different adapters (the point being discussed). Think of there being a big buffer pool in the board into which multiple adapters/chips are "plugged" in. However, at any given point in time, these buffers are allocated (handed over through command queues or whatever) to each of the individual adapters. And this is where the problem arises. The assumption that adapters can share a typical host command buffer pool on their own without host intervention is incorrect. Somesh > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Thursday, November 08, 2001 4:20 PM > To: somesh_gupta@silverbacksystems.com > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > Most folks I know have a buffer that applies to all the > connections, not to > just one buffer per connection. I fail to see the problem for most > implimentations. The window should apply to the session. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" @ece.cmu.edu on > 11/08/2001 02:22:30 PM > > Please respond to > > Sent by: owner-ips@ece.cmu.edu > > > To: "Robert D. Russell" > cc: Julian Satran/Haifa/IBM@IBMIL, > Subject: RE: iSCSI: Out of order commands > > > > Comments below > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Thursday, November 08, 2001 11:46 AM > > To: Somesh Gupta > > Cc: Julian Satran; ips@ece.cmu.edu > > Subject: RE: iSCSI: Out of order commands > > > > > > > > Somesh: > > > > > > It seems to me that if a target can only handle x new commands, > > > > then its queueing capacity is x, and in the SCSI Response PDU it > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > > > formula in section 2.2.2.1. This in turn controls the number of > > > > commands the initiator can send, and thus prevents the incoming > > > > commands from overfilling the target's queue. > > > > > > This is actually a weakness of the multiple connections over > > > multiple adapters model (it is not related to ordering). > > > The target advertises a command window on a session wide basis. > > > At the same time, it has to provide the resource to the adapter > > > to be able to DMA those commands to. Since there is no restriction > > > on which connection the commands may be received, either the > > > target has to allocate the max resources needed to each adapter > > > (thus committing n times the resources actually required), or > > > has to "lie" (it would not be a complete lie) which could > > > result in running out of resources. One way to fix that would be > > > to have the credit on a per connection basis. > > > > If I understand you correctly, you are saying that deadlock can > > occur even if we enforce ordered delivery by initiators and even > > if the advertisements are correct and even if both parties obey > > the advertisements. > > I would let Julian enlighten us on that, or whether it just > leads to wholesale command drops and retries, or TASK SET FULL > is returned (is TASK SET FULL a valid response in this case > for a LINKED task when it is not the first command?) > > > > > In other words, doesn't this mean that the standard can lead to a > > trap in implementing targets and that, in fact, the advertisements > > really should be on a per connection basis? > > However, what would be the consequences for initiators if > > the advertisements were on a per-connection basis? > > There is really no issue with adveritisement on a per > connection basis. It leads to a protocol change. However, it > also allows a much more performant flow of commands. > > > > > > > Thanks, > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 20:25:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA12828 for ; Thu, 8 Nov 2001 20:25:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA90ZUb09415 for ips-outgoing; Thu, 8 Nov 2001 19:35:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA90ZRl09411 for ; Thu, 8 Nov 2001 19:35:27 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id TAA17775; Thu, 8 Nov 2001 19:34:19 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "John Hufferd" Cc: "Robert D. Russell" , "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 16:31:29 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John, You are correct if all the connections are on a single adapter. In this case the buffer pool can be shared among all the connections on the adapter. However this is not true if the connections are on different adapters (the point being discussed). Think of there being a big buffer pool in the board into which multiple adapters/chips are "plugged" in. However, at any given point in time, these buffers are allocated (handed over through command queues or whatever) to each of the individual adapters. And this is where the problem arises. The assumption that adapters can share a typical host command buffer pool on their own without host intervention is incorrect. Somesh > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Thursday, November 08, 2001 4:20 PM > To: somesh_gupta@silverbacksystems.com > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > Most folks I know have a buffer that applies to all the > connections, not to > just one buffer per connection. I fail to see the problem for most > implimentations. The window should apply to the session. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" @ece.cmu.edu on > 11/08/2001 02:22:30 PM > > Please respond to > > Sent by: owner-ips@ece.cmu.edu > > > To: "Robert D. Russell" > cc: Julian Satran/Haifa/IBM@IBMIL, > Subject: RE: iSCSI: Out of order commands > > > > Comments below > > > -----Original Message----- > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > Sent: Thursday, November 08, 2001 11:46 AM > > To: Somesh Gupta > > Cc: Julian Satran; ips@ece.cmu.edu > > Subject: RE: iSCSI: Out of order commands > > > > > > > > Somesh: > > > > > > It seems to me that if a target can only handle x new commands, > > > > then its queueing capacity is x, and in the SCSI Response PDU it > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > > > formula in section 2.2.2.1. This in turn controls the number of > > > > commands the initiator can send, and thus prevents the incoming > > > > commands from overfilling the target's queue. > > > > > > This is actually a weakness of the multiple connections over > > > multiple adapters model (it is not related to ordering). > > > The target advertises a command window on a session wide basis. > > > At the same time, it has to provide the resource to the adapter > > > to be able to DMA those commands to. Since there is no restriction > > > on which connection the commands may be received, either the > > > target has to allocate the max resources needed to each adapter > > > (thus committing n times the resources actually required), or > > > has to "lie" (it would not be a complete lie) which could > > > result in running out of resources. One way to fix that would be > > > to have the credit on a per connection basis. > > > > If I understand you correctly, you are saying that deadlock can > > occur even if we enforce ordered delivery by initiators and even > > if the advertisements are correct and even if both parties obey > > the advertisements. > > I would let Julian enlighten us on that, or whether it just > leads to wholesale command drops and retries, or TASK SET FULL > is returned (is TASK SET FULL a valid response in this case > for a LINKED task when it is not the first command?) > > > > > In other words, doesn't this mean that the standard can lead to a > > trap in implementing targets and that, in fact, the advertisements > > really should be on a per connection basis? > > However, what would be the consequences for initiators if > > the advertisements were on a per-connection basis? > > There is really no issue with adveritisement on a per > connection basis. It leads to a protocol change. However, it > also allows a much more performant flow of commands. > > > > > > > Thanks, > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 20:37:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13008 for ; Thu, 8 Nov 2001 20:37:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA911LG11027 for ips-outgoing; Thu, 8 Nov 2001 20:01:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA911Jl11022 for ; Thu, 8 Nov 2001 20:01:19 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id UAA20906; Thu, 8 Nov 2001 20:00:29 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "John Hufferd" Cc: "Robert D. Russell" , "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 16:57:40 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I think we may be talking about different things here. Yes, there are memories that are "on the iSCSI HBA" (in the case where everything is on a single board, this would be the memories that are controlled by only the iSCSI ASIC or whatever) - and these memories are not shared among multiple HBAs. These may be used for any book-keeping purposes, contexts, eddy buffering etc etc by the adapter. Then there is memory pool that is owned by the "system". Adapters will DMA commands/data/responses directly into and out of this memory pool which is managed by the "system". However to prevent adapters from running over each other, and over the "system" processor, producer/consumer mechanisms have to be implemented. This is necessary to maintain order on the system. So on a long-term basis, the memory in the system is shared across the HBAs - absolutely yes. But at any given instance in time, it is partitioned among various usages. So all in all I agree with you. The only point of difference is that it is not possible that a given buffer (part of a buffer pool) is accessible to multiple adapters at the same instance in time. It is a necessary condition to prevent corruption and chaos. > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Thursday, November 08, 2001 4:40 PM > To: somesh_gupta@silverbacksystems.com > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > I guess I disagree with this design approach. The buffers on the iSCSI > HBAs are for Eddy buffers. Once you have the iSCSI Header should be able > to place the data directly into a Shared buffer. If you have a session > that spans HBAs, that should work just fine. But the buffers match the > Session Window, and the Buffers need to be shared across the HBAs, not in > the HBAs. Again, the HBAs should just have the Eddy Buffers. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" on 11/08/2001 04:31:29 > PM > > Please respond to > > To: John Hufferd/San Jose/IBM@IBMUS > cc: "Robert D. Russell" , Julian > Satran/Haifa/IBM@IBMIL, > Subject: RE: iSCSI: Out of order commands > > > > John, > > You are correct if all the connections are on > a single adapter. In this case the buffer pool > can be shared among all the connections on the > adapter. > > However this is not true if the connections are > on different adapters (the point being discussed). > Think of there being a big buffer pool in the board > into which multiple adapters/chips are "plugged" in. > However, at any given point in time, these buffers > are allocated (handed over through command queues or > whatever) to each of the individual adapters. And this > is where the problem arises. > > The assumption that adapters can share a typical host > command buffer pool on their own without host intervention > is incorrect. > > Somesh > > > -----Original Message----- > > From: John Hufferd [mailto:hufferd@us.ibm.com] > > Sent: Thursday, November 08, 2001 4:20 PM > > To: somesh_gupta@silverbacksystems.com > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > > Subject: RE: iSCSI: Out of order commands > > > > > > > > Most folks I know have a buffer that applies to all the > > connections, not to > > just one buffer per connection. I fail to see the problem for most > > implimentations. The window should apply to the session. > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSG San Jose Ca > > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > Home Office (408) 997-6136, Cell: (408) 499-9702 > > Internet address: hufferd@us.ibm.com > > > > > > "Somesh Gupta" @ece.cmu.edu on > > 11/08/2001 02:22:30 PM > > > > Please respond to > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > To: "Robert D. Russell" > > cc: Julian Satran/Haifa/IBM@IBMIL, > > Subject: RE: iSCSI: Out of order commands > > > > > > > > Comments below > > > > > -----Original Message----- > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > > Sent: Thursday, November 08, 2001 11:46 AM > > > To: Somesh Gupta > > > Cc: Julian Satran; ips@ece.cmu.edu > > > Subject: RE: iSCSI: Out of order commands > > > > > > > > > > > > Somesh: > > > > > > > > It seems to me that if a target can only handle x new commands, > > > > > then its queueing capacity is x, and in the SCSI Response PDU it > > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > > > > formula in section 2.2.2.1. This in turn controls the number of > > > > > commands the initiator can send, and thus prevents the incoming > > > > > commands from overfilling the target's queue. > > > > > > > > This is actually a weakness of the multiple connections over > > > > multiple adapters model (it is not related to ordering). > > > > The target advertises a command window on a session wide basis. > > > > At the same time, it has to provide the resource to the adapter > > > > to be able to DMA those commands to. Since there is no restriction > > > > on which connection the commands may be received, either the > > > > target has to allocate the max resources needed to each adapter > > > > (thus committing n times the resources actually required), or > > > > has to "lie" (it would not be a complete lie) which could > > > > result in running out of resources. One way to fix that would be > > > > to have the credit on a per connection basis. > > > > > > If I understand you correctly, you are saying that deadlock can > > > occur even if we enforce ordered delivery by initiators and even > > > if the advertisements are correct and even if both parties obey > > > the advertisements. > > > > I would let Julian enlighten us on that, or whether it just > > leads to wholesale command drops and retries, or TASK SET FULL > > is returned (is TASK SET FULL a valid response in this case > > for a LINKED task when it is not the first command?) > > > > > > > > In other words, doesn't this mean that the standard can lead to a > > > trap in implementing targets and that, in fact, the advertisements > > > really should be on a per connection basis? > > > However, what would be the consequences for initiators if > > > the advertisements were on a per-connection basis? > > > > There is really no issue with adveritisement on a per > > connection basis. It leads to a protocol change. However, it > > also allows a much more performant flow of commands. > > > > > > > > > > > Thanks, > > > > > > Bob Russell > > > InterOperability Lab > > > University of New Hampshire > > > rdr@iol.unh.edu > > > 603-862-3774 > > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 20:51:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13146 for ; Thu, 8 Nov 2001 20:51:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA91KQ612240 for ips-outgoing; Thu, 8 Nov 2001 20:20:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA91KPl12234 for ; Thu, 8 Nov 2001 20:20:25 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 8ADD32E09 for ; Thu, 8 Nov 2001 18:20:24 -0700 (MST) Received: from acropora.rose.agilent.com (acropora.rose.agilent.com [130.30.179.5]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id 1E2B3606 for ; Thu, 8 Nov 2001 18:20:24 -0700 (MST) Received: (from dbs@localhost) by acropora.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) id RAA25798 for ips@ece.cmu.edu; Thu, 8 Nov 2001 17:19:18 -0800 (PST) From: Dave Sheehy Message-Id: <200111090119.RAA25798@acropora.rose.agilent.com> Subject: RE: iSCSI: Out of order commands To: ips@ece.cmu.edu (IETF IP SAN Reflector) Date: Thu, 08 Nov 2001 17:19:17 PST X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, > Targets will most likely advertise a total (command and data) window > larger than they can accommodate on any long haul link. With the current > ordering rules nothing bad will happen. Any target that does that deserves what it gets IMHO. > The whole discussion thread seems also related to some perceived gain from > relaxing this restriction - but no one was able to show a single scenario > showing a real gain. Rod Harrison (I believe) described the scenario quite well. I recomend that you search the archive for his message. Dave Sheehy From owner-ips@ece.cmu.edu Thu Nov 8 20:54:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13171 for ; Thu, 8 Nov 2001 20:54:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA91Kgw12251 for ips-outgoing; Thu, 8 Nov 2001 20:20:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA91KQl12238 for ; Thu, 8 Nov 2001 20:20:26 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id UAA112454; Thu, 8 Nov 2001 20:17:45 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA91KK6150016; Thu, 8 Nov 2001 18:20:20 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: Out of order commands To: Cc: "Robert D. Russell" , "Julian Satran" , X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 8 Nov 2001 17:19:26 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/08/2001 06:20:20 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I understand your point, however, their are several approaches that I and others have used to mitigate this issue. We face that problem with Multi Processing all the time. One technique that I have seen use that is similar to what you describe is to allocate a set of buffers to each processor (in this case an HBA), and when they are used up, and only then, the go back to the Buffer Mother, to allocate another Buffer Set. In this way the Processors (HBA) run mostly independent, but once in awhile get a new supply. There are other techniques also, but that one clearly works wells and is very simple. Therefore, I do not see the Session Window as a problem in the protocol. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Somesh Gupta" on 11/08/2001 04:57:40 PM Please respond to To: John Hufferd/San Jose/IBM@IBMUS cc: "Robert D. Russell" , Julian Satran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: Out of order commands I think we may be talking about different things here. Yes, there are memories that are "on the iSCSI HBA" (in the case where everything is on a single board, this would be the memories that are controlled by only the iSCSI ASIC or whatever) - and these memories are not shared among multiple HBAs. These may be used for any book-keeping purposes, contexts, eddy buffering etc etc by the adapter. Then there is memory pool that is owned by the "system". Adapters will DMA commands/data/responses directly into and out of this memory pool which is managed by the "system". However to prevent adapters from running over each other, and over the "system" processor, producer/consumer mechanisms have to be implemented. This is necessary to maintain order on the system. So on a long-term basis, the memory in the system is shared across the HBAs - absolutely yes. But at any given instance in time, it is partitioned among various usages. So all in all I agree with you. The only point of difference is that it is not possible that a given buffer (part of a buffer pool) is accessible to multiple adapters at the same instance in time. It is a necessary condition to prevent corruption and chaos. > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Thursday, November 08, 2001 4:40 PM > To: somesh_gupta@silverbacksystems.com > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > I guess I disagree with this design approach. The buffers on the iSCSI > HBAs are for Eddy buffers. Once you have the iSCSI Header should be able > to place the data directly into a Shared buffer. If you have a session > that spans HBAs, that should work just fine. But the buffers match the > Session Window, and the Buffers need to be shared across the HBAs, not in > the HBAs. Again, the HBAs should just have the Eddy Buffers. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" on 11/08/2001 04:31:29 > PM > > Please respond to > > To: John Hufferd/San Jose/IBM@IBMUS > cc: "Robert D. Russell" , Julian > Satran/Haifa/IBM@IBMIL, > Subject: RE: iSCSI: Out of order commands > > > > John, > > You are correct if all the connections are on > a single adapter. In this case the buffer pool > can be shared among all the connections on the > adapter. > > However this is not true if the connections are > on different adapters (the point being discussed). > Think of there being a big buffer pool in the board > into which multiple adapters/chips are "plugged" in. > However, at any given point in time, these buffers > are allocated (handed over through command queues or > whatever) to each of the individual adapters. And this > is where the problem arises. > > The assumption that adapters can share a typical host > command buffer pool on their own without host intervention > is incorrect. > > Somesh > > > -----Original Message----- > > From: John Hufferd [mailto:hufferd@us.ibm.com] > > Sent: Thursday, November 08, 2001 4:20 PM > > To: somesh_gupta@silverbacksystems.com > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > > Subject: RE: iSCSI: Out of order commands > > > > > > > > Most folks I know have a buffer that applies to all the > > connections, not to > > just one buffer per connection. I fail to see the problem for most > > implimentations. The window should apply to the session. > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSG San Jose Ca > > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > Home Office (408) 997-6136, Cell: (408) 499-9702 > > Internet address: hufferd@us.ibm.com > > > > > > "Somesh Gupta" @ece.cmu.edu on > > 11/08/2001 02:22:30 PM > > > > Please respond to > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > To: "Robert D. Russell" > > cc: Julian Satran/Haifa/IBM@IBMIL, > > Subject: RE: iSCSI: Out of order commands > > > > > > > > Comments below > > > > > -----Original Message----- > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > > Sent: Thursday, November 08, 2001 11:46 AM > > > To: Somesh Gupta > > > Cc: Julian Satran; ips@ece.cmu.edu > > > Subject: RE: iSCSI: Out of order commands > > > > > > > > > > > > Somesh: > > > > > > > > It seems to me that if a target can only handle x new commands, > > > > > then its queueing capacity is x, and in the SCSI Response PDU it > > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > > > > formula in section 2.2.2.1. This in turn controls the number of > > > > > commands the initiator can send, and thus prevents the incoming > > > > > commands from overfilling the target's queue. > > > > > > > > This is actually a weakness of the multiple connections over > > > > multiple adapters model (it is not related to ordering). > > > > The target advertises a command window on a session wide basis. > > > > At the same time, it has to provide the resource to the adapter > > > > to be able to DMA those commands to. Since there is no restriction > > > > on which connection the commands may be received, either the > > > > target has to allocate the max resources needed to each adapter > > > > (thus committing n times the resources actually required), or > > > > has to "lie" (it would not be a complete lie) which could > > > > result in running out of resources. One way to fix that would be > > > > to have the credit on a per connection basis. > > > > > > If I understand you correctly, you are saying that deadlock can > > > occur even if we enforce ordered delivery by initiators and even > > > if the advertisements are correct and even if both parties obey > > > the advertisements. > > > > I would let Julian enlighten us on that, or whether it just > > leads to wholesale command drops and retries, or TASK SET FULL > > is returned (is TASK SET FULL a valid response in this case > > for a LINKED task when it is not the first command?) > > > > > > > > In other words, doesn't this mean that the standard can lead to a > > > trap in implementing targets and that, in fact, the advertisements > > > really should be on a per connection basis? > > > However, what would be the consequences for initiators if > > > the advertisements were on a per-connection basis? > > > > There is really no issue with adveritisement on a per > > connection basis. It leads to a protocol change. However, it > > also allows a much more performant flow of commands. > > > > > > > > > > > Thanks, > > > > > > Bob Russell > > > InterOperability Lab > > > University of New Hampshire > > > rdr@iol.unh.edu > > > 603-862-3774 > > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 20:55:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13185 for ; Thu, 8 Nov 2001 20:55:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA91SQA12727 for ips-outgoing; Thu, 8 Nov 2001 20:28:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA91SOl12722 for ; Thu, 8 Nov 2001 20:28:24 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id DA3B9C04; Thu, 8 Nov 2001 20:28:23 -0500 (EST) Received: from swathi (swathi.rose.hp.com [15.43.213.237]) by core.rose.hp.com with SMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id RAA08889; Thu, 8 Nov 2001 17:29:50 -0800 (PST) Message-ID: <00b501c168bd$d1c857c0$edd52b0f@rose.hp.com> From: "Mallikarjun C." To: "John Hufferd" Cc: References: Subject: Re: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 17:28:21 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John, Sorry, this note got a little longer than I would've liked, but.... I believe there are cases where OOO CmdSN handling is a legitimate requirement on targets due to exception events - a) retransmitting a CmdSN on a command acknowledgement timeout (within-connection recovery class). This manifests as an OOO CmdSN on the connection to a target if it didn't see the original copy due to a digest error. b) retransmitting the last few "lost" commands due to a connection failure on a new connection. If this new connection had already carried a CmdSN greater than these retransmitted commands (prior to connection failure), this again manifests as OOO CmdSN on the new connection to the target. OTOH, I believe sending OOO CmdSNs on a connection as a regular practice is counterproductive, since the target must continuously re-order the initiator "optimization" leading to a zero-sum game. I would argue that the need to dispatch CmdSNs OOO due to immediate data DMA (brought up by Rod) can be addressed by simple NIC changes to prefetch data for the next command (or more simply use unsolicited separate data PDUs, if negotiated). [ You got to deal with the case of all commands being writes anyway! ] If we allow OOO CmdSNs on a connection (I'd advocate discouraging it as a regular practice), I don't believe any of the stuff in error recovery breaks (nor does it affect the current reliance on ExpCmdSN). Julian perhaps can comment. - All the in-order assumptions are for DataSNs/R2TSNs/StatSNs, not for CmdSNs. - Any multi-connection session by definition must deal with OOO CmdSNs. - I belive that the current abort task scheme for immediate commands detailed in section 9.3 caters to OOO CmdSNs on a connection as well (we must be dealing with an immediate Abort arriving before the command today, since the command could have been hit with a digest error). To summarize, here is what I suggested to Julian in a private email - a)I suggest using a SHOULD for in-order dispatch of commands on a connection - for an initiator. b)I suggest using a SHALL handle out-of-order commands on a connection - for the target (as Barry pointed out). Hope that was useful. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization Hewlett-Packard MS 5668 Roseville CA 95747 ----- Original Message ----- From: "John Hufferd" To: Cc: Sent: Thursday, November 08, 2001 1:40 PM Subject: Re: iSCSI: Out of order commands > > Mallikarjun, > Could you comment on the concept of OOO on the ErrorRecoveryLevel>0. I had > thought that "in order delivery" was part of the detection of missing PDUs > and needed for timely Recovery. I was wondering if this changes the way we > would use the ExpCmdSN, etc. > > I think your opinions on this part of the OOO discussion would be valuable. > For example, how would you contrast the differences in detecting a problem > and recovering from that problem etc., today vrs the OOO approach (if any). > > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Mallikarjun C." @ece.cmu.edu on 11/07/2001 09:41:05 AM > > Please respond to cbm@rose.hp.com > > Sent by: owner-ips@ece.cmu.edu > > > To: Santosh Rao , ips@ece.cmu.edu > cc: > Subject: Re: iSCSI: Out of order commands > > > > Santosh, > > I have only one comment on your responses. > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > I made two convenient assumptions if you noticed, :-), one of which > is that target forces session recovery on *any* error that it sees > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > error. With that assumption, a target can afford not to implement > a scoreboard. > > As I said in a private note, I guess what primarily bothers me about > OOO commands on a connection is that it requires the receiver to > undo this "optimization" on its end - most notably on a single > connection. TCP experts may comment on how/if they dealt with a > similar issue. > > OTOH, you had some valid comments on exceptions to ordering during > connection recovery. Perhaps we can move on by making Julian's > proposed stipulation a SHOULD.... > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > Santosh Rao wrote: > > > > Mallikarjun, > > > > Some comments below. > > > > Regards, > > Santosh > > > > "Mallikarjun C." wrote: > > > > > > Rod and Julian, > > > > > > This has been an interesting thread of discussion. Some > > > comments - > > > > > > 1.My first reaction was - allowing out-of-order command > > > transmission on the same connection deprives targets of > > > an implementation choice. Targets which support only > > > single-connection sessions and only support session > > > recovery (reasonable assumptions in my mind) can no > > > longer afford *not to* implement a command scoreboard. > > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > > > Thus, there is no getting away from implementing a sequencer at the > > target. Given this, I think it is unreasonable to restrict initiator > > implementation flexibility by imposing a strict ordering requirement > > within the connection. > > > > > 2.Any end-node efficiency that is sought to be achieved > > > by transmitting CmdSNs out-of-order from the initiator > > > would be lost on the other end-node, since the target > > > now must wait for re-ordering the commands. > > > > It has to handle this situation anyway to deal with holes caused by > > digest errors. This scenario occurs even with initiators that issue > > commands in order. > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > link badwidth (albeit at the expense of end-node efficiency), > > > compared to idling the link waiting for outbound DMA. > > > We have to determine if this is a reasonable trade-off. > > > > > > 4.I can see Rod's point that prefetching all immediate > > > data can be a burden on the NIC resources. But, two > > > questions - > > > - could the NIC not use unsolicited separate data > > > PDUs in these cases? [ I realize that InitialR2T > > > has to be "no" to let it happen... ] > > > - could the NIC have a memory architecture that > > > allows data prefetching for the next command (so > > > this is a non-issue from the protocol perspective)? > > > This scheme incurs one DMA delay for every new > > > burst of commands. > > > > > > 5.Another (perhaps radical at this point) option is to do > > > away with immediate unsolicited data, to stick only with > > > separate unsolicited data. I would personally be okay > > > with the choice, particularly if this feature (that > > > helps software implementations) starts making hardware > > > design complicated/expensive. > > > > > > So, to summarize - > > > > > > option immediate allow > > > data in spec? out-of-order? > > > > > > (A) (5) above no no > > > (B) No real reason to do this. no yes > > > (C) (4) above yes no > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > >From the arguments I heard so far, I am leaning towards > > > option A, and option C in that order. > > > > > > Comments? > > > -- > > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > Rod Harrison wrote: > > > > > > > > Julian, > > > > > > > > I don't understand what you are proposing here, what do you > mean by > > > > "multiplexed" DMA? > > > > > > > > The problem is that the DMAs take some time, the more there > are > > > > queued the longer the last DMAs queued take to complete. Some > commands > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > soon a possible, which for a read after a write can be before the > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > to be sent on the HBA is expensive and redundant since the target > > > > already knows how to queue commands before committing them to its > SCSI > > > > layer. > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > change the > > > > order of commands from the OS, but the DMAs those commands need are > > > > unlikely to complete in the same order, and as I mentioned some > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > order it has to maintain an ordered queue of commands waiting to be > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > immediate data almost impossible to support. > > > > > > > > I don't see the problem with allowing out of order commands > given > > > > that the target already has to deal with very similar problems. I > > > > think we are getting in to the area of implementation choices here, > > > > which is inappropriate for a specification. > > > > > > > > - Rod > > > > From owner-ips@ece.cmu.edu Thu Nov 8 20:59:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13271 for ; Thu, 8 Nov 2001 20:59:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA91fDc13528 for ips-outgoing; Thu, 8 Nov 2001 20:41:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA91fBl13516 for ; Thu, 8 Nov 2001 20:41:11 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id UAA47646; Thu, 8 Nov 2001 20:38:33 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA91f96115016; Thu, 8 Nov 2001 18:41:09 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: Out of order commands To: "Mallikarjun C." Cc: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 8 Nov 2001 17:40:15 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/08/2001 06:41:08 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes, that was very useful. Now let me ask one other question. With in order command arrival on a connection (as a normal event) seems to provide the quick determination of an error in a Command PDU, and tell the recovery code quickly to get the missing command resent. What do you think will be the effect of not knowing if the command is delayed in the initiator or whether it was dropped because of a Header Digest Error? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Mallikarjun C." on 11/08/2001 05:28:21 PM To: John Hufferd/San Jose/IBM@IBMUS cc: Subject: Re: iSCSI: Out of order commands John, Sorry, this note got a little longer than I would've liked, but.... I believe there are cases where OOO CmdSN handling is a legitimate requirement on targets due to exception events - a) retransmitting a CmdSN on a command acknowledgement timeout (within-connection recovery class). This manifests as an OOO CmdSN on the connection to a target if it didn't see the original copy due to a digest error. b) retransmitting the last few "lost" commands due to a connection failure on a new connection. If this new connection had already carried a CmdSN greater than these retransmitted commands (prior to connection failure), this again manifests as OOO CmdSN on the new connection to the target. OTOH, I believe sending OOO CmdSNs on a connection as a regular practice is counterproductive, since the target must continuously re-order the initiator "optimization" leading to a zero-sum game. I would argue that the need to dispatch CmdSNs OOO due to immediate data DMA (brought up by Rod) can be addressed by simple NIC changes to prefetch data for the next command (or more simply use unsolicited separate data PDUs, if negotiated). [ You got to deal with the case of all commands being writes anyway! ] If we allow OOO CmdSNs on a connection (I'd advocate discouraging it as a regular practice), I don't believe any of the stuff in error recovery breaks (nor does it affect the current reliance on ExpCmdSN). Julian perhaps can comment. - All the in-order assumptions are for DataSNs/R2TSNs/StatSNs, not for CmdSNs. - Any multi-connection session by definition must deal with OOO CmdSNs. - I belive that the current abort task scheme for immediate commands detailed in section 9.3 caters to OOO CmdSNs on a connection as well (we must be dealing with an immediate Abort arriving before the command today, since the command could have been hit with a digest error). To summarize, here is what I suggested to Julian in a private email - a)I suggest using a SHOULD for in-order dispatch of commands on a connection - for an initiator. b)I suggest using a SHALL handle out-of-order commands on a connection - for the target (as Barry pointed out). Hope that was useful. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization Hewlett-Packard MS 5668 Roseville CA 95747 ----- Original Message ----- From: "John Hufferd" To: Cc: Sent: Thursday, November 08, 2001 1:40 PM Subject: Re: iSCSI: Out of order commands > > Mallikarjun, > Could you comment on the concept of OOO on the ErrorRecoveryLevel>0. I had > thought that "in order delivery" was part of the detection of missing PDUs > and needed for timely Recovery. I was wondering if this changes the way we > would use the ExpCmdSN, etc. > > I think your opinions on this part of the OOO discussion would be valuable. > For example, how would you contrast the differences in detecting a problem > and recovering from that problem etc., today vrs the OOO approach (if any). > > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Mallikarjun C." @ece.cmu.edu on 11/07/2001 09:41:05 AM > > Please respond to cbm@rose.hp.com > > Sent by: owner-ips@ece.cmu.edu > > > To: Santosh Rao , ips@ece.cmu.edu > cc: > Subject: Re: iSCSI: Out of order commands > > > > Santosh, > > I have only one comment on your responses. > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > I made two convenient assumptions if you noticed, :-), one of which > is that target forces session recovery on *any* error that it sees > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > error. With that assumption, a target can afford not to implement > a scoreboard. > > As I said in a private note, I guess what primarily bothers me about > OOO commands on a connection is that it requires the receiver to > undo this "optimization" on its end - most notably on a single > connection. TCP experts may comment on how/if they dealt with a > similar issue. > > OTOH, you had some valid comments on exceptions to ordering during > connection recovery. Perhaps we can move on by making Julian's > proposed stipulation a SHOULD.... > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > Santosh Rao wrote: > > > > Mallikarjun, > > > > Some comments below. > > > > Regards, > > Santosh > > > > "Mallikarjun C." wrote: > > > > > > Rod and Julian, > > > > > > This has been an interesting thread of discussion. Some > > > comments - > > > > > > 1.My first reaction was - allowing out-of-order command > > > transmission on the same connection deprives targets of > > > an implementation choice. Targets which support only > > > single-connection sessions and only support session > > > recovery (reasonable assumptions in my mind) can no > > > longer afford *not to* implement a command scoreboard. > > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > > > Thus, there is no getting away from implementing a sequencer at the > > target. Given this, I think it is unreasonable to restrict initiator > > implementation flexibility by imposing a strict ordering requirement > > within the connection. > > > > > 2.Any end-node efficiency that is sought to be achieved > > > by transmitting CmdSNs out-of-order from the initiator > > > would be lost on the other end-node, since the target > > > now must wait for re-ordering the commands. > > > > It has to handle this situation anyway to deal with holes caused by > > digest errors. This scenario occurs even with initiators that issue > > commands in order. > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > link badwidth (albeit at the expense of end-node efficiency), > > > compared to idling the link waiting for outbound DMA. > > > We have to determine if this is a reasonable trade-off. > > > > > > 4.I can see Rod's point that prefetching all immediate > > > data can be a burden on the NIC resources. But, two > > > questions - > > > - could the NIC not use unsolicited separate data > > > PDUs in these cases? [ I realize that InitialR2T > > > has to be "no" to let it happen... ] > > > - could the NIC have a memory architecture that > > > allows data prefetching for the next command (so > > > this is a non-issue from the protocol perspective)? > > > This scheme incurs one DMA delay for every new > > > burst of commands. > > > > > > 5.Another (perhaps radical at this point) option is to do > > > away with immediate unsolicited data, to stick only with > > > separate unsolicited data. I would personally be okay > > > with the choice, particularly if this feature (that > > > helps software implementations) starts making hardware > > > design complicated/expensive. > > > > > > So, to summarize - > > > > > > option immediate allow > > > data in spec? out-of-order? > > > > > > (A) (5) above no no > > > (B) No real reason to do this. no yes > > > (C) (4) above yes no > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > >From the arguments I heard so far, I am leaning towards > > > option A, and option C in that order. > > > > > > Comments? > > > -- > > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > Rod Harrison wrote: > > > > > > > > Julian, > > > > > > > > I don't understand what you are proposing here, what do you > mean by > > > > "multiplexed" DMA? > > > > > > > > The problem is that the DMAs take some time, the more there > are > > > > queued the longer the last DMAs queued take to complete. Some > commands > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > soon a possible, which for a read after a write can be before the > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > to be sent on the HBA is expensive and redundant since the target > > > > already knows how to queue commands before committing them to its > SCSI > > > > layer. > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > change the > > > > order of commands from the OS, but the DMAs those commands need are > > > > unlikely to complete in the same order, and as I mentioned some > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > order it has to maintain an ordered queue of commands waiting to be > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > immediate data almost impossible to support. > > > > > > > > I don't see the problem with allowing out of order commands > given > > > > that the target already has to deal with very similar problems. I > > > > think we are getting in to the area of implementation choices here, > > > > which is inappropriate for a specification. > > > > > > > > - Rod > > > > From owner-ips@ece.cmu.edu Thu Nov 8 21:45:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA14748 for ; Thu, 8 Nov 2001 21:45:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA921IW14878 for ips-outgoing; Thu, 8 Nov 2001 21:01:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA921Gl14874 for ; Thu, 8 Nov 2001 21:01:16 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id VAA18838; Thu, 8 Nov 2001 21:00:53 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "John Hufferd" Cc: "Robert D. Russell" , "Julian Satran" , Subject: RE: iSCSI: Out of order commands Date: Thu, 8 Nov 2001 17:58:03 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John, > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Thursday, November 08, 2001 5:19 PM > To: somesh_gupta@silverbacksystems.com > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > I understand your point, however, their are several approaches that I and > others have used to mitigate this issue. We face that problem with Multi > Processing all the time. > > One technique that I have seen use that is similar to what you describe is > to allocate a set of buffers to each processor (in this case an HBA), and > when they are used up, and only then, the go back to the Buffer Mother, to > allocate another Buffer Set. In this way the Processors (HBA) run mostly > independent, but once in awhile get a new supply. There are other > techniques also, but that one clearly works wells and is very simple. I agree with the mechanism above. So here is where the problem is. Let us take a simplification of M adapters and 1 connection per adapter. Let us also assume that the N buffers are given to each adapter. Total number of buffers used - M * N Buffers available to each adapter - N How large of a command window should be advertised in the case of a session wide window. If the window is N, then we are under-utilizing the buffers but we will never drop commands. If we advertise M * N, and the initiator decides to send commands in an unbalanced manner (because 1 connection is working better), then we have a potential problem where the one adapter uses up its allocation - In the meantime the system will provide it some more (but not enough), and there is an imbalance leading to a situation where commands may have to be dropped. (this could also happen if the system did not allocate buffers evenly e.g. in a scenario where the adapters are not using buffers at the same rate). Now we can try to find a balance in the middle, but again there is a loss of efficiency on one axis or the other. > > Therefore, I do not see the Session Window as a problem in the protocol. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" on 11/08/2001 04:57:40 > PM > > Please respond to > > To: John Hufferd/San Jose/IBM@IBMUS > cc: "Robert D. Russell" , Julian > Satran/Haifa/IBM@IBMIL, > Subject: RE: iSCSI: Out of order commands > > > > I think we may be talking about different things here. > Yes, there are memories that are "on the iSCSI HBA" > (in the case where everything is on a single board, > this would be the memories that are controlled by > only the iSCSI ASIC or whatever) - and these memories > are not shared among multiple HBAs. These may be used > for any book-keeping purposes, contexts, eddy buffering > etc etc by the adapter. > > Then there is memory pool that is owned by the "system". > Adapters will DMA commands/data/responses directly into > and out of this memory pool which is managed by the "system". > However to prevent adapters from running over each other, > and over the "system" processor, producer/consumer > mechanisms have to be implemented. > > This is necessary to maintain order on the system. So > on a long-term basis, the memory in the system is shared > across the HBAs - absolutely yes. But at any given > instance in time, it is partitioned among various > usages. > > So all in all I agree with you. The only point of difference > is that it is not possible that a given buffer (part of > a buffer pool) is accessible to multiple adapters at > the same instance in time. It is a necessary condition to > prevent corruption and chaos. > > > -----Original Message----- > > From: John Hufferd [mailto:hufferd@us.ibm.com] > > Sent: Thursday, November 08, 2001 4:40 PM > > To: somesh_gupta@silverbacksystems.com > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > > Subject: RE: iSCSI: Out of order commands > > > > > > > > I guess I disagree with this design approach. The buffers on the iSCSI > > HBAs are for Eddy buffers. Once you have the iSCSI Header > should be able > > to place the data directly into a Shared buffer. If you have a session > > that spans HBAs, that should work just fine. But the buffers match the > > Session Window, and the Buffers need to be shared across the > HBAs, not in > > the HBAs. Again, the HBAs should just have the Eddy Buffers. > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSG San Jose Ca > > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > Home Office (408) 997-6136, Cell: (408) 499-9702 > > Internet address: hufferd@us.ibm.com > > > > > > "Somesh Gupta" on 11/08/2001 > 04:31:29 > > PM > > > > Please respond to > > > > To: John Hufferd/San Jose/IBM@IBMUS > > cc: "Robert D. Russell" , Julian > > Satran/Haifa/IBM@IBMIL, > > Subject: RE: iSCSI: Out of order commands > > > > > > > > John, > > > > You are correct if all the connections are on > > a single adapter. In this case the buffer pool > > can be shared among all the connections on the > > adapter. > > > > However this is not true if the connections are > > on different adapters (the point being discussed). > > Think of there being a big buffer pool in the board > > into which multiple adapters/chips are "plugged" in. > > However, at any given point in time, these buffers > > are allocated (handed over through command queues or > > whatever) to each of the individual adapters. And this > > is where the problem arises. > > > > The assumption that adapters can share a typical host > > command buffer pool on their own without host intervention > > is incorrect. > > > > Somesh > > > > > -----Original Message----- > > > From: John Hufferd [mailto:hufferd@us.ibm.com] > > > Sent: Thursday, November 08, 2001 4:20 PM > > > To: somesh_gupta@silverbacksystems.com > > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > > > Subject: RE: iSCSI: Out of order commands > > > > > > > > > > > > Most folks I know have a buffer that applies to all the > > > connections, not to > > > just one buffer per connection. I fail to see the problem for most > > > implimentations. The window should apply to the session. > > > > > > . > > > . > > > . > > > John L. Hufferd > > > Senior Technical Staff Member (STSM) > > > IBM/SSG San Jose Ca > > > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > > Home Office (408) 997-6136, Cell: (408) 499-9702 > > > Internet address: hufferd@us.ibm.com > > > > > > > > > "Somesh Gupta" @ece.cmu.edu on > > > 11/08/2001 02:22:30 PM > > > > > > Please respond to > > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > > > > To: "Robert D. Russell" > > > cc: Julian Satran/Haifa/IBM@IBMIL, > > > Subject: RE: iSCSI: Out of order commands > > > > > > > > > > > > Comments below > > > > > > > -----Original Message----- > > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > > > Sent: Thursday, November 08, 2001 11:46 AM > > > > To: Somesh Gupta > > > > Cc: Julian Satran; ips@ece.cmu.edu > > > > Subject: RE: iSCSI: Out of order commands > > > > > > > > > > > > > > > > Somesh: > > > > > > > > > > It seems to me that if a target can only handle x new commands, > > > > > > then its queueing capacity is x, and in the SCSI Response PDU it > > > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > > > > > formula in section 2.2.2.1. This in turn controls the number of > > > > > > commands the initiator can send, and thus prevents the incoming > > > > > > commands from overfilling the target's queue. > > > > > > > > > > This is actually a weakness of the multiple connections over > > > > > multiple adapters model (it is not related to ordering). > > > > > The target advertises a command window on a session wide basis. > > > > > At the same time, it has to provide the resource to the adapter > > > > > to be able to DMA those commands to. Since there is no > restriction > > > > > on which connection the commands may be received, either the > > > > > target has to allocate the max resources needed to each adapter > > > > > (thus committing n times the resources actually required), or > > > > > has to "lie" (it would not be a complete lie) which could > > > > > result in running out of resources. One way to fix that would be > > > > > to have the credit on a per connection basis. > > > > > > > > If I understand you correctly, you are saying that deadlock can > > > > occur even if we enforce ordered delivery by initiators and even > > > > if the advertisements are correct and even if both parties obey > > > > the advertisements. > > > > > > I would let Julian enlighten us on that, or whether it just > > > leads to wholesale command drops and retries, or TASK SET FULL > > > is returned (is TASK SET FULL a valid response in this case > > > for a LINKED task when it is not the first command?) > > > > > > > > > > > In other words, doesn't this mean that the standard can lead to a > > > > trap in implementing targets and that, in fact, the advertisements > > > > really should be on a per connection basis? > > > > However, what would be the consequences for initiators if > > > > the advertisements were on a per-connection basis? > > > > > > There is really no issue with adveritisement on a per > > > connection basis. It leads to a protocol change. However, it > > > also allows a much more performant flow of commands. > > > > > > > > > > > > > > > Thanks, > > > > > > > > Bob Russell > > > > InterOperability Lab > > > > University of New Hampshire > > > > rdr@iol.unh.edu > > > > 603-862-3774 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 21:45:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA14760 for ; Thu, 8 Nov 2001 21:45:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9280c15339 for ips-outgoing; Thu, 8 Nov 2001 21:08:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA927wl15335 for ; Thu, 8 Nov 2001 21:07:58 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id VAA150158; Thu, 8 Nov 2001 21:05:21 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA927u6224478; Thu, 8 Nov 2001 19:07:56 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: Out of order commands To: Cc: "Robert D. Russell" , "Julian Satran" , X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 8 Nov 2001 18:07:02 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/08/2001 07:07:56 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I understand what you are saying but that is a typical buffer manager problem which most systems work with today. I have usually resolved this in the past with a statistical probability analyses, often that is adjusted in real time. Again, I see nothing unusual here. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Somesh Gupta" on 11/08/2001 05:58:03 PM Please respond to To: John Hufferd/San Jose/IBM@IBMUS cc: "Robert D. Russell" , Julian Satran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: Out of order commands John, > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > Sent: Thursday, November 08, 2001 5:19 PM > To: somesh_gupta@silverbacksystems.com > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > > I understand your point, however, their are several approaches that I and > others have used to mitigate this issue. We face that problem with Multi > Processing all the time. > > One technique that I have seen use that is similar to what you describe is > to allocate a set of buffers to each processor (in this case an HBA), and > when they are used up, and only then, the go back to the Buffer Mother, to > allocate another Buffer Set. In this way the Processors (HBA) run mostly > independent, but once in awhile get a new supply. There are other > techniques also, but that one clearly works wells and is very simple. I agree with the mechanism above. So here is where the problem is. Let us take a simplification of M adapters and 1 connection per adapter. Let us also assume that the N buffers are given to each adapter. Total number of buffers used - M * N Buffers available to each adapter - N How large of a command window should be advertised in the case of a session wide window. If the window is N, then we are under-utilizing the buffers but we will never drop commands. If we advertise M * N, and the initiator decides to send commands in an unbalanced manner (because 1 connection is working better), then we have a potential problem where the one adapter uses up its allocation - In the meantime the system will provide it some more (but not enough), and there is an imbalance leading to a situation where commands may have to be dropped. (this could also happen if the system did not allocate buffers evenly e.g. in a scenario where the adapters are not using buffers at the same rate). Now we can try to find a balance in the middle, but again there is a loss of efficiency on one axis or the other. > > Therefore, I do not see the Session Window as a problem in the protocol. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Somesh Gupta" on 11/08/2001 04:57:40 > PM > > Please respond to > > To: John Hufferd/San Jose/IBM@IBMUS > cc: "Robert D. Russell" , Julian > Satran/Haifa/IBM@IBMIL, > Subject: RE: iSCSI: Out of order commands > > > > I think we may be talking about different things here. > Yes, there are memories that are "on the iSCSI HBA" > (in the case where everything is on a single board, > this would be the memories that are controlled by > only the iSCSI ASIC or whatever) - and these memories > are not shared among multiple HBAs. These may be used > for any book-keeping purposes, contexts, eddy buffering > etc etc by the adapter. > > Then there is memory pool that is owned by the "system". > Adapters will DMA commands/data/responses directly into > and out of this memory pool which is managed by the "system". > However to prevent adapters from running over each other, > and over the "system" processor, producer/consumer > mechanisms have to be implemented. > > This is necessary to maintain order on the system. So > on a long-term basis, the memory in the system is shared > across the HBAs - absolutely yes. But at any given > instance in time, it is partitioned among various > usages. > > So all in all I agree with you. The only point of difference > is that it is not possible that a given buffer (part of > a buffer pool) is accessible to multiple adapters at > the same instance in time. It is a necessary condition to > prevent corruption and chaos. > > > -----Original Message----- > > From: John Hufferd [mailto:hufferd@us.ibm.com] > > Sent: Thursday, November 08, 2001 4:40 PM > > To: somesh_gupta@silverbacksystems.com > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > > Subject: RE: iSCSI: Out of order commands > > > > > > > > I guess I disagree with this design approach. The buffers on the iSCSI > > HBAs are for Eddy buffers. Once you have the iSCSI Header > should be able > > to place the data directly into a Shared buffer. If you have a session > > that spans HBAs, that should work just fine. But the buffers match the > > Session Window, and the Buffers need to be shared across the > HBAs, not in > > the HBAs. Again, the HBAs should just have the Eddy Buffers. > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSG San Jose Ca > > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > Home Office (408) 997-6136, Cell: (408) 499-9702 > > Internet address: hufferd@us.ibm.com > > > > > > "Somesh Gupta" on 11/08/2001 > 04:31:29 > > PM > > > > Please respond to > > > > To: John Hufferd/San Jose/IBM@IBMUS > > cc: "Robert D. Russell" , Julian > > Satran/Haifa/IBM@IBMIL, > > Subject: RE: iSCSI: Out of order commands > > > > > > > > John, > > > > You are correct if all the connections are on > > a single adapter. In this case the buffer pool > > can be shared among all the connections on the > > adapter. > > > > However this is not true if the connections are > > on different adapters (the point being discussed). > > Think of there being a big buffer pool in the board > > into which multiple adapters/chips are "plugged" in. > > However, at any given point in time, these buffers > > are allocated (handed over through command queues or > > whatever) to each of the individual adapters. And this > > is where the problem arises. > > > > The assumption that adapters can share a typical host > > command buffer pool on their own without host intervention > > is incorrect. > > > > Somesh > > > > > -----Original Message----- > > > From: John Hufferd [mailto:hufferd@us.ibm.com] > > > Sent: Thursday, November 08, 2001 4:20 PM > > > To: somesh_gupta@silverbacksystems.com > > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu > > > Subject: RE: iSCSI: Out of order commands > > > > > > > > > > > > Most folks I know have a buffer that applies to all the > > > connections, not to > > > just one buffer per connection. I fail to see the problem for most > > > implimentations. The window should apply to the session. > > > > > > . > > > . > > > . > > > John L. Hufferd > > > Senior Technical Staff Member (STSM) > > > IBM/SSG San Jose Ca > > > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > > Home Office (408) 997-6136, Cell: (408) 499-9702 > > > Internet address: hufferd@us.ibm.com > > > > > > > > > "Somesh Gupta" @ece.cmu.edu on > > > 11/08/2001 02:22:30 PM > > > > > > Please respond to > > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > > > > To: "Robert D. Russell" > > > cc: Julian Satran/Haifa/IBM@IBMIL, > > > Subject: RE: iSCSI: Out of order commands > > > > > > > > > > > > Comments below > > > > > > > -----Original Message----- > > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu] > > > > Sent: Thursday, November 08, 2001 11:46 AM > > > > To: Somesh Gupta > > > > Cc: Julian Satran; ips@ece.cmu.edu > > > > Subject: RE: iSCSI: Out of order commands > > > > > > > > > > > > > > > > Somesh: > > > > > > > > > > It seems to me that if a target can only handle x new commands, > > > > > > then its queueing capacity is x, and in the SCSI Response PDU it > > > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the > > > > > > formula in section 2.2.2.1. This in turn controls the number of > > > > > > commands the initiator can send, and thus prevents the incoming > > > > > > commands from overfilling the target's queue. > > > > > > > > > > This is actually a weakness of the multiple connections over > > > > > multiple adapters model (it is not related to ordering). > > > > > The target advertises a command window on a session wide basis. > > > > > At the same time, it has to provide the resource to the adapter > > > > > to be able to DMA those commands to. Since there is no > restriction > > > > > on which connection the commands may be received, either the > > > > > target has to allocate the max resources needed to each adapter > > > > > (thus committing n times the resources actually required), or > > > > > has to "lie" (it would not be a complete lie) which could > > > > > result in running out of resources. One way to fix that would be > > > > > to have the credit on a per connection basis. > > > > > > > > If I understand you correctly, you are saying that deadlock can > > > > occur even if we enforce ordered delivery by initiators and even > > > > if the advertisements are correct and even if both parties obey > > > > the advertisements. > > > > > > I would let Julian enlighten us on that, or whether it just > > > leads to wholesale command drops and retries, or TASK SET FULL > > > is returned (is TASK SET FULL a valid response in this case > > > for a LINKED task when it is not the first command?) > > > > > > > > > > > In other words, doesn't this mean that the standard can lead to a > > > > trap in implementing targets and that, in fact, the advertisements > > > > really should be on a per connection basis? > > > > However, what would be the consequences for initiators if > > > > the advertisements were on a per-connection basis? > > > > > > There is really no issue with adveritisement on a per > > > connection basis. It leads to a protocol change. However, it > > > also allows a much more performant flow of commands. > > > > > > > > > > > > > > > Thanks, > > > > > > > > Bob Russell > > > > InterOperability Lab > > > > University of New Hampshire > > > > rdr@iol.unh.edu > > > > 603-862-3774 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 8 21:49:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA14784 for ; Thu, 8 Nov 2001 21:49:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA92Q3m16640 for ips-outgoing; Thu, 8 Nov 2001 21:26:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA92Q2l16636 for ; Thu, 8 Nov 2001 21:26:02 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fA92PfQ06469; Thu, 8 Nov 2001 21:25:41 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fA92PZg08908; Thu, 8 Nov 2001 21:25:38 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15339.16039.876000.656074@gargle.gargle.HOWL> Date: Thu, 8 Nov 2001 21:25:43 -0500 From: Paul Koning To: Julian_Satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands References: X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 8 November 2001) by Julian Satran: > Ron, > > Targets will most likely advertise a total (command and data) window > larger than they can accommodate on any long haul link. With the current > ordering rules nothing bad will happen. This is a very bad idea. The whole notion of a window is that you advertise what you can handle. You DO NOT, EVER, advertise more than you can handle in the hope that you won't be caught. Long haul links are no excuse. The way you deal with those is by having more buffer capacity, and issuing window increases early enough to keep data flowing. In a window flow control scheme, a target that advertises resources it does not have is defective. This is all VERY old hat. TCP has understood how to do this for ages. In particular, it has always been well understood that you HAVE to have more buffers in order to run at high speed over long delay links. In essence, what you're saying is that we have an application protocol here that is so unusual that it should throw out the lessons of the past 40 years, which have taught us how you construct correct protocols. This makes no sense whatsoever to me. I think the notion of out of order commands on a single connection is somewhat strange, but your reason for opposing it is several order of magnitude stranger. paul From owner-ips@ece.cmu.edu Fri Nov 9 04:04:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA05008 for ; Fri, 9 Nov 2001 04:04:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA97wSq07510 for ips-outgoing; Fri, 9 Nov 2001 02:58:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA97wPl07501 for ; Fri, 9 Nov 2001 02:58:25 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA11588 for ; Fri, 9 Nov 2001 08:58:18 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA97wCc82416 for ; Fri, 9 Nov 2001 08:58:12 +0100 Subject: RE: iSCSI: Out of order commands To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Julian Satran" Date: Fri, 9 Nov 2001 09:39:21 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 09:58:12 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk The data generated in response to an R2T for C1 do not have to be ordered. The data can be multiplexed with commands. Ordering applies only for commands, unsolicited data and the order in which R2T are satisfied. The ordering rules exist also for targets. A target is not assumed to send a data or status PDU out of order. Julo "FRYMAN,ROBERT (A-Roseville,ex1) To: "'Barry Reinhold'" " , Rod Harrison , "Robert D. ilent.com> Russell" , Somesh Gupta 08-11-01 19:03 cc: Julian Satran/Haifa/IBM@IBMIL, Please respond to ips@ece.cmu.edu "FRYMAN,ROBERT Subject: RE: iSCSI: Out of order commands (A-Roseville,ex1) " > Thus the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry > about the fact that C1 generates data in response to the R2T. > However the command sequence of C1, C3, C2, C4 forces the iSCSI > layer to buffer C3. This is significant to a hardware accelerated > iSCSI layer. This is significant to a target hardware accelerated iSCSI layer. By forcing the initiator to order the iSCSI commands, you move the significants from the target to the initiator, but it is still a complexity added to a hardware accelerator. The problem I have is that we are trying to add a requirement that only effects a specific area, but adds complexity to a larger area. If we say that initiators must order commands on a single connection, then doesn't this mean that we have just added complexity and slowdown to the enterprise storage area? In the Fibre channel realm, a standard base configuration was to have a target (JBOD, array, etc.) with 2 ports, and an initiator with two ports. These would be connected on two seperate physical connections, so if a problem happens on one, the other will take over. We allow the same thing in iSCSI, plus specified how the customer can use both connections to increase bandwidth as long as both connections work. It seems to me that any customer that wants a fault tolerant solution, and gets increased bandwidth for free due to the ability to have multiple connections, would take advantage of that. This being the case, these targets have to have reordering capability. If we force the initiators to send the comands in order, we just added complexity and slowdown that adds no benifit. As I see it (and I could be totally off base here), we have a choice of complexity and slowdown in the enterprise market, or adding a little complexity to simple targets. Am I flawed in this thinking? Scott Fryman Agilent Technologies > -----Original Message----- > From: Barry Reinhold [mailto:bbrtrebia@mediaone.net] > Sent: Thursday, November 08, 2001 6:19 AM > To: Rod Harrison; Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Rod, > I can see your point, from the perspective of a target > that is a box > combining an iSCSI entity with a SCSI entity. However, from > the perspective > of an "iSCSI only" entity, delivery does not have to be > delayed until C1 > completes. In this case you only need to ensure that the commands are > delivered in order to the SCSI layer. The SCSI layer has to > address the > semantics of what the operation entails in terms of command > execution. Thus > the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry > about the fact > that C1 generates data in response to the R2T. However the > command sequence > of C1, C3, C2, C4 forces the iSCSI layer to buffer C3. This > is significant > to a hardware accelerated iSCSI layer. > > >-----Original Message----- > >From: Rod Harrison [mailto:rod.harrison@windriver.com] > >Sent: Wednesday, November 07, 2001 8:34 PM > >To: Barry Reinhold; Robert D. Russell; Somesh Gupta > >Cc: Julian Satran; ips@ece.cmu.edu > >Subject: RE: iSCSI: Out of order commands > > > > > >Barry, > > > > I'm curious, where do you see the extra complexity between the > >following? > > > > Assuming all the following commands are writes ... > > > >c1, c3, c2, c4 and > > > >c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. > > > > In both cases the target has to queue commands 2, 3, > and 4 whilst > >waiting for its R2T on c1 to be satisfied. > > > > Even if the target doesn't support any unsolicited data > it still has > >to queue commands 2, 3, and 4. I can't see how the target can avoid > >queuing commands, in which case the order of arrival seems to make > >little difference. > > > > - Rod > > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of > >Barry Reinhold > >Sent: Wednesday, November 07, 2001 3:10 PM > >To: Robert D. Russell; Somesh Gupta > >Cc: Julian Satran; ips@ece.cmu.edu > >Subject: RE: iSCSI: Out of order commands > > > > > >Bob, > > I can't speak for targets, but OOO commands on a session with a > >single > >connection sure increases the complexity of the code path we take. > > To me the issue is still that these types of situations > tend to be > >poorly > >tested and lead to interoperability issues. If we do go down this > >path, the > >spec. should make it very clear that this is expected behavior. Some > >statement with a shall in it should be written (for the receiver), so > >that > >there is a conformance test items to pass. > > > >>-----Original Message----- > >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > >Of > >>Robert D. Russell > >>Sent: Wednesday, November 07, 2001 4:13 PM > >>To: Somesh Gupta > >>Cc: Julian Satran; ips@ece.cmu.edu > >>Subject: RE: iSCSI: Out of order commands > >> > >> > >>Somesh, Julian: > >> > >>You state that dealing with OOO commands on the target > >>will add substantial complexity on the target. > >>Do you have any basis for that claim? My impression from the > >>plugfest is that most targets are already dealing with > >>it. Perhaps we need to hear from someone who is actually > >>building a target for which this would be a real problem. > >> > >>If anything, what we are hearing from people who really > >>are building initiators is that dealing with the requirement > >>to send commands in order will introduce substantial complexity > >>on the initiator. > >> > >>So why should we be saving complexity on (hypothetically) simple > >>targets yet requiring complexity on real initiators? > >> > >>As far as the deadlock issue is concerned, if the only way > >>that deadlock can occur with OOO commands on the same > >>connection is during the use of immediate data (which is I > >>think what Julian was saying), then shouldn't we change > >>the standard to just say that -- if the initiator sends > >>commands out of order on a single connection, then immediate > >>data MUST NOT be used on that connection in order to avoid deadlock. > >> > >>This gives everybody what they want, since initiators who find > >>it beneficial to deliver commands OOO will just negotiate > >>immediate data off. Those who really want to use immediate data > >>will have to ensure that commands are sent in order. > >>The tradeoff then becomes an implementation issue, not a > >>standards issue, which is where it belongs. > >> > >> > >>Bob Russell > >>InterOperability Lab > >>University of New Hampshire > >>rdr@iol.unh.edu > >>603-862-3774 > >> > >> > >>On Wed, 7 Nov 2001, Somesh Gupta wrote: > >> > >>> I think we should either have it as a MUST or not require > >>> it (at both ends to get the real benefit). SHOULD is one > >>> of those things that leads to implementation > >>> burden and confusion, without perhaps the feature being > >>> used. There are implementation as well as protocol > >>> considerations mixed in here. > >>> > >>> If we are to remove the restriction, we should (SHOULD) > >>> get the maximum benefit from it, rather than to > >>> accomodate an implementation choice. Out of sequence > >>> commands, combined with the possibility of digest errors, > >>> will add substantial complexity on the target side, > >>> without corrosponding benefit in performance. If we change > >>> this to SHOULD, we should also relax the requirement > >>> to present commands on the target side to a SHOULD. > >>> > >>> > >>> > >>> > -----Original Message----- > >>> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > >Behalf Of > >>> > Julian Satran > >>> > Sent: Wednesday, November 07, 2001 10:00 AM > >>> > To: ips@ece.cmu.edu > >>> > Subject: Re: iSCSI: Out of order commands > >>> > > >>> > > >>> > Mallikarjun, > >>> > > >>> > I did not see a SINGLE performance improvement that results from > >OOO > >>> > shipping. > >>> > I would be bad engineering to give away the "no-deadlock" > >mechanism we > >>> > have now for nothing. > >>> > I have also the impression that the point about deadlock that I > >keep > >>> > repeating is ignored or not understood. > >>> > As we stand today commands can be shipped with Immediate data > >>or without > >>> > and an implementer determined > >>> > to squeeze maximum bandwidth and overlap command start with > >>delivery will > >>> > choose not to work with immediate data > >>> > (as you have pointed out) while a low performance software > >>implementation > >>> > will use immediate data to minimize CPU cycles > consumed. However > >both > >>> > will be guaranteed to work without deadlock as source and sink > >use the > >>> > same ordering. > >>> > Recovery is still a low probability event and should be handled > >with a > >>> > different set of considerations in mind. > >>> > As for the strictness of the recommendation - yes we > could settle > >on > >>> > SHOULD. > >>> > > >>> > Julo > >>> > > >>> > > >>> > > >>> > > >>> > "Mallikarjun C." > >>> > Sent by: owner-ips@ece.cmu.edu > >>> > 07-11-01 19:41 > >>> > Please respond to cbm > >>> > > >>> > > >>> > To: Santosh Rao , > >ips@ece.cmu.edu > >>> > cc: > >>> > Subject: Re: iSCSI: Out of order commands > >>> > > >>> > > >>> > > >>> > Santosh, > >>> > > >>> > I have only one comment on your responses. > >>> > > >>> > > Even a single connection target *MUST* implement a scoreboard. > >The > >>> > > reason being that it can see out-of-order arrival of commands > >due to > >>> > > commands being dropped on digest errors. In such a case, it > >>must block > >>> > > further command processing until holes are filled. > >>> > > >>> > I made two convenient assumptions if you noticed, :-), one of > >which > >>> > is that target forces session recovery on *any* error that it > >sees > >>> > (ErrorRecoveryLevel=0) - including a dropped command due to a > >digest > >>> > error. With that assumption, a target can afford not to > >implement > >>> > a scoreboard. > >>> > > >>> > As I said in a private note, I guess what primarily bothers me > >about > >>> > OOO commands on a connection is that it requires the receiver to > >>> > undo this "optimization" on its end - most notably on a single > >>> > connection. TCP experts may comment on how/if they dealt with a > >>> > similar issue. > >>> > > >>> > OTOH, you had some valid comments on exceptions to ordering > >during > >>> > connection recovery. Perhaps we can move on by making Julian's > >>> > proposed stipulation a SHOULD.... > >>> > -- > >>> > Mallikarjun > >>> > > >>> > > >>> > Mallikarjun Chadalapaka > >>> > Networked Storage Architecture > >>> > Network Storage Solutions Organization > >>> > MS 5668 Hewlett-Packard, Roseville. > >>> > cbm@rose.hp.com > >>> > > >>> > > >>> > Santosh Rao wrote: > >>> > > > >>> > > Mallikarjun, > >>> > > > >>> > > Some comments below. > >>> > > > >>> > > Regards, > >>> > > Santosh > >>> > > > >>> > > "Mallikarjun C." wrote: > >>> > > > > >>> > > > Rod and Julian, > >>> > > > > >>> > > > This has been an interesting thread of discussion. Some > >>> > > > comments - > >>> > > > > >>> > > > 1.My first reaction was - allowing out-of-order command > >>> > > > transmission on the same connection deprives targets of > >>> > > > an implementation choice. Targets which support only > >>> > > > single-connection sessions and only support session > >>> > > > recovery (reasonable assumptions in my mind) can no > >>> > > > longer afford *not to* implement a command scoreboard. > >>> > > > >>> > > Even a single connection target *MUST* implement a scoreboard. > >The > >>> > > reason being that it can see out-of-order arrival of commands > >due to > >>> > > commands being dropped on digest errors. In such a case, it > >>must block > >>> > > further command processing until holes are filled. > >>> > > > >>> > > Thus, there is no getting away from implementing a > sequencer at > >the > >>> > > target. Given this, I think it is unreasonable to restrict > >initiator > >>> > > implementation flexibility by imposing a strict ordering > >requirement > >>> > > within the connection. > >>> > > > >>> > > > 2.Any end-node efficiency that is sought to be achieved > >>> > > > by transmitting CmdSNs out-of-order from the initiator > >>> > > > would be lost on the other end-node, since the target > >>> > > > now must wait for re-ordering the commands. > >>> > > > >>> > > It has to handle this situation anyway to deal with holes > >caused by > >>> > > digest errors. This scenario occurs even with initiators that > >issue > >>> > > commands in order. > >>> > > > >>> > > > > >>> > > > 3.The flipside is that out-of-order transmission saves > >>> > > > link badwidth (albeit at the expense of end-node > >efficiency), > >>> > > > compared to idling the link waiting for outbound DMA. > >>> > > > We have to determine if this is a reasonable trade-off. > >>> > > > > >>> > > > 4.I can see Rod's point that prefetching all immediate > >>> > > > data can be a burden on the NIC resources. But, two > >>> > > > questions - > >>> > > > - could the NIC not use unsolicited separate data > >>> > > > PDUs in these cases? [ I realize that InitialR2T > >>> > > > has to be "no" to let it happen... ] > >>> > > > - could the NIC have a memory architecture that > >>> > > > allows data prefetching for the next command (so > >>> > > > this is a non-issue from the protocol > perspective)? > >>> > > > This scheme incurs one DMA delay for every new > >>> > > > burst of commands. > >>> > > > > >>> > > > 5.Another (perhaps radical at this point) option is to do > >>> > > > away with immediate unsolicited data, to stick only with > >>> > > > separate unsolicited data. I would personally be okay > >>> > > > with the choice, particularly if this feature (that > >>> > > > helps software implementations) starts making hardware > >>> > > > design complicated/expensive. > >>> > > > > >>> > > > So, to summarize - > >>> > > > > >>> > > > option immediate allow > >>> > > > data in spec? > >out-of-order? > >>> > > > > >>> > > > (A) (5) above no no > >>> > > > (B) No real reason to do this. no yes > >>> > > > (C) (4) above yes no > >>> > > > (D) pros & cons (1), (2) & (3) yes yes > >>> > > > > >>> > > > >From the arguments I heard so far, I am leaning towards > >>> > > > option A, and option C in that order. > >>> > > > > >>> > > > Comments? > >>> > > > -- > >>> > > > Mallikarjun > >>> > > > > >>> > > > Mallikarjun Chadalapaka > >>> > > > Networked Storage Architecture > >>> > > > Network Storage Solutions Organization > >>> > > > MS 5668 Hewlett-Packard, Roseville. > >>> > > > cbm@rose.hp.com > >>> > > > > >>> > > > Rod Harrison wrote: > >>> > > > > > >>> > > > > Julian, > >>> > > > > > >>> > > > > I don't understand what you are proposing here, > >>what do you > >>> > mean by > >>> > > > > "multiplexed" DMA? > >>> > > > > > >>> > > > > The problem is that the DMAs take some time, the > >>more there > >>> > are > >>> > > > > queued the longer the last DMAs queued take to complete. > >Some > >>> > commands > >>> > > > > require DMAs to complete before they can be sent, i.e. > >>Writes with > >>> > > > > immediate data, some commands do not, i.e. Reads and > >>writes with no > >>> > > > > immediate data. The iSCSI HBA wants to be able to send > >>commands as > >>> > > > > soon a possible, which for a read after a write can be > >before the > >>> > > > > write's DMA has completed. Maintaining an ordered queue > >>for commands > >>> > > > > to be sent on the HBA is expensive and redundant since the > >target > >>> > > > > already knows how to queue commands before committing them > >to its > >>> > SCSI > >>> > > > > layer. > >>> > > > > > >>> > > > > The iSCSI HBA and its host driver are not at > >liberty to > >>> > change the > >>> > > > > order of commands from the OS, but the DMAs those > >>commands need are > >>> > > > > unlikely to complete in the same order, and as I mentioned > >some > >>> > > > > commands need no DMA. If the HBA can't send > commands out of > >CmdSN > >>> > > > > order it has to maintain an ordered queue of commands > >>waiting to be > >>> > > > > sent, and potentially buffer a lot of data. For > an HBA this > >makes > >>> > > > > immediate data almost impossible to support. > >>> > > > > > >>> > > > > I don't see the problem with allowing out of > >>order commands > >>> > given > >>> > > > > that the target already has to deal with very similar > >problems. I > >>> > > > > think we are getting in to the area of implementation > >>choices here, > >>> > > > > which is inappropriate for a specification. > >>> > > > > > >>> > > > > - Rod > >>> > > > > > >>> > > > > -----Original Message----- > >>> > > > > From: owner-ips@ece.cmu.edu > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > >>> > Of > >>> > > > > Julian Satran > >>> > > > > Sent: Monday, November 05, 2001 10:06 PM > >>> > > > > To: ips@ece.cmu.edu > >>> > > > > Subject: Re: iSCSI: Out of order commands, was current > >>UNH Plugfest > >>> > > > > > >>> > > > > Rod, > >>> > > > > > >>> > > > > I don't see any reason why DMA operations cant be > >>"multiplexed" with > >>> > > > > commands. > >>> > > > > If you have scheduled a long outbound DMA you are doomed > >>regardless > >>> > of > >>> > > > > the > >>> > > > > command ordering. > >>> > > > > And if you have scheduled DMA operations > piecemeal then you > >can > >>> > insert > >>> > > > > your commands in correct order. > >>> > > > > > >>> > > > > Julo > >>> > > > > > >>> > > > > "Rod Harrison" > >>> > > > > 05-11-01 20:48 > >>> > > > > Please respond to "Rod Harrison" > >>> > > > > > >>> > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > > >>> > > > > cc: > >>> > > > > Subject: iSCSI: Out of order commands, was > >current > >>> > UNH > >>> > > > > Plugfest > >>> > > > > > >>> > > > > [ Subject changed ] > >>> > > > > > >>> > > > > Julian, > >>> > > > > > >>> > > > > The ordering difference is introduced > >>between the > >>> > > > > host > >>> > > > > side driver > >>> > > > > and the iSCSI HBA. The host side driver must present > >>SCSI commands > >>> > to > >>> > > > > the HBA in the order they are received from the OS to > >>prevent read > >>> > > > > after write dependency failures. The HBA might reorder > >>the commands > >>> > > > > depending on when DMA completes. The reordering can't be > >>done ahead > >>> > of > >>> > > > > time in the host driver since it doesn't know how > long each > >DMA > >>> > might > >>> > > > > take. As long as the HBA assigns CmdSN in the order it > >receives > >>> > > > > commands the desired host ordering is preserved. > >>> > > > > > >>> > > > > - Rod > >>> > > > > > >>> > > > > -----Original Message----- > >>> > > > > From: owner-ips@ece.cmu.edu > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > >>> > Of > >>> > > > > Julian Satran > >>> > > > > Sent: Monday, November 05, 2001 12:35 AM > >>> > > > > To: ips@ece.cmu.edu > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > >>> > > > > > >>> > > > > Rod, > >>> > > > > > >>> > > > > I all examples give the point I find hard to understand is > >why is > >>> > the > >>> > > > > ordering on the wire different from the presentation order > >to the > >>> > > > > initiator. You can get as many overlaps as you want by > >>presenting > >>> > the > >>> > > > > commands to the initiator in the desired order. > >>> > > > > What we are considering here is the case in which you > >>want to ship > >>> > in > >>> > > > > an > >>> > > > > order different than the one you present the commands. > >>> > > > > > >>> > > > > Julo > >>> > > > > > >>> > > > > "Rod Harrison" > >>> > > > > Sent by: owner-ips@ece.cmu.edu > >>> > > > > 04-11-01 04:42 > >>> > > > > Please respond to "Rod Harrison" > >>> > > > > > >>> > > > > To: "Barry Reinhold" , > >"Dave > >>> > > > > Sheehy" > >>> > > > > , "IETF IP SAN Reflector" > >>> > > > > > >>> > > > > cc: > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > >>> > > > > > >>> > > > > Barry, > >>> > > > > > >>> > > > > In general I agree but I don't think this > >is as > >>> > much > >>> > > > > of a > >>> > > > > corner case > >>> > > > > as it at first appears. Targets will have code very > >>similar to that > >>> > > > > needed to handle out of order commands to deal with > >>digest errors. > >>> > > > > Targets also need to queue commands whilst > waiting for both > >>> > solicited > >>> > > > > and unsolicited data to arrive. Queuing out of order > >>commands seems > >>> > > > > little extra work. > >>> > > > > > >>> > > > > From an initiators point of view > there are > >>> > > > > efficiency, > >>> > > > > and probably > >>> > > > > performance gains to be had from sending commands out of > >>order. Bob > >>> > > > > Russell gave the example of a read being sent whilst > >>write data DMA > >>> > is > >>> > > > > happening, and a similar situation can arise with DMA for > >writes > >>> > > > > overtaking that of earlier writes if the initiator has > >>multiple DMA > >>> > > > > engines. In this case the initiator might be forced to > >>let the wire > >>> > go > >>> > > > > idle if it can't send the data from completed DMAs as soon > >as > >>> > > > > possible. > >>> > > > > > >>> > > > > We already have a command queue at the > >target to > >>> > > > > enforce > >>> > > > > correct > >>> > > > > serialisation of commands, doing the same thing at the > >>initiator is > >>> > > > > redundant. > >>> > > > > > >>> > > > > Finally, I don't believe we should be > >writing a > >>> > > > > standard > >>> > > > > to work > >>> > > > > around poor coding and test coverage, especially at the > >cost of > >>> > > > > potential efficiency gains. > >>> > > > > > >>> > > > > I agree with Dave and Santosh that > >>commands being > >>> > > > > sent > >>> > > > > out of order > >>> > > > > on a single session should be allowed by the standard. > >>> > > > > > >>> > > > > - Rod > >>> > > > > > >>> > > > > -----Original Message----- > >>> > > > > From: owner-ips@ece.cmu.edu > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > >>> > Of > >>> > > > > Barry Reinhold > >>> > > > > Sent: Friday, November 02, 2001 5:24 PM > >>> > > > > To: Dave Sheehy; IETF IP SAN Reflector > >>> > > > > Subject: RE: iSCSI: current UNH Plugfest > >>> > > > > > >>> > > > > Using features such as out of order command delivery on > >>a connection > >>> > > > > tend to > >>> > > > > be the sort of things that lead to interoperability > >>problems. It is > >>> > > > > unexpected and probably going to hit poorly tested code > >>paths even > >>> > if > >>> > > > > the > >>> > > > > standard is written to allow it. > >>> > > > > > >>> > > > > >-----Original Message----- > >>> > > > > >From: owner-ips@ece.cmu.edu > >>[mailto:owner-ips@ece.cmu.edu]On Behalf > >>> > > > > Of > >>> > > > > >Dave Sheehy > >>> > > > > >Sent: Friday, November 02, 2001 4:19 PM > >>> > > > > >To: IETF IP SAN Reflector > >>> > > > > >Subject: Re: iSCSI: current UNH Plugfest > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > >> 3. Can commands be sent out of order on the same > >connection? > >>> > > > > >> > >>> > > > > >> The behavior of targets is clearly specified in > >Section > >>> > 2.2.2.3 > >>> > > > > on > >>> > > > > >> page 25 of draft 8, which says: > >>> > > > > >> "Except for the commands marked for immediate > >>delivery the > >>> > > > > iSCSI > >>> > > > > >> target layer MUST eliver the commands for > >>execution in the > >>> > > > > order > >>> > > > > >> specified by CmdSN." > >>> > > > > >> > >>> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >>> > > > > >> "- CmdSN - the current command Sequence Number > >>advanced by 1 > >>> > > > > on > >>> > > > > >> each command shipped except for commands > marked for > >>> > immediate > >>> > > > > >> delivery." > >>> > > > > >> but the meaning of the term "shipped" is vague, > >>and does not > >>> > > > > >> necessarily > >>> > > > > >> require that the PDUs arrive on the other end of a > >TCP > >>> > > > > connection > >>> > > > > >> in the same order that the CmdSN values were > >>assigned to these > >>> > > > > PDUs. > >>> > > > > >> > >>> > > > > >> Some initiators have been designed to send commands > >out of > >>> > CmdSN > >>> > > > > >> order on one connection. Consider the situation > >>where there > >>> > is > >>> > > > > only > >>> > > > > >> one connection and a high-level dispatcher creates > >>a PDU for a > >>> > > > > SCSI > >>> > > > > >> command that involves writing immediate data to the > >target. > >>> > > > > This PDU > >>> > > > > >> is enqueued to a lower-level layer which has to > >>setup, start, > >>> > > > > and > >>> > > > > >> wait-for a DMA operation to move the immediate data > >into an > >>> > > > > onboard > >>> > > > > >> buffer before the PDU can be put onto the wire. > >>While this is > >>> > > > > >> happening, the dispatcher creates another > >>unrelated PDU for a > >>> > > > > SCSI > >>> > > > > >> read command (for example), and when this PDU is > >>passed to the > >>> > > > > >> lower-level layer it can be sent immediately, ahead > >of the > >>> > > > > previous > >>> > > > > >> write PDU and therefore out of order on this > >connection. > >>> > > > > >> > >>> > > > > >> The standard clearly allows this to happen > if the two > >PDUs > >>> > were > >>> > > > > sent > >>> > > > > >> on different connections, and seems to imply that > >this can > >>> > also > >>> > > > > happen > >>> > > > > >> when the two PDUs are sent on the same connection. > >>> > > > > >> > >>> > > > > >> The suggestion is to put in the standard an > >>explicit statement > >>> > > > > that > >>> > > > > >> this is allowed or not allowed, as appropriate. > >>> > > > > >> > >>> > > > > >> If this is allowed, such a statement would avoid > >>the erroneous > >>> > > > > >> assumption being made by some target implementers > >>that within > >>> > a > >>> > > > > single > >>> > > > > >> connection, commands will arrive in order. > >>> > > > > >> > >>> > > > > >> If this is not allowed, such a statement would avoid > >the > >>> > > > > erroneous > >>> > > > > >> assumption being made by some initiator implementers > >that > >>> > within > >>> > > > > a > >>> > > > > >> single connection, commands can be put on the wire > >out of > >>> > order. > >>> > > > > >> > >>> > > > > >> +++ > >>> > > > > >> > >>> > > > > >> will add an explicit statement saying that this > >behaviour is > >>> > > > > forbidden. > >>> > > > > >> 2.2.2.1 will contain: > >>> > > > > >> > >>> > > > > >> On any given connection, the iSCSI initiator MUST send > >the > >>> > > > > >commands in the > >>> > > > > >> order specified by CmdSN. > >>> > > > > >> > >>> > > > > >> +++ > >>> > > > > > > >>> > > > > >Why do you feel this behavior should be forbidden? > >>Targets already > >>> > > > > have to > >>> > > > > >order commands across the session. I don't see why it's > >>a problem > >>> > to > >>> > > > > extend > >>> > > > > >that to the connection as well. I, for one, believe we > >>should take > >>> > > > > >a liberal > >>> > > > > >stance on this. > >>> > > > > > > >>> > > > > >Dave Sheehy > >>> > > > > > > >>> > > > >>> > > -- > >>> > > ################################## > >>> > > Santosh Rao > >>> > > Software Design Engineer, > >>> > > HP-UX iSCSI Driver Team, > >>> > > Hewlett Packard, Cupertino. > >>> > > email : santoshr@cup.hp.com > >>> > > Phone : 408-447-3751 > >>> > > ################################## > >>> > > >>> > > >>> > > >>> > > >>> > >> > > > From owner-ips@ece.cmu.edu Fri Nov 9 05:51:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA06287 for ; Fri, 9 Nov 2001 05:51:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA99uUj13976 for ips-outgoing; Fri, 9 Nov 2001 04:56:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA99uSl13972 for ; Fri, 9 Nov 2001 04:56:29 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA125914 for ; Fri, 9 Nov 2001 10:56:22 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA99uLc116920 for ; Fri, 9 Nov 2001 10:56:21 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI over TLS X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 11:56:25 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 11:56:21, Serialize complete at 09/11/2001 11:56:21 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, It is not me you have to blame. Neither on the list nor on face-to-face the group could not reach a consensus on making it must for the sender if the receiver so wishes. Julo Dave Sheehy Sent by: owner-ips@ece.cmu.edu 08-11-01 22:42 Please respond to Dave Sheehy To: ips@ece.cmu.edu (IETF IP SAN Reflector) cc: Subject: Re: iSCSI over TLS Julo, > A group of us seriously considered TLS. The main reason for dropping it > was that it would interfere with any mechanism we could think of doing > framing and steering and we thought that framing and steering are > essential at 10Gbps and over. If framing and steering "are essential" (your words) then why is framing not a MUST in the spec? And why are so many implementers stating (or hinting) that they are NOT going to implement it? I think there is a major disconnect here. iSCSI w/o framing or markers is dead in the water IMHO. Dave Sheehy From owner-ips@ece.cmu.edu Fri Nov 9 05:53:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA06301 for ; Fri, 9 Nov 2001 05:53:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9AV9Q15826 for ips-outgoing; Fri, 9 Nov 2001 05:31:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9AV7l15822 for ; Fri, 9 Nov 2001 05:31:07 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA79056 for ; Fri, 9 Nov 2001 11:31:00 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9AUnc121348 for ; Fri, 9 Nov 2001 11:30:49 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 12:30:50 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 12:30:49, Serialize complete at 09/11/2001 12:30:49 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, There are several other mechanisms that will have to be changed to allow for OOO. Task abort relies on the fact that the task management request is delivered after the task to be aborted and on the same connection. Retry command cleaning relies on the fact that the "cleaning command" is shipped after the retried command. And I suspect we may find others. This thread is going nowhere IMHO for two reasons: the proponents of OOO (Rod, Santosh and Bob Russell ) have faced us with an issue not a design - the validity of which is hotly contested I and other authors do not seem to be willing to do all the work and make a design (or complete set of design changes) based on this sketch. Unlike other requests I could not see anything appealing enough to justify the time spent on doing it. I suggest that Rod, Santosh and whoever else is willing to work with them put together a draft based on their ideas that should: clearly state what is to be gained indicate in sufficient what changes are needed in the current draft to accommodate the new design not only in command delivery but in all other areas (task management, recovery, logout etc.). Once we have this draft I assure you that we will give it serious consideration. Julo "Mallikarjun C." Sent by: owner-ips@ece.cmu.edu 09-11-01 03:28 Please respond to "Mallikarjun C." To: John Hufferd/San Jose/IBM@IBMUS cc: Subject: Re: iSCSI: Out of order commands John, Sorry, this note got a little longer than I would've liked, but.... I believe there are cases where OOO CmdSN handling is a legitimate requirement on targets due to exception events - a) retransmitting a CmdSN on a command acknowledgement timeout (within-connection recovery class). This manifests as an OOO CmdSN on the connection to a target if it didn't see the original copy due to a digest error. b) retransmitting the last few "lost" commands due to a connection failure on a new connection. If this new connection had already carried a CmdSN greater than these retransmitted commands (prior to connection failure), this again manifests as OOO CmdSN on the new connection to the target. OTOH, I believe sending OOO CmdSNs on a connection as a regular practice is counterproductive, since the target must continuously re-order the initiator "optimization" leading to a zero-sum game. I would argue that the need to dispatch CmdSNs OOO due to immediate data DMA (brought up by Rod) can be addressed by simple NIC changes to prefetch data for the next command (or more simply use unsolicited separate data PDUs, if negotiated). [ You got to deal with the case of all commands being writes anyway! ] If we allow OOO CmdSNs on a connection (I'd advocate discouraging it as a regular practice), I don't believe any of the stuff in error recovery breaks (nor does it affect the current reliance on ExpCmdSN). Julian perhaps can comment. - All the in-order assumptions are for DataSNs/R2TSNs/StatSNs, not for CmdSNs. - Any multi-connection session by definition must deal with OOO CmdSNs. - I belive that the current abort task scheme for immediate commands detailed in section 9.3 caters to OOO CmdSNs on a connection as well (we must be dealing with an immediate Abort arriving before the command today, since the command could have been hit with a digest error). To summarize, here is what I suggested to Julian in a private email - a)I suggest using a SHOULD for in-order dispatch of commands on a connection - for an initiator. b)I suggest using a SHALL handle out-of-order commands on a connection - for the target (as Barry pointed out). Hope that was useful. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization Hewlett-Packard MS 5668 Roseville CA 95747 ----- Original Message ----- From: "John Hufferd" To: Cc: Sent: Thursday, November 08, 2001 1:40 PM Subject: Re: iSCSI: Out of order commands > > Mallikarjun, > Could you comment on the concept of OOO on the ErrorRecoveryLevel>0. I had > thought that "in order delivery" was part of the detection of missing PDUs > and needed for timely Recovery. I was wondering if this changes the way we > would use the ExpCmdSN, etc. > > I think your opinions on this part of the OOO discussion would be valuable. > For example, how would you contrast the differences in detecting a problem > and recovering from that problem etc., today vrs the OOO approach (if any). > > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "Mallikarjun C." @ece.cmu.edu on 11/07/2001 09:41:05 AM > > Please respond to cbm@rose.hp.com > > Sent by: owner-ips@ece.cmu.edu > > > To: Santosh Rao , ips@ece.cmu.edu > cc: > Subject: Re: iSCSI: Out of order commands > > > > Santosh, > > I have only one comment on your responses. > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > I made two convenient assumptions if you noticed, :-), one of which > is that target forces session recovery on *any* error that it sees > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > error. With that assumption, a target can afford not to implement > a scoreboard. > > As I said in a private note, I guess what primarily bothers me about > OOO commands on a connection is that it requires the receiver to > undo this "optimization" on its end - most notably on a single > connection. TCP experts may comment on how/if they dealt with a > similar issue. > > OTOH, you had some valid comments on exceptions to ordering during > connection recovery. Perhaps we can move on by making Julian's > proposed stipulation a SHOULD.... > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > Santosh Rao wrote: > > > > Mallikarjun, > > > > Some comments below. > > > > Regards, > > Santosh > > > > "Mallikarjun C." wrote: > > > > > > Rod and Julian, > > > > > > This has been an interesting thread of discussion. Some > > > comments - > > > > > > 1.My first reaction was - allowing out-of-order command > > > transmission on the same connection deprives targets of > > > an implementation choice. Targets which support only > > > single-connection sessions and only support session > > > recovery (reasonable assumptions in my mind) can no > > > longer afford *not to* implement a command scoreboard. > > > > Even a single connection target *MUST* implement a scoreboard. The > > reason being that it can see out-of-order arrival of commands due to > > commands being dropped on digest errors. In such a case, it must block > > further command processing until holes are filled. > > > > Thus, there is no getting away from implementing a sequencer at the > > target. Given this, I think it is unreasonable to restrict initiator > > implementation flexibility by imposing a strict ordering requirement > > within the connection. > > > > > 2.Any end-node efficiency that is sought to be achieved > > > by transmitting CmdSNs out-of-order from the initiator > > > would be lost on the other end-node, since the target > > > now must wait for re-ordering the commands. > > > > It has to handle this situation anyway to deal with holes caused by > > digest errors. This scenario occurs even with initiators that issue > > commands in order. > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > link badwidth (albeit at the expense of end-node efficiency), > > > compared to idling the link waiting for outbound DMA. > > > We have to determine if this is a reasonable trade-off. > > > > > > 4.I can see Rod's point that prefetching all immediate > > > data can be a burden on the NIC resources. But, two > > > questions - > > > - could the NIC not use unsolicited separate data > > > PDUs in these cases? [ I realize that InitialR2T > > > has to be "no" to let it happen... ] > > > - could the NIC have a memory architecture that > > > allows data prefetching for the next command (so > > > this is a non-issue from the protocol perspective)? > > > This scheme incurs one DMA delay for every new > > > burst of commands. > > > > > > 5.Another (perhaps radical at this point) option is to do > > > away with immediate unsolicited data, to stick only with > > > separate unsolicited data. I would personally be okay > > > with the choice, particularly if this feature (that > > > helps software implementations) starts making hardware > > > design complicated/expensive. > > > > > > So, to summarize - > > > > > > option immediate allow > > > data in spec? out-of-order? > > > > > > (A) (5) above no no > > > (B) No real reason to do this. no yes > > > (C) (4) above yes no > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > >From the arguments I heard so far, I am leaning towards > > > option A, and option C in that order. > > > > > > Comments? > > > -- > > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > Rod Harrison wrote: > > > > > > > > Julian, > > > > > > > > I don't understand what you are proposing here, what do you > mean by > > > > "multiplexed" DMA? > > > > > > > > The problem is that the DMAs take some time, the more there > are > > > > queued the longer the last DMAs queued take to complete. Some > commands > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > soon a possible, which for a read after a write can be before the > > > > write's DMA has completed. Maintaining an ordered queue for commands > > > > to be sent on the HBA is expensive and redundant since the target > > > > already knows how to queue commands before committing them to its > SCSI > > > > layer. > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > change the > > > > order of commands from the OS, but the DMAs those commands need are > > > > unlikely to complete in the same order, and as I mentioned some > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > order it has to maintain an ordered queue of commands waiting to be > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > immediate data almost impossible to support. > > > > > > > > I don't see the problem with allowing out of order commands > given > > > > that the target already has to deal with very similar problems. I > > > > think we are getting in to the area of implementation choices here, > > > > which is inappropriate for a specification. > > > > > > > > - Rod > > > > From owner-ips@ece.cmu.edu Fri Nov 9 05:54:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA06313 for ; Fri, 9 Nov 2001 05:54:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9ABIt14761 for ips-outgoing; Fri, 9 Nov 2001 05:11:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9ABHl14756 for ; Fri, 9 Nov 2001 05:11:17 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA45078 for ; Fri, 9 Nov 2001 11:11:10 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9AB7c91198 for ; Fri, 9 Nov 2001 11:11:08 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 12:11:11 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 12:11:08, Serialize complete at 09/11/2001 12:11:08 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Paul, I should have seen this comming. iSCSI assumes that TCP is there to take care of the transport window and the transport window is as large as it should be. However we are talking about the application window. The application has advertize it's own window in addition to the transport windows. With transport having things in order we can avoid having to have a complete or even partial duplicate of the windows. If we allow transport to deliver not in order we have to have them duplicated (or at least the control part - and whatever DMA schemes we put in place won't help). Julo Paul Koning 09-11-01 04:25 Please respond to Paul Koning To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Excerpt of message (sent 8 November 2001) by Julian Satran: > Ron, > > Targets will most likely advertise a total (command and data) window > larger than they can accommodate on any long haul link. With the current > ordering rules nothing bad will happen. This is a very bad idea. The whole notion of a window is that you advertise what you can handle. You DO NOT, EVER, advertise more than you can handle in the hope that you won't be caught. Long haul links are no excuse. The way you deal with those is by having more buffer capacity, and issuing window increases early enough to keep data flowing. In a window flow control scheme, a target that advertises resources it does not have is defective. This is all VERY old hat. TCP has understood how to do this for ages. In particular, it has always been well understood that you HAVE to have more buffers in order to run at high speed over long delay links. In essence, what you're saying is that we have an application protocol here that is so unusual that it should throw out the lessons of the past 40 years, which have taught us how you construct correct protocols. This makes no sense whatsoever to me. I think the notion of out of order commands on a single connection is somewhat strange, but your reason for opposing it is several order of magnitude stranger. paul From owner-ips@ece.cmu.edu Fri Nov 9 05:58:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA06367 for ; Fri, 9 Nov 2001 05:58:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA99q3413720 for ips-outgoing; Fri, 9 Nov 2001 04:52:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA99q1l13714 for ; Fri, 9 Nov 2001 04:52:01 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA15928 for ; Fri, 9 Nov 2001 10:51:54 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA99plB37020 for ; Fri, 9 Nov 2001 10:51:53 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 11:51:47 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 11:51:52, Serialize complete at 09/11/2001 11:51:52 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk A careful reading of the SCSI documents will clarify that SCSI does not require in order delivery of commands but the standards assume that commands will be delivered in order. iSCSI takes in fact a similar stance. The standard described mechanisms assume that the hand-over is in order and all other mechanisms will work under this assumption. However iSCSI has no mechanism to enforce this (hand-over in or out-of-order) is a local issue at the target and an implementer may choose to do whatever it deems appropriate. For a regular stream of SCSI commands things will probably work fine. Things will break when commands like as clear task set will end up clearing even commands supposed to reach the target after the clear - but with enough sophistication even this case can be handled. Julo "Rod Harrison" Sent by: owner-ips@ece.cmu.edu 08-11-01 03:57 Please respond to "Rod Harrison" To: , "Barry Reinhold" , "Robert D. Russell" cc: Julian Satran/Haifa/IBM@IBMIL, Subject: RE: iSCSI: Out of order commands Somesh, The spec prohibits processing commands simultaneously, if by processing you mean handing off to the SCSI layer. If by processing simultaneously you mean preparing data for committing to SCSI when the CmdSN allows then surely reception of commands out of order is a no brainer. By the way, I am in support of allowing the target to make the decision on when to commit the SCSI layer rather than having to use the CmdSN. That decision can be made on the basis of device type, LUN, and queue tag. Forcing CmdSN order of committal to SCSI has always seemed overly restrictive to me. - Rod -----Original Message----- From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] Sent: Wednesday, November 07, 2001 5:46 PM To: Rod Harrison; Barry Reinhold; Robert D. Russell Cc: Julian Satran; ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Rod, Sorry to butt in, but targets are not doing serialized processing. They will process many many commands simultaneously. We had prior discussions on this and unfortunately we agreed on ordered delivery at the iSCSI level. The real boost in performance is to have ordering semantics at a higher layer, so that the user code can determine whether it needs ordering or not. As an example, in the general/normal case of a computer talking to a disk, the ordering rule could be relaxed completely. Somesh > -----Original Message----- > From: Rod Harrison [mailto:rod.harrison@windriver.com] > Sent: Wednesday, November 07, 2001 5:34 PM > To: Barry Reinhold; Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Barry, > > I'm curious, where do you see the extra complexity between the > following? > > Assuming all the following commands are writes ... > > c1, c3, c2, c4 and > > c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate. > > In both cases the target has to queue commands 2, 3, and 4 whilst > waiting for its R2T on c1 to be satisfied. > > Even if the target doesn't support any unsolicited data it still has > to queue commands 2, 3, and 4. I can't see how the target can avoid > queuing commands, in which case the order of arrival seems to make > little difference. > > - Rod > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Barry Reinhold > Sent: Wednesday, November 07, 2001 3:10 PM > To: Robert D. Russell; Somesh Gupta > Cc: Julian Satran; ips@ece.cmu.edu > Subject: RE: iSCSI: Out of order commands > > > Bob, > I can't speak for targets, but OOO commands on a session with a > single > connection sure increases the complexity of the code path we take. > To me the issue is still that these types of situations tend to be > poorly > tested and lead to interoperability issues. If we do go down this > path, the > spec. should make it very clear that this is expected behavior. Some > statement with a shall in it should be written (for the receiver), so > that > there is a conformance test items to pass. > > >-----Original Message----- > >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > >Robert D. Russell > >Sent: Wednesday, November 07, 2001 4:13 PM > >To: Somesh Gupta > >Cc: Julian Satran; ips@ece.cmu.edu > >Subject: RE: iSCSI: Out of order commands > > > > > >Somesh, Julian: > > > >You state that dealing with OOO commands on the target > >will add substantial complexity on the target. > >Do you have any basis for that claim? My impression from the > >plugfest is that most targets are already dealing with > >it. Perhaps we need to hear from someone who is actually > >building a target for which this would be a real problem. > > > >If anything, what we are hearing from people who really > >are building initiators is that dealing with the requirement > >to send commands in order will introduce substantial complexity > >on the initiator. > > > >So why should we be saving complexity on (hypothetically) simple > >targets yet requiring complexity on real initiators? > > > >As far as the deadlock issue is concerned, if the only way > >that deadlock can occur with OOO commands on the same > >connection is during the use of immediate data (which is I > >think what Julian was saying), then shouldn't we change > >the standard to just say that -- if the initiator sends > >commands out of order on a single connection, then immediate > >data MUST NOT be used on that connection in order to avoid deadlock. > > > >This gives everybody what they want, since initiators who find > >it beneficial to deliver commands OOO will just negotiate > >immediate data off. Those who really want to use immediate data > >will have to ensure that commands are sent in order. > >The tradeoff then becomes an implementation issue, not a > >standards issue, which is where it belongs. > > > > > >Bob Russell > >InterOperability Lab > >University of New Hampshire > >rdr@iol.unh.edu > >603-862-3774 > > > > > >On Wed, 7 Nov 2001, Somesh Gupta wrote: > > > >> I think we should either have it as a MUST or not require > >> it (at both ends to get the real benefit). SHOULD is one > >> of those things that leads to implementation > >> burden and confusion, without perhaps the feature being > >> used. There are implementation as well as protocol > >> considerations mixed in here. > >> > >> If we are to remove the restriction, we should (SHOULD) > >> get the maximum benefit from it, rather than to > >> accomodate an implementation choice. Out of sequence > >> commands, combined with the possibility of digest errors, > >> will add substantial complexity on the target side, > >> without corrosponding benefit in performance. If we change > >> this to SHOULD, we should also relax the requirement > >> to present commands on the target side to a SHOULD. > >> > >> > >> > >> > -----Original Message----- > >> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of > >> > Julian Satran > >> > Sent: Wednesday, November 07, 2001 10:00 AM > >> > To: ips@ece.cmu.edu > >> > Subject: Re: iSCSI: Out of order commands > >> > > >> > > >> > Mallikarjun, > >> > > >> > I did not see a SINGLE performance improvement that results from > OOO > >> > shipping. > >> > I would be bad engineering to give away the "no-deadlock" > mechanism we > >> > have now for nothing. > >> > I have also the impression that the point about deadlock that I > keep > >> > repeating is ignored or not understood. > >> > As we stand today commands can be shipped with Immediate data > >or without > >> > and an implementer determined > >> > to squeeze maximum bandwidth and overlap command start with > >delivery will > >> > choose not to work with immediate data > >> > (as you have pointed out) while a low performance software > >implementation > >> > will use immediate data to minimize CPU cycles consumed. However > both > >> > will be guaranteed to work without deadlock as source and sink > use the > >> > same ordering. > >> > Recovery is still a low probability event and should be handled > with a > >> > different set of considerations in mind. > >> > As for the strictness of the recommendation - yes we could settle > on > >> > SHOULD. > >> > > >> > Julo > >> > > >> > > >> > > >> > > >> > "Mallikarjun C." > >> > Sent by: owner-ips@ece.cmu.edu > >> > 07-11-01 19:41 > >> > Please respond to cbm > >> > > >> > > >> > To: Santosh Rao , > ips@ece.cmu.edu > >> > cc: > >> > Subject: Re: iSCSI: Out of order commands > >> > > >> > > >> > > >> > Santosh, > >> > > >> > I have only one comment on your responses. > >> > > >> > > Even a single connection target *MUST* implement a scoreboard. > The > >> > > reason being that it can see out-of-order arrival of commands > due to > >> > > commands being dropped on digest errors. In such a case, it > >must block > >> > > further command processing until holes are filled. > >> > > >> > I made two convenient assumptions if you noticed, :-), one of > which > >> > is that target forces session recovery on *any* error that it > sees > >> > (ErrorRecoveryLevel=0) - including a dropped command due to a > digest > >> > error. With that assumption, a target can afford not to > implement > >> > a scoreboard. > >> > > >> > As I said in a private note, I guess what primarily bothers me > about > >> > OOO commands on a connection is that it requires the receiver to > >> > undo this "optimization" on its end - most notably on a single > >> > connection. TCP experts may comment on how/if they dealt with a > >> > similar issue. > >> > > >> > OTOH, you had some valid comments on exceptions to ordering > during > >> > connection recovery. Perhaps we can move on by making Julian's > >> > proposed stipulation a SHOULD.... > >> > -- > >> > Mallikarjun > >> > > >> > > >> > Mallikarjun Chadalapaka > >> > Networked Storage Architecture > >> > Network Storage Solutions Organization > >> > MS 5668 Hewlett-Packard, Roseville. > >> > cbm@rose.hp.com > >> > > >> > > >> > Santosh Rao wrote: > >> > > > >> > > Mallikarjun, > >> > > > >> > > Some comments below. > >> > > > >> > > Regards, > >> > > Santosh > >> > > > >> > > "Mallikarjun C." wrote: > >> > > > > >> > > > Rod and Julian, > >> > > > > >> > > > This has been an interesting thread of discussion. Some > >> > > > comments - > >> > > > > >> > > > 1.My first reaction was - allowing out-of-order command > >> > > > transmission on the same connection deprives targets of > >> > > > an implementation choice. Targets which support only > >> > > > single-connection sessions and only support session > >> > > > recovery (reasonable assumptions in my mind) can no > >> > > > longer afford *not to* implement a command scoreboard. > >> > > > >> > > Even a single connection target *MUST* implement a scoreboard. > The > >> > > reason being that it can see out-of-order arrival of commands > due to > >> > > commands being dropped on digest errors. In such a case, it > >must block > >> > > further command processing until holes are filled. > >> > > > >> > > Thus, there is no getting away from implementing a sequencer at > the > >> > > target. Given this, I think it is unreasonable to restrict > initiator > >> > > implementation flexibility by imposing a strict ordering > requirement > >> > > within the connection. > >> > > > >> > > > 2.Any end-node efficiency that is sought to be achieved > >> > > > by transmitting CmdSNs out-of-order from the initiator > >> > > > would be lost on the other end-node, since the target > >> > > > now must wait for re-ordering the commands. > >> > > > >> > > It has to handle this situation anyway to deal with holes > caused by > >> > > digest errors. This scenario occurs even with initiators that > issue > >> > > commands in order. > >> > > > >> > > > > >> > > > 3.The flipside is that out-of-order transmission saves > >> > > > link badwidth (albeit at the expense of end-node > efficiency), > >> > > > compared to idling the link waiting for outbound DMA. > >> > > > We have to determine if this is a reasonable trade-off. > >> > > > > >> > > > 4.I can see Rod's point that prefetching all immediate > >> > > > data can be a burden on the NIC resources. But, two > >> > > > questions - > >> > > > - could the NIC not use unsolicited separate data > >> > > > PDUs in these cases? [ I realize that InitialR2T > >> > > > has to be "no" to let it happen... ] > >> > > > - could the NIC have a memory architecture that > >> > > > allows data prefetching for the next command (so > >> > > > this is a non-issue from the protocol perspective)? > >> > > > This scheme incurs one DMA delay for every new > >> > > > burst of commands. > >> > > > > >> > > > 5.Another (perhaps radical at this point) option is to do > >> > > > away with immediate unsolicited data, to stick only with > >> > > > separate unsolicited data. I would personally be okay > >> > > > with the choice, particularly if this feature (that > >> > > > helps software implementations) starts making hardware > >> > > > design complicated/expensive. > >> > > > > >> > > > So, to summarize - > >> > > > > >> > > > option immediate allow > >> > > > data in spec? > out-of-order? > >> > > > > >> > > > (A) (5) above no no > >> > > > (B) No real reason to do this. no yes > >> > > > (C) (4) above yes no > >> > > > (D) pros & cons (1), (2) & (3) yes yes > >> > > > > >> > > > >From the arguments I heard so far, I am leaning towards > >> > > > option A, and option C in that order. > >> > > > > >> > > > Comments? > >> > > > -- > >> > > > Mallikarjun > >> > > > > >> > > > Mallikarjun Chadalapaka > >> > > > Networked Storage Architecture > >> > > > Network Storage Solutions Organization > >> > > > MS 5668 Hewlett-Packard, Roseville. > >> > > > cbm@rose.hp.com > >> > > > > >> > > > Rod Harrison wrote: > >> > > > > > >> > > > > Julian, > >> > > > > > >> > > > > I don't understand what you are proposing here, > >what do you > >> > mean by > >> > > > > "multiplexed" DMA? > >> > > > > > >> > > > > The problem is that the DMAs take some time, the > >more there > >> > are > >> > > > > queued the longer the last DMAs queued take to complete. > Some > >> > commands > >> > > > > require DMAs to complete before they can be sent, i.e. > >Writes with > >> > > > > immediate data, some commands do not, i.e. Reads and > >writes with no > >> > > > > immediate data. The iSCSI HBA wants to be able to send > >commands as > >> > > > > soon a possible, which for a read after a write can be > before the > >> > > > > write's DMA has completed. Maintaining an ordered queue > >for commands > >> > > > > to be sent on the HBA is expensive and redundant since the > target > >> > > > > already knows how to queue commands before committing them > to its > >> > SCSI > >> > > > > layer. > >> > > > > > >> > > > > The iSCSI HBA and its host driver are not at > liberty to > >> > change the > >> > > > > order of commands from the OS, but the DMAs those > >commands need are > >> > > > > unlikely to complete in the same order, and as I mentioned > some > >> > > > > commands need no DMA. If the HBA can't send commands out of > CmdSN > >> > > > > order it has to maintain an ordered queue of commands > >waiting to be > >> > > > > sent, and potentially buffer a lot of data. For an HBA this > makes > >> > > > > immediate data almost impossible to support. > >> > > > > > >> > > > > I don't see the problem with allowing out of > >order commands > >> > given > >> > > > > that the target already has to deal with very similar > problems. I > >> > > > > think we are getting in to the area of implementation > >choices here, > >> > > > > which is inappropriate for a specification. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Julian Satran > >> > > > > Sent: Monday, November 05, 2001 10:06 PM > >> > > > > To: ips@ece.cmu.edu > >> > > > > Subject: Re: iSCSI: Out of order commands, was current > >UNH Plugfest > >> > > > > > >> > > > > Rod, > >> > > > > > >> > > > > I don't see any reason why DMA operations cant be > >"multiplexed" with > >> > > > > commands. > >> > > > > If you have scheduled a long outbound DMA you are doomed > >regardless > >> > of > >> > > > > the > >> > > > > command ordering. > >> > > > > And if you have scheduled DMA operations piecemeal then you > can > >> > insert > >> > > > > your commands in correct order. > >> > > > > > >> > > > > Julo > >> > > > > > >> > > > > "Rod Harrison" > >> > > > > 05-11-01 20:48 > >> > > > > Please respond to "Rod Harrison" > >> > > > > > >> > > > > To: Julian Satran/Haifa/IBM@IBMIL, > > >> > > > > cc: > >> > > > > Subject: iSCSI: Out of order commands, was > current > >> > UNH > >> > > > > Plugfest > >> > > > > > >> > > > > [ Subject changed ] > >> > > > > > >> > > > > Julian, > >> > > > > > >> > > > > The ordering difference is introduced > >between the > >> > > > > host > >> > > > > side driver > >> > > > > and the iSCSI HBA. The host side driver must present > >SCSI commands > >> > to > >> > > > > the HBA in the order they are received from the OS to > >prevent read > >> > > > > after write dependency failures. The HBA might reorder > >the commands > >> > > > > depending on when DMA completes. The reordering can't be > >done ahead > >> > of > >> > > > > time in the host driver since it doesn't know how long each > DMA > >> > might > >> > > > > take. As long as the HBA assigns CmdSN in the order it > receives > >> > > > > commands the desired host ordering is preserved. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Julian Satran > >> > > > > Sent: Monday, November 05, 2001 12:35 AM > >> > > > > To: ips@ece.cmu.edu > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Rod, > >> > > > > > >> > > > > I all examples give the point I find hard to understand is > why is > >> > the > >> > > > > ordering on the wire different from the presentation order > to the > >> > > > > initiator. You can get as many overlaps as you want by > >presenting > >> > the > >> > > > > commands to the initiator in the desired order. > >> > > > > What we are considering here is the case in which you > >want to ship > >> > in > >> > > > > an > >> > > > > order different than the one you present the commands. > >> > > > > > >> > > > > Julo > >> > > > > > >> > > > > "Rod Harrison" > >> > > > > Sent by: owner-ips@ece.cmu.edu > >> > > > > 04-11-01 04:42 > >> > > > > Please respond to "Rod Harrison" > >> > > > > > >> > > > > To: "Barry Reinhold" , > "Dave > >> > > > > Sheehy" > >> > > > > , "IETF IP SAN Reflector" > >> > > > > > >> > > > > cc: > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Barry, > >> > > > > > >> > > > > In general I agree but I don't think this > is as > >> > much > >> > > > > of a > >> > > > > corner case > >> > > > > as it at first appears. Targets will have code very > >similar to that > >> > > > > needed to handle out of order commands to deal with > >digest errors. > >> > > > > Targets also need to queue commands whilst waiting for both > >> > solicited > >> > > > > and unsolicited data to arrive. Queuing out of order > >commands seems > >> > > > > little extra work. > >> > > > > > >> > > > > From an initiators point of view there are > >> > > > > efficiency, > >> > > > > and probably > >> > > > > performance gains to be had from sending commands out of > >order. Bob > >> > > > > Russell gave the example of a read being sent whilst > >write data DMA > >> > is > >> > > > > happening, and a similar situation can arise with DMA for > writes > >> > > > > overtaking that of earlier writes if the initiator has > >multiple DMA > >> > > > > engines. In this case the initiator might be forced to > >let the wire > >> > go > >> > > > > idle if it can't send the data from completed DMAs as soon > as > >> > > > > possible. > >> > > > > > >> > > > > We already have a command queue at the > target to > >> > > > > enforce > >> > > > > correct > >> > > > > serialisation of commands, doing the same thing at the > >initiator is > >> > > > > redundant. > >> > > > > > >> > > > > Finally, I don't believe we should be > writing a > >> > > > > standard > >> > > > > to work > >> > > > > around poor coding and test coverage, especially at the > cost of > >> > > > > potential efficiency gains. > >> > > > > > >> > > > > I agree with Dave and Santosh that > >commands being > >> > > > > sent > >> > > > > out of order > >> > > > > on a single session should be allowed by the standard. > >> > > > > > >> > > > > - Rod > >> > > > > > >> > > > > -----Original Message----- > >> > > > > From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > Of > >> > > > > Barry Reinhold > >> > > > > Sent: Friday, November 02, 2001 5:24 PM > >> > > > > To: Dave Sheehy; IETF IP SAN Reflector > >> > > > > Subject: RE: iSCSI: current UNH Plugfest > >> > > > > > >> > > > > Using features such as out of order command delivery on > >a connection > >> > > > > tend to > >> > > > > be the sort of things that lead to interoperability > >problems. It is > >> > > > > unexpected and probably going to hit poorly tested code > >paths even > >> > if > >> > > > > the > >> > > > > standard is written to allow it. > >> > > > > > >> > > > > >-----Original Message----- > >> > > > > >From: owner-ips@ece.cmu.edu > >[mailto:owner-ips@ece.cmu.edu]On Behalf > >> > > > > Of > >> > > > > >Dave Sheehy > >> > > > > >Sent: Friday, November 02, 2001 4:19 PM > >> > > > > >To: IETF IP SAN Reflector > >> > > > > >Subject: Re: iSCSI: current UNH Plugfest > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > >> 3. Can commands be sent out of order on the same > connection? > >> > > > > >> > >> > > > > >> The behavior of targets is clearly specified in > Section > >> > 2.2.2.3 > >> > > > > on > >> > > > > >> page 25 of draft 8, which says: > >> > > > > >> "Except for the commands marked for immediate > >delivery the > >> > > > > iSCSI > >> > > > > >> target layer MUST eliver the commands for > >execution in the > >> > > > > order > >> > > > > >> specified by CmdSN." > >> > > > > >> > >> > > > > >> Section 2.2.2.3 on page 26 of draft 8 also says: > >> > > > > >> "- CmdSN - the current command Sequence Number > >advanced by 1 > >> > > > > on > >> > > > > >> each command shipped except for commands marked for > >> > immediate > >> > > > > >> delivery." > >> > > > > >> but the meaning of the term "shipped" is vague, > >and does not > >> > > > > >> necessarily > >> > > > > >> require that the PDUs arrive on the other end of a > TCP > >> > > > > connection > >> > > > > >> in the same order that the CmdSN values were > >assigned to these > >> > > > > PDUs. > >> > > > > >> > >> > > > > >> Some initiators have been designed to send commands > out of > >> > CmdSN > >> > > > > >> order on one connection. Consider the situation > >where there > >> > is > >> > > > > only > >> > > > > >> one connection and a high-level dispatcher creates > >a PDU for a > >> > > > > SCSI > >> > > > > >> command that involves writing immediate data to the > target. > >> > > > > This PDU > >> > > > > >> is enqueued to a lower-level layer which has to > >setup, start, > >> > > > > and > >> > > > > >> wait-for a DMA operation to move the immediate data > into an > >> > > > > onboard > >> > > > > >> buffer before the PDU can be put onto the wire. > >While this is > >> > > > > >> happening, the dispatcher creates another > >unrelated PDU for a > >> > > > > SCSI > >> > > > > >> read command (for example), and when this PDU is > >passed to the > >> > > > > >> lower-level layer it can be sent immediately, ahead > of the > >> > > > > previous > >> > > > > >> write PDU and therefore out of order on this > connection. > >> > > > > >> > >> > > > > >> The standard clearly allows this to happen if the two > PDUs > >> > were > >> > > > > sent > >> > > > > >> on different connections, and seems to imply that > this can > >> > also > >> > > > > happen > >> > > > > >> when the two PDUs are sent on the same connection. > >> > > > > >> > >> > > > > >> The suggestion is to put in the standard an > >explicit statement > >> > > > > that > >> > > > > >> this is allowed or not allowed, as appropriate. > >> > > > > >> > >> > > > > >> If this is allowed, such a statement would avoid > >the erroneous > >> > > > > >> assumption being made by some target implementers > >that within > >> > a > >> > > > > single > >> > > > > >> connection, commands will arrive in order. > >> > > > > >> > >> > > > > >> If this is not allowed, such a statement would avoid > the > >> > > > > erroneous > >> > > > > >> assumption being made by some initiator implementers > that > >> > within > >> > > > > a > >> > > > > >> single connection, commands can be put on the wire > out of > >> > order. > >> > > > > >> > >> > > > > >> +++ > >> > > > > >> > >> > > > > >> will add an explicit statement saying that this > behaviour is > >> > > > > forbidden. > >> > > > > >> 2.2.2.1 will contain: > >> > > > > >> > >> > > > > >> On any given connection, the iSCSI initiator MUST send > the > >> > > > > >commands in the > >> > > > > >> order specified by CmdSN. > >> > > > > >> > >> > > > > >> +++ > >> > > > > > > >> > > > > >Why do you feel this behavior should be forbidden? > >Targets already > >> > > > > have to > >> > > > > >order commands across the session. I don't see why it's > >a problem > >> > to > >> > > > > extend > >> > > > > >that to the connection as well. I, for one, believe we > >should take > >> > > > > >a liberal > >> > > > > >stance on this. > >> > > > > > > >> > > > > >Dave Sheehy > >> > > > > > > >> > > > >> > > -- > >> > > ################################## > >> > > Santosh Rao > >> > > Software Design Engineer, > >> > > HP-UX iSCSI Driver Team, > >> > > Hewlett Packard, Cupertino. > >> > > email : santoshr@cup.hp.com > >> > > Phone : 408-447-3751 > >> > > ################################## > >> > > >> > > >> > > >> > > >> > > > > From owner-ips@ece.cmu.edu Fri Nov 9 07:19:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA07717 for ; Fri, 9 Nov 2001 07:19:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9BfL902206 for ips-outgoing; Fri, 9 Nov 2001 06:41:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9BfJl02201 for ; Fri, 9 Nov 2001 06:41:19 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA78036 for ; Fri, 9 Nov 2001 12:41:05 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9Bf4B31970 for ; Fri, 9 Nov 2001 12:41:05 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: ISCSI: Padding Preceding CRC X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 13:41:07 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 13:41:04, Serialize complete at 09/11/2001 13:41:04 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Prasenjit, Very good point. And I have always maintained that padding on the wire has no meaning at all while alignment can be achieved through wise placement of buffers. The only weak point in the argument arises when we decided to split the PDU in sub-entities and wanted to make sure that those will be also aligned and their "count" can be expressed in 4 byte terms (total AHS length). Revising those will require some more reformatting of the iSCSI PDU. Julo Prasenjit Sarkar/Almaden/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 09-11-01 01:14 Please respond to Prasenjit Sarkar To: "Williams, Jim" cc: "'ips@ece.cmu.edu'" Subject: ISCSI: Padding Preceding CRC Isnt padding independent of CRC? "Williams, Jim" mulex.com> cc: Sent by: Subject: Padding Preceding CRC owner-ips@ece.c mu.edu 11/08/2001 02:29 PM The iSCSI spec indicates that the data will be padded out to a length which is a multiple of 4 before the CRC (digest) is appended. In discussing this with our hardware designers, I was told that this provides no benefit to them, but causes a fair amount of pain. As such I would like to raise the question of whether the padding should be eliminated. Computing unaligned CRCs is fairly trivial to do, and must be done anyway. This is because the padding aligns the CRC with respect to the iSCSI PDU, but one can't in general assume that the iSCSI PDU is aligned with the TCP segment on received data, and on transmitted data the source may be a general gather list of unaligned buffers. I suppose this doesn't preclude doing a lot of work to align the data before feeding it to the CRC unit, but given how easy it is to compute unaligned CRCs in hardware, there is no reason to do this. Inserting padding is extra work, however. This creates a bubble in the data flow pipe, extra information that must be passed between hardware units indicating the number of pad bytes to be inserted, and extra logic to insert the required pad data. If someone can make a case why this padding is a good thing, then certainly the hardware problems are solvable. But it looks to me like a "feature" with cost but no benefit. From owner-ips@ece.cmu.edu Fri Nov 9 09:47:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12803 for ; Fri, 9 Nov 2001 09:47:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9E8wr11216 for ips-outgoing; Fri, 9 Nov 2001 09:08:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9E8vl11211 for ; Fri, 9 Nov 2001 09:08:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA21612 for ; Fri, 9 Nov 2001 15:08:50 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9E8nB37318 for ; Fri, 9 Nov 2001 15:08:49 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Out of order commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 16:08:53 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 16:08:49, Serialize complete at 09/11/2001 16:08:49 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Nothing wrong with your reasoning. Only that a with ordering in place a designer might commit for it's cumulative set of resources (TCP windows + application buffers) while for unordered delivery it can commit only the application resources. And I did not hear back from you about any specific reasons to do so (neither initiator nor targets get gaster). Julo "Robert D. Russell" 08-11-01 17:50 Please respond to "Robert D. Russell" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands Julian: > However allowing initiators to ship them out of order creates a > potential deadlock that does not appear otherwise. > > If a target is missing a command in a queue (and there are no errors) then > this command is bound to be first on some connection under the current set > of rules. > > If we allow OOO shipping then the missing command can be somewhere > "inside" the window on some connection and if the target has just filled > his queue and has room in the staging buffer only for the command it is > waiting for and that command happens to be the first to pass to SCSI you > have a deadlock. I must be understanding something. Your example is certainly correct if the target has no control over the number of commands sent to it by the initiator. But the target does control this number through the MaxCmdSN field. For the scenario you are describing to occur, wouldn't it be necessary for the target to advertize a MaxCmdSN value bigger than it actually has resources to handle? It seems to me that if a target can only handle x new commands, then its queueing capacity is x, and in the SCSI Response PDU it should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the formula in section 2.2.2.1. This in turn controls the number of commands the initiator can send, and thus prevents the incoming commands from overfilling the target's queue. So isn't the deadlock caused by a broken target? Isn't the target advertizing a queueing capacity greater than it actually has and isn't that the cause of the deadlock? An alternative explanation of the deadlock might be that the target is advertizing the correct MaxCmdSN, but the initiator is sending commands beyond what it is allowed to send. However, in this case the target should silently ignore any non-immediate command outside the allowed range. For the deadlock to occur, wouldn't the target have to be queuing commands outside the allowed range instead of ignoring them? So in this case too, the target is broken and that is what causes the deadlock. Where am I going wrong in this reasoning? Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 From owner-ips@ece.cmu.edu Fri Nov 9 09:50:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12954 for ; Fri, 9 Nov 2001 09:50:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9DuxM10384 for ips-outgoing; Fri, 9 Nov 2001 08:56:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9Duvl10379 for ; Fri, 9 Nov 2001 08:56:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id OAA121500 for ; Fri, 9 Nov 2001 14:56:50 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9DuoB66124 for ; Fri, 9 Nov 2001 14:56:50 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI initiator availability for Windows ? X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 15:56:53 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 15:56:50, Serialize complete at 09/11/2001 15:56:50 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, There are many windows initiators available. I know for certain that IBM has one for W2K and there are several others. I will try to find out which version is available. I know that the version tested at the UNH Plugfest conforms to 08 but I do not know if it was made available to the public. Julo "Dave Francheski" Sent by: owner-ips@ece.cmu.edu 08-11-01 01:39 Please respond to "Dave Francheski" To: cc: Subject: iSCSI initiator availability for Windows ? Hi all, This may not be the proper forum for this question, but I have exhausted almost all other options at this point. Does anybody know of a Windows based (e.g., Windows 2000) version of an iSCSI initiator implemented to Revision 6 (or greater) of the iSCSI standard? I have Cisco's version of the iSCSI Initiator for Windows but it is implemented to the first original draft of iSCSI, and doesn't interoperate with other more recent iSCSI target implementations. Thankyou for any and all support you can provide. regards, -Dave David Francheski Seven Systems Technologies 575 Menlo Drive Suite 2 Rocklin CA 916-577-1281 davef@seven-systems.com From owner-ips@ece.cmu.edu Fri Nov 9 09:52:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA13010 for ; Fri, 9 Nov 2001 09:52:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9EF2U11634 for ips-outgoing; Fri, 9 Nov 2001 09:15:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9EExl11622 for ; Fri, 9 Nov 2001 09:15:00 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id PAA146162 for ; Fri, 9 Nov 2001 15:14:53 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9EEqB67796 for ; Fri, 9 Nov 2001 15:14:53 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI over TLS X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 16:14:54 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 16:14:52, Serialize complete at 09/11/2001 16:14:52 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, The one "tiny" item you forgot to mention is that TLS records span TCP and iSCSI PDU boundaries. TLS records can't be decrypted in face of TCP packet loss and markers/alignment can't be recovered (to be more precise require a lot more tweaking of the stacks). Julo "Bill Strahm" 08-11-01 23:55 Please respond to "Bill Strahm" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: iSCSI over TLS Julian, I do not understand how TLS interferes with delivery of iSCSI packets any more than IPsec. In either case your TOE MUST decrypt the packet and deal with the results. I do not see how this changes the problem if the packet is decrypted before going to the TOE (again the hardware to do this MUST be on the NIC device) or after going through the TOE processing... Quick summary of what I think needs to happen IPsec 1) receive L2 packet 2) determine it is IP 3) Apply packet policy based on L3 header 4) Decrypt packet - verify it is covered by the SA 5) Pass to L4 (TCP) for processing 6) Verify Framing/etc. 7) Done TLS 1) Recieve L2 Packet 2) Pass to L3 3) Pass to L4 (TCP) for processing 4) Decrypt packet 5) Verify Framing/etc 6) Done It turns out the policies for TLS are much simpler than for IPsec, the application itself gets to determine if security should be turned on or not (rather than another application pushing policies into an SPD) and I don't see a difference in the security offload requirements. In many cases TLS will go through firewalls/NAT/NATP much better than IPsec, allowing for a wider deployment model. Bill Strahm +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Tuesday, November 06, 2001 10:17 PM To: ips@ece.cmu.edu Subject: Re: iSCSI over TLS Peter, A group of us seriously considered TLS. The main reason for dropping it was that it would interfere with any mechanism we could think of doing framing and steering and we thought that framing and steering are essential at 10Gbps and over. Julo "Peter Mellquist" Sent by: owner-ips@ece.cmu.edu 07-11-01 02:15 Please respond to "Peter Mellquist" To: cc: Subject: iSCSI over TLS I am aware that the ips group is leaning toward IPSEC as for the security solution but I am interested if anyone is also considering using Transport Layer Security (TLS)? I am concerned that the requirement for IPSEC might make TOEs more complex than they need to be. Can TLS be optionally used as well as defined by the specification? This could allow TOE vendors to only be concerned with providing normal IPv4 / ipv6 and leave the security to a higher layer. A TLS stack sitting above the TOE could then handle security very well. Also, I anticipate that the first generation of TOEs will not support IPSEC. With a iSCSI/TLS we could enable security solutions with the first generation of TOEs and get speed and security. Are any TOE vendors planning to support IPSEC? Can TLS or IPSEC be supported? -peter Peter Mellquist Seven Systems Technologies 575 Menlo Drive Suite 2 Rocklin CA 916-577-1275 peterm@seven-systems.com From owner-ips@ece.cmu.edu Fri Nov 9 10:01:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA13504 for ; Fri, 9 Nov 2001 10:01:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9EZId13137 for ips-outgoing; Fri, 9 Nov 2001 09:35:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars0m9.nortelnetworks.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9EZ8l13103 for ; Fri, 9 Nov 2001 09:35:12 -0500 (EST) Received: from zcars04f.ca.nortel.com (zcars04f.ca.nortel.com [47.129.242.57]) by zcars0m9.nortelnetworks.com (8.11.0/8.11.0) with ESMTP id fA9EYNS11474 for ; Fri, 9 Nov 2001 09:34:23 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04f.ca.nortel.com; Fri, 9 Nov 2001 09:34:48 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id WMPW6PW3; Fri, 9 Nov 2001 09:33:58 -0500 Received: from cse-ns-laptop.nortelnetworks.com (cse-ns-laptop.us.nortel.com [47.16.69.109]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id VW7FBVS4; Fri, 9 Nov 2001 09:34:01 -0500 Message-Id: <5.1.0.14.2.20011109090058.02886ea0@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Fri, 09 Nov 2001 09:36:51 -0500 To: Elizabeth Rodriguez , ENDL_TX , IPS Reflector From: "Franco Travostino" Subject: RE: FCencap: List ALL SOF/EOF codes Cc: Murali Rajagopal In-Reply-To: <9410A0F67DFE7F4D998D4538236498360408F1@nc8220exchange.ral. lucent.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_69338944==_.ALT" X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_69338944==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed If one cannot enumerate the legitimate SOF/EOF codes as of 00:00 on Nov. 09 2001, then there is a standard body out there (not this one :-) with a big problem on its hands. I would hate to see this problem propagating into our IETF document, or being an excuse for any vague text of ours. Should new SOF/EOF codes emerge, and should there be enough IP interest around them, it should not be a problem to reissue a new FC common encapsulation specification with just new SOF/EOF codes, and the stale RFC will be marked as obsoleted by RFCxxx. -franco At 01:05 PM 11/8/2001, Elizabeth Rodriguez wrote: >(Participant mode) > >I disagree with this motion. >We had this discussion back in January, and basically came to the >conclusion that Class 1 and Class 4 should not be included, for the >reasons that class 1 really cannot be supported across the IP network and >class 4 is not really not defined yets, so the codes are not guaranteed to >remain constant. >Even if we do decide to accept Ralph's arguments to include class 1 and >class 4 SOF/EOF codes, we cannot take ALL the codes from FC-BB and >incorporate them into the FC Encapsulation draft. >Recall, we started out by including all the SOF/EOF codes from >FC-BB-2. We reevaluated in January 2001, when we analyzed the codes >themselves. >Several that we excluded I think were valid (e.g. for class 1 and class >4), but others were completely bogus and undefined anywhere other than in >FC-BB. >We cannot just blindly accept those codes. > >If this motion is considered, we need to reopen that evaluation made in >the January interim meeting and make a determination as to what codes need >to be included in FC Common Encapsulation, and make sure not to include >invalid codes. > >Thanks, > >Elizabeth >-----Original Message----- >From: Franco Travostino [mailto:travos@nortelnetworks.com] >Sent: Wednesday, November 07, 2001 9:32 AM >To: ENDL_TX@computer.org; IPS Reflector >Cc: Murali Rajagopal; Elizabeth Rodriguez >Subject: Re: FCencap: List ALL SOF/EOF codes > >I agree with this motion. > >You wrote under another heading: "Vague may be vague but it also is not >unnecessarily constraining." What a troubling statement was this. I'm glad >we now appear to be moving towards a constructive end after all. > >thanks >-franco > >At 08:15 AM 11/7/2001, Ralph Weber wrote: >>Upon further reflection, I think the right thing to do >>is to list all the SOF/EOF codes defined in FC-BB in >>the FC Encapsulation draft. >> >>FIRST >> >>There is nothing in the FC Encapsulation draft other >>than to omission of Class 1 SOF/EOF codes that prevents >>encapsulating FC Class 1 frames for TCP transport. >>Sure, a TCP ULP that is smarter than anything anybody >>has thought about will be required to do it. BUT >>there is (or should be) nothing the the FC Encapsulation >>draft that prevents such a protocol from being invented. >>AND the FC Encapsulation draft specifically says that >>you need the wisdom of some other protocol document in >>order to get any use out of the FC Encapsulation draft. >>Why force the mad man that devises a way to transport >>Class 1 over TCP/IP to revise the FC Encapsulation >>SOF/EOF tables? >> >>SECOND >> >>It is conceivable that a future version of iFCP >>(or maybe even FCIP) might want to support Class 4. >>Again, there is nothing in the FC Encapsulation >>draft that prevents this, except the omission >>of the SOF/EOF codes. >> >>FINALLY >> >>I believe that the elimination of all SOF/EOF >>codes other than Class 2, Class 3, AND CLASS F >>is a hold over from the early FCIP work, before >>the FC Encapsulation was split into a separate >>draft. I believe that decision was right for >>FCIP but wrong for an FC Encapsulation intended >>to be used by ALL FC protocols running over >>TCP/IP. >>Thanks for your consideration. >> >>Ralph... --=====================_69338944==_.ALT Content-Type: text/html; charset="us-ascii"
If one cannot enumerate the legitimate SOF/EOF codes as of 00:00 on Nov. 09 2001, then there is a standard body out there (not this one :-) with a big problem on its hands. I would hate to see this problem propagating into our IETF document, or being an excuse for any vague text of ours. Should new SOF/EOF codes emerge, and should there be enough IP interest around them, it should not be a problem to reissue a new FC common encapsulation specification with just new SOF/EOF codes, and the stale RFC will be marked as obsoleted by RFCxxx.

-franco

At 01:05 PM 11/8/2001, Elizabeth Rodriguez wrote:
(Participant mode)
 
I disagree with this motion.
We had this discussion back in January, and basically came to the conclusion that Class 1 and Class 4 should not be included, for the reasons that class 1 really cannot be supported across the IP network and class 4 is not really not defined yets, so the codes are not guaranteed to remain constant. 
Even if we do decide to accept Ralph's arguments to include class 1 and class 4 SOF/EOF codes, we cannot take ALL the codes from FC-BB and incorporate them into the FC Encapsulation draft.
Recall, we started out by including all the SOF/EOF codes from FC-BB-2.  We reevaluated in January 2001, when we analyzed the codes themselves.
Several that we excluded I think were valid (e.g. for class 1 and class 4), but others were completely bogus and undefined anywhere other than in FC-BB.
We cannot just blindly accept those codes.
 
If this motion is considered, we need to reopen that evaluation made in the January interim meeting and make a determination as to what codes need to be included in FC Common Encapsulation, and make sure not to include invalid codes.
 
Thanks,
 
Elizabeth
-----Original Message-----
From: Franco Travostino [mailto:travos@nortelnetworks.com]
Sent: Wednesday, November 07, 2001 9:32 AM
To: ENDL_TX@computer.org; IPS Reflector
Cc: Murali Rajagopal; Elizabeth Rodriguez
Subject: Re: FCencap: List ALL SOF/EOF codes
I agree with this motion.

You wrote under another heading: "Vague may be vague but it also is not unnecessarily constraining." What a troubling statement was this. I'm glad we now appear to be moving towards a constructive end after all.

thanks
-franco

At 08:15 AM 11/7/2001, Ralph Weber wrote:
Upon further reflection, I think the right thing to do
is to list all the SOF/EOF codes defined in FC-BB in
the FC Encapsulation draft.

FIRST

There is nothing in the FC Encapsulation draft other
than to omission of Class 1 SOF/EOF codes that prevents
encapsulating FC Class 1 frames for TCP transport.
Sure, a TCP ULP that is smarter than anything anybody
has thought about will be required to do it.  BUT
there is (or should be) nothing the the FC Encapsulation
draft that prevents such a protocol from being invented.
AND the FC Encapsulation draft specifically says that
you need the wisdom of some other protocol document in
order to get any use out of the FC Encapsulation draft.
Why force the mad man that devises a way to transport
Class 1 over TCP/IP to revise the FC Encapsulation
SOF/EOF tables?

SECOND

It is conceivable that a future version of iFCP
(or maybe even FCIP) might want to support Class 4.
Again, there is nothing in the FC Encapsulation
draft that prevents this, except the omission
of the SOF/EOF codes.

FINALLY

I believe that the elimination of all SOF/EOF
codes other than Class 2, Class 3, AND CLASS F
is a hold over from the early FCIP work, before
the FC Encapsulation was split into a separate
draft. I believe that decision was right for
FCIP but wrong for an FC Encapsulation intended
to be used by ALL FC protocols running over
TCP/IP.
Thanks for your consideration.

Ralph...
--=====================_69338944==_.ALT-- From owner-ips@ece.cmu.edu Fri Nov 9 11:08:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16699 for ; Fri, 9 Nov 2001 11:08:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9FL1h16864 for ips-outgoing; Fri, 9 Nov 2001 10:21:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailnpd.hcltech.com ([203.197.145.76]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA9FKwl16852 for ; Fri, 9 Nov 2001 10:20:58 -0500 (EST) Received: from npd.hcltech.com (kdatta-pc.hcltech.com [192.168.100.63]) by mailnpd.hcltech.com (8.11.0/8.11.0) with ESMTP id fA9FIme28702; Fri, 9 Nov 2001 20:48:49 +0530 Message-ID: <3BEBF45C.6E6224CC@npd.hcltech.com> Date: Fri, 09 Nov 2001 20:51:00 +0530 From: Kaushik Datta Organization: HCL technologies X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.2-2 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu CC: kdatta@npd.hcltech.com, priya@npd.hcltech.com, pamanick@npd.hcltech.com Subject: Querry ! Content-Type: multipart/alternative; boundary="------------FD7AFD3447B73D79075134C9" Sender: owner-ips@ece.cmu.edu Precedence: bulk --------------FD7AFD3447B73D79075134C9 Content-Type: text/plain; charset=iso-8859-2 Content-Transfer-Encoding: 7bit Hello all, We are a group of Engineers working on an idea related to the scsi subsystem in FreeBSD.We have approached this project from the perspective of iscsi.The project revolves around using iscsi target stack on FreeBSD and the changes that (we feel) can be made to the scsi subsystem in FreeBSD.As we all know that the FreeBSD scsi system adheres to the CAM architecture model, wherein we have an upper layer of driver pertaining to different kinds of devices, disk, tape etc.Now this upper layer drivers or Peripheral modules(PM) takes the task of forming the scsi CDB and passes it on to the CAM transport layer or the XPT layer.This transport layer then forms a CCB and passes it further on to the lower layer SIM HBA driver. Now iscsi pdu carry scsi cdb as a part of it.But in the existing scsi subsystem in FreeBSD, the PM layer takes care of forming the scsi pdu in case of a normal I/O operation.Now assuming iscsi target stack running on the machine, won't it be feasible if we can write a pseudo driver that takes care of the fetching the scsi cdb as such and passing it on to the SIM driver ? In a way we will get away from the PM functionality which forms the scsi CDB in case of a normal I/O.Now where exactly this driver should fit ? Should this use the xpt transport layer functionality which takes care of the routing decision and device initialization? Any ideas related to the design will be of great help.Also we would like to know if any group is working on implementing iscsi stack on FreeBSD. Thanks for your patience ! Kaushik --------------FD7AFD3447B73D79075134C9 Content-Type: text/html; charset=iso-8859-2 Content-Transfer-Encoding: 7bit Hello all,

       We are a group of Engineers working on an idea related to the
   scsi subsystem in FreeBSD.We have approached this project from the
   perspective of iscsi.The project revolves around using iscsi target
   stack on FreeBSD and the changes  that (we feel) can be made to the
   scsi subsystem in FreeBSD.As we all know that the FreeBSD scsi
   system adheres to the CAM architecture model, wherein we have an
   upper layer of driver pertaining to different kinds of devices, disk,
   tape etc.Now this upper layer drivers or Peripheral modules(PM) takes
   the task of forming the scsi CDB and passes it on to the CAM transport
   layer or the XPT layer.This transport layer then forms a CCB and
   passes it further on to the lower layer SIM HBA driver.
   Now iscsi pdu carry scsi cdb as a part of it.But in the existing
   scsi subsystem in FreeBSD, the PM layer takes care of forming the
   scsi pdu in case of a normal I/O operation.Now assuming iscsi
   target stack running on the machine, won't it be feasible if we
   can write a pseudo driver that takes care of the fetching the
   scsi cdb as such and passing it on to the SIM driver ? In a
   way we will get away from the PM functionality which forms the
   scsi CDB in case of a normal I/O.Now where exactly this driver
   should fit ? Should this use the xpt transport layer functionality
   which takes care of the routing decision and device initialization?
 
   Any ideas related to the design will be of great help.Also we would
   like to know if any group is working on implementing iscsi stack
   on FreeBSD.

   Thanks for your patience !
   Kaushik
 
      


 
 
  --------------FD7AFD3447B73D79075134C9-- From owner-ips@ece.cmu.edu Fri Nov 9 11:17:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16957 for ; Fri, 9 Nov 2001 11:17:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9FUBq17587 for ips-outgoing; Fri, 9 Nov 2001 10:30:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from host32.cedant.com (host32.cedant.com [209.239.32.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9FU9l17583 for ; Fri, 9 Nov 2001 10:30:09 -0500 (EST) Received: from sahara (39.sanjose-03-04rs16rt.ca.dial-access.att.net [12.81.1.39]) by host32.cedant.com (8.10.2/8.10.2) with ESMTP id fA9FTrb25982; Fri, 9 Nov 2001 10:29:53 -0500 Message-ID: <200111090742520737.00136666@opulentsystems.com> In-Reply-To: References: X-Mailer: Calypso Version 3.30.00.00 (4) Date: Fri, 09 Nov 2001 07:42:52 -0800 Reply-To: sganguly@opulentsystems.com From: "Sukanta Ganguly" To: "Julian Satran" , "IPS" Subject: RE: iSCSI over TLS Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA9FU9l17584 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Julian, It is correct that TLS records span TCP packets, but how does that become anymore of a problem. For packets to be resend via the TCP mechanisms, the sender TLS layers prepares the TLS record and then hands it over to TCP, TCP may break that TLS layer into e.g. say 5 packets and sends them to the receiver. If the receiver does not retrieve packet number 3, it will be resend by the sender. I did not see the problem that TLS brings into the picture. Also, what tweaking of the stack are you referring to in this scenario. This is just general handlinf of packets that are done anyway. iSCSI will only make sense of the packet after TLS decrypts the packets. Did I miss something here ??? SG *********** REPLY SEPARATOR *********** On 11/9/2001 at 4:14 PM Julian Satran wrote: >Bill, > >The one "tiny" item you forgot to mention is that TLS records span TCP and >iSCSI PDU boundaries. TLS records can't be decrypted in face of TCP packet >loss and markers/alignment can't be recovered (to be more precise require >a lot more tweaking of the stacks). > >Julo > > > > >"Bill Strahm" >08-11-01 23:55 >Please respond to "Bill Strahm" > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: RE: iSCSI over TLS > > > >Julian, > >I do not understand how TLS interferes with delivery of iSCSI packets any >more than IPsec. In either case your TOE MUST decrypt the packet and deal >with the results. I do not see how this changes the problem if the packet >is decrypted before going to the TOE (again the hardware to do this MUST >be >on the NIC device) or after going through the TOE processing... >Quick summary of what I think needs to happen >IPsec >1) receive L2 packet >2) determine it is IP >3) Apply packet policy based on L3 header >4) Decrypt packet - verify it is covered by the SA >5) Pass to L4 (TCP) for processing >6) Verify Framing/etc. >7) Done >TLS >1) Recieve L2 Packet >2) Pass to L3 >3) Pass to L4 (TCP) for processing >4) Decrypt packet >5) Verify Framing/etc >6) Done > >It turns out the policies for TLS are much simpler than for IPsec, the >application itself gets to determine if security should be turned on or >not >(rather than another application pushing policies into an SPD) and I don't >see a difference in the security offload requirements. In many cases TLS >will go through firewalls/NAT/NATP much better than IPsec, allowing for a >wider deployment model. > > >Bill Strahm >+========+=========+=========+=========+=========+=========+=========+ >Bill Strahm Software Development is a race between Programmers >Member of the trying to build bigger and better idiot proof software >Technical Staff and the Universe trying to produce bigger and better >bill@sanera.net idiots. >(503) 601-0263 So far the Universe is winning --- Rich Cook > >-----Original Message----- >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >Julian Satran >Sent: Tuesday, November 06, 2001 10:17 PM >To: ips@ece.cmu.edu >Subject: Re: iSCSI over TLS > > >Peter, > >A group of us seriously considered TLS. The main reason for dropping it >was that it would interfere with any mechanism we could think of doing >framing and steering and we thought that framing and steering are >essential at 10Gbps and over. > >Julo > > > > >"Peter Mellquist" >Sent by: owner-ips@ece.cmu.edu >07-11-01 02:15 >Please respond to "Peter Mellquist" > > > To: > cc: > Subject: iSCSI over TLS > > > >I am aware that the ips group is leaning toward IPSEC as for the security >solution but I am interested if anyone is also considering using Transport >Layer Security (TLS)? > >I am concerned that the requirement for IPSEC might make TOEs more >complex >than they need to be. Can TLS be optionally used as well as defined by the >specification? This could allow TOE vendors to only be concerned with >providing normal IPv4 / ipv6 and leave the security to a higher layer. A >TLS >stack sitting above the TOE could then handle security very well. Also, I >anticipate that the first generation of TOEs will not support IPSEC. With >a >iSCSI/TLS we could enable security solutions with the first generation of >TOEs and get speed and security. > >Are any TOE vendors planning to support IPSEC? > >Can TLS or IPSEC be supported? > >-peter > > > >Peter Mellquist >Seven Systems Technologies >575 Menlo Drive Suite 2 >Rocklin CA >916-577-1275 >peterm@seven-systems.com From owner-ips@ece.cmu.edu Fri Nov 9 11:49:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17805 for ; Fri, 9 Nov 2001 11:49:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9FuBU19752 for ips-outgoing; Fri, 9 Nov 2001 10:56:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from host32.cedant.com (host32.cedant.com [209.239.32.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9Fu9l19746 for ; Fri, 9 Nov 2001 10:56:09 -0500 (EST) Received: from sahara (39.sanjose-03-04rs16rt.ca.dial-access.att.net [12.81.1.39]) by host32.cedant.com (8.10.2/8.10.2) with ESMTP id fA9Ftub06371; Fri, 9 Nov 2001 10:55:56 -0500 Message-ID: <200111090808570568.002B4704@opulentsystems.com> In-Reply-To: References: X-Mailer: Calypso Version 3.30.00.00 (4) Date: Fri, 09 Nov 2001 08:08:57 -0800 Reply-To: sganguly@opulentsystems.com From: "Sukanta Ganguly" To: "Julian Satran" Cc: "IPS" Subject: RE: iSCSI over TLS Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fA9Fu9l19747 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Julian, I agree to the philosophy of framing and steering. With TLS support you can still place incomplete TLS records in host memory, without decrpyting it. Only buffering support needs to be beefed up on the host side. And frankly speaking that it already present with out of order packet deliveries etc. I am not proposing that TLS start acting on the packet immediately as the first peice of the TLS record arrives. The same behavior is observed with iSCSI packets which span TCP packets. The host has to wait for the entire iSCSI packet to be present in the host memory before the iSCSI layer can do anything with it. Do you see my point of argument! ;-) SG *********** REPLY SEPARATOR *********** On 11/9/2001 at 5:46 PM Julian Satran wrote: >The whole reason for doing framing and steering is to be able to start >placing data in host memory without waiting for all the data to arrive. >With TLS data can't be decrypted if pieces are missing. > >Julo > > > > >"Sukanta Ganguly" >09-11-01 17:42 >Please respond to sganguly > > > To: Julian Satran/Haifa/IBM@IBMIL, "IPS" > cc: > Subject: RE: iSCSI over TLS > > > >Julian, > It is correct that TLS records span TCP packets, but how does that >become anymore of a problem. For packets to be resend via the TCP >mechanisms, the sender TLS layers prepares the TLS record and then hands >it over to TCP, TCP may break that TLS layer into e.g. say 5 packets and >sends them to the receiver. If the receiver does not retrieve packet >number 3, it will be resend by the sender. > I did not see the problem that TLS brings into the picture. Also, what >tweaking of the stack are you referring to in this scenario. This is just >general handlinf of packets that are done anyway. iSCSI will only make >sense of the packet after TLS decrypts the packets. Did I miss something >here ??? > >SG > >*********** REPLY SEPARATOR *********** > >On 11/9/2001 at 4:14 PM Julian Satran wrote: > >>Bill, >> >>The one "tiny" item you forgot to mention is that TLS records span TCP >and >>iSCSI PDU boundaries. TLS records can't be decrypted in face of TCP >packet >>loss and markers/alignment can't be recovered (to be more precise require > >>a lot more tweaking of the stacks). >> >>Julo >> >> >> >> >>"Bill Strahm" >>08-11-01 23:55 >>Please respond to "Bill Strahm" >> >> >> To: Julian Satran/Haifa/IBM@IBMIL >> cc: >> Subject: RE: iSCSI over TLS >> >> >> >>Julian, >> >>I do not understand how TLS interferes with delivery of iSCSI packets any >>more than IPsec. In either case your TOE MUST decrypt the packet and >deal >>with the results. I do not see how this changes the problem if the >packet >>is decrypted before going to the TOE (again the hardware to do this MUST >>be >>on the NIC device) or after going through the TOE processing... >>Quick summary of what I think needs to happen >>IPsec >>1) receive L2 packet >>2) determine it is IP >>3) Apply packet policy based on L3 header >>4) Decrypt packet - verify it is covered by the SA >>5) Pass to L4 (TCP) for processing >>6) Verify Framing/etc. >>7) Done >>TLS >>1) Recieve L2 Packet >>2) Pass to L3 >>3) Pass to L4 (TCP) for processing >>4) Decrypt packet >>5) Verify Framing/etc >>6) Done >> >>It turns out the policies for TLS are much simpler than for IPsec, the >>application itself gets to determine if security should be turned on or >>not >>(rather than another application pushing policies into an SPD) and I >don't >>see a difference in the security offload requirements. In many cases TLS >>will go through firewalls/NAT/NATP much better than IPsec, allowing for a >>wider deployment model. >> >> >>Bill Strahm >>+========+=========+=========+=========+=========+=========+=========+ >>Bill Strahm Software Development is a race between Programmers >>Member of the trying to build bigger and better idiot proof software >>Technical Staff and the Universe trying to produce bigger and better >>bill@sanera.net idiots. >>(503) 601-0263 So far the Universe is winning --- Rich Cook >> >>-----Original Message----- >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >>Julian Satran >>Sent: Tuesday, November 06, 2001 10:17 PM >>To: ips@ece.cmu.edu >>Subject: Re: iSCSI over TLS >> >> >>Peter, >> >>A group of us seriously considered TLS. The main reason for dropping it >>was that it would interfere with any mechanism we could think of doing >>framing and steering and we thought that framing and steering are >>essential at 10Gbps and over. >> >>Julo >> >> >> >> >>"Peter Mellquist" >>Sent by: owner-ips@ece.cmu.edu >>07-11-01 02:15 >>Please respond to "Peter Mellquist" >> >> >> To: >> cc: >> Subject: iSCSI over TLS >> >> >> >>I am aware that the ips group is leaning toward IPSEC as for the security >>solution but I am interested if anyone is also considering using >Transport >>Layer Security (TLS)? >> >>I am concerned that the requirement for IPSEC might make TOEs more >>complex >>than they need to be. Can TLS be optionally used as well as defined by >the >>specification? This could allow TOE vendors to only be concerned with >>providing normal IPv4 / ipv6 and leave the security to a higher layer. A >>TLS >>stack sitting above the TOE could then handle security very well. Also, I >>anticipate that the first generation of TOEs will not support IPSEC. With >>a >>iSCSI/TLS we could enable security solutions with the first generation of >>TOEs and get speed and security. >> >>Are any TOE vendors planning to support IPSEC? >> >>Can TLS or IPSEC be supported? >> >>-peter >> >> >> >>Peter Mellquist >>Seven Systems Technologies >>575 Menlo Drive Suite 2 >>Rocklin CA >>916-577-1275 >>peterm@seven-systems.com From owner-ips@ece.cmu.edu Fri Nov 9 12:47:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20696 for ; Fri, 9 Nov 2001 12:47:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9Gb3u23317 for ips-outgoing; Fri, 9 Nov 2001 11:37:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA9Gb1l23310 for ; Fri, 9 Nov 2001 11:37:01 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fA9GaFQ08590; Fri, 9 Nov 2001 11:36:15 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fA9GaFs04141; Fri, 9 Nov 2001 11:36:15 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15340.1544.350000.232726@gargle.gargle.HOWL> Date: Fri, 9 Nov 2001 11:36:24 -0500 From: Paul Koning To: Julian_Satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Out of order commands References: X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 9 November 2001) by Julian Satran: > Paul, > > I should have seen this comming. > iSCSI assumes that TCP is there to take care of the transport window and > the transport window is as > large as it should be. However we are talking about the application > window. The application has advertize it's own window in addition to the > transport windows. With transport having things in order we can avoid > having to have a complete or even partial duplicate of the windows. If we > allow transport to deliver not in order we have to have them duplicated > (or at least the control part - and whatever DMA schemes we put in place > won't help). I'm having trouble parsing some of this, but let me make some comments. The iSCSI spec says: The queuing capacity of the receiving iSCSI layer is MaxCmdSN - ExpCmdSN + 1. That's quite unambiguous. It means that you set the application window to indicate what resources you have, NOT to something smaller in the hope that the wire delays will save you from being caught at cheating. Each layer that has a window must use that window to control the consumption of resources. It must advertise the ability to accept incoming stuff, i.e., resources are available at that layer to handle incoming traffic. So TCP advertises a window based on its resources, and iSCSI (separately) advertises its own window based on iSCSI resources needed to do iSCSI processing. Yes, it can rely on in-order delivery from TCP, because that's the promise TCP makes to the application layer. But overcommitting because you hope there will be packets in flight is an entirely different matter. The TCP/IP stack makes no such guarantee, and assuming there is one is broken design. paul From owner-ips@ece.cmu.edu Fri Nov 9 13:22:42 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA21906 for ; Fri, 9 Nov 2001 13:22:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9HrkH00020 for ips-outgoing; Fri, 9 Nov 2001 12:53:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9Hrhl00012 for ; Fri, 9 Nov 2001 12:53:43 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id JAA22492; Fri, 9 Nov 2001 09:53:32 -0800 (PST) From: "Murali Rajagopal" To: , "IPS Reflector" Cc: "Craig W. Carlson" , "Elizabeth Rodriguez" , "Steve Wilson" Subject: RE: FCencap: List ALL SOF/EOF codes Date: Fri, 9 Nov 2001 09:54:39 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 In-Reply-To: <3BEC030A.7DB34A75@compuserve.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ralph: It is unfortunate that BB went through public comments and the full standardization process with no one catching these illegal codes at that time. But these codes will be corrected in FC-BB-2 and perhaps you may be able to adopt these updated tables. I will try getting the correct tables in the next Rev of the BB-2 document to be released before the Austin Meeting. -Murali -----Original Message----- From: ralphoweber@compuserve.com [mailto:ralphoweber@compuserve.com] Sent: Friday, November 09, 2001 8:24 AM To: IPS Reflector Cc: Elizabeth Rodriguez; Murali Rajagopal Subject: Re: FCencap: List ALL SOF/EOF codes Murali, > I also recollect that there was some discussion about some illegal > codes in that table either in that meeting or in one of the ensuing > T11 meetings. If the SOF/EOF definitions are wrong in FC-BB then someone needs to get an amendment going to fix that. If they are correct in the published FC-BB, then copying them to the FC Encapsulation will not be a problem. It makes very little sense for the FC Encapsulation draft to be the place where T11 corrects its mistakes. Elizabeth, > We had this discussion back in January, and basically came to the > conclusion that Class 1 and Class 4 should not be included, for the > reasons that class 1 really cannot be supported across the IP network > and class 4 is not really not defined yets, so the codes are not > guaranteed to remain constant. However, that discussion back in January concerned the FCIP draft. The FC Encapsulation draft was not devised until February. The argument that Class 4 should not be included because it is not defined is no longer valid because Gary Stephens has almost completed all the tasks needed to define Class 4 and none of his changes affect the SOF/EOF codes for Class 4. It is true that Gary's changes say that Class 4 frames will not transit ISLs, but the iFCP folks might choose to support them. As for Class 1, the question becomes which of the following two reasons motivated the decision last January: 1) TCP absolutely positively cannot support Class 1, no matter how well designed the protocol is; or 2) The amount of FCIP design effort needed to support Class 1 is beyond that thought sensible, so FCIP is not going to support Class 1. If choice 1 was the motivation, then it is true that Class 1 should not be added to the FC Encapsulation draft. However, this choice means telling the IESG that there is something that TCP cannot do. I see no reason why the IPS working group needs to undertake rolling that rock up that hill. If choice 2 was the motivation, then Class 1 should be listed in the FC Encapsulation draft because somebody someday might want to design a suitably inventive protocol for transporting FC Class 1 over TCP and the FC Encapsulation should not be a road block to that effort. > Even if we do decide to accept Ralph's arguments to include > class 1 and class 4 SOF/EOF codes, we cannot take ALL the codes > from FC-BB and incorporate them into the FC Encapsulation draft. > Recall, we started out by including all the SOF/EOF codes from > FC-BB-2. We reevaluated in January 2001, when we analyzed the > codes themselves. Several that we excluded I think were valid > (e.g. for class 1 and class 4), but others were completely bogus > and undefined anywhere other than in FC-BB. We cannot just blindly > accept those codes. If this motion is considered, we need to reopen > that evaluation made in the January interim meeting and make a > determination as to what codes need to be included in FC Common > Encapsulation, and make sure not to include invalid codes. It is most unfortunate that, in the ten months since all those bogus codes were discovered, not one attempt has been made in T11 to correct those errors. None the less, the lack of corrective action in T11 suggests that there are far fewer problems with bogus codes than the above diatribe suggests. However, I am not opposed to keeping demonstrably nonsense SOF/EOF codes out of the FC Encapsulation draft. SOF/EOF codes related to the various classes of service DO NOT fall into the bogus category. > (Participant mode)I disagree with this motion. Please reconsider your opposition to this proposal. Thanks. Ralph... From owner-ips@ece.cmu.edu Fri Nov 9 13:22:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA21917 for ; Fri, 9 Nov 2001 13:22:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9HaN128486 for ips-outgoing; Fri, 9 Nov 2001 12:36:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar2ab.compuserve.com (siaar2ab.compuserve.com [149.174.40.138]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9HaLl28481 for ; Fri, 9 Nov 2001 12:36:21 -0500 (EST) Received: (from mailgate@localhost) by siaar2ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id MAA14955 for ips@ece.cmu.edu; Fri, 9 Nov 2001 12:36:15 -0500 (EST) Received: from compuserve.com (sfr-tgn-sfr-vty68.as.wcom.net [216.192.37.68]) by siaar2ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id MAA14920; Fri, 9 Nov 2001 12:36:05 -0500 (EST) Message-ID: <3BEC030A.7DB34A75@compuserve.com> Date: Fri, 09 Nov 2001 10:23:38 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win98; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector CC: Elizabeth Rodriguez , Murali Rajagopal Subject: Re: FCencap: List ALL SOF/EOF codes References: <9410A0F67DFE7F4D998D4538236498360408F1@nc8220exchange.ral.lucent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Murali, > I also recollect that there was some discussion about some illegal > codes in that table either in that meeting or in one of the ensuing > T11 meetings. If the SOF/EOF definitions are wrong in FC-BB then someone needs to get an amendment going to fix that. If they are correct in the published FC-BB, then copying them to the FC Encapsulation will not be a problem. It makes very little sense for the FC Encapsulation draft to be the place where T11 corrects its mistakes. Elizabeth, > We had this discussion back in January, and basically came to the > conclusion that Class 1 and Class 4 should not be included, for the > reasons that class 1 really cannot be supported across the IP network > and class 4 is not really not defined yets, so the codes are not > guaranteed to remain constant. However, that discussion back in January concerned the FCIP draft. The FC Encapsulation draft was not devised until February. The argument that Class 4 should not be included because it is not defined is no longer valid because Gary Stephens has almost completed all the tasks needed to define Class 4 and none of his changes affect the SOF/EOF codes for Class 4. It is true that Gary's changes say that Class 4 frames will not transit ISLs, but the iFCP folks might choose to support them. As for Class 1, the question becomes which of the following two reasons motivated the decision last January: 1) TCP absolutely positively cannot support Class 1, no matter how well designed the protocol is; or 2) The amount of FCIP design effort needed to support Class 1 is beyond that thought sensible, so FCIP is not going to support Class 1. If choice 1 was the motivation, then it is true that Class 1 should not be added to the FC Encapsulation draft. However, this choice means telling the IESG that there is something that TCP cannot do. I see no reason why the IPS working group needs to undertake rolling that rock up that hill. If choice 2 was the motivation, then Class 1 should be listed in the FC Encapsulation draft because somebody someday might want to design a suitably inventive protocol for transporting FC Class 1 over TCP and the FC Encapsulation should not be a road block to that effort. > Even if we do decide to accept Ralph's arguments to include > class 1 and class 4 SOF/EOF codes, we cannot take ALL the codes > from FC-BB and incorporate them into the FC Encapsulation draft. > Recall, we started out by including all the SOF/EOF codes from > FC-BB-2. We reevaluated in January 2001, when we analyzed the > codes themselves. Several that we excluded I think were valid > (e.g. for class 1 and class 4), but others were completely bogus > and undefined anywhere other than in FC-BB. We cannot just blindly > accept those codes. If this motion is considered, we need to reopen > that evaluation made in the January interim meeting and make a > determination as to what codes need to be included in FC Common > Encapsulation, and make sure not to include invalid codes. It is most unfortunate that, in the ten months since all those bogus codes were discovered, not one attempt has been made in T11 to correct those errors. None the less, the lack of corrective action in T11 suggests that there are far fewer problems with bogus codes than the above diatribe suggests. However, I am not opposed to keeping demonstrably nonsense SOF/EOF codes out of the FC Encapsulation draft. SOF/EOF codes related to the various classes of service DO NOT fall into the bogus category. > (Participant mode)I disagree with this motion. Please reconsider your opposition to this proposal. Thanks. Ralph... From owner-ips@ece.cmu.edu Fri Nov 9 13:23:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA21936 for ; Fri, 9 Nov 2001 13:23:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9HrPO29981 for ips-outgoing; Fri, 9 Nov 2001 12:53:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9HrNl29975 for ; Fri, 9 Nov 2001 12:53:23 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA119044 for ; Fri, 9 Nov 2001 18:53:17 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9HrFB90546 for ; Fri, 9 Nov 2001 18:53:16 +0100 Importance: Normal Subject: RE: iSCSI: IPsec tunnel / transport mode decision To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "Ofer Biran" Date: Fri, 9 Nov 2001 19:53:45 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 19:53:15 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk It seems that most people prefer tunnel over transport mode and there is no real opposition for choosing tunnel mode as the MUST. In view of that we intend to add it in version 09 in the following iSCSI statements: In Section 10.3.1 Data Integrity and Authentication : "An iSCSI compliant initiator or target MUST provide data integrity and authentication by implementing IPSec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." And in Section 10.3.2 Confidentiality : "An iSCSI compliant initiator or target MUST provide confidentiality by implementing IPSec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." Any objection ? Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 01/11/2001 20:03:29 Please respond to To: Ofer Biran/Haifa/IBM@IBMIL, cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Thursday, November 01, 2001 4:31 AM To: ips@ece.cmu.edu Subject: iSCSI: IPsec tunnel / transport mode decision I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Fri Nov 9 13:26:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22132 for ; Fri, 9 Nov 2001 13:26:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9I5oU01068 for ips-outgoing; Fri, 9 Nov 2001 13:05:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9I5ll01062 for ; Fri, 9 Nov 2001 13:05:48 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id TAA137802 for ; Fri, 9 Nov 2001 19:05:42 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9I5fc55552 for ; Fri, 9 Nov 2001 19:05:41 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI over TLS X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 9 Nov 2001 20:05:42 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 20:05:40, Serialize complete at 09/11/2001 20:05:41, Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 20:05:41 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk SG, The issue is that if you are not able to decrypt you have no iSCSI packet header (or RDMA headers) and can't place data in memory (i.e. you need anonymous buffers and have to copy). With IPSec you are far better off. Julo "Sukanta Ganguly" 09-11-01 18:08 Please respond to sganguly To: Julian Satran/Haifa/IBM@IBMIL cc: "IPS" Subject: RE: iSCSI over TLS Julian, I agree to the philosophy of framing and steering. With TLS support you can still place incomplete TLS records in host memory, without decrpyting it. Only buffering support needs to be beefed up on the host side. And frankly speaking that it already present with out of order packet deliveries etc. I am not proposing that TLS start acting on the packet immediately as the first peice of the TLS record arrives. The same behavior is observed with iSCSI packets which span TCP packets. The host has to wait for the entire iSCSI packet to be present in the host memory before the iSCSI layer can do anything with it. Do you see my point of argument! ;-) SG *********** REPLY SEPARATOR *********** On 11/9/2001 at 5:46 PM Julian Satran wrote: >The whole reason for doing framing and steering is to be able to start >placing data in host memory without waiting for all the data to arrive. >With TLS data can't be decrypted if pieces are missing. > >Julo > > > > >"Sukanta Ganguly" >09-11-01 17:42 >Please respond to sganguly > > > To: Julian Satran/Haifa/IBM@IBMIL, "IPS" > cc: > Subject: RE: iSCSI over TLS > > > >Julian, > It is correct that TLS records span TCP packets, but how does that >become anymore of a problem. For packets to be resend via the TCP >mechanisms, the sender TLS layers prepares the TLS record and then hands >it over to TCP, TCP may break that TLS layer into e.g. say 5 packets and >sends them to the receiver. If the receiver does not retrieve packet >number 3, it will be resend by the sender. > I did not see the problem that TLS brings into the picture. Also, what >tweaking of the stack are you referring to in this scenario. This is just >general handlinf of packets that are done anyway. iSCSI will only make >sense of the packet after TLS decrypts the packets. Did I miss something >here ??? > >SG > >*********** REPLY SEPARATOR *********** > >On 11/9/2001 at 4:14 PM Julian Satran wrote: > >>Bill, >> >>The one "tiny" item you forgot to mention is that TLS records span TCP >and >>iSCSI PDU boundaries. TLS records can't be decrypted in face of TCP >packet >>loss and markers/alignment can't be recovered (to be more precise require > >>a lot more tweaking of the stacks). >> >>Julo >> >> >> >> >>"Bill Strahm" >>08-11-01 23:55 >>Please respond to "Bill Strahm" >> >> >> To: Julian Satran/Haifa/IBM@IBMIL >> cc: >> Subject: RE: iSCSI over TLS >> >> >> >>Julian, >> >>I do not understand how TLS interferes with delivery of iSCSI packets any >>more than IPsec. In either case your TOE MUST decrypt the packet and >deal >>with the results. I do not see how this changes the problem if the >packet >>is decrypted before going to the TOE (again the hardware to do this MUST >>be >>on the NIC device) or after going through the TOE processing... >>Quick summary of what I think needs to happen >>IPsec >>1) receive L2 packet >>2) determine it is IP >>3) Apply packet policy based on L3 header >>4) Decrypt packet - verify it is covered by the SA >>5) Pass to L4 (TCP) for processing >>6) Verify Framing/etc. >>7) Done >>TLS >>1) Recieve L2 Packet >>2) Pass to L3 >>3) Pass to L4 (TCP) for processing >>4) Decrypt packet >>5) Verify Framing/etc >>6) Done >> >>It turns out the policies for TLS are much simpler than for IPsec, the >>application itself gets to determine if security should be turned on or >>not >>(rather than another application pushing policies into an SPD) and I >don't >>see a difference in the security offload requirements. In many cases TLS >>will go through firewalls/NAT/NATP much better than IPsec, allowing for a >>wider deployment model. >> >> >>Bill Strahm >>+========+=========+=========+=========+=========+=========+=========+ >>Bill Strahm Software Development is a race between Programmers >>Member of the trying to build bigger and better idiot proof software >>Technical Staff and the Universe trying to produce bigger and better >>bill@sanera.net idiots. >>(503) 601-0263 So far the Universe is winning --- Rich Cook >> >>-----Original Message----- >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >>Julian Satran >>Sent: Tuesday, November 06, 2001 10:17 PM >>To: ips@ece.cmu.edu >>Subject: Re: iSCSI over TLS >> >> >>Peter, >> >>A group of us seriously considered TLS. The main reason for dropping it >>was that it would interfere with any mechanism we could think of doing >>framing and steering and we thought that framing and steering are >>essential at 10Gbps and over. >> >>Julo >> >> >> >> >>"Peter Mellquist" >>Sent by: owner-ips@ece.cmu.edu >>07-11-01 02:15 >>Please respond to "Peter Mellquist" >> >> >> To: >> cc: >> Subject: iSCSI over TLS >> >> >> >>I am aware that the ips group is leaning toward IPSEC as for the security >>solution but I am interested if anyone is also considering using >Transport >>Layer Security (TLS)? >> >>I am concerned that the requirement for IPSEC might make TOEs more >>complex >>than they need to be. Can TLS be optionally used as well as defined by >the >>specification? This could allow TOE vendors to only be concerned with >>providing normal IPv4 / ipv6 and leave the security to a higher layer. A >>TLS >>stack sitting above the TOE could then handle security very well. Also, I >>anticipate that the first generation of TOEs will not support IPSEC. With >>a >>iSCSI/TLS we could enable security solutions with the first generation of >>TOEs and get speed and security. >> >>Are any TOE vendors planning to support IPSEC? >> >>Can TLS or IPSEC be supported? >> >>-peter >> >> >> >>Peter Mellquist >>Seven Systems Technologies >>575 Menlo Drive Suite 2 >>Rocklin CA >>916-577-1275 >>peterm@seven-systems.com From owner-ips@ece.cmu.edu Fri Nov 9 13:54:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA23112 for ; Fri, 9 Nov 2001 13:54:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9I5aA01047 for ips-outgoing; Fri, 9 Nov 2001 13:05:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9I5Yl01042 for ; Fri, 9 Nov 2001 13:05:34 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id TAA54530 for ; Fri, 9 Nov 2001 19:05:27 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fA9I5Qc58184 for ; Fri, 9 Nov 2001 19:05:26 +0100 Importance: Normal Subject: iSCSI: IKE normative guidelines To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "Ofer Biran" Date: Fri, 9 Nov 2001 20:05:54 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 09/11/2001 20:05:26 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk In the framework of the effort being done now by the security team to sync the normative statements in the security draft with the protocol drafts, I suggest to adopt the following IKE normative guidelines that already appear in the security draft for iSCSI: ================================================================== "Conformant iSCSI, iFCP and FCIP implementations MUST support peer authentication using a pre-shared key, and MAY support certificate-based peer authentication using digital signatures. Peer authentication using the public key encryption methods outlined in IKE's sections 5.2 and 5.3[7] SHOULD NOT be used. ...Conformant iSCSI, FCIP and iFCP security implementations MUST support both IKE Main Mode and Aggressive Mode ...When digital signatures are used to achieve authentication, an IKE negotiator SHOULD use IKE Certificate Request Payload(s) to specify the certificate authority ...IKE negotiators SHOULD check the pertinent Certificate Revocation List (CRL) before accepting a PKI certificate for use in IKE's authentication procedures" ================================================================== Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Fri Nov 9 13:56:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA23400 for ; Fri, 9 Nov 2001 13:56:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9IOZs02425 for ips-outgoing; Fri, 9 Nov 2001 13:24:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9IOWl02419 for ; Fri, 9 Nov 2001 13:24:33 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id KAA23406; Fri, 9 Nov 2001 10:24:26 -0800 (PST) From: "Murali Rajagopal" To: "IPS Reflector" , Cc: "Craig W. Carlson" , "Jim Nelson" , "Steve Wilson" , "Elizabeth Rodriguez" Subject: RE: FCencap: List ALL SOF/EOF codes Date: Fri, 9 Nov 2001 10:25:33 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 In-Reply-To: <3BEC030A.7DB34A75@compuserve.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ralph: I did a quick check with the FC-FS list of codes against the ones listed in FC-BB-2 Rev. 5.1 and found that all FC-BB-2 SOF and EOF codes defined in Tables A.3 and A.4 are legal and only defined for Class 2, 3, and F. The following codes which appear in a separate OS table A.1 are not defined in FC-FS and I guess that makes them illegal: SOFc2 SOFc3 SOFcf SOFnf I guess you can still pick up tables A.3 and A.4 safely which are legal and only define Class 2, 3, and F. -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ralph Weber Sent: Friday, November 09, 2001 8:24 AM To: IPS Reflector Cc: Elizabeth Rodriguez; Murali Rajagopal Subject: Re: FCencap: List ALL SOF/EOF codes Murali, > I also recollect that there was some discussion about some illegal > codes in that table either in that meeting or in one of the ensuing > T11 meetings. If the SOF/EOF definitions are wrong in FC-BB then someone needs to get an amendment going to fix that. If they are correct in the published FC-BB, then copying them to the FC Encapsulation will not be a problem. It makes very little sense for the FC Encapsulation draft to be the place where T11 corrects its mistakes. Elizabeth, > We had this discussion back in January, and basically came to the > conclusion that Class 1 and Class 4 should not be included, for the > reasons that class 1 really cannot be supported across the IP network > and class 4 is not really not defined yets, so the codes are not > guaranteed to remain constant. However, that discussion back in January concerned the FCIP draft. The FC Encapsulation draft was not devised until February. The argument that Class 4 should not be included because it is not defined is no longer valid because Gary Stephens has almost completed all the tasks needed to define Class 4 and none of his changes affect the SOF/EOF codes for Class 4. It is true that Gary's changes say that Class 4 frames will not transit ISLs, but the iFCP folks might choose to support them. As for Class 1, the question becomes which of the following two reasons motivated the decision last January: 1) TCP absolutely positively cannot support Class 1, no matter how well designed the protocol is; or 2) The amount of FCIP design effort needed to support Class 1 is beyond that thought sensible, so FCIP is not going to support Class 1. If choice 1 was the motivation, then it is true that Class 1 should not be added to the FC Encapsulation draft. However, this choice means telling the IESG that there is something that TCP cannot do. I see no reason why the IPS working group needs to undertake rolling that rock up that hill. If choice 2 was the motivation, then Class 1 should be listed in the FC Encapsulation draft because somebody someday might want to design a suitably inventive protocol for transporting FC Class 1 over TCP and the FC Encapsulation should not be a road block to that effort. > Even if we do decide to accept Ralph's arguments to include > class 1 and class 4 SOF/EOF codes, we cannot take ALL the codes > from FC-BB and incorporate them into the FC Encapsulation draft. > Recall, we started out by including all the SOF/EOF codes from > FC-BB-2. We reevaluated in January 2001, when we analyzed the > codes themselves. Several that we excluded I think were valid > (e.g. for class 1 and class 4), but others were completely bogus > and undefined anywhere other than in FC-BB. We cannot just blindly > accept those codes. If this motion is considered, we need to reopen > that evaluation made in the January interim meeting and make a > determination as to what codes need to be included in FC Common > Encapsulation, and make sure not to include invalid codes. It is most unfortunate that, in the ten months since all those bogus codes were discovered, not one attempt has been made in T11 to correct those errors. None the less, the lack of corrective action in T11 suggests that there are far fewer problems with bogus codes than the above diatribe suggests. However, I am not opposed to keeping demonstrably nonsense SOF/EOF codes out of the FC Encapsulation draft. SOF/EOF codes related to the various classes of service DO NOT fall into the bogus category. > (Participant mode)I disagree with this motion. Please reconsider your opposition to this proposal. Thanks. Ralph... From owner-ips@ece.cmu.edu Fri Nov 9 14:00:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23689 for ; Fri, 9 Nov 2001 14:00:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9Iesd03776 for ips-outgoing; Fri, 9 Nov 2001 13:40:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9Iepl03771 for ; Fri, 9 Nov 2001 13:40:51 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA9Ieju24062 for ; Fri, 9 Nov 2001 10:40:45 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA9Ied614567 for ; Fri, 9 Nov 2001 10:40:39 -0800 From: "Bill Strahm" To: "Ips@Ece. Cmu. Edu" Subject: RE: iSCSI over TLS Date: Fri, 9 Nov 2001 10:40:29 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I am confused... Why can't I put the data into the memory location specified by the TCP sequence number ??? I am assuming that is what you are doing in the case of IPsec dropping a TCP frame in the middle of the stream. TCP stacks that I am aware of WILL NOT pass out of order stream data to applications until the intermediate frames are dealt with anyway, so what is the difference between holding encrypted frames and unencrypted frames until the stream is in order ? I believe that there are also TLS options to do per frame crypto, however at that point you have to start shipping over IVs with each frame, removing most wire advantages of TLS over IPsec... Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Friday, November 09, 2001 10:06 AM To: ips@ece.cmu.edu Subject: RE: iSCSI over TLS SG, The issue is that if you are not able to decrypt you have no iSCSI packet header (or RDMA headers) and can't place data in memory (i.e. you need anonymous buffers and have to copy). With IPSec you are far better off. Julo "Sukanta Ganguly" 09-11-01 18:08 Please respond to sganguly To: Julian Satran/Haifa/IBM@IBMIL cc: "IPS" Subject: RE: iSCSI over TLS Julian, I agree to the philosophy of framing and steering. With TLS support you can still place incomplete TLS records in host memory, without decrpyting it. Only buffering support needs to be beefed up on the host side. And frankly speaking that it already present with out of order packet deliveries etc. I am not proposing that TLS start acting on the packet immediately as the first peice of the TLS record arrives. The same behavior is observed with iSCSI packets which span TCP packets. The host has to wait for the entire iSCSI packet to be present in the host memory before the iSCSI layer can do anything with it. Do you see my point of argument! ;-) SG *********** REPLY SEPARATOR *********** On 11/9/2001 at 5:46 PM Julian Satran wrote: >The whole reason for doing framing and steering is to be able to start >placing data in host memory without waiting for all the data to arrive. >With TLS data can't be decrypted if pieces are missing. > >Julo > > > > >"Sukanta Ganguly" >09-11-01 17:42 >Please respond to sganguly > > > To: Julian Satran/Haifa/IBM@IBMIL, "IPS" > cc: > Subject: RE: iSCSI over TLS > > > >Julian, > It is correct that TLS records span TCP packets, but how does that >become anymore of a problem. For packets to be resend via the TCP >mechanisms, the sender TLS layers prepares the TLS record and then hands >it over to TCP, TCP may break that TLS layer into e.g. say 5 packets and >sends them to the receiver. If the receiver does not retrieve packet >number 3, it will be resend by the sender. > I did not see the problem that TLS brings into the picture. Also, what >tweaking of the stack are you referring to in this scenario. This is just >general handlinf of packets that are done anyway. iSCSI will only make >sense of the packet after TLS decrypts the packets. Did I miss something >here ??? > >SG > >*********** REPLY SEPARATOR *********** > >On 11/9/2001 at 4:14 PM Julian Satran wrote: > >>Bill, >> >>The one "tiny" item you forgot to mention is that TLS records span TCP >and >>iSCSI PDU boundaries. TLS records can't be decrypted in face of TCP >packet >>loss and markers/alignment can't be recovered (to be more precise require > >>a lot more tweaking of the stacks). >> >>Julo >> >> >> >> >>"Bill Strahm" >>08-11-01 23:55 >>Please respond to "Bill Strahm" >> >> >> To: Julian Satran/Haifa/IBM@IBMIL >> cc: >> Subject: RE: iSCSI over TLS >> >> >> >>Julian, >> >>I do not understand how TLS interferes with delivery of iSCSI packets any >>more than IPsec. In either case your TOE MUST decrypt the packet and >deal >>with the results. I do not see how this changes the problem if the >packet >>is decrypted before going to the TOE (again the hardware to do this MUST >>be >>on the NIC device) or after going through the TOE processing... >>Quick summary of what I think needs to happen >>IPsec >>1) receive L2 packet >>2) determine it is IP >>3) Apply packet policy based on L3 header >>4) Decrypt packet - verify it is covered by the SA >>5) Pass to L4 (TCP) for processing >>6) Verify Framing/etc. >>7) Done >>TLS >>1) Recieve L2 Packet >>2) Pass to L3 >>3) Pass to L4 (TCP) for processing >>4) Decrypt packet >>5) Verify Framing/etc >>6) Done >> >>It turns out the policies for TLS are much simpler than for IPsec, the >>application itself gets to determine if security should be turned on or >>not >>(rather than another application pushing policies into an SPD) and I >don't >>see a difference in the security offload requirements. In many cases TLS >>will go through firewalls/NAT/NATP much better than IPsec, allowing for a >>wider deployment model. >> >> >>Bill Strahm >>+========+=========+=========+=========+=========+=========+=========+ >>Bill Strahm Software Development is a race between Programmers >>Member of the trying to build bigger and better idiot proof software >>Technical Staff and the Universe trying to produce bigger and better >>bill@sanera.net idiots. >>(503) 601-0263 So far the Universe is winning --- Rich Cook >> >>-----Original Message----- >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >>Julian Satran >>Sent: Tuesday, November 06, 2001 10:17 PM >>To: ips@ece.cmu.edu >>Subject: Re: iSCSI over TLS >> >> >>Peter, >> >>A group of us seriously considered TLS. The main reason for dropping it >>was that it would interfere with any mechanism we could think of doing >>framing and steering and we thought that framing and steering are >>essential at 10Gbps and over. >> >>Julo >> >> >> >> >>"Peter Mellquist" >>Sent by: owner-ips@ece.cmu.edu >>07-11-01 02:15 >>Please respond to "Peter Mellquist" >> >> >> To: >> cc: >> Subject: iSCSI over TLS >> >> >> >>I am aware that the ips group is leaning toward IPSEC as for the security >>solution but I am interested if anyone is also considering using >Transport >>Layer Security (TLS)? >> >>I am concerned that the requirement for IPSEC might make TOEs more >>complex >>than they need to be. Can TLS be optionally used as well as defined by >the >>specification? This could allow TOE vendors to only be concerned with >>providing normal IPv4 / ipv6 and leave the security to a higher layer. A >>TLS >>stack sitting above the TOE could then handle security very well. Also, I >>anticipate that the first generation of TOEs will not support IPSEC. With >>a >>iSCSI/TLS we could enable security solutions with the first generation of >>TOEs and get speed and security. >> >>Are any TOE vendors planning to support IPSEC? >> >>Can TLS or IPSEC be supported? >> >>-peter >> >> >> >>Peter Mellquist >>Seven Systems Technologies >>575 Menlo Drive Suite 2 >>Rocklin CA >>916-577-1275 >>peterm@seven-systems.com From owner-ips@ece.cmu.edu Fri Nov 9 14:55:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA25642 for ; Fri, 9 Nov 2001 14:55:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9JVxg08019 for ips-outgoing; Fri, 9 Nov 2001 14:31:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9JVtl08006 for ; Fri, 9 Nov 2001 14:31:56 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel10.hp.com (Postfix) with ESMTP id CFC001F748; Fri, 9 Nov 2001 11:31:43 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id LAA10255; Fri, 9 Nov 2001 11:31:39 -0800 (PST) Message-ID: <3BEC3129.51854A8@cup.hp.com> Date: Fri, 09 Nov 2001 11:40:25 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Julian Satran Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian Satran wrote: > > Mallikarjun, > > There are several other mechanisms that will have to be changed to allow > for OOO. > Task abort relies on the fact that the task management request is > delivered after the task to be aborted and on the same connection. Julian, I don't agree with your example above. The initiator will only issue an Abort Task after issuing the command, [on experiencing a timeout of the command, or for other error recovery reasons]. The in-order properties of TCP ensure that the Abort Task is always delivered after the command, as long as the Abort Task is sent on the same connection as the command. The case under discussion is that multiple unrelated iscsi command pdu's may be shipped out of order on a given connection. Obviously, this does not apply for an abort task of an issued command. Again, OOO commands within a connection is not a new design. There was nothing preventing this behaviour up until now and nothing in the spec is broken by this behaviour. I have not heard a SINGLE case of some feature in the spec being broken by this behaviour. I would think the newly introduced restriction of requiring initiators to issue commands in order within a connection is a new design restriction and not the other way around ! > Retry command cleaning relies on the fact that the "cleaning command" is > shipped after the retried command. > And I suspect we may find others. What do you mean by "cleaning command" ? I see nothing called "cleaning command" in the draft. If you are referring to a re-issued command for the purpose of plugging a hole, again, the original command and the re-issued one are related and OOO will not occur in this case. I would like to see some real scenarios where the current draft is broken due to the initiator shipping multiple unrelated command OOO within a given connection. What are the REAL reasons for imposing this new design restriction ? - Santosh > > This thread is going nowhere IMHO for two reasons: > > the proponents of OOO (Rod, Santosh and Bob Russell ) have faced us with > an issue not a design - the validity of which is hotly contested > I and other authors do not seem to be willing to do all the work and make > a design (or complete set of design changes) based on this sketch. Unlike > other requests I could not see anything appealing enough to justify the > time spent on doing it. > > I suggest that Rod, Santosh and whoever else is willing to work with them > put together a draft based on their ideas that should: > > clearly state what is to be gained > indicate in sufficient what changes are needed in the current draft to > accommodate the new design not only in command delivery but in all other > areas (task management, recovery, logout etc.). > > Once we have this draft I assure you that we will give it serious > consideration. > > Julo > > "Mallikarjun C." > Sent by: owner-ips@ece.cmu.edu > 09-11-01 03:28 > Please respond to "Mallikarjun C." > > > To: John Hufferd/San Jose/IBM@IBMUS > cc: > Subject: Re: iSCSI: Out of order commands > > > > John, > > Sorry, this note got a little longer than I would've liked, but.... > > I believe there are cases where OOO CmdSN handling is a > legitimate requirement on targets due to exception events - > a) retransmitting a CmdSN on a command acknowledgement > timeout (within-connection recovery class). This manifests as > an OOO CmdSN on the connection to a target if it didn't see > the original copy due to a digest error. > b) retransmitting the last few "lost" commands due to a connection > failure on a new connection. If this new connection had already > carried a CmdSN greater than these retransmitted commands > (prior to connection failure), this again manifests as OOO CmdSN > on the new connection to the target. > > OTOH, I believe sending OOO CmdSNs on a connection as a > regular practice is counterproductive, since the target must continuously > re-order the initiator "optimization" leading to a zero-sum game. I > would argue that the need to dispatch CmdSNs OOO due to immediate > data DMA (brought up by Rod) can be addressed by simple NIC > changes to prefetch data for the next command (or more simply use > unsolicited separate data PDUs, if negotiated). [ You got to deal > with the case of all commands being writes anyway! ] > > If we allow OOO CmdSNs on a connection (I'd advocate discouraging > it as a regular practice), I don't believe any of the stuff in error > recovery > breaks (nor does it affect the current reliance on ExpCmdSN). Julian > perhaps can comment. > - All the in-order assumptions are for DataSNs/R2TSNs/StatSNs, not > for CmdSNs. > - Any multi-connection session by definition must deal with OOO > CmdSNs. > - I belive that the current abort task scheme for immediate commands > detailed in section 9.3 caters to OOO CmdSNs on a connection > as well (we must be dealing with an immediate Abort arriving before > the command today, since the command could have been hit with > a digest error). > > To summarize, here is what I suggested to Julian in a private email - > > a)I suggest using a SHOULD for in-order dispatch of > commands on a connection - for an initiator. > > b)I suggest using a SHALL handle out-of-order commands > on a connection - for the target (as Barry pointed out). > > Hope that was useful. > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > Hewlett-Packard MS 5668 > Roseville CA 95747 > > ----- Original Message ----- > From: "John Hufferd" > To: > Cc: > Sent: Thursday, November 08, 2001 1:40 PM > Subject: Re: iSCSI: Out of order commands > > > > > Mallikarjun, > > Could you comment on the concept of OOO on the ErrorRecoveryLevel>0. I > had > > thought that "in order delivery" was part of the detection of missing > PDUs > > and needed for timely Recovery. I was wondering if this changes the way > we > > would use the ExpCmdSN, etc. > > > > I think your opinions on this part of the OOO discussion would be > valuable. > > For example, how would you contrast the differences in detecting a > problem > > and recovering from that problem etc., today vrs the OOO approach (if > any). > > > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSG San Jose Ca > > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > Home Office (408) 997-6136, Cell: (408) 499-9702 > > Internet address: hufferd@us.ibm.com > > > > > > "Mallikarjun C." @ece.cmu.edu on 11/07/2001 09:41:05 AM > > > > Please respond to cbm@rose.hp.com > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > To: Santosh Rao , ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: Out of order commands > > > > > > > > Santosh, > > > > I have only one comment on your responses. > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > I made two convenient assumptions if you noticed, :-), one of which > > is that target forces session recovery on *any* error that it sees > > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > > error. With that assumption, a target can afford not to implement > > a scoreboard. > > > > As I said in a private note, I guess what primarily bothers me about > > OOO commands on a connection is that it requires the receiver to > > undo this "optimization" on its end - most notably on a single > > connection. TCP experts may comment on how/if they dealt with a > > similar issue. > > > > OTOH, you had some valid comments on exceptions to ordering during > > connection recovery. Perhaps we can move on by making Julian's > > proposed stipulation a SHOULD.... > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > Santosh Rao wrote: > > > > > > Mallikarjun, > > > > > > Some comments below. > > > > > > Regards, > > > Santosh > > > > > > "Mallikarjun C." wrote: > > > > > > > > Rod and Julian, > > > > > > > > This has been an interesting thread of discussion. Some > > > > comments - > > > > > > > > 1.My first reaction was - allowing out-of-order command > > > > transmission on the same connection deprives targets of > > > > an implementation choice. Targets which support only > > > > single-connection sessions and only support session > > > > recovery (reasonable assumptions in my mind) can no > > > > longer afford *not to* implement a command scoreboard. > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > > > Thus, there is no getting away from implementing a sequencer at the > > > target. Given this, I think it is unreasonable to restrict initiator > > > implementation flexibility by imposing a strict ordering requirement > > > within the connection. > > > > > > > 2.Any end-node efficiency that is sought to be achieved > > > > by transmitting CmdSNs out-of-order from the initiator > > > > would be lost on the other end-node, since the target > > > > now must wait for re-ordering the commands. > > > > > > It has to handle this situation anyway to deal with holes caused by > > > digest errors. This scenario occurs even with initiators that issue > > > commands in order. > > > > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > > link badwidth (albeit at the expense of end-node efficiency), > > > > compared to idling the link waiting for outbound DMA. > > > > We have to determine if this is a reasonable trade-off. > > > > > > > > 4.I can see Rod's point that prefetching all immediate > > > > data can be a burden on the NIC resources. But, two > > > > questions - > > > > - could the NIC not use unsolicited separate data > > > > PDUs in these cases? [ I realize that InitialR2T > > > > has to be "no" to let it happen... ] > > > > - could the NIC have a memory architecture that > > > > allows data prefetching for the next command (so > > > > this is a non-issue from the protocol perspective)? > > > > This scheme incurs one DMA delay for every new > > > > burst of commands. > > > > > > > > 5.Another (perhaps radical at this point) option is to do > > > > away with immediate unsolicited data, to stick only with > > > > separate unsolicited data. I would personally be okay > > > > with the choice, particularly if this feature (that > > > > helps software implementations) starts making hardware > > > > design complicated/expensive. > > > > > > > > So, to summarize - > > > > > > > > option immediate allow > > > > data in spec? out-of-order? > > > > > > > > (A) (5) above no no > > > > (B) No real reason to do this. no yes > > > > (C) (4) above yes no > > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > > > >From the arguments I heard so far, I am leaning towards > > > > option A, and option C in that order. > > > > > > > > Comments? > > > > -- > > > > Mallikarjun > > > > > > > > Mallikarjun Chadalapaka > > > > Networked Storage Architecture > > > > Network Storage Solutions Organization > > > > MS 5668 Hewlett-Packard, Roseville. > > > > cbm@rose.hp.com > > > > > > > > Rod Harrison wrote: > > > > > > > > > > Julian, > > > > > > > > > > I don't understand what you are proposing here, what do > you > > mean by > > > > > "multiplexed" DMA? > > > > > > > > > > The problem is that the DMAs take some time, the more > there > > are > > > > > queued the longer the last DMAs queued take to complete. Some > > commands > > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > > immediate data, some commands do not, i.e. Reads and writes with > no > > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > > soon a possible, which for a read after a write can be before the > > > > > write's DMA has completed. Maintaining an ordered queue for > commands > > > > > to be sent on the HBA is expensive and redundant since the target > > > > > already knows how to queue commands before committing them to its > > SCSI > > > > > layer. > > > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > > change the > > > > > order of commands from the OS, but the DMAs those commands need > are > > > > > unlikely to complete in the same order, and as I mentioned some > > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > > order it has to maintain an ordered queue of commands waiting to > be > > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > > immediate data almost impossible to support. > > > > > > > > > > I don't see the problem with allowing out of order > commands > > given > > > > > that the target already has to deal with very similar problems. I > > > > > think we are getting in to the area of implementation choices > here, > > > > > which is inappropriate for a specification. > > > > > > > > > > - Rod > > > > > -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Fri Nov 9 14:55:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA25669 for ; Fri, 9 Nov 2001 14:55:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9JJBA06908 for ips-outgoing; Fri, 9 Nov 2001 14:19:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from goliath.xo.com (goliath.xo.com [207.155.252.78]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9JJ9l06902 for ; Fri, 9 Nov 2001 14:19:09 -0500 (EST) Received: from rooster ([66.89.96.162]) by goliath.xo.com id OAA06013; Fri, 9 Nov 2001 14:19:04 -0500 (EST) [ConcentricHost SMTP Relay 1.14] From: "Dave Francheski" To: "Julian Satran" , Subject: RE: iSCSI initiator availability for Windows ? Date: Fri, 9 Nov 2001 11:09:35 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi Julian, Thankyou for your effort in locating a Windows initiator. I do know that IBM's currently released Windows version of the initiator is coded to the original Version 0 (July 2000). I don't think they've released a newer version as of yet. Best regards, -Dave -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Friday, November 09, 2001 5:57 AM To: ips@ece.cmu.edu Subject: Re: iSCSI initiator availability for Windows ? Dave, There are many windows initiators available. I know for certain that IBM has one for W2K and there are several others. I will try to find out which version is available. I know that the version tested at the UNH Plugfest conforms to 08 but I do not know if it was made available to the public. Julo "Dave Francheski" Sent by: owner-ips@ece.cmu.edu 08-11-01 01:39 Please respond to "Dave Francheski" To: cc: Subject: iSCSI initiator availability for Windows ? Hi all, This may not be the proper forum for this question, but I have exhausted almost all other options at this point. Does anybody know of a Windows based (e.g., Windows 2000) version of an iSCSI initiator implemented to Revision 6 (or greater) of the iSCSI standard? I have Cisco's version of the iSCSI Initiator for Windows but it is implemented to the first original draft of iSCSI, and doesn't interoperate with other more recent iSCSI target implementations. Thankyou for any and all support you can provide. regards, -Dave David Francheski Seven Systems Technologies 575 Menlo Drive Suite 2 Rocklin CA 916-577-1281 davef@seven-systems.com From owner-ips@ece.cmu.edu Fri Nov 9 15:07:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA25972 for ; Fri, 9 Nov 2001 15:07:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9J6gO05874 for ips-outgoing; Fri, 9 Nov 2001 14:06:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9J6cl05867 for ; Fri, 9 Nov 2001 14:06:38 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA216932; Fri, 9 Nov 2001 14:04:00 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fA9J6ZG172158; Fri, 9 Nov 2001 12:06:36 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI over TLS To: "Bill Strahm" Cc: "Ips@Ece. Cmu. Edu" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Fri, 9 Nov 2001 11:05:41 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/09/2001 12:06:35 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, What you are missing here is that many (most) folks that are combining the iSCSI functions and TCP/IP Offload Engines (TOEs) together on the same chip (or HBA), get to see the packets as they arrive, if they can see the iSCSI header (which is usually but not always first) it can begin the placement of the PDU segments directly in the final memory locations and not into anonymous buffers, which will require another move within the Hosting System. (The reason they can do that is they are able to have their own internal interfaces into the TCP/IP stacks that they have on their Chips or HBAs). It is this key reason why the various vendors think they can have performance close to or equivalent to FC adapters. They will not give that up for TLS. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Bill Strahm" @ece.cmu.edu on 11/09/2001 10:40:29 AM Sent by: owner-ips@ece.cmu.edu To: "Ips@Ece. Cmu. Edu" cc: Subject: RE: iSCSI over TLS I am confused... Why can't I put the data into the memory location specified by the TCP sequence number ??? I am assuming that is what you are doing in the case of IPsec dropping a TCP frame in the middle of the stream. TCP stacks that I am aware of WILL NOT pass out of order stream data to applications until the intermediate frames are dealt with anyway, so what is the difference between holding encrypted frames and unencrypted frames until the stream is in order ? I believe that there are also TLS options to do per frame crypto, however at that point you have to start shipping over IVs with each frame, removing most wire advantages of TLS over IPsec... Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Friday, November 09, 2001 10:06 AM To: ips@ece.cmu.edu Subject: RE: iSCSI over TLS SG, The issue is that if you are not able to decrypt you have no iSCSI packet header (or RDMA headers) and can't place data in memory (i.e. you need anonymous buffers and have to copy). With IPSec you are far better off. Julo "Sukanta Ganguly" 09-11-01 18:08 Please respond to sganguly To: Julian Satran/Haifa/IBM@IBMIL cc: "IPS" Subject: RE: iSCSI over TLS Julian, I agree to the philosophy of framing and steering. With TLS support you can still place incomplete TLS records in host memory, without decrpyting it. Only buffering support needs to be beefed up on the host side. And frankly speaking that it already present with out of order packet deliveries etc. I am not proposing that TLS start acting on the packet immediately as the first peice of the TLS record arrives. The same behavior is observed with iSCSI packets which span TCP packets. The host has to wait for the entire iSCSI packet to be present in the host memory before the iSCSI layer can do anything with it. Do you see my point of argument! ;-) SG *********** REPLY SEPARATOR *********** On 11/9/2001 at 5:46 PM Julian Satran wrote: >The whole reason for doing framing and steering is to be able to start >placing data in host memory without waiting for all the data to arrive. >With TLS data can't be decrypted if pieces are missing. > >Julo > > > > >"Sukanta Ganguly" >09-11-01 17:42 >Please respond to sganguly > > > To: Julian Satran/Haifa/IBM@IBMIL, "IPS" > cc: > Subject: RE: iSCSI over TLS > > > >Julian, > It is correct that TLS records span TCP packets, but how does that >become anymore of a problem. For packets to be resend via the TCP >mechanisms, the sender TLS layers prepares the TLS record and then hands >it over to TCP, TCP may break that TLS layer into e.g. say 5 packets and >sends them to the receiver. If the receiver does not retrieve packet >number 3, it will be resend by the sender. > I did not see the problem that TLS brings into the picture. Also, what >tweaking of the stack are you referring to in this scenario. This is just >general handlinf of packets that are done anyway. iSCSI will only make >sense of the packet after TLS decrypts the packets. Did I miss something >here ??? > >SG > >*********** REPLY SEPARATOR *********** > >On 11/9/2001 at 4:14 PM Julian Satran wrote: > >>Bill, >> >>The one "tiny" item you forgot to mention is that TLS records span TCP >and >>iSCSI PDU boundaries. TLS records can't be decrypted in face of TCP >packet >>loss and markers/alignment can't be recovered (to be more precise require > >>a lot more tweaking of the stacks). >> >>Julo >> >> >> >> >>"Bill Strahm" >>08-11-01 23:55 >>Please respond to "Bill Strahm" >> >> >> To: Julian Satran/Haifa/IBM@IBMIL >> cc: >> Subject: RE: iSCSI over TLS >> >> >> >>Julian, >> >>I do not understand how TLS interferes with delivery of iSCSI packets any >>more than IPsec. In either case your TOE MUST decrypt the packet and >deal >>with the results. I do not see how this changes the problem if the >packet >>is decrypted before going to the TOE (again the hardware to do this MUST >>be >>on the NIC device) or after going through the TOE processing... >>Quick summary of what I think needs to happen >>IPsec >>1) receive L2 packet >>2) determine it is IP >>3) Apply packet policy based on L3 header >>4) Decrypt packet - verify it is covered by the SA >>5) Pass to L4 (TCP) for processing >>6) Verify Framing/etc. >>7) Done >>TLS >>1) Recieve L2 Packet >>2) Pass to L3 >>3) Pass to L4 (TCP) for processing >>4) Decrypt packet >>5) Verify Framing/etc >>6) Done >> >>It turns out the policies for TLS are much simpler than for IPsec, the >>application itself gets to determine if security should be turned on or >>not >>(rather than another application pushing policies into an SPD) and I >don't >>see a difference in the security offload requirements. In many cases TLS >>will go through firewalls/NAT/NATP much better than IPsec, allowing for a >>wider deployment model. >> >> >>Bill Strahm >>+========+=========+=========+=========+=========+=========+=========+ >>Bill Strahm Software Development is a race between Programmers >>Member of the trying to build bigger and better idiot proof software >>Technical Staff and the Universe trying to produce bigger and better >>bill@sanera.net idiots. >>(503) 601-0263 So far the Universe is winning --- Rich Cook >> >>-----Original Message----- >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >>Julian Satran >>Sent: Tuesday, November 06, 2001 10:17 PM >>To: ips@ece.cmu.edu >>Subject: Re: iSCSI over TLS >> >> >>Peter, >> >>A group of us seriously considered TLS. The main reason for dropping it >>was that it would interfere with any mechanism we could think of doing >>framing and steering and we thought that framing and steering are >>essential at 10Gbps and over. >> >>Julo >> >> >> >> >>"Peter Mellquist" >>Sent by: owner-ips@ece.cmu.edu >>07-11-01 02:15 >>Please respond to "Peter Mellquist" >> >> >> To: >> cc: >> Subject: iSCSI over TLS >> >> >> >>I am aware that the ips group is leaning toward IPSEC as for the security >>solution but I am interested if anyone is also considering using >Transport >>Layer Security (TLS)? >> >>I am concerned that the requirement for IPSEC might make TOEs more >>complex >>than they need to be. Can TLS be optionally used as well as defined by >the >>specification? This could allow TOE vendors to only be concerned with >>providing normal IPv4 / ipv6 and leave the security to a higher layer. A >>TLS >>stack sitting above the TOE could then handle security very well. Also, I >>anticipate that the first generation of TOEs will not support IPSEC. With >>a >>iSCSI/TLS we could enable security solutions with the first generation of >>TOEs and get speed and security. >> >>Are any TOE vendors planning to support IPSEC? >> >>Can TLS or IPSEC be supported? >> >>-peter >> >> >> >>Peter Mellquist >>Seven Systems Technologies >>575 Menlo Drive Suite 2 >>Rocklin CA >>916-577-1275 >>peterm@seven-systems.com From owner-ips@ece.cmu.edu Fri Nov 9 16:01:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA27383 for ; Fri, 9 Nov 2001 16:01:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9Kaof13340 for ips-outgoing; Fri, 9 Nov 2001 15:36:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9Kail13321 for ; Fri, 9 Nov 2001 15:36:48 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id PAA00881; Fri, 9 Nov 2001 15:36:40 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "IPS" Subject: Clarification (again) for Task Management Commands Date: Fri, 9 Nov 2001 12:33:52 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit On page 67 of the 8-92.txt draft (section 3.5.1), it says "For all the tasks covered by the task management response (i.e., with CmdSN not higher than the task management command CmdSN), additional responses MUST NOT be delivered to the SCSI layer after the task management response." If there is a multiple connection session, a status for a command impacted by the task management command (say ABORT TASK SET) could be stuck in the pipe on one connection, while the ABORT TASK SET completes on another connection. How does the initiator iSCSI enforce the rule above? Seems to be the equivalent of sending the impacted commands on other connections in a zombie state, and not having a very good idea of how to get out. Similarly Section 9.4 provides additional rules, but seems to leave a hole open with regards to status already in flight on other connections. Any clarifications would be appreciated. Somesh From owner-ips@ece.cmu.edu Fri Nov 9 16:03:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA27422 for ; Fri, 9 Nov 2001 16:03:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9KGpu11702 for ips-outgoing; Fri, 9 Nov 2001 15:16:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9KGol11698 for ; Fri, 9 Nov 2001 15:16:50 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id E28C9E36; Fri, 9 Nov 2001 15:16:48 -0500 (EST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id MAA15260; Fri, 9 Nov 2001 12:18:15 -0800 (PST) Message-Id: <200111092018.MAA15260@core.rose.hp.com> Date: Fri, 09 Nov 2001 12:29:10 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: John Hufferd Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Out of order commands References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John Hufferd wrote: .... > With in order command arrival on a connection (as a normal event) seems to > provide the quick determination of an error in a Command PDU, and tell the > recovery code quickly to get the missing command resent. What do you think > will be the effect of not knowing if the command is delayed in the > initiator or whether it was dropped because of a Header Digest Error? > On single-connection sessions, yes, there's additional non-determinism on the target. But keep in mind though that the target can at best send a NOP to prompt the initiator to retransmit, even if it knows for sure - it can not definitively communicate its knowledge about the missing to the initiator. On multi-connection sessions, nothing really changes. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > "Mallikarjun C." on 11/08/2001 05:28:21 PM > > To: John Hufferd/San Jose/IBM@IBMUS > cc: > Subject: Re: iSCSI: Out of order commands > > John, > > Sorry, this note got a little longer than I would've liked, but.... > > I believe there are cases where OOO CmdSN handling is a > legitimate requirement on targets due to exception events - > a) retransmitting a CmdSN on a command acknowledgement > timeout (within-connection recovery class). This manifests as > an OOO CmdSN on the connection to a target if it didn't see > the original copy due to a digest error. > b) retransmitting the last few "lost" commands due to a connection > failure on a new connection. If this new connection had already > carried a CmdSN greater than these retransmitted commands > (prior to connection failure), this again manifests as OOO CmdSN > on the new connection to the target. > > OTOH, I believe sending OOO CmdSNs on a connection as a > regular practice is counterproductive, since the target must continuously > re-order the initiator "optimization" leading to a zero-sum game. I > would argue that the need to dispatch CmdSNs OOO due to immediate > data DMA (brought up by Rod) can be addressed by simple NIC > changes to prefetch data for the next command (or more simply use > unsolicited separate data PDUs, if negotiated). [ You got to deal > with the case of all commands being writes anyway! ] > > If we allow OOO CmdSNs on a connection (I'd advocate discouraging > it as a regular practice), I don't believe any of the stuff in error > recovery > breaks (nor does it affect the current reliance on ExpCmdSN). Julian > perhaps can comment. > - All the in-order assumptions are for DataSNs/R2TSNs/StatSNs, not > for CmdSNs. > - Any multi-connection session by definition must deal with OOO CmdSNs. > - I belive that the current abort task scheme for immediate commands > detailed in section 9.3 caters to OOO CmdSNs on a connection > as well (we must be dealing with an immediate Abort arriving before > the command today, since the command could have been hit with > a digest error). > > To summarize, here is what I suggested to Julian in a private email - > > a)I suggest using a SHOULD for in-order dispatch of > commands on a connection - for an initiator. > > b)I suggest using a SHALL handle out-of-order commands > on a connection - for the target (as Barry pointed out). > > Hope that was useful. > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > Hewlett-Packard MS 5668 > Roseville CA 95747 > > ----- Original Message ----- > From: "John Hufferd" > To: > Cc: > Sent: Thursday, November 08, 2001 1:40 PM > Subject: Re: iSCSI: Out of order commands > > > > > Mallikarjun, > > Could you comment on the concept of OOO on the ErrorRecoveryLevel>0. I > had > > thought that "in order delivery" was part of the detection of missing > PDUs > > and needed for timely Recovery. I was wondering if this changes the way > we > > would use the ExpCmdSN, etc. > > > > I think your opinions on this part of the OOO discussion would be > valuable. > > For example, how would you contrast the differences in detecting a > problem > > and recovering from that problem etc., today vrs the OOO approach (if > any). > > > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSG San Jose Ca > > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > > Home Office (408) 997-6136, Cell: (408) 499-9702 > > Internet address: hufferd@us.ibm.com > > > > > > "Mallikarjun C." @ece.cmu.edu on 11/07/2001 09:41:05 AM > > > > Please respond to cbm@rose.hp.com > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > To: Santosh Rao , ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: Out of order commands > > > > > > > > Santosh, > > > > I have only one comment on your responses. > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > I made two convenient assumptions if you noticed, :-), one of which > > is that target forces session recovery on *any* error that it sees > > (ErrorRecoveryLevel=0) - including a dropped command due to a digest > > error. With that assumption, a target can afford not to implement > > a scoreboard. > > > > As I said in a private note, I guess what primarily bothers me about > > OOO commands on a connection is that it requires the receiver to > > undo this "optimization" on its end - most notably on a single > > connection. TCP experts may comment on how/if they dealt with a > > similar issue. > > > > OTOH, you had some valid comments on exceptions to ordering during > > connection recovery. Perhaps we can move on by making Julian's > > proposed stipulation a SHOULD.... > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > Santosh Rao wrote: > > > > > > Mallikarjun, > > > > > > Some comments below. > > > > > > Regards, > > > Santosh > > > > > > "Mallikarjun C." wrote: > > > > > > > > Rod and Julian, > > > > > > > > This has been an interesting thread of discussion. Some > > > > comments - > > > > > > > > 1.My first reaction was - allowing out-of-order command > > > > transmission on the same connection deprives targets of > > > > an implementation choice. Targets which support only > > > > single-connection sessions and only support session > > > > recovery (reasonable assumptions in my mind) can no > > > > longer afford *not to* implement a command scoreboard. > > > > > > Even a single connection target *MUST* implement a scoreboard. The > > > reason being that it can see out-of-order arrival of commands due to > > > commands being dropped on digest errors. In such a case, it must block > > > further command processing until holes are filled. > > > > > > Thus, there is no getting away from implementing a sequencer at the > > > target. Given this, I think it is unreasonable to restrict initiator > > > implementation flexibility by imposing a strict ordering requirement > > > within the connection. > > > > > > > 2.Any end-node efficiency that is sought to be achieved > > > > by transmitting CmdSNs out-of-order from the initiator > > > > would be lost on the other end-node, since the target > > > > now must wait for re-ordering the commands. > > > > > > It has to handle this situation anyway to deal with holes caused by > > > digest errors. This scenario occurs even with initiators that issue > > > commands in order. > > > > > > > > > > > 3.The flipside is that out-of-order transmission saves > > > > link badwidth (albeit at the expense of end-node efficiency), > > > > compared to idling the link waiting for outbound DMA. > > > > We have to determine if this is a reasonable trade-off. > > > > > > > > 4.I can see Rod's point that prefetching all immediate > > > > data can be a burden on the NIC resources. But, two > > > > questions - > > > > - could the NIC not use unsolicited separate data > > > > PDUs in these cases? [ I realize that InitialR2T > > > > has to be "no" to let it happen... ] > > > > - could the NIC have a memory architecture that > > > > allows data prefetching for the next command (so > > > > this is a non-issue from the protocol perspective)? > > > > This scheme incurs one DMA delay for every new > > > > burst of commands. > > > > > > > > 5.Another (perhaps radical at this point) option is to do > > > > away with immediate unsolicited data, to stick only with > > > > separate unsolicited data. I would personally be okay > > > > with the choice, particularly if this feature (that > > > > helps software implementations) starts making hardware > > > > design complicated/expensive. > > > > > > > > So, to summarize - > > > > > > > > option immediate allow > > > > data in spec? out-of-order? > > > > > > > > (A) (5) above no no > > > > (B) No real reason to do this. no yes > > > > (C) (4) above yes no > > > > (D) pros & cons (1), (2) & (3) yes yes > > > > > > > > >From the arguments I heard so far, I am leaning towards > > > > option A, and option C in that order. > > > > > > > > Comments? > > > > -- > > > > Mallikarjun > > > > > > > > Mallikarjun Chadalapaka > > > > Networked Storage Architecture > > > > Network Storage Solutions Organization > > > > MS 5668 Hewlett-Packard, Roseville. > > > > cbm@rose.hp.com > > > > > > > > Rod Harrison wrote: > > > > > > > > > > Julian, > > > > > > > > > > I don't understand what you are proposing here, what do you > > mean by > > > > > "multiplexed" DMA? > > > > > > > > > > The problem is that the DMAs take some time, the more there > > are > > > > > queued the longer the last DMAs queued take to complete. Some > > commands > > > > > require DMAs to complete before they can be sent, i.e. Writes with > > > > > immediate data, some commands do not, i.e. Reads and writes with no > > > > > immediate data. The iSCSI HBA wants to be able to send commands as > > > > > soon a possible, which for a read after a write can be before the > > > > > write's DMA has completed. Maintaining an ordered queue for > commands > > > > > to be sent on the HBA is expensive and redundant since the target > > > > > already knows how to queue commands before committing them to its > > SCSI > > > > > layer. > > > > > > > > > > The iSCSI HBA and its host driver are not at liberty to > > change the > > > > > order of commands from the OS, but the DMAs those commands need are > > > > > unlikely to complete in the same order, and as I mentioned some > > > > > commands need no DMA. If the HBA can't send commands out of CmdSN > > > > > order it has to maintain an ordered queue of commands waiting to be > > > > > sent, and potentially buffer a lot of data. For an HBA this makes > > > > > immediate data almost impossible to support. > > > > > > > > > > I don't see the problem with allowing out of order commands > > given > > > > > that the target already has to deal with very similar problems. I > > > > > think we are getting in to the area of implementation choices here, > > > > > which is inappropriate for a specification. > > > > > > > > > > - Rod > > > > > From owner-ips@ece.cmu.edu Fri Nov 9 16:04:48 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA27466 for ; Fri, 9 Nov 2001 16:04:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9KNst12282 for ips-outgoing; Fri, 9 Nov 2001 15:23:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hermes.fm.intel.com ([192.55.52.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9KNpl12277 for ; Fri, 9 Nov 2001 15:23:51 -0500 (EST) Received: from petasus.fm.intel.com (petasus.fm.intel.com [10.1.192.37]) by hermes.fm.intel.com (8.11.6/8.11.6/d: outer.mc,v 1.18 2001/11/06 18:55:09 root Exp $) with ESMTP id fA9KOMv02758 for ; Fri, 9 Nov 2001 20:24:22 GMT Received: from fmsmsxvs041.fm.intel.com (fmsmsxv041-1.fm.intel.com [132.233.48.109]) by petasus.fm.intel.com (8.11.6/8.11.6/d: inner.mc,v 1.10 2001/11/06 00:25:50 root Exp $) with SMTP id fA9KOL928226 for ; Fri, 9 Nov 2001 20:24:21 GMT Received: from fmsmsx29.FM.INTEL.COM ([132.233.42.29]) by fmsmsxvs041.fm.intel.com (NAVGW 2.5.1.6) with SMTP id M2001110912250812361 ; Fri, 09 Nov 2001 12:25:08 -0800 Received: by fmsmsx29.fm.intel.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 12:23:58 -0800 Message-ID: <3677E033A5F3D211AC4000A0C96B53EB05C2ED31@FMSMSX94> From: "Herbert, Howard C" To: "'Ofer Biran'" , ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Fri, 9 Nov 2001 12:24:43 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Does this mean that transport mode will be a "MAY" or a "SHOULD"? Howard -----Original Message----- From: Ofer Biran [mailto:BIRAN@il.ibm.com] Sent: Friday, November 09, 2001 10:54 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision It seems that most people prefer tunnel over transport mode and there is no real opposition for choosing tunnel mode as the MUST. In view of that we intend to add it in version 09 in the following iSCSI statements: In Section 10.3.1 Data Integrity and Authentication : "An iSCSI compliant initiator or target MUST provide data integrity and authentication by implementing IPSec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." And in Section 10.3.2 Confidentiality : "An iSCSI compliant initiator or target MUST provide confidentiality by implementing IPSec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." Any objection ? Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 01/11/2001 20:03:29 Please respond to To: Ofer Biran/Haifa/IBM@IBMIL, cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Thursday, November 01, 2001 4:31 AM To: ips@ece.cmu.edu Subject: iSCSI: IPsec tunnel / transport mode decision I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Fri Nov 9 17:03:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28335 for ; Fri, 9 Nov 2001 17:03:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9Lb0U18274 for ips-outgoing; Fri, 9 Nov 2001 16:37:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9Laxl18270 for ; Fri, 9 Nov 2001 16:36:59 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id 09E43C56 for ; Fri, 9 Nov 2001 16:36:58 -0500 (EST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id NAA28844 for ; Fri, 9 Nov 2001 13:38:24 -0800 (PST) Message-Id: <200111092138.NAA28844@core.rose.hp.com> Date: Fri, 09 Nov 2001 13:49:19 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: iSCSI: update on OOO CmdSNs/connection Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit To those interested in this discussion: Julian and I had a phone conversation on the topic of OOO CmdSNs on a connection. An update follows. Julian agrees that there are valid error recovery scenarios where CmdSNs will legitimately end up OOO on a given connection. OTOH, I agree with two of Julian's scenarios that he pointed out right away - the "cleaning command" (command required to be sent after a retry copy to ensure flushing within 2^31 -1), and an immediate Logout posted with unacknowledged commands. Neither of this can be shipped OOO - since the former undoes the flushing intent, and the latter breaks the rule that nothing more follows a Logout on the connection (and troublesome in other ways, see below). In general, I share the concern with Julian that we have not closely scrutinized all possibilities. With that said, something along the following lines seemed reasonable - A)Initiator MUST send commands in increasing order of CmdSN on a connection if both the following are true - - operational ErrorRecoveryLevel is 0, - MaxConnections is negotiated to 1. B)In all the other cases, initiator SHOULD send commands in increasing order of CmdSN on a connection. It is strongly encouraged that commands with out-of-order CmdSNs be sent on a connection only if they are retransmitted commands due to digest error recovery and connection recovery. I also suggest the following upon further reflection- C)Add wording in section 2.2.2.1 to mandate that the cleaning command MUST be sent in-order after the retried command. D)Warn clearly that sending an immediate Logout command in the presence of other unacknowledged commands MAY create inadvertent discarding of certain commands (even if it is a recovery Logout), and MAY cause protocol errors leading to ungraceful shutdown of the connection. Hopefully A will bring the determinism that Somesh was looking for certain design points. B describes the more general n-connection session case. C & D are fixes for two identfied areas (so far) which will break. Comments? -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Fri Nov 9 17:05:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28381 for ; Fri, 9 Nov 2001 17:05:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9LikB18878 for ips-outgoing; Fri, 9 Nov 2001 16:44:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA9Liil18874 for ; Fri, 9 Nov 2001 16:44:44 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fA9LiSQ10980; Fri, 9 Nov 2001 16:44:28 -0500 Received: from pkoning.dev.equallogic.com.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with SMTP id fA9LiRB10556; Fri, 9 Nov 2001 16:44:27 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15340.20037.424321.501414@pkoning.dev.equallogic.com> Date: Fri, 9 Nov 2001 16:44:37 -0500 (EST) From: Paul Koning To: BIRAN@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: IKE normative guidelines References: X-Mailer: VM 6.75 under 21.1 (patch 11) "Carlsbad Caverns" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit >>>>> "Ofer" == Ofer Biran writes: Ofer> In the framework of the effort being done now by the security Ofer> team to sync the normative statements in the security draft Ofer> with the protocol drafts, I suggest to adopt the following IKE Ofer> normative guidelines that already appear in the security draft Ofer> for iSCSI: ... Does any of this differ from what's in RFC 2408/2409? If so, why? If not, why not just refer to that standard and say nothing further? If we're going to be different, I'd suggest dropping aggressive mode since main mode is at least as good. paul From owner-ips@ece.cmu.edu Fri Nov 9 17:20:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28729 for ; Fri, 9 Nov 2001 17:20:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9LGf016643 for ips-outgoing; Fri, 9 Nov 2001 16:16:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9LGdl16639 for ; Fri, 9 Nov 2001 16:16:39 -0500 (EST) Received: from hq-ex-c2.brocade.com (hq-ex-c2 [192.168.126.35]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id NAA20806; Fri, 9 Nov 2001 13:16:18 -0800 (PST) Received: by hq-ex-c2.brocade.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 13:16:16 -0800 Message-ID: <176B85B0D4E0D411BA5200508B692E0A2E6102@hq-ex-1.brocade.com> From: Kha Sin Teow To: "'Black_David@emc.com'" , kzm@cisco.com, ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Date: Fri, 9 Nov 2001 13:16:14 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, I just got back from the T10 meeting and saw a flurry of discussion on this topic. I think Dave's comments are mostly in line with mine. Some more in-line comments below ... | -----Original Message----- | From: Black_David@emc.com [mailto:Black_David@emc.com] | Sent: Thursday, November 08, 2001 12:51 PM | To: kzm@cisco.com; ips@ece.cmu.edu | Subject: RE: FC Management MIB - proposed changes | | | Folks, | | For simplicity, I would propose keeping the initial revision focused | on structural and format changes only - i.e., make no functional | changes that aren't required to bring the MIB into line with the | overall architecture of MIBs, use of SMIv2, and the like. Absolutely. It's not a good idea to start opening up at this stage for more functionalities. | Comments on | this, keyed to Keith's issue numbers: | | 1. The trap table doesn't match up with RFC 2573 - I presume | correcting this is part of the conversion to SMIv2. | | 3. Let's keep the replacement of RFC 2837 as (a) a proposal | and (b) Keith's recommendation to the WG as MIB Advisor, | but not formally adopt that approach until we have a version | of the MIB draft that would replace RFC 2837 available for | review by the WG. agree. let me add Keith's point 3 here (preceded with ">") with my comments. > - the reason that we now have the issue of how it relates to the > FC-Mgmt MIB is because it was written as a Fabric Element MIB; > this issue would not have arisen if it had been written as a > Fibre Channel interface MIB. I'm not sure if I follow the argument above as a good reason to replace RFC2837. It started out as an effort to define MIB for FC switches (not FC hub, HBA or other FC interconnect elements as in to be re-worked FC-Mgmt MIB). I'm open to suggestion that if RFC2837 needs revision to line up with other MIBs due to overlaps (such as Entity MIB). If that means that RFC2837 will be obsoleted some time in future by the new FC-Mgmt MIB, I'm ok with it. However, we need to be careful that we don't try to fold everything into the new MIB in one go. If we need to fix the sink, please don't think about doing a new roof.-) > - it should have extended the ifTable, rather than overlapping with it, Unfortunately, at the time, the adivsor didn't raise the issue with the working group; I agree it's an area to improve. > - it should not have defined the fcFeModuleTable, which overlaps with > the Entity MIB. At the time, the Entity MIB was work in progress. I didn't quite see how it fits in then. Given the state of Entity MIB, I agree that this is another area to improve. > - and it should have specified (at least) its octet counters as Counter64. as Predrag pointed out, FC standards doesn't specify 64bit counters. As far as I'm aware, most current FC implementations only support 32-bit counters. Therefore we should be cautious about specifying 64bit counter (which would require an agent to maintain a set of software counters). ... | 2, 4, and 6 look fine to me, as does the new draft name. | | Comments? | | Thanks, | --David | | --------------------------------------------------- | David L. Black, Senior Technologist | EMC Corporation, 42 South St., Hopkinton, MA 01748 | +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 | black_david@emc.com Mobile: +1 (978) 394-7754 | --------------------------------------------------- | | > -----Original Message----- | > From: Keith McCloghrie [mailto:kzm@cisco.com] | > Sent: Thursday, November 08, 2001 1:43 PM | > To: ips@ece.cmu.edu | > Cc: kzm@cisco.com | > Subject: FC Management MIB - proposed changes | > ... Kha Sin/ From owner-ips@ece.cmu.edu Fri Nov 9 17:21:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28793 for ; Fri, 9 Nov 2001 17:21:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9Lgrk18732 for ips-outgoing; Fri, 9 Nov 2001 16:42:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA9Lgql18728 for ; Fri, 9 Nov 2001 16:42:52 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fA9LgjQ10968; Fri, 9 Nov 2001 16:42:45 -0500 Received: from pkoning.dev.equallogic.com.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with SMTP id fA9Lgj710206; Fri, 9 Nov 2001 16:42:45 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15340.19934.570689.7766@pkoning.dev.equallogic.com> Date: Fri, 9 Nov 2001 16:42:54 -0500 (EST) From: Paul Koning To: BIRAN@il.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision References: X-Mailer: VM 6.75 under 21.1 (patch 11) "Carlsbad Caverns" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I'm not sure if this is the sort of answer your question was looking for, but I don't want to let silence be taken for agreement, so... I'm opposed to the presence of "MUST" in sections 10.3.1 and 10.3.2 for reasons stated in mail to this list a week or two ago. paul From owner-ips@ece.cmu.edu Fri Nov 9 17:57:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29418 for ; Fri, 9 Nov 2001 17:57:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9LvXM19993 for ips-outgoing; Fri, 9 Nov 2001 16:57:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9LvUl19986 for ; Fri, 9 Nov 2001 16:57:31 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id QAA13209; Fri, 9 Nov 2001 16:57:25 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "IPS" Subject: iSCSI: Clarification (again) for Task Management Commands Date: Fri, 9 Nov 2001 13:54:33 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Resend to add iSCSI tag. Sorry for missing it. On page 67 of the 8-92.txt draft (section 3.5.1), it says "For all the tasks covered by the task management response (i.e., with CmdSN not higher than the task management command CmdSN), additional responses MUST NOT be delivered to the SCSI layer after the task management response." If there is a multiple connection session, a status for a command impacted by the task management command (say ABORT TASK SET) could be stuck in the pipe on one connection, while the ABORT TASK SET completes on another connection. How does the initiator iSCSI enforce the rule above? Seems to be the equivalent of sending the impacted commands on other connections in a zombie state, and not having a very good idea of how to get out. Similarly Section 9.4 provides additional rules, but seems to leave a hole open with regards to status already in flight on other connections. Any clarifications would be appreciated. Somesh From owner-ips@ece.cmu.edu Fri Nov 9 18:18:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00009 for ; Fri, 9 Nov 2001 18:18:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9Mw6p24630 for ips-outgoing; Fri, 9 Nov 2001 17:58:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9Mw5l24626 for ; Fri, 9 Nov 2001 17:58:05 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 17:57:59 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937149@CORPMX14> From: Black_David@emc.com To: ENDL_TX@computer.org, ips@ece.cmu.edu Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Fri, 9 Nov 2001 17:48:22 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Those who were at the Irvine Interim meeting will remember that > the problem with FCIP and NAPTS is a reliance on IP address in > the determination of which incoming TCP connections belong in a > given FCIP Link. This proposal solves that problem by requiring > that FC Entity World Wide Name be transmitted in the first bytes > sent by the FCIP Entity that initiates a TCP Connect request. > This allows the FCIP Entity that receives a TCP Connect request > to match it with any previously received TCP Connect requests > from the same source. Since the transmitted World Wide Name is > required to be unique within Fibre Channel, the FCIP Entity > that receives this information can correctly assign FCIP Link > relationships without relying on IP Addresses. From a functional standpoint, this works, but it opens up a security issue. The problem is that on the second TCP connection (and subsequent connections) that claim to be from the same FCIP Entity, the WWN that is initially sent (and whatever extension is used) is functioning as an authentication to allow that connection to join the first TCP connection, but that authentication is unsecured -- the sender announces the WWN, and the receiver does not (and has no way to) check it. There's a fairly obvious denial of service attack here involving the attacker joining a new connection to an existing one and then bit-bucketing all the frames sent over the new connection. Limiting FCIP to one TCP connection among any pair of FCIP entity identifiers would help, but is not sufficient. The attack of concern in this situation involves the attacker crashing the real entity and opening up a connection in its name, thereby locking out the real entity when the real entity restarts. This may be headed in the direction of needing in-band authentication which I know the FCIP community has been doing their best to avoid. Sorry to be the bearer of bad news, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Nov 9 18:18:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00023 for ; Fri, 9 Nov 2001 18:18:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9MdTd23206 for ips-outgoing; Fri, 9 Nov 2001 17:39:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pop.mainstreet.net (smtp.mainstreet.net [207.5.0.46]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9MaRl22938 for ; Fri, 9 Nov 2001 17:36:27 -0500 (EST) Received: from vip (saqibj.mainstreet.net [207.5.1.168]) by pop.mainstreet.net (8.11.3/8.11.3) with SMTP id fA9MWJP21884; Fri, 9 Nov 2001 14:32:19 -0800 (PST) Reply-To: From: "Saqib Jang" To: "Ofer Biran" , Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Fri, 9 Nov 2001 14:41:59 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Two pts. First, I believe that either flavor of IPsec has to be a MUST-to-implement requirement for iSCSI to be viewed as a 'secure' protocol, so I'm in favor of the proposed statements to that extent. Second, my vote is in favor of transport mode to avoid folks playing 'marketing' games with positioning their products as iSCSI compliant. Saqib -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Friday, November 09, 2001 9:54 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision It seems that most people prefer tunnel over transport mode and there is no real opposition for choosing tunnel mode as the MUST. In view of that we intend to add it in version 09 in the following iSCSI statements: In Section 10.3.1 Data Integrity and Authentication : "An iSCSI compliant initiator or target MUST provide data integrity and authentication by implementing IPSec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." And in Section 10.3.2 Confidentiality : "An iSCSI compliant initiator or target MUST provide confidentiality by implementing IPSec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." Any objection ? Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 01/11/2001 20:03:29 Please respond to To: Ofer Biran/Haifa/IBM@IBMIL, cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Thursday, November 01, 2001 4:31 AM To: ips@ece.cmu.edu Subject: iSCSI: IPsec tunnel / transport mode decision I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Fri Nov 9 18:28:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00235 for ; Fri, 9 Nov 2001 18:28:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9MgXi23518 for ips-outgoing; Fri, 9 Nov 2001 17:42:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12506.mail.yahoo.com (web12506.mail.yahoo.com [216.136.173.198]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA9J71l05899 for ; Fri, 9 Nov 2001 14:07:01 -0500 (EST) Message-ID: <20011109190700.94551.qmail@web12506.mail.yahoo.com> Received: from [66.122.195.194] by web12506.mail.yahoo.com via HTTP; Fri, 09 Nov 2001 11:07:00 PST Date: Fri, 9 Nov 2001 11:07:00 -0800 (PST) From: Sukanta ganguly Subject: RE: iSCSI: IPsec tunnel / transport mode decision To: Ofer Biran , ips@ece.cmu.edu In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk By doing this we are forcing IPSec. No flexibility of going transport over tunnel. I think we were still having a discussion of whether transport can also be supported and hence instead of forcing with IPSec can't we allow both mechanisms to a MAY. In that scenario one could opt for transport mode with tunnel and still have a good implementation running. What do other think? SG --- Ofer Biran wrote: > > It seems that most people prefer tunnel over > transport mode > and there is no real opposition for choosing tunnel > mode as > the MUST. In view of that we intend to add it in > version 09 > in the following iSCSI statements: > > In Section 10.3.1 Data Integrity and Authentication > : > > "An iSCSI compliant initiator or target MUST provide > data > integrity and authentication by implementing IPSec > [RFC2401] > with ESP in tunnel mode [RFC2406] with the > following..." > > And in Section 10.3.2 Confidentiality : > > "An iSCSI compliant initiator or target MUST provide > confidentiality by implementing IPSec [RFC2401] with > ESP in tunnel mode [RFC2406] with the following..." > > Any objection ? > > Regards, > Ofer > > > Ofer Biran > Storage and Systems Technology > IBM Research Lab in Haifa > biran@il.ibm.com 972-4-8296253 > > > "Saqib Jang" on 01/11/2001 > 20:03:29 > > Please respond to > > To: Ofer Biran/Haifa/IBM@IBMIL, > cc: > Subject: RE: iSCSI: IPsec tunnel / transport mode > decision > > > > > -----Original Message----- > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ofer Biran > Sent: Thursday, November 01, 2001 4:31 AM > To: ips@ece.cmu.edu > Subject: iSCSI: IPsec tunnel / transport mode > decision > > > I'd like to drive this open issue into group > consensus. It seems to > me that the tendency was more toward making tunnel > mode a MUST as iFCP > and FCIP did, mainly due the option of integrating > an existing IPsec > chip/box with the iSCSI implementation offering. If > we reach this decision, > we may choose even not to mention transport mode (as > MAY or some other > recommending text). > > There is an excellent analysis made by Bernard Aboba > in Section > "5.1. Transport mode versus tunnel mode" of > draft-ietf-ips-security-04 > ( > http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt > ) > that can help us with this decision (also Section > "5.2. NAT traversal" is > relevant). > > Regards, > Ofer > > Ofer Biran > Storage and Systems Technology > IBM Research Lab in Haifa > biran@il.ibm.com 972-4-8296253 > > > > __________________________________________________ Do You Yahoo!? Find a job, post your resume. http://careers.yahoo.com From owner-ips@ece.cmu.edu Fri Nov 9 18:32:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00349 for ; Fri, 9 Nov 2001 18:32:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9MgvN23559 for ips-outgoing; Fri, 9 Nov 2001 17:42:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12504.mail.yahoo.com (web12504.mail.yahoo.com [216.136.173.196]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA9JNZl07324 for ; Fri, 9 Nov 2001 14:23:35 -0500 (EST) Message-ID: <20011109192334.23935.qmail@web12504.mail.yahoo.com> Received: from [66.122.195.194] by web12504.mail.yahoo.com via HTTP; Fri, 09 Nov 2001 11:23:34 PST Date: Fri, 9 Nov 2001 11:23:34 -0800 (PST) From: Sukanta ganguly Subject: RE: iSCSI over TLS To: Bill Strahm , "Ips@Ece. Cmu. Edu" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, I think what Julian is trying to say is that since the TLS record is not completed, one would have to have seperate buffers so that first the entire TLS record is collected in the host memory and the TLS works on decrypting it. After that iSCSI can start working on the packet received. SG --- Bill Strahm wrote: > I am confused... > > Why can't I put the data into the memory location > specified by the TCP > sequence number ??? > > I am assuming that is what you are doing in the case > of IPsec dropping a TCP > frame in the middle of the stream. TCP stacks that > I am aware of WILL NOT > pass out of order stream data to applications until > the intermediate frames > are dealt with anyway, so what is the difference > between holding encrypted > frames and unencrypted frames until the stream is in > order ? > > I believe that there are also TLS options to do per > frame crypto, however at > that point you have to start shipping over IVs with > each frame, removing > most wire advantages of TLS over IPsec... > > Bill > +========+=========+=========+=========+=========+=========+=========+ > Bill Strahm Software Development is a race > between Programmers > Member of the trying to build bigger and better > idiot proof software > Technical Staff and the Universe trying to produce > bigger and better > bill@sanera.net idiots. > (503) 601-0263 So far the Universe is winning --- > Rich Cook > > > -----Original Message----- > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Friday, November 09, 2001 10:06 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI over TLS > > > SG, > > The issue is that if you are not able to decrypt you > have no iSCSI packet > header (or RDMA headers) and can't place data in > memory (i.e. you need > anonymous buffers and have to copy). > With IPSec you are far better off. > > Julo > > > > > "Sukanta Ganguly" > 09-11-01 18:08 > Please respond to sganguly > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: "IPS" > Subject: RE: iSCSI over TLS > > > > Julian, > I agree to the philosophy of framing and > steering. With TLS support you > can still place incomplete TLS records in host > memory, without decrpyting > it. Only buffering support needs to be beefed up on > the host side. And > frankly speaking that it already present with out of > order packet > deliveries etc. I am not proposing that TLS start > acting on the packet > immediately as the first peice of the TLS record > arrives. > The same behavior is observed with iSCSI packets > which span TCP > packets. The host has to wait for the entire iSCSI > packet to be present in > the host memory before the iSCSI layer can do > anything with it. > Do you see my point of argument! ;-) > > SG > > *********** REPLY SEPARATOR *********** > > On 11/9/2001 at 5:46 PM Julian Satran wrote: > > >The whole reason for doing framing and steering is > to be able to start > >placing data in host memory without waiting for all > the data to arrive. > >With TLS data can't be decrypted if pieces are > missing. > > > >Julo > > > > > > > > > >"Sukanta Ganguly" > >09-11-01 17:42 > >Please respond to sganguly > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > "IPS" > > cc: > > Subject: RE: iSCSI over TLS > > > > > > > >Julian, > > It is correct that TLS records span TCP packets, > but how does that > >become anymore of a problem. For packets to be > resend via the TCP > >mechanisms, the sender TLS layers prepares the TLS > record and then hands > >it over to TCP, TCP may break that TLS layer into > e.g. say 5 packets and > >sends them to the receiver. If the receiver does > not retrieve packet > >number 3, it will be resend by the sender. > > I did not see the problem that TLS brings into > the picture. Also, what > >tweaking of the stack are you referring to in this > scenario. This is just > > >general handlinf of packets that are done anyway. > iSCSI will only make > >sense of the packet after TLS decrypts the packets. > Did I miss something > >here ??? > > > >SG > > > >*********** REPLY SEPARATOR *********** > > > >On 11/9/2001 at 4:14 PM Julian Satran wrote: > > > >>Bill, > >> > >>The one "tiny" item you forgot to mention is that > TLS records span TCP > >and > >>iSCSI PDU boundaries. TLS records can't be > decrypted in face of TCP > >packet > >>loss and markers/alignment can't be recovered (to > be more precise > require > > > >>a lot more tweaking of the stacks). > >> > >>Julo > >> > >> > >> > >> > >>"Bill Strahm" > >>08-11-01 23:55 > >>Please respond to "Bill Strahm" > >> > >> > >> To: Julian Satran/Haifa/IBM@IBMIL > >> cc: > >> Subject: RE: iSCSI over TLS > >> > >> > >> > >>Julian, > >> > >>I do not understand how TLS interferes with > delivery of iSCSI packets > any > >>more than IPsec. In either case your TOE MUST > decrypt the packet and > >deal > >>with the results. I do not see how this changes > the problem if the > >packet > >>is decrypted before going to the TOE (again the > hardware to do this MUST > > >>be > >>on the NIC device) or after going through the TOE > processing... > >>Quick summary of what I think needs to happen > >>IPsec > >>1) receive L2 packet > >>2) determine it is IP > === message truncated === __________________________________________________ Do You Yahoo!? Find a job, post your resume. http://careers.yahoo.com From owner-ips@ece.cmu.edu Fri Nov 9 18:34:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00419 for ; Fri, 9 Nov 2001 18:34:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9MgqH23550 for ips-outgoing; Fri, 9 Nov 2001 17:42:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12501.mail.yahoo.com (web12501.mail.yahoo.com [216.136.173.193]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fA9JKml07061 for ; Fri, 9 Nov 2001 14:20:48 -0500 (EST) Message-ID: <20011109192047.75482.qmail@web12501.mail.yahoo.com> Received: from [66.122.195.194] by web12501.mail.yahoo.com via HTTP; Fri, 09 Nov 2001 11:20:47 PST Date: Fri, 9 Nov 2001 11:20:47 -0800 (PST) From: Sukanta ganguly Subject: RE: iSCSI over TLS To: Julian Satran , ips@ece.cmu.edu In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I see what you are saying. I also not just blindly opposing IPSec. An optimal security for a protocol can best be done at that layer. That would be the most optimal solution. Obviously, we do not have an security developed inherently for the iSCSI at the iSCSI layer. So we are debating whether we should transport i.e. TLS or tunnel i.e. IPSec. Given that if we go lower in the layer than the protocol itself, the knowledge of the portocol does not exist, i.e. at IPSec the semantics of iSCSI is not present at all. So to some extext integrity is questionable, for lack of a better term to explain. At a layer higher than the protocol atleast we maintain the semantics of the protocol, i.e. a TLS record would have the entire iSCSI packet. Thats seems like the best way to maintain the integrity of the protocol (Don't chew me up on the use of the work integrity as I am just trying to explain it in terms of the semantics of the iSCSI protocol, so bear with my explaination). Defining a standard with dependencies like the one we are doing with IPSec involved is not necessarily bad, but does spill the working of a protocol to multiple layers, just to get it working. This is not a very good design (Again, I am just explaining my views and not pointing fingers at anybody. If there is some short coming that I am stating, then it is me who does not yet have a better solution and hence I am the part to the problem. I am not trying to sit on the sideline and point out issues. This being a public forum and people are defining standards that others will be using in the future, we have an obligation to think through the whole picture throughly. I had a pretty nasty response from one of the iSCSI veteran, when I was trying to point out some problems with TCP and iSCSI and hence I am putting this disclaimer.) Hope, my comments are taken constructively :-) SG --- Julian Satran wrote: > SG, > > The issue is that if you are not able to decrypt you > have no iSCSI packet > header (or RDMA headers) and can't place data in > memory (i.e. you need > anonymous buffers and have to copy). > With IPSec you are far better off. > > Julo > > > > > "Sukanta Ganguly" > 09-11-01 18:08 > Please respond to sganguly > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: "IPS" > Subject: RE: iSCSI over TLS > > > > Julian, > I agree to the philosophy of framing and > steering. With TLS support you > can still place incomplete TLS records in host > memory, without decrpyting > it. Only buffering support needs to be beefed up on > the host side. And > frankly speaking that it already present with out of > order packet > deliveries etc. I am not proposing that TLS start > acting on the packet > immediately as the first peice of the TLS record > arrives. > The same behavior is observed with iSCSI packets > which span TCP > packets. The host has to wait for the entire iSCSI > packet to be present in > the host memory before the iSCSI layer can do > anything with it. > Do you see my point of argument! ;-) > > SG > > *********** REPLY SEPARATOR *********** > > On 11/9/2001 at 5:46 PM Julian Satran wrote: > > >The whole reason for doing framing and steering is > to be able to start > >placing data in host memory without waiting for all > the data to arrive. > >With TLS data can't be decrypted if pieces are > missing. > > > >Julo > > > > > > > > > >"Sukanta Ganguly" > >09-11-01 17:42 > >Please respond to sganguly > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, > "IPS" > > cc: > > Subject: RE: iSCSI over TLS > > > > > > > >Julian, > > It is correct that TLS records span TCP packets, > but how does that > >become anymore of a problem. For packets to be > resend via the TCP > >mechanisms, the sender TLS layers prepares the TLS > record and then hands > >it over to TCP, TCP may break that TLS layer into > e.g. say 5 packets and > >sends them to the receiver. If the receiver does > not retrieve packet > >number 3, it will be resend by the sender. > > I did not see the problem that TLS brings into > the picture. Also, what > >tweaking of the stack are you referring to in this > scenario. This is just > > >general handlinf of packets that are done anyway. > iSCSI will only make > >sense of the packet after TLS decrypts the packets. > Did I miss something > >here ??? > > > >SG > > > >*********** REPLY SEPARATOR *********** > > > >On 11/9/2001 at 4:14 PM Julian Satran wrote: > > > >>Bill, > >> > >>The one "tiny" item you forgot to mention is that > TLS records span TCP > >and > >>iSCSI PDU boundaries. TLS records can't be > decrypted in face of TCP > >packet > >>loss and markers/alignment can't be recovered (to > be more precise > require > > > >>a lot more tweaking of the stacks). > >> > >>Julo > >> > >> > >> > >> > >>"Bill Strahm" > >>08-11-01 23:55 > >>Please respond to "Bill Strahm" > >> > >> > >> To: Julian Satran/Haifa/IBM@IBMIL > >> cc: > >> Subject: RE: iSCSI over TLS > >> > >> > >> > >>Julian, > >> > >>I do not understand how TLS interferes with > delivery of iSCSI packets > any > >>more than IPsec. In either case your TOE MUST > decrypt the packet and > >deal > >>with the results. I do not see how this changes > the problem if the > >packet > >>is decrypted before going to the TOE (again the > hardware to do this MUST > > >>be > >>on the NIC device) or after going through the TOE > processing... > >>Quick summary of what I think needs to happen > >>IPsec > >>1) receive L2 packet > >>2) determine it is IP > >>3) Apply packet policy based on L3 header > >>4) Decrypt packet - verify it is covered by the SA > >>5) Pass to L4 (TCP) for processing > >>6) Verify Framing/etc. > >>7) Done > >>TLS > >>1) Recieve L2 Packet > >>2) Pass to L3 > >>3) Pass to L4 (TCP) for processing > >>4) Decrypt packet > >>5) Verify Framing/etc > >>6) Done > >> > >>It turns out the policies for TLS are much simpler > than for IPsec, the > >>application itself gets to determine if security > should be turned on or > >>not > >>(rather than another application pushing policies > into an SPD) and I > >don't > >>see a difference in the security offload > requirements. In many cases > TLS > >>will go through firewalls/NAT/NATP much better > than IPsec, allowing for > a > >>wider deployment model. > >> > >> > >>Bill Strahm > >>+========+=========+=========+=========+=========+=========+=========+ > >>Bill Strahm Software Development is a race > between Programmers > >>Member of the trying to build bigger and better > idiot proof software > >>Technical Staff and the Universe trying to produce > bigger and better > >>bill@sanera.net idiots. > >>(503) 601-0263 So far the Universe is winning --- > Rich Cook > >> > >>-----Original Message----- > >>From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > === message truncated === __________________________________________________ Do You Yahoo!? Find a job, post your resume. http://careers.yahoo.com From owner-ips@ece.cmu.edu Fri Nov 9 19:13:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01020 for ; Fri, 9 Nov 2001 19:13:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9Nd1W27386 for ips-outgoing; Fri, 9 Nov 2001 18:39:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sentinel.sanera.net (ns1.sanera.net [208.253.254.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9Ncwl27377 for ; Fri, 9 Nov 2001 18:38:58 -0500 (EST) Received: from icarus.sanera.net (icarus [192.168.254.11]) by sentinel.sanera.net (8.11.2/8.11.2) with ESMTP id fA9Ncqu26235; Fri, 9 Nov 2001 15:38:52 -0800 Received: from strahmw2k (dhcp-pc-100-165.sanera.net [192.168.100.165]) by icarus.sanera.net (8.11.6/8.11.2) with SMTP id fA9Nck624210; Fri, 9 Nov 2001 15:38:47 -0800 From: "Bill Strahm" To: "Sukanta ganguly" , "Ofer Biran" , Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Fri, 9 Nov 2001 15:38:05 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <20011109190700.94551.qmail@web12506.mail.yahoo.com> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I don't understand what you are really asking for... Do you want both Transport & Tunnel mode to be a MAY ? Do you want the option to not have either ? Do you expect to run Transport mode ESP through a Tunnel Mode ESP transform ? Do you expect to run another security protocol (for example TLS) ? I think we should just say, we require (a MUST) a 2401 IPsec implementation (and all the other random IPsec RFCs as well) (This answers the first three questions above) I think we should allow TLS rather than IPsec (this has lost a long time in the WG, so I am pretty much just giving up) (answers the 4th question) Bill -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Sukanta ganguly Sent: Friday, November 09, 2001 11:07 AM To: Ofer Biran; ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision By doing this we are forcing IPSec. No flexibility of going transport over tunnel. I think we were still having a discussion of whether transport can also be supported and hence instead of forcing with IPSec can't we allow both mechanisms to a MAY. In that scenario one could opt for transport mode with tunnel and still have a good implementation running. What do other think? SG --- Ofer Biran wrote: > > It seems that most people prefer tunnel over > transport mode > and there is no real opposition for choosing tunnel > mode as > the MUST. In view of that we intend to add it in > version 09 > in the following iSCSI statements: > > In Section 10.3.1 Data Integrity and Authentication > : > > "An iSCSI compliant initiator or target MUST provide > data > integrity and authentication by implementing IPSec > [RFC2401] > with ESP in tunnel mode [RFC2406] with the > following..." > > And in Section 10.3.2 Confidentiality : > > "An iSCSI compliant initiator or target MUST provide > confidentiality by implementing IPSec [RFC2401] with > ESP in tunnel mode [RFC2406] with the following..." > > Any objection ? > > Regards, > Ofer > > > Ofer Biran > Storage and Systems Technology > IBM Research Lab in Haifa > biran@il.ibm.com 972-4-8296253 > > > "Saqib Jang" on 01/11/2001 > 20:03:29 > > Please respond to > > To: Ofer Biran/Haifa/IBM@IBMIL, > cc: > Subject: RE: iSCSI: IPsec tunnel / transport mode > decision > > > > > -----Original Message----- > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ofer Biran > Sent: Thursday, November 01, 2001 4:31 AM > To: ips@ece.cmu.edu > Subject: iSCSI: IPsec tunnel / transport mode > decision > > > I'd like to drive this open issue into group > consensus. It seems to > me that the tendency was more toward making tunnel > mode a MUST as iFCP > and FCIP did, mainly due the option of integrating > an existing IPsec > chip/box with the iSCSI implementation offering. If > we reach this decision, > we may choose even not to mention transport mode (as > MAY or some other > recommending text). > > There is an excellent analysis made by Bernard Aboba > in Section > "5.1. Transport mode versus tunnel mode" of > draft-ietf-ips-security-04 > ( > http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt > ) > that can help us with this decision (also Section > "5.2. NAT traversal" is > relevant). > > Regards, > Ofer > > Ofer Biran > Storage and Systems Technology > IBM Research Lab in Haifa > biran@il.ibm.com 972-4-8296253 > > > > __________________________________________________ Do You Yahoo!? Find a job, post your resume. http://careers.yahoo.com From owner-ips@ece.cmu.edu Fri Nov 9 19:14:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01066 for ; Fri, 9 Nov 2001 19:14:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fA9NTPM26769 for ips-outgoing; Fri, 9 Nov 2001 18:29:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fA9NTHl26710 for ; Fri, 9 Nov 2001 18:29:23 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id SAA27267; Fri, 9 Nov 2001 18:29:10 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "ips" Subject: RE: iSCSI: update on OOO CmdSNs/connection Date: Fri, 9 Nov 2001 15:26:18 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <200111092138.NAA28844@core.rose.hp.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Mallikarjun, I never liked the SHOULD. It is not a design point. If we really want to allow it, perhaps a negotiation parameter is a better choice (which is only marginally better). Error detection and recovery have completely different design requirements than the data path. So ideally we say MUST or nothing or we negotiate it Also on your point A, it "MUST" be a MUST on a single connection case except for when required for error recovery (i.e. the very very rare - case of digest error). Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Mallikarjun C. > Sent: Friday, November 09, 2001 1:49 PM > To: ips > Subject: iSCSI: update on OOO CmdSNs/connection > > > To those interested in this discussion: > > Julian and I had a phone conversation on the topic > of OOO CmdSNs on a connection. An update follows. > > Julian agrees that there are valid error recovery > scenarios where CmdSNs will legitimately end up OOO > on a given connection. > > OTOH, I agree with two of Julian's scenarios that he > pointed out right away - the "cleaning command" (command > required to be sent after a retry copy to ensure flushing > within 2^31 -1), and an immediate Logout posted with > unacknowledged commands. Neither of this can be shipped > OOO - since the former undoes the flushing intent, and > the latter breaks the rule that nothing more follows a > Logout on the connection (and troublesome in other ways, > see below). > > In general, I share the concern with Julian that we > have not closely scrutinized all possibilities. > > With that said, something along the following lines > seemed reasonable - > > A)Initiator MUST send commands in increasing order of > CmdSN on a connection if both the following are true - > - operational ErrorRecoveryLevel is 0, > - MaxConnections is negotiated to 1. > B)In all the other cases, initiator SHOULD send commands > in increasing order of CmdSN on a connection. It is > strongly encouraged that commands with out-of-order > CmdSNs be sent on a connection only if they are > retransmitted commands due to digest error recovery > and connection recovery. > > I also suggest the following upon further reflection- > > C)Add wording in section 2.2.2.1 to mandate that > the cleaning command MUST be sent in-order after > the retried command. > D)Warn clearly that sending an immediate Logout command > in the presence of other unacknowledged commands MAY > create inadvertent discarding of certain commands (even > if it is a recovery Logout), and MAY cause protocol > errors leading to ungraceful shutdown of the connection. > > Hopefully A will bring the determinism that Somesh was > looking for certain design points. B describes the more > general n-connection session case. C & D are fixes for > two identfied areas (so far) which will break. > > Comments? > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > From owner-ips@ece.cmu.edu Fri Nov 9 19:57:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01563 for ; Fri, 9 Nov 2001 19:57:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA05TI29053 for ips-outgoing; Fri, 9 Nov 2001 19:05:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA05Nl29041 for ; Fri, 9 Nov 2001 19:05:23 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W487Q39>; Fri, 9 Nov 2001 19:01:44 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293714C@CORPMX14> From: Black_David@emc.com To: knicholson@corp.iready.com, ips@ece.cmu.edu Subject: iSCSI RE: Phase collapse Date: Fri, 9 Nov 2001 18:55:37 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C1697A.074D0820" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C1697A.074D0820 Content-Type: text/plain; charset="iso-8859-1" Yes, most of the time: - If the INQ succeeds, a single Data-in PDU can carry the successful status without a Response PDU. - If the INQ fails without transferring any data, a single Response PDU carries the status and autosense info. Recall that autosense is REQUIRED by iSCSI. But - If the INQ transfers data and then fails, a Data-In PDU is needed to transfer the data and a Response PDU is needed to transfer the status and autosense info. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- -----Original Message----- From: Ken Nicholson [mailto:knicholson@corp.iready.com] Sent: Wednesday, October 31, 2001 9:45 AM To: 'ips@ece.cmu.edu' Subject: Phase collapse Julian and David: Could you clarify the information below? It seems to us that it should be possible for a target to phase collapse the response to an INQ in a single PDU which is both the first and last PDU of that response. If this true: >A command and its associated data may be > shipped together from initiator to target and data and responses > may be shipped together from targets." Why is this true: "...the above text makes it quite clear that data transferred by the action of a SCSI command (including Read, Inquiry, Mode Sense, and Read Capacity) cannot be put in the Response PDU. " Ken Nicholson iReady Corp. ---------------------------------------------------------------------------- ------------- To: ips@ece.cmu.edu Subject: RE: iSCSI: Inquiry, Mode Sense, Read Capacity From: "Julian Satran" Date: Wed, 3 Oct 2001 22:46:44 +0300 Content-Type: multipart/alternative; boundary="=_alternative 006CB109C2256ADA_=" Sender: owner-ips@ece.cmu.edu David is right. Response doe not contain data proper. Phase collapse is for the last DataPDU in which a target can fit a GOOD status (and thus avoid an additional response) and some residual counts (when those are not considered errors). Julo Black_David@emc.com Sent by: owner-ips@ece.cmu.edu 03-10-01 20:21 Please respond to Black_David To: lxing@Crossroads.com, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Inquiry, Mode Sense, Read Capacity The governing text is the following from p.62 of -08 on the contents of the Response PDU: 3.4.6 Data Segment - Sense and Response Data Segment iSCSI targets MUST support and enable autosense. If Status is CHECK CONDITION (0x02), then the Data Segment contains sense data for the failed command. For some iSCSI responses, the response data segment MAY contain some response related information, (e.g., for a target failure it may contain a vendor specific detailed description of the failure). Julian may be able to shed more light on the intended meaning of "phase collapse" for the response case, but the above text makes it quite clear that data transferred by the action of a SCSI command (including Read, Inquiry, Mode Sense, and Read Capacity) cannot be put in the Response PDU. A target can send the Response PDU immediately following the last Data-In PDU for the associated command, which is similar to the ability of an Initiator to send unsolicited Data-Out PDUs. There is no Target analog to the Initiator's ability to send Immediate Data in the same PDU as the SCSI command. Thanks, --David ------_=_NextPart_001_01C1697A.074D0820 Content-Type: text/html; charset="iso-8859-1" Phase collapse

Yes, most of the time:
- If the INQ succeeds, a single Data-in PDU can carry the
    successful status without a Response PDU.
- If the INQ fails without transferring any data, a single
    Response PDU carries the status and autosense info.
    Recall that autosense is REQUIRED by iSCSI.
But
- If the INQ transfers data and then fails, a Data-In PDU
    is needed to transfer the data and a Response PDU is
    needed to transfer the status and autosense info.
 
--David
 

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
black_david@emc.com       Mobile: +1 (978) 394-7754
---------------------------------------------------

 
-----Original Message-----
From: Ken Nicholson [mailto:knicholson@corp.iready.com]
Sent: Wednesday, October 31, 2001 9:45 AM
To: 'ips@ece.cmu.edu'
Subject: Phase collapse

Julian and David:

Could you clarify the information below?

It seems to us that it should be possible for a target to phase collapse the response to an INQ in a single PDU which is both the first and last PDU of that response.

If this true:
>A command and its associated data may be
> shipped together from initiator to target and data and responses
> may be shipped together from targets."


Why is this true:
"...the above text makes
it quite clear that data transferred by the action of a SCSI command
(including Read, Inquiry, Mode Sense, and Read Capacity) cannot be put in
the Response PDU. "





Ken Nicholson
iReady Corp.

----------------------------------------------------------------------------
-------------
To: ips@ece.cmu.edu
Subject: RE: iSCSI: Inquiry, Mode Sense, Read Capacity
From: "Julian Satran" <Julian_Satran@il.ibm.com>
Date: Wed, 3 Oct 2001 22:46:44 +0300
Content-Type: multipart/alternative; boundary="=_alternative
006CB109C2256ADA_="
Sender: owner-ips@ece.cmu.edu




David is right. Response doe not contain data proper.
Phase collapse is for the last DataPDU in which a target can fit a GOOD
status (and thus avoid an additional response) and some residual counts
(when those are not considered errors).

Julo



Black_David@emc.com
Sent by: owner-ips@ece.cmu.edu
03-10-01 20:21
Please respond to Black_David

        To:        lxing@Crossroads.com, ips@ece.cmu.edu
        cc:
        Subject:        RE: iSCSI: Inquiry, Mode Sense, Read Capacity





The governing text is the following from p.62 of -08 on the contents of
the Response PDU:

    3.4.6 Data Segment - Sense and Response Data Segment

       iSCSI targets MUST support and enable autosense.  If Status is CHECK

       CONDITION (0x02), then the Data Segment contains sense data for the
       failed command.

       For some iSCSI responses, the response data segment MAY contain some

       response related information, (e.g., for a target failure it may
       contain a vendor specific detailed description of the failure).

Julian may be able to shed more light on the intended meaning of
"phase collapse" for the response case, but the above text makes
it quite clear that data transferred by the action of a SCSI command
(including Read, Inquiry, Mode Sense, and Read Capacity) cannot be
put in the Response PDU.  A target can send the Response PDU immediately
following the last Data-In PDU for the associated command, which is
similar to the ability of an Initiator to send unsolicited Data-Out
PDUs.  There is no Target analog to the Initiator's ability to send
Immediate Data in the same PDU as the SCSI command.

Thanks,
--David

------_=_NextPart_001_01C1697A.074D0820-- From owner-ips@ece.cmu.edu Fri Nov 9 20:56:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA02122 for ; Fri, 9 Nov 2001 20:56:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA16kc02906 for ips-outgoing; Fri, 9 Nov 2001 20:06:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA16hl02897 for ; Fri, 9 Nov 2001 20:06:43 -0500 (EST) Received: from hq-ex-c2.brocade.com (hq-bes-c2 [192.168.126.35]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id RAA08657; Fri, 9 Nov 2001 17:06:27 -0800 (PST) Received: by hq-bes-c2.brocade.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 17:06:26 -0800 Message-ID: From: Robert Snively To: "'Black_David@emc.com'" , ENDL_TX@computer.org, ips@ece.cmu.edu Cc: Robert Snively Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Fri, 9 Nov 2001 17:06:22 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk David, For those cases where a secure environment is required, the new connection comes up through the normal IPSec authentication and encryption processes. As a result, the transmission of the identifying world wide names is already transmitted in an encrypted format, creatable only by the authenticated and certified source and interpretable only by the authenticated and certified destination. If the Fibre Channel fabric is also operating in a secure mode, subsequent Fibre Channel authentication and certification is performed using the standard FC SLAP mechanisms. In addition to this, there are a whole bunch of policy restrictions that are verified as part of the creation of subsequent connections. While these are not necessarily part of the security steps, they prevent the formation of connections which do not meet the security rules. Assuming security mechanisms are properly implemented, where, then, is the security hole? Bob Snively e-mail: rsnively@brocade.com Brocade Communications Systems phone: 408 487 8135 1745 Technology Drive San Jose, CA 95110 > > Those who were at the Irvine Interim meeting will remember that > > the problem with FCIP and NAPTS is a reliance on IP address in > > the determination of which incoming TCP connections belong in a > > given FCIP Link. This proposal solves that problem by requiring > > that FC Entity World Wide Name be transmitted in the first bytes > > sent by the FCIP Entity that initiates a TCP Connect request. > > This allows the FCIP Entity that receives a TCP Connect request > > to match it with any previously received TCP Connect requests > > from the same source. Since the transmitted World Wide Name is > > required to be unique within Fibre Channel, the FCIP Entity > > that receives this information can correctly assign FCIP Link > > relationships without relying on IP Addresses. > > From a functional standpoint, this works, but it opens up a security > issue. The problem is that on the second TCP connection (and > subsequent > connections) that claim to be from the same FCIP Entity, the WWN that > is initially sent (and whatever extension is used) is functioning > as an authentication to allow that connection to join the first > TCP connection, but that authentication is unsecured -- the sender > announces the WWN, and the receiver does not (and has no way to) > check it. > > There's a fairly obvious denial of service attack here involving > the attacker joining a new connection to an existing one > and then bit-bucketing all the frames sent over the new connection. > > Limiting FCIP to one TCP connection among any pair of FCIP entity > identifiers would help, but is not sufficient. The attack of concern > in this situation involves the attacker crashing the real entity > and opening up a connection in its name, thereby locking out the > real entity when the real entity restarts. > > This may be headed in the direction of needing in-band authentication > which I know the FCIP community has been doing their best to avoid. > > Sorry to be the bearer of bad news, > --David From owner-ips@ece.cmu.edu Fri Nov 9 20:58:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA02149 for ; Fri, 9 Nov 2001 20:58:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA1gYL04984 for ips-outgoing; Fri, 9 Nov 2001 20:42:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA1gXl04979 for ; Fri, 9 Nov 2001 20:42:33 -0500 (EST) Received: by ariel.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 17:42:26 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Cc: "David Black (E-mail)" , "Franco Travostino (E-mail)" , "Elizabeth Rodriguez (E-mail)" Subject: FW: Application for port-number (3420) Date: Fri, 9 Nov 2001 17:42:16 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: IANA [mailto:iana@icann.org] Sent: Friday, November 09, 2001 5:33 PM To: cmonia@nishansystems.com Subject: RE: Application for port-number (3420) Dear Charles, We have assigned the following user port number with you as the point of contact: ifcp-port 3420/tcp iFCP User Port ifcp-port 3420/udp iFCP User Port # Charles Monia Please notify the IANA if there is a change in contact information. Thank you, Michelle S. Cotton IANA Administrator *************************************************************** Internet Assigned Numbers Authority (IANA) 4676 Admiralty Way, Suite 330 Marina del Rey, California 90292 Voice: (310) 823-9358 FAX: (310) 823-8649 email: iana@iana.org *************************************************************** From owner-ips@ece.cmu.edu Fri Nov 9 20:59:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA02164 for ; Fri, 9 Nov 2001 20:59:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA1cA104718 for ips-outgoing; Fri, 9 Nov 2001 20:38:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA1bpl04700 for ; Fri, 9 Nov 2001 20:37:51 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel10.hp.com (Postfix) with ESMTP id 7A2C71F704; Fri, 9 Nov 2001 17:37:45 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id RAA20120; Fri, 9 Nov 2001 17:39:12 -0800 (PST) Message-Id: <200111100139.RAA20120@core.rose.hp.com> Date: Fri, 09 Nov 2001 17:50:07 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: somesh_gupta@silverbacksystems.com Cc: ips Subject: Re: iSCSI: update on OOO CmdSNs/connection References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Somesh, I am in general saying the same thing as you describe, we are perhaps debating on how to phrase the sentiment. >Error detection and recovery > have completely different design requirements than > the data path. Completely agreed, as I have been saying all along. But we're faced with the task of crafting wording that works for both - my wording was B. Please suggest alternate wording that you propose. > So ideally we say MUST or nothing > or we negotiate it In what way saying nothing would help the performance/ interoperability issues on hand? > Also on your point A, it "MUST" be a MUST on > a single connection case except for when required > for error recovery (i.e. the very very rare - > case of digest error). Please see the two conditions that distinguish A from B. Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Somesh Gupta wrote: > > Mallikarjun, > > I never liked the SHOULD. It is not a design point. > If we really want to allow it, perhaps a negotiation > parameter is a better choice (which is only > marginally better). Error detection and recovery > have completely different design requirements than > the data path. > > So ideally we say MUST or nothing > or we negotiate it > > Also on your point A, it "MUST" be a MUST on > a single connection case except for when required > for error recovery (i.e. the very very rare - > case of digest error). > > Somesh > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Mallikarjun C. > > Sent: Friday, November 09, 2001 1:49 PM > > To: ips > > Subject: iSCSI: update on OOO CmdSNs/connection > > > > > > To those interested in this discussion: > > > > Julian and I had a phone conversation on the topic > > of OOO CmdSNs on a connection. An update follows. > > > > Julian agrees that there are valid error recovery > > scenarios where CmdSNs will legitimately end up OOO > > on a given connection. > > > > OTOH, I agree with two of Julian's scenarios that he > > pointed out right away - the "cleaning command" (command > > required to be sent after a retry copy to ensure flushing > > within 2^31 -1), and an immediate Logout posted with > > unacknowledged commands. Neither of this can be shipped > > OOO - since the former undoes the flushing intent, and > > the latter breaks the rule that nothing more follows a > > Logout on the connection (and troublesome in other ways, > > see below). > > > > In general, I share the concern with Julian that we > > have not closely scrutinized all possibilities. > > > > With that said, something along the following lines > > seemed reasonable - > > > > A)Initiator MUST send commands in increasing order of > > CmdSN on a connection if both the following are true - > > - operational ErrorRecoveryLevel is 0, > > - MaxConnections is negotiated to 1. > > B)In all the other cases, initiator SHOULD send commands > > in increasing order of CmdSN on a connection. It is > > strongly encouraged that commands with out-of-order > > CmdSNs be sent on a connection only if they are > > retransmitted commands due to digest error recovery > > and connection recovery. > > > > I also suggest the following upon further reflection- > > > > C)Add wording in section 2.2.2.1 to mandate that > > the cleaning command MUST be sent in-order after > > the retried command. > > D)Warn clearly that sending an immediate Logout command > > in the presence of other unacknowledged commands MAY > > create inadvertent discarding of certain commands (even > > if it is a recovery Logout), and MAY cause protocol > > errors leading to ungraceful shutdown of the connection. > > > > Hopefully A will bring the determinism that Somesh was > > looking for certain design points. B describes the more > > general n-connection session case. C & D are fixes for > > two identfied areas (so far) which will break. > > > > Comments? > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > From owner-ips@ece.cmu.edu Fri Nov 9 21:52:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA03683 for ; Fri, 9 Nov 2001 21:52:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA229p06160 for ips-outgoing; Fri, 9 Nov 2001 21:02:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA227l06154 for ; Fri, 9 Nov 2001 21:02:07 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W487SNB>; Fri, 9 Nov 2001 20:58:28 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293714E@CORPMX14> From: Black_David@emc.com To: rsnively@Brocade.COM, ips@ece.cmu.edu Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Fri, 9 Nov 2001 20:52:21 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > For those cases where a secure environment is required, the > new connection comes up through the normal IPSec authentication > and encryption processes. As a result, the transmission of > the identifying world wide names is already transmitted in an > encrypted format, creatable only by the authenticated and > certified source and interpretable only by the authenticated > and certified destination. The assumption that there's a big on/off switch for security is not correct (e.g., suppose IPsec is running with integrity on and confidentiality off) and "authenticated and certified" is a peculiar concept for group pre-shared keys, a very likely deployment scenario. Also, IPsec has no clue about the WWN, and hence the IKE authentication is not linked to the WWN that has to be presented (a particular problem with group pre-shared keys, but also a risk even with certificates). In other words, "authenticated and certified" doesn't place any restrictions on what WWN is presented. Finally, solving an inband authentication problem by requiring that IPsec be used to encrypt the WWN is wildly excessive, and besides, the WWN has no secret contents- it's a public piece of configuration information. There is a solution in this general area, but it involves linking the WWN to the IKE identity that is authenticated. That's probably going to break any attempt to use existing off-the-shelf external IPsec gateways because gateways don't grok WWNs, and the FCIP implementations won't know what identity was used in the IKE authentication to the gateway. This is all a bit peculiar because FC currently trust WWNs passed in various FC Login frames without authenticating them, but that's T11's area of responsibility putting this sort of thing into an IETF standard isn't going to be acceptable. > If the Fibre Channel fabric is also operating in a secure mode, > subsequent Fibre Channel authentication and certification > is performed using the standard FC SLAP mechanisms. Only on the first TCP connection, unless you plan to require taking the FCIP link down when the second TCP connection is added (which strikes me as highly counterproductive). The second connection just inherits the results of the SLAP performed on the first one, whether its entitled to or not. > In addition to this, there are a whole bunch of policy > restrictions that are verified as part of the creation > of subsequent connections. While these are not necessarily > part of the security steps, they prevent the formation of > connections which do not meet the security rules. I'm not sure what you're referring to, but I suspect they don't help much here. > Assuming security mechanisms are properly implemented, where, > then, is the security hole? The fact that the WWN is not authenticated means that anyone who can set up an FCIP connection to an FCIP entity can tap into any existing FC traffic flowing on any connection through that FCIP entity. --David > -----Original Message----- > From: Robert Snively [mailto:rsnively@brocade.com] > Sent: Friday, November 09, 2001 8:06 PM > To: 'Black_David@emc.com'; ENDL_TX@computer.org; ips@ece.cmu.edu > Cc: Robert Snively > Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA > Interim meeting) > > > > David, > > For those cases where a secure environment is required, the > new connection comes up through the normal IPSec authentication > and encryption processes. As a result, the transmission of > the identifying world wide names is already transmitted in an > encrypted format, creatable only by the authenticated and > certified source and interpretable only by the authenticated > and certified destination. > > If the Fibre Channel fabric is also operating in a secure mode, > subsequent Fibre Channel authentication and certification > is performed using the standard FC SLAP mechanisms. > > In addition to this, there are a whole bunch of policy > restrictions that are verified as part of the creation > of subsequent connections. While these are not necessarily > part of the security steps, they prevent the formation of > connections which do not meet the security rules. > > Assuming security mechanisms are properly implemented, where, > then, is the security hole? > > Bob Snively e-mail: rsnively@brocade.com > Brocade Communications Systems phone: 408 487 8135 > 1745 Technology Drive > San Jose, CA 95110 > > > > > > Those who were at the Irvine Interim meeting will remember that > > > the problem with FCIP and NAPTS is a reliance on IP address in > > > the determination of which incoming TCP connections belong in a > > > given FCIP Link. This proposal solves that problem by requiring > > > that FC Entity World Wide Name be transmitted in the first bytes > > > sent by the FCIP Entity that initiates a TCP Connect request. > > > This allows the FCIP Entity that receives a TCP Connect request > > > to match it with any previously received TCP Connect requests > > > from the same source. Since the transmitted World Wide Name is > > > required to be unique within Fibre Channel, the FCIP Entity > > > that receives this information can correctly assign FCIP Link > > > relationships without relying on IP Addresses. > > > > From a functional standpoint, this works, but it opens up a security > > issue. The problem is that on the second TCP connection (and > > subsequent > > connections) that claim to be from the same FCIP Entity, > the WWN that > > is initially sent (and whatever extension is used) is functioning > > as an authentication to allow that connection to join the first > > TCP connection, but that authentication is unsecured -- the sender > > announces the WWN, and the receiver does not (and has no way to) > > check it. > > > > There's a fairly obvious denial of service attack here involving > > the attacker joining a new connection to an existing one > > and then bit-bucketing all the frames sent over the new connection. > > > > Limiting FCIP to one TCP connection among any pair of FCIP entity > > identifiers would help, but is not sufficient. The attack > of concern > > in this situation involves the attacker crashing the real entity > > and opening up a connection in its name, thereby locking out the > > real entity when the real entity restarts. > > > > This may be headed in the direction of needing in-band > authentication > > which I know the FCIP community has been doing their best to avoid. > > > > Sorry to be the bearer of bad news, > > --David > From owner-ips@ece.cmu.edu Fri Nov 9 22:39:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA06188 for ; Fri, 9 Nov 2001 22:39:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA36ND09950 for ips-outgoing; Fri, 9 Nov 2001 22:06:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA36Ml09946 for ; Fri, 9 Nov 2001 22:06:22 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 22:08:42 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937151@CORPMX14> From: Black_David@emc.com To: muralir@lightsand.com, ips@ece.cmu.edu, ENDL_TX@computer.org Cc: craig.carlson@qlogic.com, Jim.Nelson@Vixel.com, swilson@Brocade.COM, egrodriguez@lucent.com Subject: RE: FCencap: List ALL SOF/EOF codes Date: Fri, 9 Nov 2001 21:56:38 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk FC-MI was going to prohibit Class 1 last time I checked. Since the I in FC-MI stands for "Interoperability", this seems like a reasonable rationale for excluding Class 1 service. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Nov 9 22:40:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA06274 for ; Fri, 9 Nov 2001 22:40:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA3NJ610892 for ips-outgoing; Fri, 9 Nov 2001 22:23:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA3NHl10882 for ; Fri, 9 Nov 2001 22:23:17 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 22:23:12 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937152@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iSCSI security clarifications Date: Fri, 9 Nov 2001 22:13:34 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Gathering up a bunch of security topics/questions that need to be addressed from the last few weeks of email. -- SRP Intellectual Property Yes, this situation is confusing and unclear. It is also evolving (e.g., the zero-cost Stanford license is a recent development). I will endeavor to obtain clarity and explain it at the Salt Lake City meeting. If clarity is not obtainable, it would be reasonable to remove the "MUST" requirement for implementing SRP at that time. -- Substituting CHAP for SRP as the REQUIRED mechanism Not a good idea. Situations can be expected in which IPsec is turned off by an administrator who relies on authentication. CHAP is considerably weaker than SRP in this situation because (all too common) weak passwords for CHAP are vulnerable to off-line dictionary attacks, whereas SRP does not have this vulnerability. -- IPsec requirements A quick reminder that IPsec is "MUST implement" but "MAY use" for all of our protocols. Arguments that start from assuming the use of IPsec could lead to having to strengthen the "MAY use" requirement, and this should be considered by folks making such arguments. We are subsetting IPsec (e.g., AH is NOT REQUIRED) for all of our protocols. While quoting requirements from IPsec RFCs is illuminating and useful to understand what was originally done in IPsec and why, those requirements are not necessarily binding on us. We do have to exercise good engineering and security judgment in picking our subset (e.g., leaving out AH is an example of doing so). This approach of picking an appropriate subset of IPsec does have the approval of the IETF Security area, as long as we don't do anything obviously wrong. One other specific example is that "MUST implement DES" (as required by the current IPsec RFCs) will only go into IPS WG documents over my dead body ;-). -- TLS for iSCSI Those interested in standardizing the use of TLS for iSCSI should write and submit an Internet-Draft. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Nov 9 22:41:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA06364 for ; Fri, 9 Nov 2001 22:41:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA2vSw09403 for ips-outgoing; Fri, 9 Nov 2001 21:57:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA2vKl09394 for ; Fri, 9 Nov 2001 21:57:20 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W487T13>; Fri, 9 Nov 2001 21:53:41 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293714F@CORPMX14> From: Black_David@emc.com To: ni1d@arrl.net Cc: ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision Date: Fri, 9 Nov 2001 21:47:36 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk With my WG chair hat firmly on, I have to say that Paul Koning is mistaken in asserting the absence of WG rough consensus for use of IPsec with iSCSI. The rough consensus in question is at least 6 months old having been established across 2 interim meetings plus the meeting in London, all of whose minutes have been reported to the mailing list and all of which have been accepted without significant objection on the list. Paul's objection to this consensus is noted, but the consensus stands. The two "MUST"s that Ofer proposed are not only appropriate, but required - the IESG can be expected to return a document that does not contain them to the WG in short order. More on security is coming shortly, but this seemed important enough to put in a message by itself. Thanks, --David > -----Original Message----- > From: Paul Koning [mailto:ni1d@arrl.net] > Sent: Friday, November 09, 2001 4:43 PM > To: BIRAN@il.ibm.com > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: IPsec tunnel / transport mode decision > > > I'm not sure if this is the sort of answer your question was looking > for, but I don't want to let silence be taken for agreement, so... > > I'm opposed to the presence of "MUST" in sections 10.3.1 and 10.3.2 > for reasons stated in mail to this list a week or two ago. > > paul > From owner-ips@ece.cmu.edu Fri Nov 9 22:42:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA06610 for ; Fri, 9 Nov 2001 22:42:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA339f09744 for ips-outgoing; Fri, 9 Nov 2001 22:03:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA338l09738 for ; Fri, 9 Nov 2001 22:03:08 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W487THB>; Fri, 9 Nov 2001 21:59:29 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937150@CORPMX14> From: Black_David@emc.com To: somesh_gupta@silverbacksystems.com, ips@ece.cmu.edu Subject: RE: iSCSI: Clarification (again) for Task Management Commands Date: Fri, 9 Nov 2001 21:53:25 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, The language in question reflects fairly direct requirements language to be found in SAM-2's description of SCSI Task Management. FCP goes to serious lengths with FC sequence aborts to make sure this behaves as required. For iSCSI, if responses to the aborted commands show up unexpectedly, they have to be discarded. How the Initiator keeps track of that is the Initiator's problem - keeping track of the CmdSN of the Abort Task Set may be useful. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > Sent: Friday, November 09, 2001 4:55 PM > To: IPS > Subject: iSCSI: Clarification (again) for Task Management Commands > > > Resend to add iSCSI tag. Sorry for missing it. > > On page 67 of the 8-92.txt draft (section 3.5.1), it > says > > "For all the tasks covered by the task > management response (i.e., with CmdSN not higher than the task > management command CmdSN), additional responses MUST NOT be delivered > to the SCSI layer after the task management response." > > If there is a multiple connection session, > a status for a command impacted by the task > management command (say ABORT TASK SET) could > be stuck in the pipe on one connection, while > the ABORT TASK SET completes on another > connection. > > How does the initiator iSCSI enforce the rule above? > Seems to be the equivalent of sending the impacted > commands on other connections in a zombie state, > and not having a very good idea of how to get out. > > Similarly Section 9.4 provides additional rules, > but seems to leave a hole open with regards to > status already in flight on other connections. > > Any clarifications would be appreciated. > > Somesh > From owner-ips@ece.cmu.edu Fri Nov 9 22:45:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA06800 for ; Fri, 9 Nov 2001 22:45:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA3ZOC11561 for ips-outgoing; Fri, 9 Nov 2001 22:35:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA3ZOl11557 for ; Fri, 9 Nov 2001 22:35:24 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W487TTY>; Fri, 9 Nov 2001 22:31:45 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937154@CORPMX14> From: Black_David@emc.com To: ni1d@arrl.net Cc: ips@ece.cmu.edu Subject: RE: iSCSI: IKE normative guidelines Date: Fri, 9 Nov 2001 22:25:39 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes, these guidelines differ from those RFCs in a number of ways, based on things learned since those RFCs were written. Many of these are things that would go into revised versions of the IPsec RFCs, but revising the IPsec RFCs turns out to be intractable to the point of near-impossibility for the security folks - you'll have to ask them why. See the IPS security draft for some further explanation of these changes. Main mode is not "at least as good" as aggressive mode when group pre-shared keys are used; see the security draft and the ipsec WG "improveike" draft for details. Group pre- shared keys are often used in practice because they greatly simplify key management. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Paul Koning [mailto:ni1d@arrl.net] > Sent: Friday, November 09, 2001 4:45 PM > To: BIRAN@il.ibm.com > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI: IKE normative guidelines > > > >>>>> "Ofer" == Ofer Biran writes: > > Ofer> In the framework of the effort being done now by the security > Ofer> team to sync the normative statements in the security draft > Ofer> with the protocol drafts, I suggest to adopt the following IKE > Ofer> normative guidelines that already appear in the security draft > Ofer> for iSCSI: ... > > Does any of this differ from what's in RFC 2408/2409? If so, why? If > not, why not just refer to that standard and say nothing further? > > If we're going to be different, I'd suggest dropping aggressive mode > since main mode is at least as good. > > paul > From owner-ips@ece.cmu.edu Fri Nov 9 22:46:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA06937 for ; Fri, 9 Nov 2001 22:46:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA3TIZ11222 for ips-outgoing; Fri, 9 Nov 2001 22:29:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA3TGl11216 for ; Fri, 9 Nov 2001 22:29:16 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id WAA15614; Fri, 9 Nov 2001 22:29:03 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: , Subject: RE: iSCSI: Clarification (again) for Task Management Commands Date: Fri, 9 Nov 2001 19:26:11 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <277DD60FB639D511AC0400B0D068B71E02937150@CORPMX14> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, In iSCSI the multiple connection/session construct adds significant complexity in determining whether a response for a command (on a different connection than the one on which the task management command was sent) impacted by the task management command will be received or not - and when? On a single connection, or similar links, when the response for the task management command is received (or after a fixed additional time), no responses will be received for the commands aborted by the task management command. However, with multiple connections there is no such "flushing" event on connections other than the one on which the task management command was sent. I would hope that the protocol would address this situation - the current response seems to be be to put the task tag and some associated resources in a zombie state for an unspecified amount of time. Somesh > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, November 09, 2001 6:53 PM > To: somesh_gupta@silverbacksystems.com; ips@ece.cmu.edu > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > Somesh, > > The language in question reflects fairly direct requirements > language to be found in SAM-2's description of SCSI Task Management. > FCP goes to serious lengths with FC sequence aborts to make sure > this behaves as required. > > For iSCSI, if responses to the aborted commands show up unexpectedly, > they have to be discarded. How the Initiator keeps track of that > is the Initiator's problem - keeping track of the CmdSN of the > Abort Task Set may be useful. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > -----Original Message----- > > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > > Sent: Friday, November 09, 2001 4:55 PM > > To: IPS > > Subject: iSCSI: Clarification (again) for Task Management Commands > > > > > > Resend to add iSCSI tag. Sorry for missing it. > > > > On page 67 of the 8-92.txt draft (section 3.5.1), it > > says > > > > "For all the tasks covered by the task > > management response (i.e., with CmdSN not higher than the task > > management command CmdSN), additional responses MUST NOT be delivered > > to the SCSI layer after the task management response." > > > > If there is a multiple connection session, > > a status for a command impacted by the task > > management command (say ABORT TASK SET) could > > be stuck in the pipe on one connection, while > > the ABORT TASK SET completes on another > > connection. > > > > How does the initiator iSCSI enforce the rule above? > > Seems to be the equivalent of sending the impacted > > commands on other connections in a zombie state, > > and not having a very good idea of how to get out. > > > > Similarly Section 9.4 provides additional rules, > > but seems to leave a hole open with regards to > > status already in flight on other connections. > > > > Any clarifications would be appreciated. > > > > Somesh > > > From owner-ips@ece.cmu.edu Fri Nov 9 22:47:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA07029 for ; Fri, 9 Nov 2001 22:47:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA3YqZ11527 for ips-outgoing; Fri, 9 Nov 2001 22:34:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA3Ypl11522 for ; Fri, 9 Nov 2001 22:34:51 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W487TTT>; Fri, 9 Nov 2001 22:31:12 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937153@CORPMX14> From: Black_David@emc.com To: somesh_gupta@silverbacksystems.com, ips@ece.cmu.edu Subject: RE: iSCSI: Clarification (again) for Task Management Commands Date: Fri, 9 Nov 2001 22:25:07 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, What do you have in mind? In practice, completion of any command sent after the aborted "zombies" serves as a "flushing" event for the connection on which that command is sent; if the Initiator is reasonably busy, I think the "flushing" occurs fairly quickly. An initiator can also selectively fall back to the FCP-like technique of explicitly issuing a task abort for the "zombie" it wants to recover on the appropriate connection. Thanks, --David > -----Original Message----- > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > Sent: Friday, November 09, 2001 10:26 PM > To: Black_David@emc.com; ips@ece.cmu.edu > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > David, > > In iSCSI the multiple connection/session construct > adds significant complexity in determining whether > a response for a command (on a different connection > than the one on which the task management command > was sent) impacted by the task management command will > be received or not - and when? > > On a single connection, or similar links, when the > response for the task management command is > received (or after a fixed additional time), no > responses will be received for the commands aborted > by the task management command. > > However, with multiple connections there is no > such "flushing" event on connections other than the > one on which the task management command was sent. > > I would hope that the protocol would address this > situation - the current response seems to be be to > put the task tag and some associated resources in > a zombie state for an unspecified amount of time. > > Somesh > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Friday, November 09, 2001 6:53 PM > > To: somesh_gupta@silverbacksystems.com; ips@ece.cmu.edu > > Subject: RE: iSCSI: Clarification (again) for Task > Management Commands > > > > > > Somesh, > > > > The language in question reflects fairly direct requirements > > language to be found in SAM-2's description of SCSI Task Management. > > FCP goes to serious lengths with FC sequence aborts to make sure > > this behaves as required. > > > > For iSCSI, if responses to the aborted commands show up > unexpectedly, > > they have to be discarded. How the Initiator keeps track of that > > is the Initiator's problem - keeping track of the CmdSN of the > > Abort Task Set may be useful. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > -----Original Message----- > > > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > > > Sent: Friday, November 09, 2001 4:55 PM > > > To: IPS > > > Subject: iSCSI: Clarification (again) for Task Management Commands > > > > > > > > > Resend to add iSCSI tag. Sorry for missing it. > > > > > > On page 67 of the 8-92.txt draft (section 3.5.1), it > > > says > > > > > > "For all the tasks covered by the task > > > management response (i.e., with CmdSN not higher than the task > > > management command CmdSN), additional responses MUST NOT > be delivered > > > to the SCSI layer after the task management response." > > > > > > If there is a multiple connection session, > > > a status for a command impacted by the task > > > management command (say ABORT TASK SET) could > > > be stuck in the pipe on one connection, while > > > the ABORT TASK SET completes on another > > > connection. > > > > > > How does the initiator iSCSI enforce the rule above? > > > Seems to be the equivalent of sending the impacted > > > commands on other connections in a zombie state, > > > and not having a very good idea of how to get out. > > > > > > Similarly Section 9.4 provides additional rules, > > > but seems to leave a hole open with regards to > > > status already in flight on other connections. > > > > > > Any clarifications would be appreciated. > > > > > > Somesh > > > > > > From owner-ips@ece.cmu.edu Fri Nov 9 23:25:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08080 for ; Fri, 9 Nov 2001 23:25:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA3tJH12824 for ips-outgoing; Fri, 9 Nov 2001 22:55:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA3tIl12820 for ; Fri, 9 Nov 2001 22:55:18 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 22:55:13 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937157@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iSCSI reserved bits and fields Date: Fri, 9 Nov 2001 22:45:35 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk "MUST be ignored by the receiver" is definitely the correct language for the receive side. On the send side, I believe "SHOULD be set to zero" is sufficient; given the "MUST ignore" on the receive side, I don't think another "MUST" is needed. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Nov 9 23:26:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08104 for ; Fri, 9 Nov 2001 23:26:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA3ffk11980 for ips-outgoing; Fri, 9 Nov 2001 22:41:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA3fel11976 for ; Fri, 9 Nov 2001 22:41:40 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 22:43:59 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937156@CORPMX14> From: Black_David@emc.com To: cbm@rose.hp.com, ips@ece.cmu.edu Subject: RE: iSCSI: new state diagrams for rev09 Date: Fri, 9 Nov 2001 22:31:56 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Oops, "A bit tardy" refers to me taking a week to get this message out, not to the time taken to produce these diagrams. Apologies for any confusion. --David > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, November 09, 2001 10:29 PM > To: cbm@rose.hp.com; ips@ece.cmu.edu > Subject: RE: iSCSI: new state diagrams for rev09 > > > A bit tardy, but I'd like to thank everyone whose efforts > have contributed to these diagrams and explanations. Getting > error behavior completely specified is not easy, and very > important to the overall specification of iSCSI. Keep up > the good work. > > Many Thanks, > --David > > > -----Original Message----- > > From: Mallikarjun C. [mailto:cbm@rose.hp.com] > > Sent: Friday, November 02, 2001 1:33 PM > > To: ips > > Subject: iSCSI: new state diagrams for rev09 > > > > > > All: > > > > Consequent to the feedback I received both online and > > offline, iSCSI state diagrams are considerably revamped. > > Thanks to all those who provided the feedback. The new > > state diagrams are posted at: > > > > http://storage-arch.hp.com/iscsi_states_rev09.pdf > > > > The ASCII forms of the above are currently part of rev 08-90 > > draft that Julian posted on his web. > > > > The major changes are - > > > > - The initiator and target state machines are split for > > both standard connection diagram and session state > > diagram. It is hoped that this separation makes it > > easier to follow. > > > > - Several (informative, but) non-essential states were > > eliminated, since they were waiting for internal events > > (like acquiring resources) that are likely non-existent > > in good implementations. That leaves us with a 7-state > > machine for both initiator and target. > > > > - The reduction of states has a salutary effect on the ASCII > > representation. I was able to translate all the diagrams > > into ASCII! > > > > - The state descriptions are more descriptive now, specifically > > calling out the event(s) that the state machine is waiting > > for in that state. > > > > - The transition descriptions are also more descriptive, > > listing the set of events that causes the given transition > > for each of the roles. > > > > - The "connection recovery state diagram" has been renamed > > as "connection cleanup state diagram" since the word "recovery" > > was inadvertently overloaded. Connection recovery (the > > act of reassigning the task allegiance to a different connection) > > happens outside the scope of this state machine depending on > > the operational ErrorRecoveryLevel - all this diagram > > specifies is the dynamics of gracefully closing an active > > iSCSI connection, perhaps replacing with a new connection. > > > > - Consequent to the above and for general clarity, some > > states have been renamed to reflect the "wait reason" better. > > > > - Certain missing/incorrect transitions are now fixed - > > like the "close the session" Logout handling etc. > > > > - I added some text on the first slide that describes the > > design philosophy behind the current model, perhaps also > > can be called design assumptions. > > > > > > I have only one (likely) open issue on my list. I received > > feedback from one reviewer that the _description_ of the diagrams > > is not "traditional" - for example, the current tabular format > > captures the transitions (each caused by a set of events) in > > a state-state matrix, but does not describe them in terms of > > an event-state matrix for each state (which would run into > > several pages). The current description scheme seemed quite > > acceptable to myself (and also to Julian among others) - but > > I am open on this. Please comment if you strongly feel about > > this issue. > > > > Thanks! > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > From owner-ips@ece.cmu.edu Fri Nov 9 23:29:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08128 for ; Fri, 9 Nov 2001 23:29:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA3ct711779 for ips-outgoing; Fri, 9 Nov 2001 22:38:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA3csl11775 for ; Fri, 9 Nov 2001 22:38:54 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 9 Nov 2001 22:41:14 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937155@CORPMX14> From: Black_David@emc.com To: cbm@rose.hp.com, ips@ece.cmu.edu Subject: RE: iSCSI: new state diagrams for rev09 Date: Fri, 9 Nov 2001 22:29:10 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk A bit tardy, but I'd like to thank everyone whose efforts have contributed to these diagrams and explanations. Getting error behavior completely specified is not easy, and very important to the overall specification of iSCSI. Keep up the good work. Many Thanks, --David > -----Original Message----- > From: Mallikarjun C. [mailto:cbm@rose.hp.com] > Sent: Friday, November 02, 2001 1:33 PM > To: ips > Subject: iSCSI: new state diagrams for rev09 > > > All: > > Consequent to the feedback I received both online and > offline, iSCSI state diagrams are considerably revamped. > Thanks to all those who provided the feedback. The new > state diagrams are posted at: > > http://storage-arch.hp.com/iscsi_states_rev09.pdf > > The ASCII forms of the above are currently part of rev 08-90 > draft that Julian posted on his web. > > The major changes are - > > - The initiator and target state machines are split for > both standard connection diagram and session state > diagram. It is hoped that this separation makes it > easier to follow. > > - Several (informative, but) non-essential states were > eliminated, since they were waiting for internal events > (like acquiring resources) that are likely non-existent > in good implementations. That leaves us with a 7-state > machine for both initiator and target. > > - The reduction of states has a salutary effect on the ASCII > representation. I was able to translate all the diagrams > into ASCII! > > - The state descriptions are more descriptive now, specifically > calling out the event(s) that the state machine is waiting > for in that state. > > - The transition descriptions are also more descriptive, > listing the set of events that causes the given transition > for each of the roles. > > - The "connection recovery state diagram" has been renamed > as "connection cleanup state diagram" since the word "recovery" > was inadvertently overloaded. Connection recovery (the > act of reassigning the task allegiance to a different connection) > happens outside the scope of this state machine depending on > the operational ErrorRecoveryLevel - all this diagram > specifies is the dynamics of gracefully closing an active > iSCSI connection, perhaps replacing with a new connection. > > - Consequent to the above and for general clarity, some > states have been renamed to reflect the "wait reason" better. > > - Certain missing/incorrect transitions are now fixed - > like the "close the session" Logout handling etc. > > - I added some text on the first slide that describes the > design philosophy behind the current model, perhaps also > can be called design assumptions. > > > I have only one (likely) open issue on my list. I received > feedback from one reviewer that the _description_ of the diagrams > is not "traditional" - for example, the current tabular format > captures the transitions (each caused by a set of events) in > a state-state matrix, but does not describe them in terms of > an event-state matrix for each state (which would run into > several pages). The current description scheme seemed quite > acceptable to myself (and also to Julian among others) - but > I am open on this. Please comment if you strongly feel about > this issue. > > Thanks! > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > From owner-ips@ece.cmu.edu Sat Nov 10 04:51:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA25543 for ; Sat, 10 Nov 2001 04:51:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA8isu28730 for ips-outgoing; Sat, 10 Nov 2001 03:44:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA8iql28726 for ; Sat, 10 Nov 2001 03:44:52 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA119282 for ; Sat, 10 Nov 2001 09:44:46 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAA8ijt96984 for ; Sat, 10 Nov 2001 09:44:45 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI Clarification (again) for Task Management Commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 10 Nov 2001 10:44:43 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 10/11/2001 10:44:45, Serialize complete at 10/11/2001 10:44:45 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh - see 9.4 - Regards, Julo "Somesh Gupta" Sent by: owner-ips@ece.cmu.edu 09-11-01 22:33 Please respond to somesh_gupta To: "IPS" cc: Subject: iSCSI - Clarification (again) for Task Management Commands On page 67 of the 8-92.txt draft (section 3.5.1), it says "For all the tasks covered by the task management response (i.e., with CmdSN not higher than the task management command CmdSN), additional responses MUST NOT be delivered to the SCSI layer after the task management response." If there is a multiple connection session, a status for a command impacted by the task management command (say ABORT TASK SET) could be stuck in the pipe on one connection, while the ABORT TASK SET completes on another connection. How does the initiator iSCSI enforce the rule above? Seems to be the equivalent of sending the impacted commands on other connections in a zombie state, and not having a very good idea of how to get out. Similarly Section 9.4 provides additional rules, but seems to leave a hole open with regards to status already in flight on other connections. Any clarifications would be appreciated. Somesh From owner-ips@ece.cmu.edu Sat Nov 10 04:55:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA25574 for ; Sat, 10 Nov 2001 04:55:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA8png29095 for ips-outgoing; Sat, 10 Nov 2001 03:51:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA8pml29090 for ; Sat, 10 Nov 2001 03:51:48 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA45944 for ; Sat, 10 Nov 2001 09:51:42 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAA8pet45848 for ; Sat, 10 Nov 2001 09:51:40 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: IPsec tunnel / transport mode decision X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 10 Nov 2001 10:51:38 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 10/11/2001 10:51:41, Serialize complete at 10/11/2001 10:51:41 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk SG, You must have one of them MUST to ensure interoperability. Julo Sukanta ganguly Sent by: owner-ips@ece.cmu.edu 09-11-01 21:07 Please respond to Sukanta ganguly To: Ofer Biran , ips@ece.cmu.edu cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision By doing this we are forcing IPSec. No flexibility of going transport over tunnel. I think we were still having a discussion of whether transport can also be supported and hence instead of forcing with IPSec can't we allow both mechanisms to a MAY. In that scenario one could opt for transport mode with tunnel and still have a good implementation running. What do other think? SG --- Ofer Biran wrote: > > It seems that most people prefer tunnel over > transport mode > and there is no real opposition for choosing tunnel > mode as > the MUST. In view of that we intend to add it in > version 09 > in the following iSCSI statements: > > In Section 10.3.1 Data Integrity and Authentication > : > > "An iSCSI compliant initiator or target MUST provide > data > integrity and authentication by implementing IPSec > [RFC2401] > with ESP in tunnel mode [RFC2406] with the > following..." > > And in Section 10.3.2 Confidentiality : > > "An iSCSI compliant initiator or target MUST provide > confidentiality by implementing IPSec [RFC2401] with > ESP in tunnel mode [RFC2406] with the following..." > > Any objection ? > > Regards, > Ofer > > > Ofer Biran > Storage and Systems Technology > IBM Research Lab in Haifa > biran@il.ibm.com 972-4-8296253 > > > "Saqib Jang" on 01/11/2001 > 20:03:29 > > Please respond to > > To: Ofer Biran/Haifa/IBM@IBMIL, > cc: > Subject: RE: iSCSI: IPsec tunnel / transport mode > decision > > > > > -----Original Message----- > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ofer Biran > Sent: Thursday, November 01, 2001 4:31 AM > To: ips@ece.cmu.edu > Subject: iSCSI: IPsec tunnel / transport mode > decision > > > I'd like to drive this open issue into group > consensus. It seems to > me that the tendency was more toward making tunnel > mode a MUST as iFCP > and FCIP did, mainly due the option of integrating > an existing IPsec > chip/box with the iSCSI implementation offering. If > we reach this decision, > we may choose even not to mention transport mode (as > MAY or some other > recommending text). > > There is an excellent analysis made by Bernard Aboba > in Section > "5.1. Transport mode versus tunnel mode" of > draft-ietf-ips-security-04 > ( > http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt > ) > that can help us with this decision (also Section > "5.2. NAT traversal" is > relevant). > > Regards, > Ofer > > Ofer Biran > Storage and Systems Technology > IBM Research Lab in Haifa > biran@il.ibm.com 972-4-8296253 > > > > __________________________________________________ Do You Yahoo!? Find a job, post your resume. http://careers.yahoo.com From owner-ips@ece.cmu.edu Sat Nov 10 04:55:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA25586 for ; Sat, 10 Nov 2001 04:55:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAA8haN28663 for ips-outgoing; Sat, 10 Nov 2001 03:43:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA8hYl28656 for ; Sat, 10 Nov 2001 03:43:34 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA37088 for ; Sat, 10 Nov 2001 09:43:27 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAA8hNt99124 for ; Sat, 10 Nov 2001 09:43:27 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI over TLS X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 10 Nov 2001 10:43:19 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 10/11/2001 10:43:27, Serialize complete at 10/11/2001 10:43:27 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, The archive has long discussions about it. There are several other docs you may want to go through (a memo from June 2000 on my site and a doc+presentation by Randy Haagens in the archive. Julo "Bill Strahm" 09-11-01 20:40 Please respond to "Bill Strahm" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: iSCSI over TLS I am confused... Why can't I put the data into the memory location specified by the TCP sequence number ??? I am assuming that is what you are doing in the case of IPsec dropping a TCP frame in the middle of the stream. TCP stacks that I am aware of WILL NOT pass out of order stream data to applications until the intermediate frames are dealt with anyway, so what is the difference between holding encrypted frames and unencrypted frames until the stream is in order ? I believe that there are also TLS options to do per frame crypto, however at that point you have to start shipping over IVs with each frame, removing most wire advantages of TLS over IPsec... Bill +========+=========+=========+=========+=========+=========+=========+ Bill Strahm Software Development is a race between Programmers Member of the trying to build bigger and better idiot proof software Technical Staff and the Universe trying to produce bigger and better bill@sanera.net idiots. (503) 601-0263 So far the Universe is winning --- Rich Cook -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran Sent: Friday, November 09, 2001 10:06 AM To: ips@ece.cmu.edu Subject: RE: iSCSI over TLS SG, The issue is that if you are not able to decrypt you have no iSCSI packet header (or RDMA headers) and can't place data in memory (i.e. you need anonymous buffers and have to copy). With IPSec you are far better off. Julo "Sukanta Ganguly" 09-11-01 18:08 Please respond to sganguly To: Julian Satran/Haifa/IBM@IBMIL cc: "IPS" Subject: RE: iSCSI over TLS Julian, I agree to the philosophy of framing and steering. With TLS support you can still place incomplete TLS records in host memory, without decrpyting it. Only buffering support needs to be beefed up on the host side. And frankly speaking that it already present with out of order packet deliveries etc. I am not proposing that TLS start acting on the packet immediately as the first peice of the TLS record arrives. The same behavior is observed with iSCSI packets which span TCP packets. The host has to wait for the entire iSCSI packet to be present in the host memory before the iSCSI layer can do anything with it. Do you see my point of argument! ;-) SG *********** REPLY SEPARATOR *********** On 11/9/2001 at 5:46 PM Julian Satran wrote: >The whole reason for doing framing and steering is to be able to start >placing data in host memory without waiting for all the data to arrive. >With TLS data can't be decrypted if pieces are missing. > >Julo > > > > >"Sukanta Ganguly" >09-11-01 17:42 >Please respond to sganguly > > > To: Julian Satran/Haifa/IBM@IBMIL, "IPS" > cc: > Subject: RE: iSCSI over TLS > > > >Julian, > It is correct that TLS records span TCP packets, but how does that >become anymore of a problem. For packets to be resend via the TCP >mechanisms, the sender TLS layers prepares the TLS record and then hands >it over to TCP, TCP may break that TLS layer into e.g. say 5 packets and >sends them to the receiver. If the receiver does not retrieve packet >number 3, it will be resend by the sender. > I did not see the problem that TLS brings into the picture. Also, what >tweaking of the stack are you referring to in this scenario. This is just >general handlinf of packets that are done anyway. iSCSI will only make >sense of the packet after TLS decrypts the packets. Did I miss something >here ??? > >SG > >*********** REPLY SEPARATOR *********** > >On 11/9/2001 at 4:14 PM Julian Satran wrote: > >>Bill, >> >>The one "tiny" item you forgot to mention is that TLS records span TCP >and >>iSCSI PDU boundaries. TLS records can't be decrypted in face of TCP >packet >>loss and markers/alignment can't be recovered (to be more precise require > >>a lot more tweaking of the stacks). >> >>Julo >> >> >> >> >>"Bill Strahm" >>08-11-01 23:55 >>Please respond to "Bill Strahm" >> >> >> To: Julian Satran/Haifa/IBM@IBMIL >> cc: >> Subject: RE: iSCSI over TLS >> >> >> >>Julian, >> >>I do not understand how TLS interferes with delivery of iSCSI packets any >>more than IPsec. In either case your TOE MUST decrypt the packet and >deal >>with the results. I do not see how this changes the problem if the >packet >>is decrypted before going to the TOE (again the hardware to do this MUST >>be >>on the NIC device) or after going through the TOE processing... >>Quick summary of what I think needs to happen >>IPsec >>1) receive L2 packet >>2) determine it is IP >>3) Apply packet policy based on L3 header >>4) Decrypt packet - verify it is covered by the SA >>5) Pass to L4 (TCP) for processing >>6) Verify Framing/etc. >>7) Done >>TLS >>1) Recieve L2 Packet >>2) Pass to L3 >>3) Pass to L4 (TCP) for processing >>4) Decrypt packet >>5) Verify Framing/etc >>6) Done >> >>It turns out the policies for TLS are much simpler than for IPsec, the >>application itself gets to determine if security should be turned on or >>not >>(rather than another application pushing policies into an SPD) and I >don't >>see a difference in the security offload requirements. In many cases TLS >>will go through firewalls/NAT/NATP much better than IPsec, allowing for a >>wider deployment model. >> >> >>Bill Strahm >>+========+=========+=========+=========+=========+=========+=========+ >>Bill Strahm Software Development is a race between Programmers >>Member of the trying to build bigger and better idiot proof software >>Technical Staff and the Universe trying to produce bigger and better >>bill@sanera.net idiots. >>(503) 601-0263 So far the Universe is winning --- Rich Cook >> >>-----Original Message----- >>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of >>Julian Satran >>Sent: Tuesday, November 06, 2001 10:17 PM >>To: ips@ece.cmu.edu >>Subject: Re: iSCSI over TLS >> >> >>Peter, >> >>A group of us seriously considered TLS. The main reason for dropping it >>was that it would interfere with any mechanism we could think of doing >>framing and steering and we thought that framing and steering are >>essential at 10Gbps and over. >> >>Julo >> >> >> >> >>"Peter Mellquist" >>Sent by: owner-ips@ece.cmu.edu >>07-11-01 02:15 >>Please respond to "Peter Mellquist" >> >> >> To: >> cc: >> Subject: iSCSI over TLS >> >> >> >>I am aware that the ips group is leaning toward IPSEC as for the security >>solution but I am interested if anyone is also considering using >Transport >>Layer Security (TLS)? >> >>I am concerned that the requirement for IPSEC might make TOEs more >>complex >>than they need to be. Can TLS be optionally used as well as defined by >the >>specification? This could allow TOE vendors to only be concerned with >>providing normal IPv4 / ipv6 and leave the security to a higher layer. A >>TLS >>stack sitting above the TOE could then handle security very well. Also, I >>anticipate that the first generation of TOEs will not support IPSEC. With >>a >>iSCSI/TLS we could enable security solutions with the first generation of >>TOEs and get speed and security. >> >>Are any TOE vendors planning to support IPSEC? >> >>Can TLS or IPSEC be supported? >> >>-peter >> >> >> >>Peter Mellquist >>Seven Systems Technologies >>575 Menlo Drive Suite 2 >>Rocklin CA >>916-577-1275 >>peterm@seven-systems.com From owner-ips@ece.cmu.edu Sat Nov 10 11:09:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA28982 for ; Sat, 10 Nov 2001 11:09:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAAFNg600439 for ips-outgoing; Sat, 10 Nov 2001 10:23:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f237.law8.hotmail.com [216.33.241.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAA0nkl01871 for ; Fri, 9 Nov 2001 19:49:46 -0500 (EST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Fri, 9 Nov 2001 16:49:40 -0800 Received: from 194.117.133.196 by lw8fd.law8.hotmail.msn.com with HTTP; Sat, 10 Nov 2001 00:49:40 GMT X-Originating-IP: [194.117.133.196] From: "Alison and Ken Thomas" To: ips@ece.cmu.edu Cc: ksandars@eurologic.com Subject: iSCSI:Negotiating a maximum SCSI data phase Date: Sat, 10 Nov 2001 00:49:40 +0000 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 10 Nov 2001 00:49:40.0555 (UTC) FILETIME=[947619B0:01C16981] Sender: owner-ips@ece.cmu.edu Precedence: bulk Evening all, One parameter which is trivial to specify but would make a big difference to the target is to be able to negotiate the maximum SCSI data phase any individual SCSI Command PDU will have associated with it. The suggested words are: 35 MaxSCSIDataPhase Use: LO Who can send: Initiator and Target MaxSCSIDataPhase= Default is 4294967295. Initiator and target negotiate the maximum number of bytes that can be specified in the Expected Data Transfer Length field of a SCSI Command PDU. The lower of the 2 numbers is selected. Basically, the default case is no change to the status quo, but in the field of resource management, this assists the target by knowing what may be thrown at it. It may be beneficial for initiators too as one day when 1GB transfers are the norm, a legacy target can advertise its age and indicate that it stands not a snowflake's chance in Australia of actually servicing such requests. Cheers, Ken Sandars ksandars@eurologic.com Eurologic Systems +44 117 930 9616 _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp From owner-ips@ece.cmu.edu Sat Nov 10 11:09:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA28997 for ; Sat, 10 Nov 2001 11:09:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAAFMmk00390 for ips-outgoing; Sat, 10 Nov 2001 10:22:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAAFMkl00385 for ; Sat, 10 Nov 2001 10:22:47 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA44498 for ; Sat, 10 Nov 2001 16:22:39 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAAFMet20976 for ; Sat, 10 Nov 2001 16:22:40 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Clarification (again) for Task Management Commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 10 Nov 2001 17:22:36 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 10/11/2001 17:22:40, Serialize complete at 10/11/2001 17:22:40 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I think that readers will want to consider 9.4 as an implementation alternative. Julo Black_David@emc.com Sent by: owner-ips@ece.cmu.edu 10-11-01 04:53 Please respond to Black_David To: somesh_gupta@silverbacksystems.com, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Clarification (again) for Task Management Commands Somesh, The language in question reflects fairly direct requirements language to be found in SAM-2's description of SCSI Task Management. FCP goes to serious lengths with FC sequence aborts to make sure this behaves as required. For iSCSI, if responses to the aborted commands show up unexpectedly, they have to be discarded. How the Initiator keeps track of that is the Initiator's problem - keeping track of the CmdSN of the Abort Task Set may be useful. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > Sent: Friday, November 09, 2001 4:55 PM > To: IPS > Subject: iSCSI: Clarification (again) for Task Management Commands > > > Resend to add iSCSI tag. Sorry for missing it. > > On page 67 of the 8-92.txt draft (section 3.5.1), it > says > > "For all the tasks covered by the task > management response (i.e., with CmdSN not higher than the task > management command CmdSN), additional responses MUST NOT be delivered > to the SCSI layer after the task management response." > > If there is a multiple connection session, > a status for a command impacted by the task > management command (say ABORT TASK SET) could > be stuck in the pipe on one connection, while > the ABORT TASK SET completes on another > connection. > > How does the initiator iSCSI enforce the rule above? > Seems to be the equivalent of sending the impacted > commands on other connections in a zombie state, > and not having a very good idea of how to get out. > > Similarly Section 9.4 provides additional rules, > but seems to leave a hole open with regards to > status already in flight on other connections. > > Any clarifications would be appreciated. > > Somesh > From owner-ips@ece.cmu.edu Sat Nov 10 11:12:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29030 for ; Sat, 10 Nov 2001 11:12:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAAFNII00420 for ips-outgoing; Sat, 10 Nov 2001 10:23:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12504.mail.yahoo.com (web12504.mail.yahoo.com [216.136.173.196]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAA0Ial29925 for ; Fri, 9 Nov 2001 19:18:36 -0500 (EST) Message-ID: <20011110001835.68690.qmail@web12504.mail.yahoo.com> Received: from [66.122.195.194] by web12504.mail.yahoo.com via HTTP; Fri, 09 Nov 2001 16:18:35 PST Date: Fri, 9 Nov 2001 16:18:35 -0800 (PST) From: Sukanta ganguly Subject: RE: iSCSI: IPsec tunnel / transport mode decision To: Bill Strahm , Ofer Biran , ips@ece.cmu.edu In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, I was very clear in my email. Let me put this is simple words. I was expecting a MAY for IPSec and a MAY for TLS. A MUST for IPSec i.e. 2401 rules out everything. Seems like IPSec has already been decided already. SG --- Bill Strahm wrote: > I don't understand what you are really asking for... > Do you want both Transport & Tunnel mode to be a MAY > ? > Do you want the option to not have either ? > Do you expect to run Transport mode ESP through a > Tunnel Mode ESP transform > ? > Do you expect to run another security protocol (for > example TLS) ? > > I think we should just say, we require (a MUST) a > 2401 IPsec implementation > (and all the other random IPsec RFCs as well) (This > answers the first three > questions above) > > I think we should allow TLS rather than IPsec (this > has lost a long time in > the WG, so I am pretty much just giving up) (answers > the 4th question) > > Bill > -----Original Message----- > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Sukanta ganguly > Sent: Friday, November 09, 2001 11:07 AM > To: Ofer Biran; ips@ece.cmu.edu > Subject: RE: iSCSI: IPsec tunnel / transport mode > decision > > > By doing this we are forcing IPSec. No flexibility > of > going transport over tunnel. I think we were still > having a discussion of whether transport can also be > supported and hence instead of forcing with IPSec > can't we allow both mechanisms to a MAY. > > In that scenario one could opt for transport mode > with > tunnel and still have a good implementation running. > What do other think? > > SG > > --- Ofer Biran wrote: > > > > It seems that most people prefer tunnel over > > transport mode > > and there is no real opposition for choosing > tunnel > > mode as > > the MUST. In view of that we intend to add it in > > version 09 > > in the following iSCSI statements: > > > > In Section 10.3.1 Data Integrity and > Authentication > > : > > > > "An iSCSI compliant initiator or target MUST > provide > > data > > integrity and authentication by implementing IPSec > > [RFC2401] > > with ESP in tunnel mode [RFC2406] with the > > following..." > > > > And in Section 10.3.2 Confidentiality : > > > > "An iSCSI compliant initiator or target MUST > provide > > confidentiality by implementing IPSec [RFC2401] > with > > ESP in tunnel mode [RFC2406] with the > following..." > > > > Any objection ? > > > > Regards, > > Ofer > > > > > > Ofer Biran > > Storage and Systems Technology > > IBM Research Lab in Haifa > > biran@il.ibm.com 972-4-8296253 > > > > > > "Saqib Jang" on > 01/11/2001 > > 20:03:29 > > > > Please respond to > > > > To: Ofer Biran/Haifa/IBM@IBMIL, > > > cc: > > Subject: RE: iSCSI: IPsec tunnel / transport mode > > decision > > > > > > > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Ofer Biran > > Sent: Thursday, November 01, 2001 4:31 AM > > To: ips@ece.cmu.edu > > Subject: iSCSI: IPsec tunnel / transport mode > > decision > > > > > > I'd like to drive this open issue into group > > consensus. It seems to > > me that the tendency was more toward making tunnel > > mode a MUST as iFCP > > and FCIP did, mainly due the option of integrating > > an existing IPsec > > chip/box with the iSCSI implementation offering. > If > > we reach this decision, > > we may choose even not to mention transport mode > (as > > MAY or some other > > recommending text). > > > > There is an excellent analysis made by Bernard > Aboba > > in Section > > "5.1. Transport mode versus tunnel mode" of > > draft-ietf-ips-security-04 > > ( > > > http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt > > ) > > that can help us with this decision (also Section > > "5.2. NAT traversal" is > > relevant). > > > > Regards, > > Ofer > > > > Ofer Biran > > Storage and Systems Technology > > IBM Research Lab in Haifa > > biran@il.ibm.com 972-4-8296253 > > > > > > > > > > > __________________________________________________ > Do You Yahoo!? > Find a job, post your resume. > http://careers.yahoo.com > __________________________________________________ Do You Yahoo!? Find a job, post your resume. http://careers.yahoo.com From owner-ips@ece.cmu.edu Sat Nov 10 11:12:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29041 for ; Sat, 10 Nov 2001 11:12:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAAFRlp00704 for ips-outgoing; Sat, 10 Nov 2001 10:27:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAAFRjl00698 for ; Sat, 10 Nov 2001 10:27:45 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA69128 for ; Sat, 10 Nov 2001 16:27:39 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAAFRct25128 for ; Sat, 10 Nov 2001 16:27:38 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Clarification (again) for Task Management Commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 10 Nov 2001 17:27:35 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 10/11/2001 17:27:38, Serialize complete at 10/11/2001 17:27:38 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk ----- Forwarded by Julian Satran/Haifa/IBM on 10-11-01 17:28 ----- Julian Satran 10-11-01 17:26 To: cc: From: Julian Satran/Haifa/IBM@IBMIL Subject: RE: iSCSI: Clarification (again) for Task Management Commands Read 9.4 for one implementation - Julo "Somesh Gupta" Sent by: owner-ips@ece.cmu.edu 10-11-01 05:26 Please respond to somesh_gupta To: , cc: Subject: RE: iSCSI: Clarification (again) for Task Management Commands David, In iSCSI the multiple connection/session construct adds significant complexity in determining whether a response for a command (on a different connection than the one on which the task management command was sent) impacted by the task management command will be received or not - and when? On a single connection, or similar links, when the response for the task management command is received (or after a fixed additional time), no responses will be received for the commands aborted by the task management command. However, with multiple connections there is no such "flushing" event on connections other than the one on which the task management command was sent. I would hope that the protocol would address this situation - the current response seems to be be to put the task tag and some associated resources in a zombie state for an unspecified amount of time. Somesh > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, November 09, 2001 6:53 PM > To: somesh_gupta@silverbacksystems.com; ips@ece.cmu.edu > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > Somesh, > > The language in question reflects fairly direct requirements > language to be found in SAM-2's description of SCSI Task Management. > FCP goes to serious lengths with FC sequence aborts to make sure > this behaves as required. > > For iSCSI, if responses to the aborted commands show up unexpectedly, > they have to be discarded. How the Initiator keeps track of that > is the Initiator's problem - keeping track of the CmdSN of the > Abort Task Set may be useful. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > > -----Original Message----- > > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > > Sent: Friday, November 09, 2001 4:55 PM > > To: IPS > > Subject: iSCSI: Clarification (again) for Task Management Commands > > > > > > Resend to add iSCSI tag. Sorry for missing it. > > > > On page 67 of the 8-92.txt draft (section 3.5.1), it > > says > > > > "For all the tasks covered by the task > > management response (i.e., with CmdSN not higher than the task > > management command CmdSN), additional responses MUST NOT be delivered > > to the SCSI layer after the task management response." > > > > If there is a multiple connection session, > > a status for a command impacted by the task > > management command (say ABORT TASK SET) could > > be stuck in the pipe on one connection, while > > the ABORT TASK SET completes on another > > connection. > > > > How does the initiator iSCSI enforce the rule above? > > Seems to be the equivalent of sending the impacted > > commands on other connections in a zombie state, > > and not having a very good idea of how to get out. > > > > Similarly Section 9.4 provides additional rules, > > but seems to leave a hole open with regards to > > status already in flight on other connections. > > > > Any clarifications would be appreciated. > > > > Somesh > > > From owner-ips@ece.cmu.edu Sat Nov 10 11:12:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29052 for ; Sat, 10 Nov 2001 11:12:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAAFjJt01778 for ips-outgoing; Sat, 10 Nov 2001 10:45:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAAFjIl01774 for ; Sat, 10 Nov 2001 10:45:18 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA125814 for ; Sat, 10 Nov 2001 16:45:11 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAAFjBt42546 for ; Sat, 10 Nov 2001 16:45:12 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI:Negotiating a maximum SCSI data phase X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 10 Nov 2001 17:45:08 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 10/11/2001 17:45:12, Serialize complete at 10/11/2001 17:45:12 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ken, It looks to me that this is realy a SCSI issue and as such it should be handled by T10. iSCSI is not the end-source/end-sink and as far as stating its own limitations it has all it needs in MaxBurstSize/FirstBurstSize and MaxRcvPDU. Julo "Alison and Ken Thomas" Sent by: owner-ips@ece.cmu.edu 10-11-01 02:49 Please respond to "Alison and Ken Thomas" To: ips@ece.cmu.edu cc: ksandars@eurologic.com Subject: iSCSI:Negotiating a maximum SCSI data phase Evening all, One parameter which is trivial to specify but would make a big difference to the target is to be able to negotiate the maximum SCSI data phase any individual SCSI Command PDU will have associated with it. The suggested words are: 35 MaxSCSIDataPhase Use: LO Who can send: Initiator and Target MaxSCSIDataPhase= Default is 4294967295. Initiator and target negotiate the maximum number of bytes that can be specified in the Expected Data Transfer Length field of a SCSI Command PDU. The lower of the 2 numbers is selected. Basically, the default case is no change to the status quo, but in the field of resource management, this assists the target by knowing what may be thrown at it. It may be beneficial for initiators too as one day when 1GB transfers are the norm, a legacy target can advertise its age and indicate that it stands not a snowflake's chance in Australia of actually servicing such requests. Cheers, Ken Sandars ksandars@eurologic.com Eurologic Systems +44 117 930 9616 _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp From owner-ips@ece.cmu.edu Sat Nov 10 12:12:19 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA29623 for ; Sat, 10 Nov 2001 12:12:19 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAAGG4R03521 for ips-outgoing; Sat, 10 Nov 2001 11:16:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAAGG2l03517 for ; Sat, 10 Nov 2001 11:16:02 -0500 (EST) Received: from hq-ex-c2.brocade.com (hq-bes-c2 [192.168.126.35]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id IAA22028; Sat, 10 Nov 2001 08:15:47 -0800 (PST) Received: by hq-bes-c2.brocade.com with Internet Mail Service (5.5.2653.19) id ; Sat, 10 Nov 2001 08:15:47 -0800 Message-ID: From: Robert Snively To: Black_David@emc.com, rsnively@Brocade.COM, ips@ece.cmu.edu Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Sat, 10 Nov 2001 08:15:46 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Let me try this again. (E-mail sure is a slow medium of information exchange. We need some more face-to-face meetings on this subject.) If secure IPSec connections can be made, I will use those connections. The information about the boxes and the permissions to connect them are part of the administrative set up necessary to bring them into a secure environment. If that is an exposed connection and carries information that is important enough to be worth protecting, some degree of encryption will be used on ALL data transmitted on the connection, including the initial information passed to the opposite end point. If you did that, the WWN would be never be publicly available on the network, but only available through the administrative mechanisms (like walking up to the device through your bullet proof glass doors and examining its labels). If you truly truly cared about the connection, you would have gone to the trouble of using keys (perhaps pre-established) that were not group pre-shared keys. A lot of this is of the nature "if it hurts, don't do it (unless it is worth the pain)". Who would allow his SAN to pass important data without providing security mechanisms appropriate to the value of the data? If you care, you would not use group pre-shared keys, but some other kind of keying. If you care, you would use confidentiality protection. It really does not make much sense to criticize a security approach by making the assumption that you are going to throw away your first lines of defense. Since the identity of the box is very much part of the same administrative setup, and because (in the latest revision of our proposal) the forward information exchange to the remote point includes both the known source and the expected destination, either side has the right to terminate the connection if the information is incorrect. If all the administrative information necessary to form a secure connection and all the administrative information necessary to control access to FCIP devices is known to a device, it is by definition authorized to perform the connection. You must deny untrustworthy devices some part of the necessary information for any security program to work. As the first connection is formed, it indicates what type of information may flow across it. Typically, the first connection will permit only class F information (and no other connection will permit class F information). Any attempt to create a second class F information connection will terminate the entire set of connections, since it is a protocol violation as well as a security violation. You are correct in saying that the first is the connection that will perform the SLAP authentication. Note that some part of this protocol is specified by T11 in FC-BB-2, not in FCIP. Subsequent connections can only be created from the same device and are assigned classes of information and other usage characteristics by that same device. If any other device tries to do so, and expects to be successful, it must have all the administrative information necessary to create a second secure connection and must have knowledge of what connections the first device's previous connections and FC information exchanges would permit to be set up. And if it has all that information, it is by definition permitted to establish the connection. I really don't see how you can create a bothersome second connection unless you have chosen to allow violation of the security of your administration procedures. And if you allow violation of the security of your administration procedures, I don't see how you can prevent a malicious connection using any mechanism at all. > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, November 09, 2001 5:52 PM > To: rsnively@brocade.com; ips@ece.cmu.edu > Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA > Interim meeting) > > > > For those cases where a secure environment is required, the > > new connection comes up through the normal IPSec authentication > > and encryption processes. As a result, the transmission of > > the identifying world wide names is already transmitted in an > > encrypted format, creatable only by the authenticated and > > certified source and interpretable only by the authenticated > > and certified destination. > > The assumption that there's a big on/off switch for > security is not correct (e.g., suppose IPsec is running > with integrity on and confidentiality off) and > "authenticated and certified" is a peculiar concept for > group pre-shared keys, a very likely deployment scenario. > Also, IPsec has no clue about the WWN, and hence the IKE > authentication is not linked to the WWN that has to be > presented (a particular problem with group pre-shared keys, > but also a risk even with certificates). In other words, > "authenticated and certified" doesn't place any restrictions > on what WWN is presented. Finally, solving > an inband authentication problem by requiring that IPsec be > used to encrypt the WWN is wildly excessive, and besides, > the WWN has no secret contents- it's a public piece of > configuration information. > > There is a solution in this general area, but it involves > linking the WWN to the IKE identity that is authenticated. > That's probably going to break any attempt to use existing > off-the-shelf external IPsec gateways because gateways > don't grok WWNs, and the FCIP implementations won't > know what identity was used in the IKE authentication > to the gateway. This is all a bit peculiar because FC > currently trust WWNs passed in various FC Login frames > without authenticating them, but that's T11's area of > responsibility putting this sort of thing into an IETF > standard isn't going to be acceptable. > > > If the Fibre Channel fabric is also operating in a secure mode, > > subsequent Fibre Channel authentication and certification > > is performed using the standard FC SLAP mechanisms. > > Only on the first TCP connection, unless you plan to require > taking the FCIP link down when the second TCP connection > is added (which strikes me as highly counterproductive). > The second connection just inherits the results of the SLAP > performed on the first one, whether its entitled to or not. > > > In addition to this, there are a whole bunch of policy > > restrictions that are verified as part of the creation > > of subsequent connections. While these are not necessarily > > part of the security steps, they prevent the formation of > > connections which do not meet the security rules. > > I'm not sure what you're referring to, but I suspect they > don't help much here. > > > Assuming security mechanisms are properly implemented, where, > > then, is the security hole? > > The fact that the WWN is not authenticated means that anyone > who can set up an FCIP connection to an FCIP entity can tap > into any existing FC traffic flowing on any connection through > that FCIP entity. > > --David > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@brocade.com] > > Sent: Friday, November 09, 2001 8:06 PM > > To: 'Black_David@emc.com'; ENDL_TX@computer.org; ips@ece.cmu.edu > > Cc: Robert Snively > > Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA > > Interim meeting) > > > > > > > > David, > > > > For those cases where a secure environment is required, the > > new connection comes up through the normal IPSec authentication > > and encryption processes. As a result, the transmission of > > the identifying world wide names is already transmitted in an > > encrypted format, creatable only by the authenticated and > > certified source and interpretable only by the authenticated > > and certified destination. > > > > If the Fibre Channel fabric is also operating in a secure mode, > > subsequent Fibre Channel authentication and certification > > is performed using the standard FC SLAP mechanisms. > > > > In addition to this, there are a whole bunch of policy > > restrictions that are verified as part of the creation > > of subsequent connections. While these are not necessarily > > part of the security steps, they prevent the formation of > > connections which do not meet the security rules. > > > > Assuming security mechanisms are properly implemented, where, > > then, is the security hole? > > > > Bob Snively e-mail: rsnively@brocade.com > > Brocade Communications Systems phone: 408 487 8135 > > 1745 Technology Drive > > San Jose, CA 95110 > > > > > > > > > > Those who were at the Irvine Interim meeting will remember that > > > > the problem with FCIP and NAPTS is a reliance on IP address in > > > > the determination of which incoming TCP connections belong in a > > > > given FCIP Link. This proposal solves that problem by requiring > > > > that FC Entity World Wide Name be transmitted in the first bytes > > > > sent by the FCIP Entity that initiates a TCP Connect request. > > > > This allows the FCIP Entity that receives a TCP Connect request > > > > to match it with any previously received TCP Connect requests > > > > from the same source. Since the transmitted World Wide Name is > > > > required to be unique within Fibre Channel, the FCIP Entity > > > > that receives this information can correctly assign FCIP Link > > > > relationships without relying on IP Addresses. > > > > > > From a functional standpoint, this works, but it opens up > a security > > > issue. The problem is that on the second TCP connection (and > > > subsequent > > > connections) that claim to be from the same FCIP Entity, > > the WWN that > > > is initially sent (and whatever extension is used) is functioning > > > as an authentication to allow that connection to join the first > > > TCP connection, but that authentication is unsecured -- the sender > > > announces the WWN, and the receiver does not (and has no way to) > > > check it. > > > > > > There's a fairly obvious denial of service attack here involving > > > the attacker joining a new connection to an existing one > > > and then bit-bucketing all the frames sent over the new > connection. > > > > > > Limiting FCIP to one TCP connection among any pair of FCIP entity > > > identifiers would help, but is not sufficient. The attack > > of concern > > > in this situation involves the attacker crashing the real entity > > > and opening up a connection in its name, thereby locking out the > > > real entity when the real entity restarts. > > > > > > This may be headed in the direction of needing in-band > > authentication > > > which I know the FCIP community has been doing their best > to avoid. > > > > > > Sorry to be the bearer of bad news, > > > --David > > > From owner-ips@ece.cmu.edu Sat Nov 10 12:16:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA29652 for ; Sat, 10 Nov 2001 12:16:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAAGlGq05350 for ips-outgoing; Sat, 10 Nov 2001 11:47:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar2aa.compuserve.com (siaar2aa.compuserve.com [149.174.40.137]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAAGlDl05341 for ; Sat, 10 Nov 2001 11:47:13 -0500 (EST) Received: (from mailgate@localhost) by siaar2aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id LAA08072 for ips@ece.cmu.edu; Sat, 10 Nov 2001 11:47:08 -0500 (EST) Received: from compuserve.com (dal-tgn-tkq-vty30.as.wcom.net [216.192.228.30]) by siaar2aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id LAA07990; Sat, 10 Nov 2001 11:47:00 -0500 (EST) Message-ID: <3BED5B42.2A9A283C@compuserve.com> Date: Sat, 10 Nov 2001 10:52:18 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector CC: Black_David@emc.com Subject: Re: FCencap: List ALL SOF/EOF codes References: <277DD60FB639D511AC0400B0D068B71E02937151@CORPMX14> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) prohibits Class 1 and I can find no letter ballot comments asking that it be reinstated. Therefore, I am forced to agree with David. Class 1 MUST NOT be mentioned in the FC Encapsulation draft. If necessary, a note discussing interoperability and FC-MI can be added. Thanks. Ralph... Black_David@emc.com wrote: > FC-MI was going to prohibit Class 1 last time I checked. Since the > I in FC-MI stands for "Interoperability", this seems like a reasonable > rationale for excluding Class 1 service. > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ece.cmu.edu Sun Nov 11 05:23:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA23384 for ; Sun, 11 Nov 2001 05:23:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAB9dYa11543 for ips-outgoing; Sun, 11 Nov 2001 04:39:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAB9dWl11538 for ; Sun, 11 Nov 2001 04:39:32 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA04684; Sun, 11 Nov 2001 10:39:24 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAB9dOI76770; Sun, 11 Nov 2001 10:39:24 +0100 Importance: Normal Subject: RE: iSCSI: IPsec tunnel / transport mode decision To: "Herbert, Howard C" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Ofer Biran" Date: Sun, 11 Nov 2001 11:40:40 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 11/11/2001 11:39:25 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Howard, In my opinion we only need a MUST on one mode and algorithm to ensure interoperability. This doesn't rule out implementation and use of transport mode. I don't think putting a MAY is necessary - the only reason for MAY (RFC2119) is forcing each side to expect that the other side might offer this option. Transport mode offering is well defined in the IKE negotiation and the responder chooses the desired SA, a one which it supports of course, or denies. So I think we have an implicit MAY for transport mode and any other (and future) algorithm by the IKE negotiation and by not saying MUST NOT or SHOULD NOT use. Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Herbert, Howard C" on 09/11/2001 22:24:43 Please respond to "Herbert, Howard C" To: Ofer Biran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision Does this mean that transport mode will be a "MAY" or a "SHOULD"? Howard -----Original Message----- From: Ofer Biran [mailto:BIRAN@il.ibm.com] Sent: Friday, November 09, 2001 10:54 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: IPsec tunnel / transport mode decision It seems that most people prefer tunnel over transport mode and there is no real opposition for choosing tunnel mode as the MUST. In view of that we intend to add it in version 09 in the following iSCSI statements: In Section 10.3.1 Data Integrity and Authentication : "An iSCSI compliant initiator or target MUST provide data integrity and authentication by implementing IPSec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." And in Section 10.3.2 Confidentiality : "An iSCSI compliant initiator or target MUST provide confidentiality by implementing IPSec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." Any objection ? Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Saqib Jang" on 01/11/2001 20:03:29 Please respond to To: Ofer Biran/Haifa/IBM@IBMIL, cc: Subject: RE: iSCSI: IPsec tunnel / transport mode decision -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ofer Biran Sent: Thursday, November 01, 2001 4:31 AM To: ips@ece.cmu.edu Subject: iSCSI: IPsec tunnel / transport mode decision I'd like to drive this open issue into group consensus. It seems to me that the tendency was more toward making tunnel mode a MUST as iFCP and FCIP did, mainly due the option of integrating an existing IPsec chip/box with the iSCSI implementation offering. If we reach this decision, we may choose even not to mention transport mode (as MAY or some other recommending text). There is an excellent analysis made by Bernard Aboba in Section "5.1. Transport mode versus tunnel mode" of draft-ietf-ips-security-04 ( http://www.ietf.org/internet-drafts/draft-ietf-ips-security-04.txt ) that can help us with this decision (also Section "5.2. NAT traversal" is relevant). Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Sun Nov 11 15:36:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29214 for ; Sun, 11 Nov 2001 15:36:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fABJib013540 for ips-outgoing; Sun, 11 Nov 2001 14:44:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from NetworkRelay ([63.119.175.39]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAB1UDl05049 for ; Sat, 10 Nov 2001 20:30:13 -0500 (EST) Received: from quicksall.com [63.119.175.45] by NetworkRelay with ESMTP (SMTPD32-7.00) id A33018140042; Sat, 10 Nov 2001 19:24:00 -0600 Received: from eddydesktop [24.61.64.49] by quicksall.com with ESMTP (SMTPD32-6.06) id A49C15E700F2; Sat, 10 Nov 2001 19:30:04 -0600 From: "Eddy Quicksall" To: "ips@ece. cmu. edu \(E-mail\)" Subject: test message Date: Sat, 10 Nov 2001 20:30:07 -0500 Message-ID: <006c01c16a50$66da1220$0202a8c0@eddydesktop> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_006D_01C16A26.7E040A20" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_006D_01C16A26.7E040A20 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit This is a test to see if this is getting through. Eddy ------=_NextPart_000_006D_01C16A26.7E040A20 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

This is a test to see if this is getting = through.

 

Eddy<= /span>

------=_NextPart_000_006D_01C16A26.7E040A20-- From owner-ips@ece.cmu.edu Sun Nov 11 16:45:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29705 for ; Sun, 11 Nov 2001 16:45:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fABKsM917473 for ips-outgoing; Sun, 11 Nov 2001 15:54:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls20.mediaone.net (chmls20.mediaone.net [24.147.1.156]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fABKsKl17467 for ; Sun, 11 Nov 2001 15:54:20 -0500 (EST) Received: from cs.uchicago.edu (h005018030f43.ne.mediaone.net [66.31.16.32]) by chmls20.mediaone.net (8.11.1/8.11.1) with ESMTP id fABKtMx19209 for ; Sun, 11 Nov 2001 15:55:22 -0500 (EST) Message-Id: <200111112055.fABKtMx19209@chmls20.mediaone.net> To: ips@ece.cmu.edu Subject: Re: iSCSI over TLS In-Reply-To: Message from "Bill Strahm" of "Fri, 09 Nov 2001 10:40:29 PST." References: Date: Sun, 11 Nov 2001 15:52:23 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, > I believe that there are also TLS options to do per frame crypto, however at > that point you have to start shipping over IVs with each frame, removing > most wire advantages of TLS over IPsec... You have a reference for this? Sounds interesting, but I've never heard of it before (which means nothing). Thanks, Steph From owner-ips@ece.cmu.edu Sun Nov 11 16:45:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29716 for ; Sun, 11 Nov 2001 16:45:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fABLOZk19291 for ips-outgoing; Sun, 11 Nov 2001 16:24:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls16.mediaone.net (chmls16.mediaone.net [24.147.1.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fABLOYl19287 for ; Sun, 11 Nov 2001 16:24:34 -0500 (EST) Received: from cs.uchicago.edu (h005018030f43.ne.mediaone.net [66.31.16.32]) by chmls16.mediaone.net (8.11.1/8.11.1) with ESMTP id fABLOXT05082 for ; Sun, 11 Nov 2001 16:24:33 -0500 (EST) Message-Id: <200111112124.fABLOXT05082@chmls16.mediaone.net> To: ips@ece.cmu.edu Subject: Re: Querry ! In-Reply-To: Message from Kaushik Datta of "Fri, 09 Nov 2001 20:51:00 +0530." <3BEBF45C.6E6224CC@npd.hcltech.com> References: <3BEBF45C.6E6224CC@npd.hcltech.com> Date: Sun, 11 Nov 2001 16:22:40 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk Kaushik, I'm not quite sure what your proposing here, but something seems wrong. Just fyi, I have worked on several CAM-based initiators (who was it that was saying CAM was never implemented?). You have mentioned `iSCSI target stack' several times. Are you trying to make FreeBSD behave like a storage target? If so, you should probably not be using CAM at all. CAM is about implementing initiators. It does define a target mode path, but it is relatively limited in scope. CAM target mode is good for implementing a SCSI processor target (for things like cluster communication and bus pinging), but I don't think it was intended for a richer target (e.g. a storage target). It's defined for connectivity, but not performance in the target role. If you are trying to make a FreeBSD iSCSI initiator, then what your saying here: > won't it be feasible if we can write a pseudo driver that takes care > of the fetching the scsi cdb as such and passing it on to the SIM > driver ? In a way we will get away from the PM functionality which > forms the scsi CDB in case of a normal I/O. doesn't make sense to me. You don't want to `get away' from the periperal drivers, since the are your ultimate clients. The whole design of CAM is to allow peripheral drivers (disk, tape, etc, for the unCAM-oriented) and SIMs (SCSI Interface Modules, aka adapter drivers) to be independent so that the same PDs work no matter what SIM. To make a FreeBSDI iSCSI initiator, you need to write an iSCSI SIM. To do that, you probably shouldn't start with an iSCSI target stack. One trick doing this with CAM is that it didn't really anticipate the different addressing characteristics of the various SCSI transports (a problem not unique to CAM). CAM was designed anticipating lots of different ||SCSI adapters, rather than adapters with completely different transports. The standard solution to this (done in FCP, for example), is to store some transport-specific translation state in your SIM to map, say, target xx to a particular transport address (an FC WWN, or iSCSI socket address). Another trick will be getting to the network stack from the SCSI stack, and vice versa. Not impossible, but you may have to be careful calling the network stack from the storage stack, as they're typically in different synchronization domains. Sometimes trying to get synchronized on both domains simultaneously can lead to surprising deadlocks. Steph From owner-ips@ece.cmu.edu Sun Nov 11 16:54:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29807 for ; Sun, 11 Nov 2001 16:54:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fABKxQv17765 for ips-outgoing; Sun, 11 Nov 2001 15:59:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from chmls20.mediaone.net (chmls20.mediaone.net [24.147.1.156]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fABKxKl17759 for ; Sun, 11 Nov 2001 15:59:25 -0500 (EST) Received: from cs.uchicago.edu (h005018030f43.ne.mediaone.net [66.31.16.32]) by chmls20.mediaone.net (8.11.1/8.11.1) with ESMTP id fABL0Fx24610 for ; Sun, 11 Nov 2001 16:00:16 -0500 (EST) Message-Id: <200111112100.fABL0Fx24610@chmls20.mediaone.net> To: ips@ece.cmu.edu Subject: Re: iSCSI over TLS In-Reply-To: Message from Sukanta ganguly of "Fri, 09 Nov 2001 11:20:47 PST." <20011109192047.75482.qmail@web12501.mail.yahoo.com> References: <20011109192047.75482.qmail@web12501.mail.yahoo.com> Date: Sun, 11 Nov 2001 15:57:20 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk Sukanta, > a TLS record would have the entire iSCSI packet. How do you figure? TLS has exactly the same stream semantics as TCP. A ULP has no control over how many octets go in a TLS record. More importantly, even if a sender TLS implementation does control this (agreed you could build an implementation which provided this property), a receiver can not assume the property, since the recordization (like segmentation) is a free choice of the TLS layer. Steph From owner-ips@ece.cmu.edu Sun Nov 11 19:01:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA00908 for ; Sun, 11 Nov 2001 19:01:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fABNBBf25912 for ips-outgoing; Sun, 11 Nov 2001 18:11:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fABNB9l25903 for ; Sun, 11 Nov 2001 18:11:09 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id PAA29694; Sun, 11 Nov 2001 15:10:57 -0800 (PST) From: Keith McCloghrie Message-Id: <200111112310.PAA29694@cisco.com> Subject: Re: FC Management MIB - proposed changes To: Black_David@emc.com Date: Sun, 11 Nov 2001 15:10:57 -0800 (PST) Cc: kzm@cisco.com, ips@ece.cmu.edu In-Reply-To: <277DD60FB639D511AC0400B0D068B71ECAD8FB@CORPMX14> from "Black_David@emc.com" at Nov 08, 2001 03:51:23 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > For simplicity, I would propose keeping the initial revision focused > on structural and format changes only - i.e., make no functional > changes that aren't required to bring the MIB into line with the > overall architecture of MIBs, use of SMIv2, and the like. Agreed. Even so, almost everything in the MIB will be affected. > Comments on this, keyed to Keith's issue numbers: > > 1. The trap table doesn't match up with RFC 2573 - I presume > correcting this is part of the conversion to SMIv2. With SNMPv2c/SNMPv3, a Trap became one of the two forms by which a notification is transmitted; the other form is an InformRequest. SMIv2 MIBs don't define traps - they define notifications (with the NOTIFICATION-TYPE macro). With SNMPv2c/SNMPv3, agents generate notifications, not traps. The destinations to which a notification is sent is determined by the tables in the two MIBs defined in RFC 2573: the SNMP-NOTIFICATION-MIB and SNMP-TARGET-MIB. The snmpNotifyType object in RFC 2573 determines in which form the notification is tranmsitted to a destination; it's possible for the same notification generated by an agent to be sent as a trap to one destination, and as an inform to another destination. Thus, the trap table is a relic of the SNMPv1-only days; it is: a) insufficient for SNMPv2c/SNMPv3, and b) a duplicate of a subset of RFC 2573. It should be deleted without replacement. > 3. Let's keep the replacement of RFC 2837 as (a) a proposal > and (b) Keith's recommendation to the WG as MIB Advisor, > but not formally adopt that approach until we have a version > of the MIB draft that would replace RFC 2837 available for > review by the WG. We can postpone the decision on what happens to 2837. However, 2837 contains information on FC interfaces. The FC-MGMT MIB also contains information on FC interfaces. Most of the information will be the same, and should be defined once; I propose in the new FC-MGMT MIB. There are a few pieces of information which are specific to Fabric Elements. I'll include those in the FC-MGMT MIB for the time being, until the postponed decision is made. > 5. I think there's a misunderstanding here. There is only one > name service in any FC fabric, period. The term "Simple Name > Service" is historical and refers to the current approach > to Fibre Channel fabric name service by contrast to the > originally-proposed X.500 directory-based approach (see the > original FC-GS) - the meaning of "Simple" should now be > obvious, as it's by comparison to X.500 ;-). > > Both the Zoned and Unzoned name services are simple name services. > The same objects can be used to represent both, as only one is > active at any time, and the communication interfaces/protocols > are identical. Client access to the name service is the same > in both cases; zoned vs. unzoned only affects server behavior > in terms of what is returned to a query. In addition the name > service objects are described to represent only entities > "known to this unit". So, in a zoned fabric, I would expect > the table in a switch to be completely populated, but the table > in an HBA to contain only the entries in that HBA's zone > because the HBA doesn't "know" about any others. > > The name server objects need to be retained - these are quite > important for fabric management visibility. Whether the fabric > is zoned or not and how the nameserver behaves can be determined > from the zone objects (i.e., for a switch, the nameserver objects > contain everything and the zone objects have to be consulted to > figure out what a query from a particular client to the nameserver > will return). OK. > 7. I would definitely take out Class 1 support, and reference FC-MI > (which prohibits Class 1 service) as an authority for doing so. OK. > 8. For the initial version, I would only carry forward the zone > objects existing in the current MIB and not define any new > functionality - that can be done in revisions after -00. OK. > 9. I'd prefer that the -00 version contain no additional functional > changes beyond those required to support the necessary > structure and format changes. Once we have that -00 baseline, > functional changes can be made from there. OK. > 2, 4, and 6 look fine to me, as does the new draft name. Thnaks, Keith. From owner-ips@ece.cmu.edu Sun Nov 11 19:08:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA00988 for ; Sun, 11 Nov 2001 19:08:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fABNc7X27688 for ips-outgoing; Sun, 11 Nov 2001 18:38:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fABNc5l27683 for ; Sun, 11 Nov 2001 18:38:05 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id PAA00811; Sun, 11 Nov 2001 15:37:58 -0800 (PST) From: Keith McCloghrie Message-Id: <200111112337.PAA00811@cisco.com> Subject: Re: FC Management MIB - proposed changes To: predrag_spasic@hp.com ("SPASIC,PREDRAG (HP-Roseville,ex1)") Date: Sun, 11 Nov 2001 15:37:58 -0800 (PST) Cc: kzm@cisco.com ('Keith McCloghrie'), ips@ece.cmu.edu In-Reply-To: <028FE7141C79D511B65100D0B74FE875BCA5C7@xrose01.rose.hp.com> from "SPASIC,PREDRAG (HP-Roseville,ex1)" at Nov 08, 2001 02:09:12 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > Use Counter64 only if necessary. Counter32 would be the > preferred solution. Right. The SMI (RFC 2578) says: A requirement on "standard" MIB modules is that the Counter64 type may be used only if the information being modeled would wrap in less than one hour if the Counter32 type was used instead. > -There are no Counter64s in FC standards that we need in the MIB A Counter32 which counts octets at 1Gbs can wrap in about 34 seconds !! If we assume the minimum packet size is approx 50 bytes (e.g., RFC 2625 says that an FC frame containing an ARP packet is 52 bytes long), then a Counter32 which counts packets at 1Gbs can wrap in 28 minutes. Thus, every counter which can increment at a sustained rate of at least one increment per packet needs to be a Counter64. Note that it's common that instrumentation (e.g., hardware) doesn't keep 64-bit counters; in such cases, the SNMP agent needs to maintain the Counter64 variable by polling the instrumentation at whatever rate is necessary to avoid loss of information. > -Some existing tools have problems with Counter64 and Unsigned64 Such tools should have been upgraded to support SNMPv2c/SNMPv3 long ago. > -No Gauge64 in SMIv2 See RFC 2856. > > I assume you mean the inadequate security of SNMPv1 and SNMPv2c, > > i.e., nobody perceives that SNMPv3 is insecure, right ?? > Yes, SNMPv3 is secure, but not that many implementations. Actually, there are lots of implementations. I believe the issue is how many customers feel security is important enough to deploy SNMPv3, and in today's climate, I suspect that number is rising. Keith. From owner-ips@ece.cmu.edu Sun Nov 11 19:09:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01000 for ; Sun, 11 Nov 2001 19:09:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fABNRig26979 for ips-outgoing; Sun, 11 Nov 2001 18:27:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from NetworkRelay ([63.119.175.39]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fABNRgl26974 for ; Sun, 11 Nov 2001 18:27:42 -0500 (EST) Received: from quicksall.com [63.119.175.45] by NetworkRelay with ESMTP (SMTPD32-7.00) id A7F6135D0044; Sun, 11 Nov 2001 17:21:26 -0600 Received: from eddydesktop [24.61.64.49] by quicksall.com with ESMTP (SMTPD32-6.06) id A961C6E0036; Sun, 11 Nov 2001 17:27:29 -0600 From: "Eddy Quicksall" To: "ips@ece. cmu. edu \(E-mail\)" Subject: test message Date: Sun, 11 Nov 2001 18:27:30 -0500 Message-ID: <000501c16b08$72ed28a0$0202a8c0@eddydesktop> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0006_01C16ADE.8A1720A0" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0006_01C16ADE.8A1720A0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Test message to see if I am getting through. ------=_NextPart_000_0006_01C16ADE.8A1720A0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Test message to see if I am getting = through.

 

------=_NextPart_000_0006_01C16ADE.8A1720A0-- From owner-ips@ece.cmu.edu Mon Nov 12 00:26:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA06269 for ; Mon, 12 Nov 2001 00:26:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAC4YOf15069 for ips-outgoing; Sun, 11 Nov 2001 23:34:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAC4YNl15065 for ; Sun, 11 Nov 2001 23:34:23 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by crufty; Sun Nov 11 23:30:00 EST 2001 Received: from aura.research.bell-labs.com (aura.research.bell-labs.com [135.104.46.10]) by grubby.research.bell-labs.com (8.11.6/8.11.6) with ESMTP id fAC4YKo34835; Sun, 11 Nov 2001 23:34:20 -0500 (EST) Received: (from sandeepj@localhost) by aura.research.bell-labs.com (8.9.1/8.9.1) id XAA11227; Sun, 11 Nov 2001 23:34:20 -0500 (EST) Date: Sun, 11 Nov 2001 23:34:20 -0500 (EST) Message-Id: <200111120434.XAA11227@aura.research.bell-labs.com> From: sandeepj@research.bell-labs.com (Sandeep Joshi) To: kdatta@npd.hcltech.com, steph@cs.uchicago.edu Cc: ips@ece.cmu.edu Subject: Re: iSCSI Querry ! Sender: owner-ips@ece.cmu.edu Precedence: bulk (added iSCSI to the subject line..) Steph, I believe what is being suggested is similar to running nfsd in the kernel and making it hook up directly to the native filesystem layer. Its feasible provided you can solve all the associated problems ... N initiators, N connections, N LUNs, etc :-) Kaushik, > we would like to know if any group is working on > implementing iscsi stack on FreeBSD. yes, we have a FreeBSD-based iSCSI stack for target as well as initiator. Some of it may be released to the community in the near future. -Sandeep > Kaushik, > > > won't it be feasible if we can write a pseudo driver that takes care > > of the fetching the scsi cdb as such and passing it on to the SIM > > driver ? In a way we will get away from the PM functionality which > > forms the scsi CDB in case of a normal I/O. > > I'm not quite sure what your proposing here, but something seems > wrong. Just fyi, I have worked on several CAM-based initiators (who > was it that was saying CAM was never implemented?). > > You have mentioned `iSCSI target stack' several times. Are you trying > to make FreeBSD behave like a storage target? If so, you should > probably not be using CAM at all. CAM is about implementing > initiators. It does define a target mode path, but it is relatively > limited in scope. CAM target mode is good for implementing a SCSI > processor target (for things like cluster communication and bus > pinging), but I don't think it was intended for a richer target > (e.g. a storage target). It's defined for connectivity, but not > performance in the target role. > From owner-ips@ece.cmu.edu Mon Nov 12 08:24:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA28234 for ; Mon, 12 Nov 2001 08:24:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACCnQf22658 for ips-outgoing; Mon, 12 Nov 2001 07:49:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1ab.compuserve.com (siaar1ab.compuserve.com [149.174.40.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACCnOl22654 for ; Mon, 12 Nov 2001 07:49:25 -0500 (EST) Received: (from mailgate@localhost) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id HAA18198 for ips@ece.cmu.edu; Mon, 12 Nov 2001 07:49:18 -0500 (EST) Received: from compuserve.com (dal-tgn-tkx-vty14.as.wcom.net [216.192.233.14]) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id HAA18143; Mon, 12 Nov 2001 07:49:10 -0500 (EST) Message-ID: <3BEFC681.F2DC9803@compuserve.com> Date: Mon, 12 Nov 2001 06:54:25 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector CC: Black_David@emc.com Subject: Re: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) References: <277DD60FB639D511AC0400B0D068B71E0293714E@CORPMX14> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, You appear to be saying one of two things: 1) The NAPTs solution, as proposed, requires that FCIP eschew the use of group pre-shared keys, or 2) The problems with the best on offer FCIP NAPTs solution design are sufficient to convince the IESG that FCIP is justified in prohibiting the use of NAPTs. Although time may have altered opinions, I will note that option 2) conforms to the position of several people on the FCIP development team during the Irvine meeting. Certainly, option 2) greatly simplifies the changes required in FCIP. It is clear that you will object to ANY form of identification that an FCIP Entity offers for one of the following reasons: a) Offering IP addresses is unacceptable because of NAPTs, and b) Offering any other value is unacceptable because it is not secure with group pre-shared keys. Unless option 1) above is your real concern, it appears that the FCIP development team has spent two months on a near fruitless exercise, whose only benefit is a strong proof that FCIP should not support NAPTs. Ralph... Black_David@emc.com wrote: > > For those cases where a secure environment is required, the > > new connection comes up through the normal IPSec authentication > > and encryption processes. As a result, the transmission of > > the identifying world wide names is already transmitted in an > > encrypted format, creatable only by the authenticated and > > certified source and interpretable only by the authenticated > > and certified destination. > > The assumption that there's a big on/off switch for > security is not correct (e.g., suppose IPsec is running > with integrity on and confidentiality off) and > "authenticated and certified" is a peculiar concept for > group pre-shared keys, a very likely deployment scenario. > Also, IPsec has no clue about the WWN, and hence the IKE > authentication is not linked to the WWN that has to be > presented (a particular problem with group pre-shared keys, > but also a risk even with certificates). In other words, > "authenticated and certified" doesn't place any restrictions > on what WWN is presented. Finally, solving > an inband authentication problem by requiring that IPsec be > used to encrypt the WWN is wildly excessive, and besides, > the WWN has no secret contents- it's a public piece of > configuration information. > > There is a solution in this general area, but it involves > linking the WWN to the IKE identity that is authenticated. > That's probably going to break any attempt to use existing > off-the-shelf external IPsec gateways because gateways > don't grok WWNs, and the FCIP implementations won't > know what identity was used in the IKE authentication > to the gateway. This is all a bit peculiar because FC > currently trust WWNs passed in various FC Login frames > without authenticating them, but that's T11's area of > responsibility putting this sort of thing into an IETF > standard isn't going to be acceptable. > > > If the Fibre Channel fabric is also operating in a secure mode, > > subsequent Fibre Channel authentication and certification > > is performed using the standard FC SLAP mechanisms. > > Only on the first TCP connection, unless you plan to require > taking the FCIP link down when the second TCP connection > is added (which strikes me as highly counterproductive). > The second connection just inherits the results of the SLAP > performed on the first one, whether its entitled to or not. > > > In addition to this, there are a whole bunch of policy > > restrictions that are verified as part of the creation > > of subsequent connections. While these are not necessarily > > part of the security steps, they prevent the formation of > > connections which do not meet the security rules. > > I'm not sure what you're referring to, but I suspect they > don't help much here. > > > Assuming security mechanisms are properly implemented, where, > > then, is the security hole? > > The fact that the WWN is not authenticated means that anyone > who can set up an FCIP connection to an FCIP entity can tap > into any existing FC traffic flowing on any connection through > that FCIP entity. > > --David From owner-ips@ece.cmu.edu Mon Nov 12 08:27:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA28292 for ; Mon, 12 Nov 2001 08:27:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACCV2d21645 for ips-outgoing; Mon, 12 Nov 2001 07:31:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACCV0l21636 for ; Mon, 12 Nov 2001 07:31:00 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id NAA61308 for ; Mon, 12 Nov 2001 13:30:52 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fACCUkk90506 for ; Mon, 12 Nov 2001 13:30:52 +0100 To: ips@ece.cmu.edu Importance: High MIME-Version: 1.0 Subject: iSCSI - change proposal LUN field definition on every PDU X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 12 Nov 2001 14:30:38 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 12/11/2001 14:30:52, Serialize complete at 12/11/2001 14:30:52 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, A colleague interested in instrumentation approached me with a question about stateless logging of specific LU activity. With the current iSCSI PDU formats this is not possible. We have consistently avoided having fields that are redundant and will need consistency checking. However I think we should consider including the LU field in all PDUs that are referencing a specific LU to simplify this type of instrumentation (as we did with the direction bit in the op-code). As I am already in "count-down" mode for version 09 I would appreciate your comments ASAP. Julo From owner-ips@ece.cmu.edu Mon Nov 12 12:43:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA12732 for ; Mon, 12 Nov 2001 12:43:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACGkfL09235 for ips-outgoing; Mon, 12 Nov 2001 11:46:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fACGkdl09230 for ; Mon, 12 Nov 2001 11:46:40 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fACGkPQ27051; Mon, 12 Nov 2001 11:46:25 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fACGkOH12387; Mon, 12 Nov 2001 11:46:25 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15343.64748.925000.330096@gargle.gargle.HOWL> Date: Mon, 12 Nov 2001 11:46:36 -0500 From: Paul Koning To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: IKE normative guidelines References: <277DD60FB639D511AC0400B0D068B71E02937154@CORPMX14> X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 9 November 2001) by Black_David@emc.com: > Yes, these guidelines differ from those RFCs in a number of > ways, based on things learned since those RFCs were written. > Many of these are things that would go into revised versions > of the IPsec RFCs, but revising the IPsec RFCs turns out to > be intractable to the point of near-impossibility for the > security folks - you'll have to ask them why. See the IPS > security draft for some further explanation of these changes. That's all well and good, but I see a major process problem with WG XYZ overriding the requirements created in another protocol that belongs to WG ABC on the grounds that WG ABC can't get the work done itself. If that's what we want to do, we should be clear about it and not call the lower layer protocol we're relying on "IPSec" but rather something like "IPS-sec" to make it clear we're defining an IPS specific variant forked from the original standard. Perhaps IKE is less problematic than, say, not yet settled on modes such as XCBC, but the same principle applies in both places. > Main mode is not "at least as good" as aggressive mode when > group pre-shared keys are used; see the security draft and > the ipsec WG "improveike" draft for details. Group pre- > shared keys are often used in practice because they > greatly simplify key management. Yes, that's covered by text I supplied a while back. That case applies to dynamic IP addresses, which is an obscure corner case at best for IPS. With static addresses, group keys are not needed and are very much not normal practice. paul From owner-ips@ece.cmu.edu Mon Nov 12 12:45:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA12792 for ; Mon, 12 Nov 2001 12:45:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACH2eZ10507 for ips-outgoing; Mon, 12 Nov 2001 12:02:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fACH2cl10501 for ; Mon, 12 Nov 2001 12:02:38 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fACH2HQ27128; Mon, 12 Nov 2001 12:02:17 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fACH2Gs19542; Mon, 12 Nov 2001 12:02:16 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15344.164.694000.71328@gargle.gargle.HOWL> Date: Mon, 12 Nov 2001 12:02:28 -0500 From: Paul Koning To: Julian_Satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI - change proposal LUN field definition on every PDU References: X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 12 November 2001) by Julian Satran: > Dear colleagues, > > A colleague interested in instrumentation approached me with a question > about stateless logging of specific LU activity. > With the current iSCSI PDU formats this is not possible. > We have consistently avoided having fields that are redundant and will > need consistency checking. > However I think we should consider including the LU field in all PDUs that > are referencing a specific LU to simplify this type of instrumentation (as > we did with the direction bit in the op-code). NO. I can see no good reason to add work to an implementation fastpath for this. It slightly simplifies instrumentation, but instrumentation is certainly possible without it. That is sufficient. paul From owner-ips@ece.cmu.edu Mon Nov 12 12:50:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA12965 for ; Mon, 12 Nov 2001 12:50:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACGoGo09535 for ips-outgoing; Mon, 12 Nov 2001 11:50:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fACGoEl09528 for ; Mon, 12 Nov 2001 11:50:14 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fACGo3Q27055; Mon, 12 Nov 2001 11:50:03 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fACGo2813089; Mon, 12 Nov 2001 11:50:03 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15343.64967.299000.664394@gargle.gargle.HOWL> Date: Mon, 12 Nov 2001 11:50:15 -0500 From: Paul Koning To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI reserved bits and fields References: <277DD60FB639D511AC0400B0D068B71E02937157@CORPMX14> X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 9 November 2001) by Black_David@emc.com: > "MUST be ignored by the receiver" is definitely the correct > language for the receive side. > On the send side, I believe "SHOULD be set to zero" is sufficient; > given the "MUST ignore" on the receive side, I don't > think another "MUST" is needed. "MUST" is better on the send side, because then those bits can safely be used in a later protocol version. If you say "SHOULD" then V1 implementations can have non-zero values (which V>1 wants to assign meaning to) in those fields and still be compliant. In other words, "MUST set to zero" keeps the fields available, "SHOULD set to zero" makes them unavailable for later use. paul From owner-ips@ece.cmu.edu Mon Nov 12 13:25:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA14219 for ; Mon, 12 Nov 2001 13:25:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACHYv513047 for ips-outgoing; Mon, 12 Nov 2001 12:34:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACHYtl13043 for ; Mon, 12 Nov 2001 12:34:55 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W489HR3>; Mon, 12 Nov 2001 12:31:14 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293715F@CORPMX14> From: Black_David@emc.com To: ENDL_TX@computer.org, ips@ece.cmu.edu Subject: RE: FCencap: List ALL SOF/EOF codes Date: Mon, 12 Nov 2001 12:25:05 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Please add the note of explanation with a pointer to FC-MI so the next person who wonders about the absence of Class 1 is presented with a pointer to the answer. Thanks, --David > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Saturday, November 10, 2001 11:52 AM > To: IPS Reflector > Cc: Black_David@emc.com > Subject: Re: FCencap: List ALL SOF/EOF codes > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. > > Therefore, I am forced to agree with David. Class 1 MUST NOT > be mentioned in the FC Encapsulation draft. If necessary, a > note discussing interoperability and FC-MI can be added. > > Thanks. > > Ralph... > > Black_David@emc.com wrote: > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > I in FC-MI stands for "Interoperability", this seems like a > reasonable > > rationale for excluding Class 1 service. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > From owner-ips@ece.cmu.edu Mon Nov 12 13:29:27 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA14401 for ; Mon, 12 Nov 2001 13:29:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACHWNd12836 for ips-outgoing; Mon, 12 Nov 2001 12:32:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACHWLl12832 for ; Mon, 12 Nov 2001 12:32:21 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W489HNH>; Mon, 12 Nov 2001 12:28:40 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293715E@CORPMX14> From: Black_David@emc.com To: rsnively@Brocade.COM, ips@ece.cmu.edu Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Mon, 12 Nov 2001 12:22:31 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, > Let me try this again. (E-mail sure is a slow medium > of information exchange. We need some more face-to-face > meetings on this subject.) The next opportunity is Salt Lake City. > If secure IPSec connections can be made, I will use > those connections. The information about > the boxes and the permissions to connect them are part of the > administrative set up necessary to bring them into a secure > environment. > > If that is an exposed connection and carries information that > is important enough to be worth protecting, some degree of > encryption will be used on ALL data transmitted on the > connection, including the initial information passed to the > opposite end point. If you did that, the WWN would be > never be publicly available on the network, but only available > through the administrative mechanisms (like walking up to the > device through your bullet proof glass doors and examining its > labels). In other words, FCIP relies on the presence of IPsec to provide adequate security to a mandatory authentication element of FCIP. I think that makes IPsec at least "SHOULD use" for FCIP, and possibly "MUST use". This would come with some environmental caveats, but the upshot is that FCIP goes directly from no security to cryptographic connection integrity without the possibility of just securing the authentication (unlike iSCSI) - that jump makes me uncomfortable. FWIW, only cryptographic connection integrity is required - WWNs are not secret, and hence confidentiality is not in principle necessary to protect them. > If you truly truly cared about the connection, you would have > gone to the trouble of using keys (perhaps pre-established) > that were not group pre-shared keys. > > A lot of this is of the nature "if it hurts, don't do it > (unless it is worth the pain)". Who would allow his SAN > to pass important data without providing security mechanisms > appropriate to the value of the data? If you care, you would > not use group pre-shared keys, but some other kind of keying. > If you care, you would use confidentiality protection. > It really does not make much sense to criticize a security > approach by making the assumption that you are going to > throw away your first lines of defense. And this makes group pre-shared keys at least "SHOULD NOT use", probably "MUST NOT use". Thou shalt not present the user with security mechanisms that provide only the appearance of security as opposed to actual security - such mechanisms are more dangerous than omitting security entirely. FWIW, I could see a more complex discussion about making sure that the domain of sharing of pre-shared keys matches the domain of trust among systems, but this is a slippery slope, as the concept of "domain of trust" is inherently not arbitrarily scalable. > Since the identity of the box is very much part of the same > administrative setup, and because (in the latest revision of > our proposal) the forward information exchange to the remote > point includes both the known source and the expected destination, > either side has the right to terminate the connection if > the information is incorrect. If all the administrative > information necessary to form a secure connection and all the > administrative information necessary to control access to > FCIP devices is known to a device, it is by definition > authorized to perform the connection. You must deny untrustworthy > devices some part of the necessary information for any security > program to work. I agree with this as far as it goes, but note that WWNs are not considered secret, and hence any rationale for security that depends on them being secret is ill-founded (another way of arriving at your comment about not using group pre-shared keys if they don't provide enough security). In turn, this means that if a WWN is part of the "administrative information necessary to form a secure connection", it has to be securely associated with something that is authenticated based on information that really is secret. If that authentication is performed by IKE, the result is at least the "SHOULD use" and "SHOULD NOT use" indicated above. > As the first connection is formed, it indicates what type of > information may flow across it. Typically, the first connection > will permit only class F information (and no other connection will > permit class F information). Any attempt to create a second > class F information connection will terminate the entire set of > connections, since it is a protocol violation as well as a security > violation. You are correct in saying that the first is the connection > that will perform the SLAP authentication. Note that some part > of this protocol is specified by T11 in FC-BB-2, not in FCIP. Notice the denial of service attack opening here - if the attacker can form a second class F connection, the entire FCIP inter-switch link goes down, and SLAP can't do anything about it. Another opportunity would be setting up a Class 2 and/or Class 3 connection and bit-bucketing some portion of the traffic, which will lead to severe indigestion at the FC switches. This leads directly to ... > Subsequent connections can only be created from the same device > and are assigned classes of information and other usage > characteristics by that same device. If any other device tries > to do so, and expects to be successful, it must have all the > administrative information necessary to create a second secure > connection and must have knowledge of what connections the > first device's previous connections and FC information exchanges > would permit to be set up. And if it has all that information, > it is by definition permitted to establish the connection. > > I really don't see how you can create a bothersome second connection > unless you have chosen to allow violation of the security of your > administration procedures. And if you allow violation of the > security of your administration procedures, I don't see how you > can prevent a malicious connection using any mechanism at all. In order to get here, three things appear to have to happen to the spec: (1) IPsec, at least the cryptographic integrity (I don't believe that confidentiality/encryption, is necessary to solve this problem) becomes at least "SHOULD use", possibly with an exception for environments in which authentication is not needed. (2) Group pre-shared keys become "MUST NOT use" - the potential they create for denial of service is just too dangerous. Similar words need to be added about certificate usage. (3) Something at the FCIP level MUST contain an authorization table that maps WWNs to whatever IKE is authenticating (probably an IP address via either a pre-shared key or a certificate). Item (2) will become a deployment obstacle in practice due to the need to manage IPsec authentication material for every node. Item (3) will become an issue for use of external IPsec gateways, because the IKE information has to be obtained from that gateway promptly when the IPsec SA is setup in order to check that the WWN is being presented from a node that is allowed to present that WWN. Failing to do this results in authorization without authentication which is a no-no. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Nov 12 14:04:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA15444 for ; Mon, 12 Nov 2001 14:04:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACIK2v16492 for ips-outgoing; Mon, 12 Nov 2001 13:20:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from best.eurologic.com ([193.120.246.5]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACIJwl16482 for ; Mon, 12 Nov 2001 13:19:59 -0500 (EST) Received: from wombat (wombat [192.168.7.41]) by best.eurologic.com (8.9.3+Sun/8.9.3) with SMTP id SAA23402; Mon, 12 Nov 2001 18:15:38 GMT Message-ID: <082301c16ba6$8d9770e0$2907a8c0@wombat> From: "Ken Sandars" To: , "Julian Satran" References: Subject: Re: iSCSI - change proposal LUN field definition on every PDU Date: Mon, 12 Nov 2001 18:19:22 -0000 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi Julo, Anything which helps get diagnostic and measurement equipment out is a GOOD thing, especially when there is a kick back to those who speak up in support ;-) Are the only changes to the SCSI Response, the Task Management Response and the Data-In PDUs? Cheers, Ken ksandars@eurologic.com Eurologic Systems +44 117 930 9616 From owner-ips@ece.cmu.edu Mon Nov 12 14:07:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA15538 for ; Mon, 12 Nov 2001 14:07:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACIHbY16273 for ips-outgoing; Mon, 12 Nov 2001 13:17:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACIHZl16268 for ; Mon, 12 Nov 2001 13:17:35 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W489JTN>; Mon, 12 Nov 2001 13:13:54 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937160@CORPMX14> From: Black_David@emc.com To: kzm@cisco.com Cc: ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Date: Mon, 12 Nov 2001 13:07:44 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk A couple more comments on the traps: - The thing that's going away is the trap registration table. The existing traps become notifications. This is what I was expecting. - One of the things hiding in the existing trap registration table is the ability to register for traps by severity. Since the existing traps have severity specified statically (trap X always has severity Y), this looks like it can be handled by the filters in the notification MIB, although filter configuration will be somewhat more complex (but also much more flexible) than the existing severity mechanism. Does that sound right? Proposed course of action on RFC 2837 (include sufficient elements in new MIB to replace it, even though decision as to whether to replace has not been made) is fine. Thanks, --David > -----Original Message----- > From: Keith McCloghrie [mailto:kzm@cisco.com] > Sent: Sunday, November 11, 2001 6:11 PM > To: Black_David@emc.com > Cc: kzm@cisco.com; ips@ece.cmu.edu > Subject: Re: FC Management MIB - proposed changes > > > > For simplicity, I would propose keeping the initial revision focused > > on structural and format changes only - i.e., make no functional > > changes that aren't required to bring the MIB into line with the > > overall architecture of MIBs, use of SMIv2, and the like. > > Agreed. Even so, almost everything in the MIB will be affected. > > > Comments on this, keyed to Keith's issue numbers: > > > > 1. The trap table doesn't match up with RFC 2573 - I presume > > correcting this is part of the conversion to SMIv2. > > With SNMPv2c/SNMPv3, a Trap became one of the two forms by which a > notification is transmitted; the other form is an InformRequest. > SMIv2 MIBs don't define traps - they define notifications (with the > NOTIFICATION-TYPE macro). With SNMPv2c/SNMPv3, agents generate > notifications, not traps. The destinations to which a notification is > sent is determined by the tables in the two MIBs defined in RFC 2573: > the SNMP-NOTIFICATION-MIB and SNMP-TARGET-MIB. The snmpNotifyType > object in RFC 2573 determines in which form the notification is > tranmsitted to a destination; it's possible for the same notification > generated by an agent to be sent as a trap to one destination, and as > an inform to another destination. Thus, the trap table is a > relic of the SNMPv1-only days; it is: a) insufficient for > SNMPv2c/SNMPv3, and b) a duplicate of a subset of RFC 2573. It should > be deleted without replacement. > > > 3. Let's keep the replacement of RFC 2837 as (a) a proposal > > and (b) Keith's recommendation to the WG as MIB Advisor, > > but not formally adopt that approach until we have a version > > of the MIB draft that would replace RFC 2837 available for > > review by the WG. > > We can postpone the decision on what happens to 2837. However, 2837 > contains information on FC interfaces. The FC-MGMT MIB also contains > information on FC interfaces. Most of the information will be the > same, and should be defined once; I propose in the new FC-MGMT MIB. > There are a few pieces of information which are specific to Fabric > Elements. I'll include those in the FC-MGMT MIB for the time being, > until the postponed decision is made. > > > 5. I think there's a misunderstanding here. There is only one > > name service in any FC fabric, period. The term "Simple Name > > Service" is historical and refers to the current approach > > to Fibre Channel fabric name service by contrast to the > > originally-proposed X.500 directory-based approach (see the > > original FC-GS) - the meaning of "Simple" should now be > > obvious, as it's by comparison to X.500 ;-). > > > > Both the Zoned and Unzoned name services are simple > name services. > > The same objects can be used to represent both, as only one is > > active at any time, and the communication interfaces/protocols > > are identical. Client access to the name service is the same > > in both cases; zoned vs. unzoned only affects server behavior > > in terms of what is returned to a query. In addition the name > > service objects are described to represent only entities > > "known to this unit". So, in a zoned fabric, I would expect > > the table in a switch to be completely populated, but the table > > in an HBA to contain only the entries in that HBA's zone > > because the HBA doesn't "know" about any others. > > > > The name server objects need to be retained - these are quite > > important for fabric management visibility. Whether the fabric > > is zoned or not and how the nameserver behaves can be determined > > from the zone objects (i.e., for a switch, the > nameserver objects > > contain everything and the zone objects have to be consulted to > > figure out what a query from a particular client to the > nameserver > > will return). > > OK. > > > 7. I would definitely take out Class 1 support, and reference FC-MI > > (which prohibits Class 1 service) as an authority for doing so. > > OK. > > > 8. For the initial version, I would only carry forward the zone > > objects existing in the current MIB and not define any new > > functionality - that can be done in revisions after -00. > > OK. > > > 9. I'd prefer that the -00 version contain no additional functional > > changes beyond those required to support the necessary > > structure and format changes. Once we have that -00 baseline, > > functional changes can be made from there. > > OK. > > > 2, 4, and 6 look fine to me, as does the new draft name. > > Thnaks, > Keith. > From owner-ips@ece.cmu.edu Mon Nov 12 14:07:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA15549 for ; Mon, 12 Nov 2001 14:07:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACIkM918675 for ips-outgoing; Mon, 12 Nov 2001 13:46:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACIkJl18666 for ; Mon, 12 Nov 2001 13:46:20 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel12.hp.com (Postfix) with ESMTP id 842A31F8AC; Mon, 12 Nov 2001 10:46:10 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA10844; Mon, 12 Nov 2001 10:46:06 -0800 (PST) Message-ID: <3BF01B05.CA2A3AB4@cup.hp.com> Date: Mon, 12 Nov 2001 10:55:01 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Julian Satran Cc: ips@ece.cmu.edu Subject: Re: iSCSI - change proposal LUN field definition on every PDU References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Can you explain further the extent of PDUs that are intended to be affected by this change ? Also, can you clarify why it is not a problem for other SCSI transport protocols like FC where the LUN field is not included in every FC IU ? IMO, the LUN knowledge is best kept out of the SCSI transport as far as possible. All LUN level logging/tracing belongs to the SCSI ULP, which is given a LUN context on each call made to it from the LLP. At the SCSI LLP layer, tracing/logging as well as state information should be at the I-T nexus level (session). I don't (yet) see a sufficient justification for this change. Perhaps, more information on the motivation for this request may change that. - Santosh Julian Satran wrote: > > Dear colleagues, > > A colleague interested in instrumentation approached me with a question > about stateless logging of specific LU activity. > With the current iSCSI PDU formats this is not possible. > We have consistently avoided having fields that are redundant and will > need consistency checking. > However I think we should consider including the LU field in all PDUs that > are referencing a specific LU to simplify this type of instrumentation (as > we did with the direction bit in the op-code). > > As I am already in "count-down" mode for version 09 I would appreciate > your comments ASAP. > > Julo -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Mon Nov 12 14:12:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA15646 for ; Mon, 12 Nov 2001 14:12:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACIWoK17542 for ips-outgoing; Mon, 12 Nov 2001 13:32:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACIWnl17538 for ; Mon, 12 Nov 2001 13:32:49 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W489KNQ>; Mon, 12 Nov 2001 13:29:08 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937161@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Cc: Black_David@emc.com Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Mon, 12 Nov 2001 13:22:56 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, See my response to Bob's mail. The situation is 1) with a few additions - the particular way in which FCIP has designed this identity mechanism makes group pre-shared keys a bad idea, and saying they SHOULD NOT or MUST NOT be used is part of the solution. I would prefer that FCIP use an FC level identifier rather than an IP address for identifying FC entities, but in the current design, that identifier is being used as an unsecured means of authentication, and hence some way of securing that authentication needs to be found. The reply to Bob outlines how to go about getting IKE and IPsec to solve this problem; the alternative of having FC try to solve this problem doesn't seem tractable to me because the problem occurs with the TCP connections after the first one, and FC is not going to want to rerun SLAP just because a second TCP connection has been set up (and in fact may not be able to). Thanks, --David > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Monday, November 12, 2001 7:54 AM > To: IPS Reflector > Cc: Black_David@emc.com > Subject: Re: FCIP: NAPTs Solution Proposal (issue from Irvine, CA > Interim meeting) > > > David, > > You appear to be saying one of two things: > > 1) The NAPTs solution, as proposed, requires that FCIP eschew the > use of group pre-shared keys, or > 2) The problems with the best on offer FCIP NAPTs solution design > are sufficient to convince the IESG that FCIP is justified in > prohibiting the use of NAPTs. > > Although time may have altered opinions, I will note that option > 2) conforms to the position of several people on the FCIP development > team during the Irvine meeting. Certainly, option 2) greatly > simplifies the changes required in FCIP. > > It is clear that you will object to ANY form of identification > that an FCIP Entity offers for one of the following reasons: > > a) Offering IP addresses is unacceptable because of NAPTs, and > b) Offering any other value is unacceptable because it is not > secure with group pre-shared keys. > > Unless option 1) above is your real concern, it appears that the > FCIP development team has spent two months on a near fruitless > exercise, whose only benefit is a strong proof that FCIP should > not support NAPTs. > > Ralph... > > Black_David@emc.com wrote: > > > > For those cases where a secure environment is required, the > > > new connection comes up through the normal IPSec authentication > > > and encryption processes. As a result, the transmission of > > > the identifying world wide names is already transmitted in an > > > encrypted format, creatable only by the authenticated and > > > certified source and interpretable only by the authenticated > > > and certified destination. > > > > The assumption that there's a big on/off switch for > > security is not correct (e.g., suppose IPsec is running > > with integrity on and confidentiality off) and > > "authenticated and certified" is a peculiar concept for > > group pre-shared keys, a very likely deployment scenario. > > Also, IPsec has no clue about the WWN, and hence the IKE > > authentication is not linked to the WWN that has to be > > presented (a particular problem with group pre-shared keys, > > but also a risk even with certificates). In other words, > > "authenticated and certified" doesn't place any restrictions > > on what WWN is presented. Finally, solving > > an inband authentication problem by requiring that IPsec be > > used to encrypt the WWN is wildly excessive, and besides, > > the WWN has no secret contents- it's a public piece of > > configuration information. > > > > There is a solution in this general area, but it involves > > linking the WWN to the IKE identity that is authenticated. > > That's probably going to break any attempt to use existing > > off-the-shelf external IPsec gateways because gateways > > don't grok WWNs, and the FCIP implementations won't > > know what identity was used in the IKE authentication > > to the gateway. This is all a bit peculiar because FC > > currently trust WWNs passed in various FC Login frames > > without authenticating them, but that's T11's area of > > responsibility putting this sort of thing into an IETF > > standard isn't going to be acceptable. > > > > > If the Fibre Channel fabric is also operating in a secure mode, > > > subsequent Fibre Channel authentication and certification > > > is performed using the standard FC SLAP mechanisms. > > > > Only on the first TCP connection, unless you plan to require > > taking the FCIP link down when the second TCP connection > > is added (which strikes me as highly counterproductive). > > The second connection just inherits the results of the SLAP > > performed on the first one, whether its entitled to or not. > > > > > In addition to this, there are a whole bunch of policy > > > restrictions that are verified as part of the creation > > > of subsequent connections. While these are not necessarily > > > part of the security steps, they prevent the formation of > > > connections which do not meet the security rules. > > > > I'm not sure what you're referring to, but I suspect they > > don't help much here. > > > > > Assuming security mechanisms are properly implemented, where, > > > then, is the security hole? > > > > The fact that the WWN is not authenticated means that anyone > > who can set up an FCIP connection to an FCIP entity can tap > > into any existing FC traffic flowing on any connection through > > that FCIP entity. > > > > --David > From owner-ips@ece.cmu.edu Mon Nov 12 14:51:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA16452 for ; Mon, 12 Nov 2001 14:51:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACJ0LO19836 for ips-outgoing; Mon, 12 Nov 2001 14:00:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACJ0Kl19830 for ; Mon, 12 Nov 2001 14:00:20 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 12 Nov 2001 14:02:40 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937162@CORPMX14> From: Black_David@emc.com To: ni1d@arrl.net Cc: ips@ece.cmu.edu Subject: RE: iSCSI: IKE normative guidelines Date: Mon, 12 Nov 2001 13:50:29 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Paul, > > Yes, these guidelines differ from those RFCs in a number of > > ways, based on things learned since those RFCs were written. > > Many of these are things that would go into revised versions > > of the IPsec RFCs, but revising the IPsec RFCs turns out to > > be intractable to the point of near-impossibility for the > > security folks - you'll have to ask them why. See the IPS > > security draft for some further explanation of these changes. > > That's all well and good, but I see a major process problem with WG > XYZ overriding the requirements created in another protocol that > belongs to WG ABC on the grounds that WG ABC can't get the work done > itself. Please take this up with the Security ADs, as opposed to pursuing it here. AFAIK, both the IPsec WG and the IETF Security area (e.g., saag meeting in London) are aware of the approach that the ips WG is taking here and do not have a problem with it. We (including yours truly) are tracking what goes on in the security area and trying not to get ahead of it, but the fact that DES is the only REQUIRED cipher in the current documents is embarrassing. > If that's what we want to do, we should be clear about it and not call > the lower layer protocol we're relying on "IPSec" but rather something > like "IPS-sec" to make it clear we're defining an IPS specific variant > forked from the original standard. I have no problem with the use of "IPsec" along with a word like "subset". The fact that we are taking this subset approach does need to be crystal clear in our drafts. FWIW, the "forking" is primarily in the area of deletion - we are not (and do not have the license to) extend the protocols in any fashion. The approach is to remove requirements for things like AH and DES and strengthen requirements for things that replace them where replacements are needed (e.g., 3DES). > Perhaps IKE is less problematic than, say, not yet settled on modes > such as XCBC, but the same principle applies in both places. Any reference to XCBC will be a normative reference to an RFC that goes via the IPsec WG. We cannot do work on XCBC here. If the IPsec WG does not approve XCBC, we'll have to drop it. > > Main mode is not "at least as good" as aggressive mode when > > group pre-shared keys are used; see the security draft and > > the ipsec WG "improveike" draft for details. Group pre- > > shared keys are often used in practice because they > > greatly simplify key management. > > Yes, that's covered by text I supplied a while back. That case > applies to dynamic IP addresses, which is an obscure corner case at > best for IPS. With static addresses, group keys are not needed and > are very much not normal practice. At least for iSCSI, I definitely disagree with the "obscure corner case" characterization. I'm less certain about FCIP and iFCP. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Mon Nov 12 15:46:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18022 for ; Mon, 12 Nov 2001 15:46:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACJVxh22455 for ips-outgoing; Mon, 12 Nov 2001 14:31:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACJVvl22451 for ; Mon, 12 Nov 2001 14:31:57 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel1.hp.com (Postfix) with ESMTP id 3CF8D93B for ; Mon, 12 Nov 2001 11:31:55 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id LAA13333 for ; Mon, 12 Nov 2001 11:33:22 -0800 (PST) Message-Id: <200111121933.LAA13333@core.rose.hp.com> Date: Mon, 12 Nov 2001 11:44:21 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI - change proposal LUN field definition on every PDU References: <3BF01B05.CA2A3AB4@cup.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > IMO, the LUN knowledge is best kept out of the SCSI transport as far as > possible. I agree. I also agree with Paul Koning about the perf path implications for software implementations. In addition, more significantly, I am concerned about the consistency checking aspects of this change. Apart from the performance costs incurred because of checking, what if the LUN doesn't match the active ITT? In general, more errors (and possibly Reject reasons). Lastly, it appears to me that the general philosophy of relying only on ITT for instantiated tasks is not applied here (given that ITT is unique per session across all LUNs). To summarize, IMHO, these disadvantages outweigh the benefit of stateless logging of LUN traffic. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Santosh Rao wrote: > > Julian, > > Can you explain further the extent of PDUs that are intended to be > affected by this change ? Also, can you clarify why it is not a problem > for other SCSI transport protocols like FC where the LUN field is not > included in every FC IU ? > > IMO, the LUN knowledge is best kept out of the SCSI transport as far as > possible. All LUN level logging/tracing belongs to the SCSI ULP, which > is given a LUN context on each call made to it from the LLP. > > At the SCSI LLP layer, tracing/logging as well as state information > should be at the I-T nexus level (session). > > I don't (yet) see a sufficient justification for this change. Perhaps, > more information on the motivation for this request may change that. > > - Santosh > > Julian Satran wrote: > > > > Dear colleagues, > > > > A colleague interested in instrumentation approached me with a question > > about stateless logging of specific LU activity. > > With the current iSCSI PDU formats this is not possible. > > We have consistently avoided having fields that are redundant and will > > need consistency checking. > > However I think we should consider including the LU field in all PDUs that > > are referencing a specific LU to simplify this type of instrumentation (as > > we did with the direction bit in the op-code). > > > > As I am already in "count-down" mode for version 09 I would appreciate > > your comments ASAP. > > > > Julo > > -- > ################################## > Santosh Rao > Software Design Engineer, > HP-UX iSCSI Driver Team, > Hewlett Packard, Cupertino. > email : santoshr@cup.hp.com > Phone : 408-447-3751 > ################################## From owner-ips@ece.cmu.edu Mon Nov 12 18:57:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA24023 for ; Mon, 12 Nov 2001 18:57:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACNXYl12672 for ips-outgoing; Mon, 12 Nov 2001 18:33:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACNXWl12667 for ; Mon, 12 Nov 2001 18:33:33 -0500 (EST) Received: by ARIEL with Internet Mail Service (5.5.2653.19) id ; Mon, 12 Nov 2001 15:33:26 -0800 Message-ID: From: Charles Monia To: IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Date: Mon, 12 Nov 2001 15:33:26 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Folks: > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. The last time I checked, the FC-MI spec was not a "standards track" document (to use IETF terminology). If that's still the case, is FC-MI's prohibition of class 1 a sufficient basis for precluding class 1 support in the encapsulation spec? Charles > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Saturday, November 10, 2001 8:52 AM > To: IPS Reflector > Cc: Black_David@emc.com > Subject: Re: FCencap: List ALL SOF/EOF codes > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. > > Therefore, I am forced to agree with David. Class 1 MUST NOT > be mentioned in the FC Encapsulation draft. If necessary, a > note discussing interoperability and FC-MI can be added. > > Thanks. > > Ralph... > > Black_David@emc.com wrote: > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > I in FC-MI stands for "Interoperability", this seems like a > reasonable > > rationale for excluding Class 1 service. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > From owner-ips@ece.cmu.edu Mon Nov 12 18:57:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA24036 for ; Mon, 12 Nov 2001 18:57:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fACMwIT09606 for ips-outgoing; Mon, 12 Nov 2001 17:58:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel6.hp.com (atlrel6.hp.com [192.151.27.8]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACMwHl09599 for ; Mon, 12 Nov 2001 17:58:17 -0500 (EST) Received: from xatlrelay1.atl.hp.com (xatlrelay1.atl.hp.com [15.45.89.190]) by atlrel6.hp.com (Postfix) with ESMTP id 6CBF71F7DB; Mon, 12 Nov 2001 17:58:11 -0500 (EST) Received: from xatlbh1.atl.hp.com (xatlbh1.atl.hp.com [15.45.89.186]) by xatlrelay1.atl.hp.com (Postfix) with ESMTP id D26A61F519; Mon, 12 Nov 2001 17:58:10 -0500 (EST) Received: by xatlbh1.atl.hp.com with Internet Mail Service (5.5.2653.19) id ; Mon, 12 Nov 2001 17:58:10 -0500 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87805CCF20F@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "'Keith McCloghrie'" , ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Date: Mon, 12 Nov 2001 17:58:00 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Keith, A little background on the intent of these two MIBs - the FC Mgmt MIB was originally intended to represent "everything that had Fibre Channel Ports". In that sense, it is analogous to the Interfaces Group MIB. RFC 2837 was intended to represent FC Fabric Switch information - more like a "router MIB" - intended to represent the information specific to the functions of a Fibre Channel fabric switch. That doesn't seem to me to be information that should be in an "interfaces MIB". It doesn't seem logical to replace RFC 2837 with a new FC Mgmt MIB, the intent of both MIBs is/should be much different. The name server stuff never belonged in the FC Mgmt MIB, it (and other Fabric Services information) belongs in the RFC 2837 MIB. Thoughts? Marjorie > 3. With 20x20 hindsight, the following observations can be made with > respect to RFC 2837: > > - the reason that we now have the issue of how it relates to the > FC-Mgmt MIB is because it was written as a Fabric Element MIB; > this issue would not have arisen if it had been written as a > Fibre Channel interface MIB. > - it should have extended the ifTable, rather than > overlapping with it, > - it should not have defined the fcFeModuleTable, which overlaps with > the Entity MIB. > - and it should have specified (at least) its octet counters > as Counter64. > > Given these problems, I propose that the new FC-Mgmt MIB be specified > as a replacement for RFC 2837. > From owner-ips@ece.cmu.edu Mon Nov 12 20:00:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25747 for ; Mon, 12 Nov 2001 20:00:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAD00ef14645 for ips-outgoing; Mon, 12 Nov 2001 19:00:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAD00dl14637 for ; Mon, 12 Nov 2001 19:00:39 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id QAA07100; Mon, 12 Nov 2001 16:00:22 -0800 (PST) From: "Murali Rajagopal" To: "Charles Monia" , "IPS Reflector" Subject: RE: FCencap: List ALL SOF/EOF codes Date: Mon, 12 Nov 2001 16:01:29 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit On the specific topic of supported SOF and EOF codes the ietf documents should be driven by the specification provided in the *most relevant * document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI should be kept out of this. If we simply accept to adopt the SOF and EOF codes listed for BB-2 the problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. I don't see why we are making a big deal about this. -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Charles Monia Sent: Monday, November 12, 2001 3:33 PM To: IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Hi Folks: > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. The last time I checked, the FC-MI spec was not a "standards track" document (to use IETF terminology). If that's still the case, is FC-MI's prohibition of class 1 a sufficient basis for precluding class 1 support in the encapsulation spec? Charles > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Saturday, November 10, 2001 8:52 AM > To: IPS Reflector > Cc: Black_David@emc.com > Subject: Re: FCencap: List ALL SOF/EOF codes > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. > > Therefore, I am forced to agree with David. Class 1 MUST NOT > be mentioned in the FC Encapsulation draft. If necessary, a > note discussing interoperability and FC-MI can be added. > > Thanks. > > Ralph... > > Black_David@emc.com wrote: > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > I in FC-MI stands for "Interoperability", this seems like a > reasonable > > rationale for excluding Class 1 service. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > From owner-ips@ece.cmu.edu Mon Nov 12 21:39:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA28246 for ; Mon, 12 Nov 2001 21:39:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAD1niW21863 for ips-outgoing; Mon, 12 Nov 2001 20:49:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antigonus.hosting.pacbell.net (antigonus.hosting.pacbell.net [216.100.98.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAD1nbl21852 for ; Mon, 12 Nov 2001 20:49:42 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by antigonus.hosting.pacbell.net id UAA00869; Mon, 12 Nov 2001 20:49:06 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "IPS" Subject: RE: iSCSI: Clarification (again) for Task Management Commands Date: Mon, 12 Nov 2001 17:46:05 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Section 9.4 does (or so I can figure) leave the sort of hole mentioned below uncovered. On a multiple connection session, the status for any of the commands effected by the task management command could already be in transit on any of the connections by the time the task management command is received by the target, and its reception is acknowledged - and then received by the initiator. The barrier works well for a single connection because the command sequence number for the task management command makes sure that there is nothing stuck in the "pipe". However, there could be status "stuck" in the other pipes. The mechanism David indicated might be the only way to be sure, which however might require a NOP on each connection to ensure there is nothing in the pipe. Somesh > -----Original Message----- > From: Julian Satran [mailto:Julian_Satran@il.ibm.com] > Sent: Saturday, November 10, 2001 7:26 AM > To: somesh_gupta@silverbacksystems.com > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > Read 9.4 for one implementation - Julo > > > > > "Somesh Gupta" > Sent by: owner-ips@ece.cmu.edu > 10-11-01 05:26 > Please respond to somesh_gupta > > > To: , > cc: > Subject: RE: iSCSI: Clarification (again) for Task > Management Commands > > > > David, > > In iSCSI the multiple connection/session construct > adds significant complexity in determining whether > a response for a command (on a different connection > than the one on which the task management command > was sent) impacted by the task management command will > be received or not - and when? > > On a single connection, or similar links, when the > response for the task management command is > received (or after a fixed additional time), no > responses will be received for the commands aborted > by the task management command. > > However, with multiple connections there is no > such "flushing" event on connections other than the > one on which the task management command was sent. > > I would hope that the protocol would address this > situation - the current response seems to be be to > put the task tag and some associated resources in > a zombie state for an unspecified amount of time. > > Somesh > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Friday, November 09, 2001 6:53 PM > > To: somesh_gupta@silverbacksystems.com; ips@ece.cmu.edu > > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > > > > Somesh, > > > > The language in question reflects fairly direct requirements > > language to be found in SAM-2's description of SCSI Task Management. > > FCP goes to serious lengths with FC sequence aborts to make sure > > this behaves as required. > > > > For iSCSI, if responses to the aborted commands show up unexpectedly, > > they have to be discarded. How the Initiator keeps track of that > > is the Initiator's problem - keeping track of the CmdSN of the > > Abort Task Set may be useful. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > -----Original Message----- > > > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > > > Sent: Friday, November 09, 2001 4:55 PM > > > To: IPS > > > Subject: iSCSI: Clarification (again) for Task Management Commands > > > > > > > > > Resend to add iSCSI tag. Sorry for missing it. > > > > > > On page 67 of the 8-92.txt draft (section 3.5.1), it > > > says > > > > > > "For all the tasks covered by the task > > > management response (i.e., with CmdSN not higher than the task > > > management command CmdSN), additional responses MUST NOT be delivered > > > to the SCSI layer after the task management response." > > > > > > If there is a multiple connection session, > > > a status for a command impacted by the task > > > management command (say ABORT TASK SET) could > > > be stuck in the pipe on one connection, while > > > the ABORT TASK SET completes on another > > > connection. > > > > > > How does the initiator iSCSI enforce the rule above? > > > Seems to be the equivalent of sending the impacted > > > commands on other connections in a zombie state, > > > and not having a very good idea of how to get out. > > > > > > Similarly Section 9.4 provides additional rules, > > > but seems to leave a hole open with regards to > > > status already in flight on other connections. > > > > > > Any clarifications would be appreciated. > > > > > > Somesh > > > > > > > > > From owner-ips@ece.cmu.edu Mon Nov 12 21:42:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA28294 for ; Mon, 12 Nov 2001 21:42:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAD1pbB22003 for ips-outgoing; Mon, 12 Nov 2001 20:51:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from med.corp.rhapsodynetworks.com (64-160-62-201.rhapsodynetworks.com [64.160.62.201] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fACNqil14121 for ; Mon, 12 Nov 2001 18:52:44 -0500 (EST) Received: by med.corp.rhapsodynetworks.com with Internet Mail Service (5.5.2653.19) id ; Mon, 12 Nov 2001 15:52:01 -0800 Message-ID: <45BEF1D68145D51186C100B0D0AABE6543E0B3@med.corp.rhapsodynetworks.com> From: Venkat Rangan To: "'Black_David@emc.com'" , ips@ece.cmu.edu Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Mon, 12 Nov 2001 15:51:54 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Section 5.8.2 of the draft-ietf-ips-security-04.txt covers the use pre-shared keys. In that section, we state that: "This vulnerability is typically not of concern with iFCP and FCIP, since iFCP and FCIP devices typically use statically defined addresses, so that individual pre-shared keys are possible within IKE main mode." If statically defined addresses is not workable due to the presence of NAPTs, we state that: "As a result, where iSCSI, iFCP, or FCIP Entities utilize dynamic address assignment along with pre-shared key authentication, IKE Agressive Mode MUST be used." These two statements taken together would appear to provide the necessary tools to implement a secure FCIP deployment with both static and dynamic address assignment. On your statement about the implicit assumption about the identity of an FCIP Entity based on its IP Address, Section 2.4 states the following: "The usage of SLPv2 by FCIP is described in [64]. FCIP Entities assume that once the IKE identity of a peer is established, the FCIP Entity Name carried in FCIP Short Frame is also implicity accepted as the authenticated peer. Any such association between the IKE identity and the FCIP Entitiy Name is administratively established." We also state in Section 2.3: "When pre-shared keys are used for authentication, IKE Aggressive Mode SHOULD be used, and IKE Main Mode SHOULD NOT be used." Thus, if IPSec IKE Phase 1 negotiation has succeeded between a pair of IP Addresses, the FCIP Entity referred to in the FCIP Short Frame which carries the remote FCIP Switch WWN and FCIP Identifier is also validated as a trusted FCIP Entity. This allows for an external gateway to deal with the IKE Phase 1, and the receving FCIP Entity is assured that the FCIP Short Frame is really from a trusted peer (in that configuration). There is a question of how one would set up an IKE Phase 1 validation of IP Address-based idenity in the presence of NAPTs, but that is a general problem fo which IPSec WG needs to provide the right set of tools (which is on its way.) I believe the draft-aboba-nat-ipsec-04.txt is just one proposal being considered. There may be others as well. Do you see any further clarification required in this area? Also, is there any conflict with the FCIP Short Frame proposal (the NAPTs) solution? Regards, Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Monday, November 12, 2001 10:23 AM To: ips@ece.cmu.edu Cc: Black_David@emc.com Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Ralph, See my response to Bob's mail. The situation is 1) with a few additions - the particular way in which FCIP has designed this identity mechanism makes group pre-shared keys a bad idea, and saying they SHOULD NOT or MUST NOT be used is part of the solution. I would prefer that FCIP use an FC level identifier rather than an IP address for identifying FC entities, but in the current design, that identifier is being used as an unsecured means of authentication, and hence some way of securing that authentication needs to be found. The reply to Bob outlines how to go about getting IKE and IPsec to solve this problem; the alternative of having FC try to solve this problem doesn't seem tractable to me because the problem occurs with the TCP connections after the first one, and FC is not going to want to rerun SLAP just because a second TCP connection has been set up (and in fact may not be able to). Thanks, --David > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Monday, November 12, 2001 7:54 AM > To: IPS Reflector > Cc: Black_David@emc.com > Subject: Re: FCIP: NAPTs Solution Proposal (issue from Irvine, CA > Interim meeting) > > > David, > > You appear to be saying one of two things: > > 1) The NAPTs solution, as proposed, requires that FCIP eschew the > use of group pre-shared keys, or > 2) The problems with the best on offer FCIP NAPTs solution design > are sufficient to convince the IESG that FCIP is justified in > prohibiting the use of NAPTs. > > Although time may have altered opinions, I will note that option > 2) conforms to the position of several people on the FCIP development > team during the Irvine meeting. Certainly, option 2) greatly > simplifies the changes required in FCIP. > > It is clear that you will object to ANY form of identification > that an FCIP Entity offers for one of the following reasons: > > a) Offering IP addresses is unacceptable because of NAPTs, and > b) Offering any other value is unacceptable because it is not > secure with group pre-shared keys. > > Unless option 1) above is your real concern, it appears that the > FCIP development team has spent two months on a near fruitless > exercise, whose only benefit is a strong proof that FCIP should > not support NAPTs. > > Ralph... > > Black_David@emc.com wrote: > > > > For those cases where a secure environment is required, the > > > new connection comes up through the normal IPSec authentication > > > and encryption processes. As a result, the transmission of > > > the identifying world wide names is already transmitted in an > > > encrypted format, creatable only by the authenticated and > > > certified source and interpretable only by the authenticated > > > and certified destination. > > > > The assumption that there's a big on/off switch for > > security is not correct (e.g., suppose IPsec is running > > with integrity on and confidentiality off) and > > "authenticated and certified" is a peculiar concept for > > group pre-shared keys, a very likely deployment scenario. > > Also, IPsec has no clue about the WWN, and hence the IKE > > authentication is not linked to the WWN that has to be > > presented (a particular problem with group pre-shared keys, > > but also a risk even with certificates). In other words, > > "authenticated and certified" doesn't place any restrictions > > on what WWN is presented. Finally, solving > > an inband authentication problem by requiring that IPsec be > > used to encrypt the WWN is wildly excessive, and besides, > > the WWN has no secret contents- it's a public piece of > > configuration information. > > > > There is a solution in this general area, but it involves > > linking the WWN to the IKE identity that is authenticated. > > That's probably going to break any attempt to use existing > > off-the-shelf external IPsec gateways because gateways > > don't grok WWNs, and the FCIP implementations won't > > know what identity was used in the IKE authentication > > to the gateway. This is all a bit peculiar because FC > > currently trust WWNs passed in various FC Login frames > > without authenticating them, but that's T11's area of > > responsibility putting this sort of thing into an IETF > > standard isn't going to be acceptable. > > > > > If the Fibre Channel fabric is also operating in a secure mode, > > > subsequent Fibre Channel authentication and certification > > > is performed using the standard FC SLAP mechanisms. > > > > Only on the first TCP connection, unless you plan to require > > taking the FCIP link down when the second TCP connection > > is added (which strikes me as highly counterproductive). > > The second connection just inherits the results of the SLAP > > performed on the first one, whether its entitled to or not. > > > > > In addition to this, there are a whole bunch of policy > > > restrictions that are verified as part of the creation > > > of subsequent connections. While these are not necessarily > > > part of the security steps, they prevent the formation of > > > connections which do not meet the security rules. > > > > I'm not sure what you're referring to, but I suspect they > > don't help much here. > > > > > Assuming security mechanisms are properly implemented, where, > > > then, is the security hole? > > > > The fact that the WWN is not authenticated means that anyone > > who can set up an FCIP connection to an FCIP entity can tap > > into any existing FC traffic flowing on any connection through > > that FCIP entity. > > > > --David > From owner-ips@ece.cmu.edu Tue Nov 13 04:52:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA18693 for ; Tue, 13 Nov 2001 04:52:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAD8Yjw16458 for ips-outgoing; Tue, 13 Nov 2001 03:34:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAD8Yhl16453 for ; Tue, 13 Nov 2001 03:34:43 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA70124 for ; Tue, 13 Nov 2001 09:34:31 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAD8YXi34280 for ; Tue, 13 Nov 2001 09:34:33 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI - change proposal LUN field definition on every PDU X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Tue, 13 Nov 2001 10:34:33 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 13/11/2001 10:34:34, Serialize complete at 13/11/2001 10:34:34 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Paul, I would appreciate if you could expand on what your concern about the fastpath is. And I would appreciate also you telling us what sufficient refers to. As for NO (uppercase "n" and "o") I do not find it in any "common acronym" list so I assume it is a way (impolite) of saying "we should not consider" it. Julo Paul Koning 12-11-01 19:02 Please respond to Paul Koning To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI - change proposal LUN field definition on every PDU Excerpt of message (sent 12 November 2001) by Julian Satran: > Dear colleagues, > > A colleague interested in instrumentation approached me with a question > about stateless logging of specific LU activity. > With the current iSCSI PDU formats this is not possible. > We have consistently avoided having fields that are redundant and will > need consistency checking. > However I think we should consider including the LU field in all PDUs that > are referencing a specific LU to simplify this type of instrumentation (as > we did with the direction bit in the op-code). NO. I can see no good reason to add work to an implementation fastpath for this. It slightly simplifies instrumentation, but instrumentation is certainly possible without it. That is sufficient. paul From owner-ips@ece.cmu.edu Tue Nov 13 04:56:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA18718 for ; Tue, 13 Nov 2001 04:56:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAD986B18323 for ips-outgoing; Tue, 13 Nov 2001 04:08:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAD984l18318 for ; Tue, 13 Nov 2001 04:08:05 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA42508 for ; Tue, 13 Nov 2001 10:07:57 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAD97xi45602 for ; Tue, 13 Nov 2001 10:07:59 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI - change proposal LUN field definition on every PDU X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Tue, 13 Nov 2001 11:07:58 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 13/11/2001 11:08:01, Serialize complete at 13/11/2001 11:08:01 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ken, Yes - that should affect only the PDUs mentioned plus in the Data-Out it will be thee even when the data is not in response to an R2T. Needles to say that this field being mainly for instrumentation targets are not supposed to check it for conformance. Julo "Ken Sandars" 12-11-01 20:19 Please respond to "Ken Sandars" To: , Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: iSCSI - change proposal LUN field definition on every PDU Hi Julo, Anything which helps get diagnostic and measurement equipment out is a GOOD thing, especially when there is a kick back to those who speak up in support ;-) Are the only changes to the SCSI Response, the Task Management Response and the Data-In PDUs? Cheers, Ken ksandars@eurologic.com Eurologic Systems +44 117 930 9616 From owner-ips@ece.cmu.edu Tue Nov 13 07:24:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA21120 for ; Tue, 13 Nov 2001 07:24:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADB12g06756 for ips-outgoing; Tue, 13 Nov 2001 06:01:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADB0xl06743 for ; Tue, 13 Nov 2001 06:01:00 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA63636 for ; Tue, 13 Nov 2001 12:00:51 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fADB0km24250 for ; Tue, 13 Nov 2001 12:00:47 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI - change proposal LUN field definition on every PDU X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Tue, 13 Nov 2001 13:00:44 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 13/11/2001 13:00:50, Serialize complete at 13/11/2001 13:00:50 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Santosh, We are talking about instrumentation that could work based only on what is on-the-wire. For an instrument (software I assume) that sits between SCSI and iSCSI this is obviously not an issue. It is not a major issue for an instrument on the wire (it must maintain some state anyhow) but telling something about traffic based on the field might simplify it. We can also make this field "not mandatory to test by receiver" to avoid having the receiver pay a penalty. As for the PDUs - SCSI Response, Data-Out & In and Task Mangement request/response. I am not sure what other uses or side-effects this change may have. Julo Santosh Rao Sent by: santoshr@cup.hp.com 12-11-01 20:55 Please respond to Santosh Rao To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI - change proposal LUN field definition on every PDU Julian, Can you explain further the extent of PDUs that are intended to be affected by this change ? Also, can you clarify why it is not a problem for other SCSI transport protocols like FC where the LUN field is not included in every FC IU ? IMO, the LUN knowledge is best kept out of the SCSI transport as far as possible. All LUN level logging/tracing belongs to the SCSI ULP, which is given a LUN context on each call made to it from the LLP. At the SCSI LLP layer, tracing/logging as well as state information should be at the I-T nexus level (session). I don't (yet) see a sufficient justification for this change. Perhaps, more information on the motivation for this request may change that. - Santosh Julian Satran wrote: > > Dear colleagues, > > A colleague interested in instrumentation approached me with a question > about stateless logging of specific LU activity. > With the current iSCSI PDU formats this is not possible. > We have consistently avoided having fields that are redundant and will > need consistency checking. > However I think we should consider including the LU field in all PDUs that > are referencing a specific LU to simplify this type of instrumentation (as > we did with the direction bit in the op-code). > > As I am already in "count-down" mode for version 09 I would appreciate > your comments ASAP. > > Julo -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Tue Nov 13 08:44:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA24030 for ; Tue, 13 Nov 2001 08:44:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADCoVo12701 for ips-outgoing; Tue, 13 Nov 2001 07:50:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1ab.compuserve.com (siaar1ab.compuserve.com [149.174.40.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADCoRl12694 for ; Tue, 13 Nov 2001 07:50:27 -0500 (EST) Received: (from mailgate@localhost) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id HAA28142 for ips@ece.cmu.edu; Tue, 13 Nov 2001 07:50:18 -0500 (EST) Received: from compuserve.com (dal-tgn-tkw-vty11.as.wcom.net [216.192.232.11]) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id HAA28106 for ; Tue, 13 Nov 2001 07:50:09 -0500 (EST) Message-ID: <3BF1183C.A6014852@compuserve.com> Date: Tue, 13 Nov 2001 06:55:25 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: FCIP: draft-ietf-ips-fcovertcpip-06b Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit An interim revision of the FCIP draft, draft-ietf-ips-fcovertcpip-06b is available on the T11 web site as: ftp://ftp.t11.org/t11/pub/fc/bb-2/01-482v2.pdf ftp://ftp.t11.org/t11/pub/fc/bb-2/01-482v2.txt This draft is awaiting review by the FCIP team before the r07 draft is sent to the Internet Drafts office, so changes are still possible/ likely. However, the FCIP authors will consider any requests for changes received before 5 pm EST on Wednesday, 7 November. It is my hope that the final r07 draft can be sent shortly after that. FYI r06b includes the following changes: a) Clarification of the TCP_NODELAY requirement in 8.4.3. b) Changes to figures 2 and 3 to clarify the nature of the FCIP Link. c) Incorporate the NAPTs solution agreed by the FCIP team. Enjoy. Ralph... From owner-ips@ece.cmu.edu Tue Nov 13 10:57:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA01017 for ; Tue, 13 Nov 2001 10:57:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADFEuA22964 for ips-outgoing; Tue, 13 Nov 2001 10:14:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fADFErl22958 for ; Tue, 13 Nov 2001 10:14:53 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fADF9aQ30638; Tue, 13 Nov 2001 10:09:36 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fADG9aq07890; Tue, 13 Nov 2001 11:09:36 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15345.14267.951000.951292@gargle.gargle.HOWL> Date: Tue, 13 Nov 2001 10:09:47 -0500 From: Paul Koning To: Julian_Satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI - change proposal LUN field definition on every PDU References: X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 13 November 2001) by Julian Satran: > Paul, > > I would appreciate if you could expand on what your concern about the > fastpath is. "fastpath" is a commonly used term to describe the performance critical parts of an implementation -- typically a software implementation. In designing a fastpath, a major consideration is the avoidance of *any* extra operations not strictly necessary for successful operation of the protocol in question. Supplying LUNs for monitoring devices is not needed for successful communication between initiator and target, and it consumes additional cycles. Not many, admittedly. But in fastpath design, you don't want to accept unnecessary overhead just because it's "not much". > And I would appreciate also you telling us what sufficient refers to. "sufficient" refers to the current design. Instrumentation can figure out what LUN each message relates to. It requires state to do so, but that is not a problem. > As for NO (uppercase "n" and "o") I do not find it in any "common acronym" > list so I assume it is a way (impolite) of saying "we should not consider" > it. Capitalizing a word is a common way to indicate emphasis, which is what I intended. It is not, as far as I know, a common way to indicate impoliteness, and indeed I did not intend any. paul From owner-ips@ece.cmu.edu Tue Nov 13 11:00:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01232 for ; Tue, 13 Nov 2001 11:00:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADFBF322641 for ips-outgoing; Tue, 13 Nov 2001 10:11:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar2aa.compuserve.com (siaar2aa.compuserve.com [149.174.40.137]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADFBAl22631 for ; Tue, 13 Nov 2001 10:11:11 -0500 (EST) Received: (from mailgate@localhost) by siaar2aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id KAA13994 for ips@ece.cmu.edu; Tue, 13 Nov 2001 10:11:04 -0500 (EST) Received: from compuserve.com (dal-tgn-tkr-vty53.as.wcom.net [216.192.229.53]) by siaar2aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id KAA13912 for ; Tue, 13 Nov 2001 10:10:56 -0500 (EST) Message-ID: <3BF1393B.79D0D2DD@compuserve.com> Date: Tue, 13 Nov 2001 09:16:12 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCIP: draft-ietf-ips-fcovertcpip-06b References: <08DC7D0912B46D4EBF224A34371D7BC2DE1952@cxoexc12.americas.cpqcorp.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit "Fraser, Don" wrote: > Did you mean Wed Nov 14th instead of the 7th? Yes. > > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Tuesday, November 13, 2001 5:55 AM > To: IPS Reflector > Subject: FCIP: draft-ietf-ips-fcovertcpip-06b > > An interim revision of the FCIP draft, > draft-ietf-ips-fcovertcpip-06b is available > on the T11 web site as: > > ftp://ftp.t11.org/t11/pub/fc/bb-2/01-482v2.pdf > ftp://ftp.t11.org/t11/pub/fc/bb-2/01-482v2.txt > > This draft is awaiting review by the FCIP team > before the r07 draft is sent to the Internet > Drafts office, so changes are still possible/ > likely. > > However, the FCIP authors will consider any > requests for changes received before 5 pm EST > on Wednesday, 7 November. It is my hope that > the final r07 draft can be sent shortly after > that. > > FYI r06b includes the following changes: > > a) Clarification of the TCP_NODELAY requirement in 8.4.3. > b) Changes to figures 2 and 3 to clarify the nature of > the FCIP Link. > c) Incorporate the NAPTs solution agreed by the FCIP > team. > > Enjoy. > > Ralph... From owner-ips@ece.cmu.edu Tue Nov 13 12:39:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07431 for ; Tue, 13 Nov 2001 12:39:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADGgtW00675 for ips-outgoing; Tue, 13 Nov 2001 11:42:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADGgpl00667 for ; Tue, 13 Nov 2001 11:42:51 -0500 (EST) Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.99.140.22]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA50686 for ; Tue, 13 Nov 2001 11:40:12 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay01.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fADGgnS49602 for ; Tue, 13 Nov 2001 09:42:49 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI reserved bits and fields To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Tue, 13 Nov 2001 08:41:57 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/13/2001 09:42:49 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I am sending a response to my note, to Paul Koning, onto the list, since I failed to send my note there to begin with. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com ---------------------- Forwarded by John Hufferd/San Jose/IBM on 11/13/2001 08:39 AM --------------------------- Paul Koning on 11/13/2001 06:56:47 AM To: John Hufferd/San Jose/IBM@IBMUS cc: Subject: Re: iSCSI reserved bits and fields Excerpt of message (sent 12 November 2001) by John Hufferd: > > Paul, > The reason for the wording was to permit unique vendor versions to be > produced before things are made standard or for vendors to add key value > (at their own risk until they get standardized. Good point, but such implementations are operating outside the standard, so the rules of the standard don't apply then. > I understood what you meant however, the must on the sender would normally > require the target to check. Not at all, and that's exactly what I did NOT intend. Sender behavior and receiver behavior are a separate matter. As a principle of design, receivers should check *only* what is necessary for correct results (i.e., for interoperability -- or sometimes other considerations, for example security protocols tend to check more because of a need to detect attacks). > Perhaps a compromise is a "MUST leave zero" on the sending side and "MUST > not check" on the target side. This would overtly tell implementers that > they are NOT a valid iSCSI implementation, but not cause things to break in > early implementations of features not yet standardized. That is exactly what I meant to propose; apparently I did not make it clear enough. > However, I really think the "SHOULD leave zero" on the Initiator side and > "MUST not check" on the target side, is sufficient. This is how folks have > generally built (at their own risk) new features. It is not clear we would > have the Jumbo Frames Features, on any products if this type of wordage was > not operative. I'm not sure about the jumbo frame example. In any case, I'll accept "should" though I'd prefer the "compromise" wording you gave. By the way, did you mean to send your note to the list? It seems that you did not. paul From owner-ips@ece.cmu.edu Tue Nov 13 12:42:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07768 for ; Tue, 13 Nov 2001 12:42:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADGNLx29049 for ips-outgoing; Tue, 13 Nov 2001 11:23:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADGNGl29035 for ; Tue, 13 Nov 2001 11:23:16 -0500 (EST) Received: from hq-ex-c2.brocade.com (hq-ex-c2 [192.168.126.35]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id IAA20297; Tue, 13 Nov 2001 08:23:05 -0800 (PST) Received: by hq-bes-c2.brocade.com with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Nov 2001 08:23:03 -0800 Message-ID: From: Robert Snively To: "'Julian Satran'" , ips@ece.cmu.edu Subject: RE: iSCSI - change proposal LUN field definition on every PDU Date: Tue, 13 Nov 2001 08:22:59 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, I have followed this thread up to the current time, and I have to agree with the people who think that this is probably not necessary and has undesirable consequences. One reason that it is not necessary is that as we write, the SCSI MIB people are defining the properties and statistic gathering capabilities of SCSI logical units. These capabilities are clearly going to require logical units to capture additional performance information using SCSI logging commands (which already capture a huge amount of error information). A second reason that it is not necessary is that the command and data counts can be extracted from the command PDU's CDB field, which is labeled with a logical unit. Given that the command completes correctly, the counting is already done and the count is in the command PDU. A third reason that it is not necessary is that any decent instrumentation on the line has to already be capable of sorting through the TCP/IP mapping, the security mappings, and the basic PDU identification. All the monitors that we have today in the SCSI environment apply a little simple postprocessing to the data they have captured and presto, they not only identify PDU's related to particular logical units, but to particular commands, and they have interpreted the commands and verified that the responses are appropriate to the commands. If the instrumentation is inline with the receiving code, it is by definition aware of the states and identifications already available to it and does not need this additional information. As for undesirable consequences: Paul Koning's objection to the additional functional steps required to provide the LUN is a valid concern. Mallikarjun Chadalapaka's concern about the requirement for additional consistency checking (and a whole new class of non-SCSI protocol error checking) is also a valid issue. Let's talk to your colleague and explain to him/her how there are better ways to achieve his/her goals. Bob Snively e-mail: rsnively@brocade.com Brocade Communications Systems phone: 408 487 8135 1745 Technology Drive San Jose, CA 95110 > -----Original Message----- > From: Julian Satran [mailto:Julian_Satran@il.ibm.com] > Sent: Monday, November 12, 2001 4:31 AM > To: ips@ece.cmu.edu > Subject: iSCSI - change proposal LUN field definition on every PDU > Importance: High > > > Dear colleagues, > > A colleague interested in instrumentation approached me with > a question > about stateless logging of specific LU activity. > With the current iSCSI PDU formats this is not possible. > We have consistently avoided having fields that are redundant > and will > need consistency checking. > However I think we should consider including the LU field in > all PDUs that > are referencing a specific LU to simplify this type of > instrumentation (as > we did with the direction bit in the op-code). > > As I am already in "count-down" mode for version 09 I would > appreciate > your comments ASAP. > > Julo > From owner-ips@ece.cmu.edu Tue Nov 13 13:44:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA11605 for ; Tue, 13 Nov 2001 13:44:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADHV0204820 for ips-outgoing; Tue, 13 Nov 2001 12:31:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADHUvl04814 for ; Tue, 13 Nov 2001 12:30:57 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id JAA01612; Tue, 13 Nov 2001 09:30:48 -0800 (PST) From: Keith McCloghrie Message-Id: <200111131730.JAA01612@cisco.com> Subject: Re: FC Management MIB - proposed changes To: marjorie_krueger@hp.com ("KRUEGER,MARJORIE (HP-Roseville,ex1)") Date: Tue, 13 Nov 2001 09:30:47 -0800 (PST) Cc: kzm@cisco.com ('Keith McCloghrie'), ips@ece.cmu.edu In-Reply-To: <6BD67FFB937FD411A04F00D0B74FE87805CCF20F@xrose06.rose.hp.com> from "KRUEGER,MARJORIE (HP-Roseville,ex1)" at Nov 12, 2001 05:58:00 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Marjorie, > A little background on the intent of these two MIBs - the FC Mgmt MIB was > originally intended to represent "everything that had Fibre Channel Ports". > In that sense, it is analogous to the Interfaces Group MIB. Your analogy doesn't (to me) seem very accurate. (Only RFC 2863, not the FC Mgmt MIB, has many other MIBs which extend its functionality with more specific information.) > RFC 2837 was > intended to represent FC Fabric Switch information - more like a "router > MIB" - intended to represent the information specific to the functions of a > Fibre Channel fabric switch. Well, the IETF has not specified a "router MIB". It's better to specify MIBs in smaller functional increments (e.g., OSPF MIB, BGP MIB, etc.), rather than as "type of box" MIBs. This allows functions which are common to multiple types of boxes to be specified in one MIB and implemented by all the types, e.g., the ATM interface MIB applies to ATM hosts and ATM switches. Moral: the definition of "type of box" MIBs can be a contributory factor to the kinds of problems that we have now. > That doesn't seem to me to be information that > should be in an "interfaces MIB". It doesn't seem logical to replace RFC > 2837 with a new FC Mgmt MIB, the intent of both MIBs is/should be much > different. All MIB information which is relevant to an FC interface should have been specified as an extension of the ifTable. This applies to both the FC Mgmt MIB and to RFC 2837. That information should be specified only once. I was focussing on the significant overlap between them. Above you seem to be saying there are significant differences between them. Perhaps, both the overlap *and* the differences are significant. Let me work on it and see what emerges. > The name server stuff never belonged in the FC Mgmt MIB, it (and > other Fabric Services information) belongs in the RFC 2837 MIB. You're right that the connUnitSnsTable describes itself as: "This table contains an entry for each object registered with this port [sic] in the switch." However, see comments above on "type of box" MIBs; it would be better for it to be in a Name Server MIB. Keith. > Thoughts? > Marjorie > > > > 3. With 20x20 hindsight, the following observations can be made with > > respect to RFC 2837: > > > > - the reason that we now have the issue of how it relates to the > > FC-Mgmt MIB is because it was written as a Fabric Element MIB; > > this issue would not have arisen if it had been written as a > > Fibre Channel interface MIB. > > - it should have extended the ifTable, rather than > > overlapping with it, > > - it should not have defined the fcFeModuleTable, which overlaps with > > the Entity MIB. > > - and it should have specified (at least) its octet counters > > as Counter64. > > > > Given these problems, I propose that the new FC-Mgmt MIB be specified > > as a replacement for RFC 2837. > > > > > From owner-ips@ece.cmu.edu Tue Nov 13 13:45:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA11708 for ; Tue, 13 Nov 2001 13:45:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADIF6008418 for ips-outgoing; Tue, 13 Nov 2001 13:15:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADIEwl08390 for ; Tue, 13 Nov 2001 13:14:58 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id TAA66936 for ; Tue, 13 Nov 2001 19:14:48 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fADIEom89832 for ; Tue, 13 Nov 2001 19:14:50 +0100 Importance: Normal Subject: iSCSI: The new security normative statements in 09 To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Ofer Biran" Date: Tue, 13 Nov 2001 20:15:53 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 13/11/2001 20:14:50 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Following are the new security normative statements which we intend to add in iSCSI version 09. They are result of the sync effort with the security draft, the tunnel / transport mode discussion and the SRP groups discussion in the security team. For the mode statements - there is a clear preference for tunnel mode, and at this point we have to put a MUST. Actually there was only one supporter of transport mode (Saqib Jang) with the argument: "to avoid folks playing 'marketing' games with positioning their products as iSCSI compliant". This is not really a technical argument, in anyway, an implementation that doesn't provide IKE, ESP in tunnel mode with 3DES in CBC mode... will not be iSCSI compliant, no matter what marketing people say. ====================================================================== 10.2 In-band Initiator-Target Authentication (SRP) "As described in [RFC2945], N is required to be a Sophie-German prime (of the form N = 2q + 1, where q is also prime) and the generator g is a primitive root of GF(n). For use in iSCSI authentication, the prime modulus N MUST be at least 768 bits. Upon receiving N and g from the Target, the Initiator MUST verify that they satisfy the above requirements (and abort the connection otherwise). This verification MAY start by trying to match them with a well-known group that satisfies the above requirements. SRP well-known groups are given in [SEC-IPS]." 10.3.1 Data Integrity and Authentication "An iSCSI compliant initiator or target MUST provide data integrity and authentication by implementing IPsec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." "At the high speeds iSCSI is expected to operate, a single IPsec SA could rapidly cycle through the 32-bit IPsec sequence number space. In view of this, iSCSI implementation operating at speeds of 1 Gbps or less MAY implement the IPsec sequence number extension [SEQ-EXT]. Implementation operating at speeds of 10 Gbps or faster SHOULD implement the sequence number extension." 10.3.2 Confidentiality "An iSCSI compliant initiator or target MUST provide confidentiality by implementing IPsec [RFC2401] with ESP in tunnel mode [RFC2406] with the following..." "DES in CBC mode SHOULD NOT be used due to its inherent weakness." 10.3.3 Security Associations and Key Management "Authentication, security association negotiation and key management MUST be provided by implementing IKE [RFC2409] using the IPsec DOI [RFC2407] with the following iSCSI specific requirements: - Peer authentication using a pre-shared key MUST be supported, and certificate-based peer authentication using digital signatures MAY be supported. Peer authentication using the public key encryption methods outlined in IKE's sections 5.2 and 5.3[7] SHOULD NOT be used. - When digital signatures are used to achieve authentication, an IKE negotiator SHOULD use IKE Certificate Request Payload(s) to specify the certificate authority. IKE negotiators SHOULD check the pertinent Certificate Revocation List (CRL) before accepting a PKI certificate for use in IKE's authentication procedures. - Both IKE Main Mode and Aggressive Mode MUST be supported. IKE main mode with pre-shared key authentication method SHOULD NOT be used when either the initiator or the target uses dynamically assigned IP addresses (while pre-shared keys in many cases offer good security, situations where dynamically assigned addresses are used force the use of a group pre-shared key which creates vulnerability to man-in-the-middle attack)." "Manual keying MUST NOT be used since it does not provide the necessary rekeying support." ====================================================================== Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Tue Nov 13 13:53:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA12214 for ; Tue, 13 Nov 2001 13:53:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADHwgE07041 for ips-outgoing; Tue, 13 Nov 2001 12:58:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADHwcl07032 for ; Tue, 13 Nov 2001 12:58:38 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA74928 for ; Tue, 13 Nov 2001 18:58:32 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fADHwYm43908 for ; Tue, 13 Nov 2001 18:58:34 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Clarification (again) for Task Management Commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Tue, 13 Nov 2001 19:58:33 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 13/11/2001 19:58:35, Serialize complete at 13/11/2001 19:58:35 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, I assume you may want to reread 9.4. It works well on any number of connections and it handles all commands for which there is a "delivery uncertainty". For commands for which the delivery is certain - i.e, they have an instantiated Task Block at both initiator and target there is no uncertainty either and the implementation of "no more statuses" does not raise any special issues wherever their statuses are. The initiator portion of the state has to be marked as aborted and kept in place until the status arrives. To clarify this implementation detail I have added some words to the relevant part of the Task Management request: For all these functions, if executed, the Task Management Function Response MUST be returned using the Initiator Task Tag to identify the operation for which it is responding. All those functions apply to the referenced tasks regardless if they are proper SCSI tasks or tagged iSCSI operations. Task management commands must be executed as if all the commands having a CmdSN lower or equal to the task management CmdSN have been received by the target (i.e., have to be executed as if received for ordered delivery even when marked for immediate delivery). For all the tasks covered by the task management response (i.e., with CmdSN not higher than the task management command CmdSN), additional responses MUST NOT be delivered to the SCSI layer after the task management response. This requirement implies that the initiator must keep around state until the status is received from the target for an aborted task and the target MUST deliver to the initiator good status for an aborted task. Julo "Somesh Gupta" Sent by: owner-ips@ece.cmu.edu 13-11-01 03:46 Please respond to somesh_gupta To: "IPS" cc: Subject: RE: iSCSI: Clarification (again) for Task Management Commands Julian, Section 9.4 does (or so I can figure) leave the sort of hole mentioned below uncovered. On a multiple connection session, the status for any of the commands effected by the task management command could already be in transit on any of the connections by the time the task management command is received by the target, and its reception is acknowledged - and then received by the initiator. The barrier works well for a single connection because the command sequence number for the task management command makes sure that there is nothing stuck in the "pipe". However, there could be status "stuck" in the other pipes. The mechanism David indicated might be the only way to be sure, which however might require a NOP on each connection to ensure there is nothing in the pipe. Somesh > -----Original Message----- > From: Julian Satran [mailto:Julian_Satran@il.ibm.com] > Sent: Saturday, November 10, 2001 7:26 AM > To: somesh_gupta@silverbacksystems.com > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > Read 9.4 for one implementation - Julo > > > > > "Somesh Gupta" > Sent by: owner-ips@ece.cmu.edu > 10-11-01 05:26 > Please respond to somesh_gupta > > > To: , > cc: > Subject: RE: iSCSI: Clarification (again) for Task > Management Commands > > > > David, > > In iSCSI the multiple connection/session construct > adds significant complexity in determining whether > a response for a command (on a different connection > than the one on which the task management command > was sent) impacted by the task management command will > be received or not - and when? > > On a single connection, or similar links, when the > response for the task management command is > received (or after a fixed additional time), no > responses will be received for the commands aborted > by the task management command. > > However, with multiple connections there is no > such "flushing" event on connections other than the > one on which the task management command was sent. > > I would hope that the protocol would address this > situation - the current response seems to be be to > put the task tag and some associated resources in > a zombie state for an unspecified amount of time. > > Somesh > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Friday, November 09, 2001 6:53 PM > > To: somesh_gupta@silverbacksystems.com; ips@ece.cmu.edu > > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > > > > Somesh, > > > > The language in question reflects fairly direct requirements > > language to be found in SAM-2's description of SCSI Task Management. > > FCP goes to serious lengths with FC sequence aborts to make sure > > this behaves as required. > > > > For iSCSI, if responses to the aborted commands show up unexpectedly, > > they have to be discarded. How the Initiator keeps track of that > > is the Initiator's problem - keeping track of the CmdSN of the > > Abort Task Set may be useful. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > -----Original Message----- > > > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > > > Sent: Friday, November 09, 2001 4:55 PM > > > To: IPS > > > Subject: iSCSI: Clarification (again) for Task Management Commands > > > > > > > > > Resend to add iSCSI tag. Sorry for missing it. > > > > > > On page 67 of the 8-92.txt draft (section 3.5.1), it > > > says > > > > > > "For all the tasks covered by the task > > > management response (i.e., with CmdSN not higher than the task > > > management command CmdSN), additional responses MUST NOT be delivered > > > to the SCSI layer after the task management response." > > > > > > If there is a multiple connection session, > > > a status for a command impacted by the task > > > management command (say ABORT TASK SET) could > > > be stuck in the pipe on one connection, while > > > the ABORT TASK SET completes on another > > > connection. > > > > > > How does the initiator iSCSI enforce the rule above? > > > Seems to be the equivalent of sending the impacted > > > commands on other connections in a zombie state, > > > and not having a very good idea of how to get out. > > > > > > Similarly Section 9.4 provides additional rules, > > > but seems to leave a hole open with regards to > > > status already in flight on other connections. > > > > > > Any clarifications would be appreciated. > > > > > > Somesh > > > > > > > > > From owner-ips@ece.cmu.edu Tue Nov 13 13:53:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA12254 for ; Tue, 13 Nov 2001 13:53:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADI58f07598 for ips-outgoing; Tue, 13 Nov 2001 13:05:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADI54l07589 for ; Tue, 13 Nov 2001 13:05:04 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W49AAWV>; Tue, 13 Nov 2001 13:01:19 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293717C@CORPMX14> From: Black_David@emc.com To: vrangan@rhapsodynetworks.com, ips@ece.cmu.edu Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Tue, 13 Nov 2001 12:55:06 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Venkat, Let me try to summarize and explain without quoting your entire message: -- Pre-shared keys The existing text in the IPS security draft on pre-shared keys and IKE authentication modes is fine in the absence of the recent WWN short frame addition. That addition changes the security properties of FCIP by introducing an inband authentication/authorization for which it is necessary to provide security. Doing so without using an inband authentication protocol impacts group pre-shared keys and other things. -- WWN short frame and administration > "The usage of SLPv2 by FCIP is described in [64]. FCIP Entities assume > that once the IKE identity of a peer is established, the FCIP Entity > Name carried in FCIP Short Frame is also implicity accepted as the > authenticated peer. Any such association between the IKE identity and > the FCIP Entitiy Name is administratively established." > Do you see any further clarification required in this area? > Also, is there > any conflict with the FCIP Short Frame proposal (the NAPTs) solution? Work is definitely required here, because that short frame is serving an authentication/authorization purpose and hence the means need to be provided to adequately secure it. The assumption in the second sentence above isn't sufficient because it opens up nasty attacks including the denial of service ones I described earlier. In addition, that assumption makes IKE and ESP cryptographic integrity at least "SHOULD use" for FCIP, and I can't promise that the Security ADs will settle for "SHOULD use" as opposed to "MUST use". The reason for this change from the "MAY use" that applied prior to the introduction of the WWN short frame is that the authentication/authorization performed by that short frame is a class of function that is far more important, expected, and widely used than cryptographic integrity - the assumption uses cryptographic integrity to secure a mandatory authentication mechanism and hence increases the requirement for cryptographic integrity. And as things currently stand, the "administrative establishment" of that association will need to be done not only at the sender of the short frame, but also at the recipient. When IKE is in use, both establishment of the association and the check at the receiver (IKE identity for IPsec SA and WWN in short frame that arrived on that SA are associated) will need to be "MUST"s. Group pre-shared keys make these sort of checks difficult to specify and use properly - the fastest way to resolve this is to make group pre-shared keys "MUST NOT use" for FCIP. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Nov 13 14:40:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14316 for ; Tue, 13 Nov 2001 14:40:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADJBiO13391 for ips-outgoing; Tue, 13 Nov 2001 14:11:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADJBel13379 for ; Tue, 13 Nov 2001 14:11:40 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id LAA19182; Tue, 13 Nov 2001 11:11:22 -0800 (PST) From: "Murali Rajagopal" To: "Elizabeth Rodriguez" , "Charles Monia" , "IPS Reflector" Subject: RE: FCencap: List ALL SOF/EOF codes Date: Tue, 13 Nov 2001 11:12:31 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 Importance: Normal In-Reply-To: <9410A0F67DFE7F4D998D4538236498360408F5@nc8220exchange.ral.lucent.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I would like to add another option to Elizabeth's list : Option 5: Add Class 4 to the tables in FC-BB-2 and reference these codes. I will vote for Option 5. -Murali -----Original Message----- From: Elizabeth Rodriguez [mailto:egrodriguez@lucent.com] Sent: Tuesday, November 13, 2001 11:05 AM To: Murali Rajagopal; Charles Monia; IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes (Chair hat on) Let me jump back in here, and summarize as I see it: Codes from FC-BB cannot be taken intact, since there are invalid codes in that specification. Codes initially trimmed in January of this year, but Ralph correctly points out, that was prior to the formation of the FC encapsulation draft (e.g. discussion pertained directly to FCIP; did not take into consideration iFCP) That said, we can modify the current FC encapsulation draft to include other classes of service, if we so choose. We can choose to 1) Keep the table as is. Note: The current SOF/EOF codes defined in the FC encapsulation draft match the currently defined FC-BB-2 subset. 2) Add Class 1 codes Note: FC-MI (technical report, not a standard) does not support Class 1. Note2: Practicality of Class 1 over IP questioned. 3) Add Class 4 codes Note: Class 4 work is ongoing, but SOF/EOF codes currently defined for class 4 unchanged. 4) Add both class 1 and class 4 codes I would like to solicit input from everyone on what codes people feel should be included in the FC encapsulation draft. Note: Drafts following FC encapsulation may restrict classes of service that draft supports. Thanks, Elizabeth -----Original Message----- From: Murali Rajagopal [mailto:muralir@lightsand.com] Sent: Monday, November 12, 2001 6:01 PM To: Charles Monia; IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes On the specific topic of supported SOF and EOF codes the ietf documents should be driven by the specification provided in the *most relevant * document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI should be kept out of this. If we simply accept to adopt the SOF and EOF codes listed for BB-2 the problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. I don't see why we are making a big deal about this. -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Charles Monia Sent: Monday, November 12, 2001 3:33 PM To: IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Hi Folks: > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. The last time I checked, the FC-MI spec was not a "standards track" document (to use IETF terminology). If that's still the case, is FC-MI's prohibition of class 1 a sufficient basis for precluding class 1 support in the encapsulation spec? Charles > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Saturday, November 10, 2001 8:52 AM > To: IPS Reflector > Cc: Black_David@emc.com > Subject: Re: FCencap: List ALL SOF/EOF codes > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. > > Therefore, I am forced to agree with David. Class 1 MUST NOT > be mentioned in the FC Encapsulation draft. If necessary, a > note discussing interoperability and FC-MI can be added. > > Thanks. > > Ralph... > > Black_David@emc.com wrote: > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > I in FC-MI stands for "Interoperability", this seems like a > reasonable > > rationale for excluding Class 1 service. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > From owner-ips@ece.cmu.edu Tue Nov 13 14:40:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14329 for ; Tue, 13 Nov 2001 14:40:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADIeT110569 for ips-outgoing; Tue, 13 Nov 2001 13:40:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from auemail2.firewall.lucent.com (auemail2.lucent.com [192.11.223.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADIeHl10529 for ; Tue, 13 Nov 2001 13:40:17 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by auemail2.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fADIeA528863 for ; Tue, 13 Nov 2001 13:40:10 -0500 (EST) Content-Class: urn:content-classes:message Subject: RE: FCencap: List ALL SOF/EOF codes MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Date: Tue, 13 Nov 2001 13:40:06 -0500 Message-ID: <9410A0F67DFE7F4D998D45382364983617B2EE@nc8220exchange.ral.lucent.com> Thread-Topic: FCencap: List ALL SOF/EOF codes Thread-Index: AcFr2VEJolJTOzDJQZCUv0fH74sROwAmJ7bA From: "Elizabeth Rodriguez" To: "Murali Rajagopal" , "Charles Monia" , "IPS Reflector" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fADIeNl10554 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Charles, Franco (and others interested in iFCP) Murali's statement below does apply to the FCIP draft. Do you feel the statement also applies to iFCP? Elizabeth -----Original Message----- From: Murali Rajagopal [mailto:muralir@lightsand.com] Sent: Monday, November 12, 2001 6:01 PM To: Charles Monia; IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes On the specific topic of supported SOF and EOF codes the ietf documents should be driven by the specification provided in the *most relevant * document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI should be kept out of this. If we simply accept to adopt the SOF and EOF codes listed for BB-2 the problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. I don't see why we are making a big deal about this. -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Charles Monia Sent: Monday, November 12, 2001 3:33 PM To: IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Hi Folks: > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. The last time I checked, the FC-MI spec was not a "standards track" document (to use IETF terminology). If that's still the case, is FC-MI's prohibition of class 1 a sufficient basis for precluding class 1 support in the encapsulation spec? Charles > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Saturday, November 10, 2001 8:52 AM > To: IPS Reflector > Cc: Black_David@emc.com > Subject: Re: FCencap: List ALL SOF/EOF codes > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. > > Therefore, I am forced to agree with David. Class 1 MUST NOT > be mentioned in the FC Encapsulation draft. If necessary, a > note discussing interoperability and FC-MI can be added. > > Thanks. > > Ralph... > > Black_David@emc.com wrote: > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > I in FC-MI stands for "Interoperability", this seems like a > reasonable > > rationale for excluding Class 1 service. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > From owner-ips@ece.cmu.edu Tue Nov 13 14:46:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14475 for ; Tue, 13 Nov 2001 14:46:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADJ4wn12809 for ips-outgoing; Tue, 13 Nov 2001 14:04:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ihemail1.firewall.lucent.com (ihemail1.lucent.com [192.11.222.161]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADJ4rl12799 for ; Tue, 13 Nov 2001 14:04:53 -0500 (EST) Received: from nc8220exchange.ral.lucent.com (h135-92-100-21.lucent.com [135.92.100.21]) by ihemail1.firewall.lucent.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id fADJ4o525161 for ; Tue, 13 Nov 2001 14:04:51 -0500 (EST) Content-Class: urn:content-classes:message Subject: RE: FCencap: List ALL SOF/EOF codes MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Date: Tue, 13 Nov 2001 14:04:50 -0500 Message-ID: <9410A0F67DFE7F4D998D4538236498360408F5@nc8220exchange.ral.lucent.com> Thread-Topic: FCencap: List ALL SOF/EOF codes Thread-Index: AcFr2VEJolJTOzDJQZCUv0fH74sROwAl+RrQ From: "Elizabeth Rodriguez" To: "Murali Rajagopal" , "Charles Monia" , "IPS Reflector" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fADJ4sl12802 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit (Chair hat on) Let me jump back in here, and summarize as I see it: Codes from FC-BB cannot be taken intact, since there are invalid codes in that specification. Codes initially trimmed in January of this year, but Ralph correctly points out, that was prior to the formation of the FC encapsulation draft (e.g. discussion pertained directly to FCIP; did not take into consideration iFCP) That said, we can modify the current FC encapsulation draft to include other classes of service, if we so choose. We can choose to 1) Keep the table as is. Note: The current SOF/EOF codes defined in the FC encapsulation draft match the currently defined FC-BB-2 subset. 2) Add Class 1 codes Note: FC-MI (technical report, not a standard) does not support Class 1. Note2: Practicality of Class 1 over IP questioned. 3) Add Class 4 codes Note: Class 4 work is ongoing, but SOF/EOF codes currently defined for class 4 unchanged. 4) Add both class 1 and class 4 codes I would like to solicit input from everyone on what codes people feel should be included in the FC encapsulation draft. Note: Drafts following FC encapsulation may restrict classes of service that draft supports. Thanks, Elizabeth -----Original Message----- From: Murali Rajagopal [mailto:muralir@lightsand.com] Sent: Monday, November 12, 2001 6:01 PM To: Charles Monia; IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes On the specific topic of supported SOF and EOF codes the ietf documents should be driven by the specification provided in the *most relevant * document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI should be kept out of this. If we simply accept to adopt the SOF and EOF codes listed for BB-2 the problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. I don't see why we are making a big deal about this. -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Charles Monia Sent: Monday, November 12, 2001 3:33 PM To: IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Hi Folks: > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. The last time I checked, the FC-MI spec was not a "standards track" document (to use IETF terminology). If that's still the case, is FC-MI's prohibition of class 1 a sufficient basis for precluding class 1 support in the encapsulation spec? Charles > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Saturday, November 10, 2001 8:52 AM > To: IPS Reflector > Cc: Black_David@emc.com > Subject: Re: FCencap: List ALL SOF/EOF codes > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > prohibits Class 1 and I can find no letter ballot comments > asking that it be reinstated. > > Therefore, I am forced to agree with David. Class 1 MUST NOT > be mentioned in the FC Encapsulation draft. If necessary, a > note discussing interoperability and FC-MI can be added. > > Thanks. > > Ralph... > > Black_David@emc.com wrote: > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > I in FC-MI stands for "Interoperability", this seems like a > reasonable > > rationale for excluding Class 1 service. > > > > --David > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- > From owner-ips@ece.cmu.edu Tue Nov 13 14:52:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14612 for ; Tue, 13 Nov 2001 14:52:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADJGUg13798 for ips-outgoing; Tue, 13 Nov 2001 14:16:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADJGMl13773 for ; Tue, 13 Nov 2001 14:16:22 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Nov 2001 14:16:17 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293717E@CORPMX14> From: Black_David@emc.com To: Ken.Hirata@Vixel.com Cc: ips@ece.cmu.edu Subject: RE: FC Management MIB - proposed changes Date: Tue, 13 Nov 2001 14:06:27 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ah, now I see the confusion. The "nameservice" that is actually used by a port attached to a Fibre Channel fabric in order to locate other ports with which to communicate is the Directory Service described in Section 5 of FC-GS3. That's the service for which management visibility via the MIB is needed. There's only one instance of such a service in a fabric, and the interface from a port to that service is the same independent of whether zoning is in use. Section 6 of FC-GS3 defines the Management Service which is an inband Fibre Channel analogue to SNMP and MIBs (it is entirely for management). When the FC management MIB is implemented in a fabric switch, I would expect the name service objects in it to report information equivalent to the Unzoned Name service. One then figures out the behavior of the directory service by retrieving this information plus the zone information. OTOH, if the MIB is implemented in another device (e.g., FC to SCSI bridge that does not contain an embedded FC switch), the MIB may report only the zoned view of the directory service that's visible to that device. Thanks, --David > -----Original Message----- > From: Ken Hirata [mailto:Ken.Hirata@Vixel.com] > Sent: Tuesday, November 13, 2001 12:52 PM > To: Black_David@emc.com > Cc: kzm@cisco.com; ips@ece.cmu.edu > Subject: Re: FC Management MIB - proposed changes > > > David, > > One nit (or misunderstanding). In item 5 below, Zoned and Unzoned > name services are simultaneously active. > > My understanding is that Zoned name services present a view to a > device of all other devices with which it may communicate (same as > your explanation). > > Unzoned name services are used by a Management application to > retrieve information about all devices in the Fabric. > > Ken > > Black_David@emc.com wrote: > > > Folks, > > > > For simplicity, I would propose keeping the initial revision focused > > on structural and format changes only - i.e., make no functional > > changes that aren't required to bring the MIB into line with the > > overall architecture of MIBs, use of SMIv2, and the like. > Comments on > > this, keyed to Keith's issue numbers: > > > > 1. The trap table doesn't match up with RFC 2573 - I presume > > correcting this is part of the conversion to SMIv2. > > > > 3. Let's keep the replacement of RFC 2837 as (a) a proposal > > and (b) Keith's recommendation to the WG as MIB Advisor, > > but not formally adopt that approach until we have a version > > of the MIB draft that would replace RFC 2837 available for > > review by the WG. > > > > 5. I think there's a misunderstanding here. There is only one > > name service in any FC fabric, period. The term > "Simple Name > > Service" is historical and refers to the current approach > > to Fibre Channel fabric name service by contrast to the > > originally-proposed X.500 directory-based approach (see the > > original FC-GS) - the meaning of "Simple" should now be > > obvious, as it's by comparison to X.500 ;-). > > > > Both the Zoned and Unzoned name services are simple > name services. > > The same objects can be used to represent both, as > only one is > > active at any time, and the communication > interfaces/protocols > > are identical. Client access to the name service > is the same > > in both cases; zoned vs. unzoned only affects > server behavior > > in terms of what is returned to a query. In > addition the name > > service objects are described to represent only entities > > "known to this unit". So, in a zoned fabric, I would expect > > the table in a switch to be completely populated, > but the table > > in an HBA to contain only the entries in that HBA's zone > > because the HBA doesn't "know" about any others. > > > > The name server objects need to be retained - these > are quite > > important for fabric management visibility. > Whether the fabric > > is zoned or not and how the nameserver behaves can > be determined > > from the zone objects (i.e., for a switch, the > nameserver objects > > contain everything and the zone objects have to be > consulted to > > figure out what a query from a particular client to > the nameserver > > will return). > > > > 7. I would definitely take out Class 1 support, and reference FC-MI > > (which prohibits Class 1 service) as an authority > for doing so. > > > > 8. For the initial version, I would only carry forward the zone > > objects existing in the current MIB and not define any new > > functionality - that can be done in revisions after -00. > > > > 9. I'd prefer that the -00 version contain no additional functional > > changes beyond those required to support the necessary > > structure and format changes. Once we have that > -00 baseline, > > functional changes can be made from there. > > > > 2, 4, and 6 look fine to me, as does the new draft name. > > > > Comments? > > > > Thanks, > > --David > > -- > Kenneth Hirata > Vixel Corporation > Irvine, CA 92618 > Phone: (949) 450-6100 > Email: Ken.Hirata@Vixel.com > > From owner-ips@ece.cmu.edu Tue Nov 13 14:54:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14715 for ; Tue, 13 Nov 2001 14:54:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADJ92g13146 for ips-outgoing; Tue, 13 Nov 2001 14:09:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel2.hp.com (palrel2.hp.com [156.153.255.234]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADJ8xl13140 for ; Tue, 13 Nov 2001 14:08:59 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel2.hp.com (Postfix) with ESMTP id 3F0CDA4B for ; Tue, 13 Nov 2001 11:08:58 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id LAA28330 for ; Tue, 13 Nov 2001 11:10:26 -0800 (PST) Message-Id: <200111131910.LAA28330@core.rose.hp.com> Date: Tue, 13 Nov 2001 11:21:25 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: iSCSI: SCSI timeout handling change Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit All: Currently, if a command is not acknowledged by the ULP timeout, iSCSI mandates the initiators to tear up the session. The rationale behind this is that if the initiator could not get the command through (in possibly multiple retries) even by the ULP timeout, there's a serious problem with the session. But there are some drawbacks to this approach - - tearing up a session due to a NIC failure is disruptive to potentially several other active tasks on other NICs. - this puts those initiator implementations not wanting to do within-connection recovery (i.e. no retries) at a disadvantage, since one digest error would cause potentially several active I/Os to be terminated. - (albeit not very serious, ) this behavior is different from today's storage stacks' expectations - of being able to selectively abort one I/O on a timeout (with no command retransmissions). To address these issues, and also to simplify the current Task Management request PDU, I propose the following changes to handling SCSI timeouts - Following changes to section 3.5: - Abort Task MUST always be sent immediate. - Abort Task task management function request MUST be sent with its CmdSN equal to the CmdSN of the task to be aborted, and the Referenced Task Tag initialized to the ITT of the task to be aborted. - Consequent to the above, drop the RefCmdSN field in the Task Management command payload that is currently only used by the Abort Task function. Following changes to section 8.6: Propose the following text to replace the current - An iSCSI initiator MAY attempt to plug a command sequence gap on the target end (in the absence of an acknowledgement of the command by way of ExpCmdSN) before the ULP timeout by retrying the unacknowledged command, as described in section 8.1. On a ULP timeout for a command that carried a CmdSN of n, if the ExpCmdSN is still less than (n+1) on ULP timeout, the iSCSI initiator MUST abort the command using the Abort Task task management function request. In this process, the target may see the abort request before the original command itself due to one of the three reasons - - the original command was dropped due to digest error, or - the Abort Task request was shipped out-of-order on the same connection, or - the connection the original command sent on was successfully logged out. If the abort request is received prior to the original command, targets MUST consider the original command with that CmdSN to be received and discard the original command if and when received - i.e. treating it as a duplicate CmdSN. Initiators desirous of maintaining command ordering while maintaining the same session MUST NOT issue Abort Task on an unacknowledged command because of this reason. Following changes to section 2.2.2.1: - The above approach exposes the possibility that some stale (aborted from target's perspective) commands could be stuck in the TCP connection long enough for the CmdSN wrap - similar to the issue we dealt with for command retries. So, aborting unacknowledged commands should require the same flushing actions described for command retries. [ I almost would prefer at this point to require flushing all connections every 2^31 -1 commands starting from InitCmdSN, than enumerating these cases individually...] Comments? -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Tue Nov 13 15:36:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15920 for ; Tue, 13 Nov 2001 15:36:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADJjKQ16268 for ips-outgoing; Tue, 13 Nov 2001 14:45:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from eumenes.hosting.pacbell.net (eumenes.hosting.pacbell.net [216.100.98.24]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADJjCl16258 for ; Tue, 13 Nov 2001 14:45:17 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by eumenes.hosting.pacbell.net id OAA25644; Tue, 13 Nov 2001 14:44:57 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: Subject: RE: iSCSI - change proposal LUN field definition on every PDU Date: Tue, 13 Nov 2001 11:41:55 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I agree with all the folks who oppose this change. The approach should be to question usefulness of each and every field, not add more. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Robert Snively > Sent: Tuesday, November 13, 2001 8:23 AM > To: 'Julian Satran'; ips@ece.cmu.edu > Subject: RE: iSCSI - change proposal LUN field definition on every PDU > > > Folks, > > I have followed this thread up to the current time, and I have > to agree with the people who think that this is probably > not necessary and has undesirable consequences. > > One reason that it is not necessary is that as we write, the > SCSI MIB people are defining the properties and statistic > gathering capabilities of SCSI logical units. These capabilities > are clearly going to require logical units to capture additional > performance information using SCSI logging commands (which > already capture a huge amount of error information). > > A second reason that it is not necessary is that the command > and data counts can be extracted from the command PDU's > CDB field, which is labeled with a logical unit. Given that > the command completes correctly, the counting is already done > and the count is in the command PDU. > > A third reason that it is not necessary is that any decent > instrumentation on the line has to already be capable of sorting > through the TCP/IP mapping, the security mappings, and the > basic PDU identification. All the monitors that we have today > in the SCSI environment apply a little simple postprocessing to > the data they have captured and presto, they not only identify > PDU's related to particular logical units, but to particular > commands, and they have interpreted the commands and verified > that the responses are appropriate to the commands. > > If the instrumentation is inline with the receiving code, > it is by definition aware of the states and identifications > already available to it and does not need this additional information. > > As for undesirable consequences: > > Paul Koning's objection to the additional functional steps > required to provide the LUN is a valid concern. > > Mallikarjun Chadalapaka's concern about the requirement for > additional consistency checking (and a whole new class of > non-SCSI protocol error checking) is also a valid issue. > > Let's talk to your colleague and explain to him/her how there > are better ways to achieve his/her goals. > > Bob Snively e-mail: rsnively@brocade.com > Brocade Communications Systems phone: 408 487 8135 > 1745 Technology Drive > San Jose, CA 95110 > > > > > -----Original Message----- > > From: Julian Satran [mailto:Julian_Satran@il.ibm.com] > > Sent: Monday, November 12, 2001 4:31 AM > > To: ips@ece.cmu.edu > > Subject: iSCSI - change proposal LUN field definition on every PDU > > Importance: High > > > > > > Dear colleagues, > > > > A colleague interested in instrumentation approached me with > > a question > > about stateless logging of specific LU activity. > > With the current iSCSI PDU formats this is not possible. > > We have consistently avoided having fields that are redundant > > and will > > need consistency checking. > > However I think we should consider including the LU field in > > all PDUs that > > are referencing a specific LU to simplify this type of > > instrumentation (as > > we did with the direction bit in the op-code). > > > > As I am already in "count-down" mode for version 09 I would > > appreciate > > your comments ASAP. > > > > Julo > > > From owner-ips@ece.cmu.edu Tue Nov 13 17:04:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA18738 for ; Tue, 13 Nov 2001 17:04:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADLH3a24351 for ips-outgoing; Tue, 13 Nov 2001 16:17:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1ab.compuserve.com (siaar1ab.compuserve.com [149.174.40.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADLH0l24338 for ; Tue, 13 Nov 2001 16:17:00 -0500 (EST) Received: (from mailgate@localhost) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id QAA17570 for ips@ece.cmu.edu; Tue, 13 Nov 2001 16:16:54 -0500 (EST) Received: from compuserve.com (dal-tgn-tkw-vty1.as.wcom.net [216.192.232.1]) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id QAA17526 for ; Tue, 13 Nov 2001 16:16:46 -0500 (EST) Message-ID: <3BF18ED4.A3495BAD@compuserve.com> Date: Tue, 13 Nov 2001 15:21:24 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: Re: FCencap: List ALL SOF/EOF codes References: <9410A0F67DFE7F4D998D4538236498360408F5@nc8220exchange.ral.lucent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I support option 3. Option 1 fails to reflect the reality of current T11 work on Class 4. Option 2 fails to consider the recent debate over iSCSI profiles. The IETF position on profiles (as stated by the ADs during last summer's debate) is that profiles must be stated as requirements in RFCs. Therefore, the T11 FC-MI carries the weight of a standard in the IETF and the FC-MI interoperability prohibition for Class 1 must be stated in the FC Encapsulation. (Either that or those folks wanting to write iSCSI profiles can resharpen their pens for round two.) Option 4 attempts to link the good works of option 3 with the bogus concept of profiles in option 2, which makes it every bit as bogus. Murali's option 5 guarantees that there will be no RFC on FC Encapsulation until 2003 since there will be no FC-BB-2 standard to reference until then. Thanks. Ralph... Elizabeth Rodriguez wrote: > > (Chair hat on) > > Let me jump back in here, and summarize as I see it: > > Codes from FC-BB cannot be taken intact, since there are invalid codes > in that specification. > Codes initially trimmed in January of this year, but Ralph correctly > points out, that was prior to the formation of the FC encapsulation > draft (e.g. discussion pertained directly to FCIP; did not take into > consideration iFCP) > > That said, we can modify the current FC encapsulation draft to include > other classes of service, if we so choose. > > We can choose to > 1) Keep the table as is. > Note: The current SOF/EOF codes defined in the FC encapsulation draft > match the currently defined FC-BB-2 subset. > > 2) Add Class 1 codes > Note: FC-MI (technical report, not a standard) does not support Class > 1. > Note2: Practicality of Class 1 over IP questioned. > > 3) Add Class 4 codes > Note: Class 4 work is ongoing, but SOF/EOF codes currently defined for > class 4 unchanged. > > 4) Add both class 1 and class 4 codes > > I would like to solicit input from everyone on what codes people feel > should be included in the FC encapsulation draft. > Note: Drafts following FC encapsulation may restrict classes of service > that draft supports. > > Thanks, > > Elizabeth > > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Monday, November 12, 2001 6:01 PM > To: Charles Monia; IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > On the specific topic of supported SOF and EOF codes the ietf documents > should be driven by the specification provided in the *most relevant * > document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI > should > be kept out of this. > > If we simply accept to adopt the SOF and EOF codes listed for BB-2 the > problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. > I don't see why we are making a big deal about this. > > -Murali > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Monia > Sent: Monday, November 12, 2001 3:33 PM > To: IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > Hi Folks: > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > The last time I checked, the FC-MI spec was not a "standards track" > document > (to use IETF terminology). If that's still the case, is FC-MI's > prohibition > of class 1 a sufficient basis for precluding class 1 support in the > encapsulation spec? > > Charles > > -----Original Message----- > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > Sent: Saturday, November 10, 2001 8:52 AM > > To: IPS Reflector > > Cc: Black_David@emc.com > > Subject: Re: FCencap: List ALL SOF/EOF codes > > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > > > Therefore, I am forced to agree with David. Class 1 MUST NOT > > be mentioned in the FC Encapsulation draft. If necessary, a > > note discussing interoperability and FC-MI can be added. > > > > Thanks. > > > > Ralph... > > > > Black_David@emc.com wrote: > > > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > > I in FC-MI stands for "Interoperability", this seems like a > > reasonable > > > rationale for excluding Class 1 service. > > > > > > --David > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Tue Nov 13 17:04:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA18749 for ; Tue, 13 Nov 2001 17:04:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADKuWx22513 for ips-outgoing; Tue, 13 Nov 2001 15:56:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from eumenes.hosting.pacbell.net (eumenes.hosting.pacbell.net [216.100.98.24]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADKuPl22491 for ; Tue, 13 Nov 2001 15:56:25 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by eumenes.hosting.pacbell.net id PAA16790; Tue, 13 Nov 2001 15:56:13 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: "ips" Subject: RE: iSCSI: update on OOO CmdSNs/connection Date: Tue, 13 Nov 2001 12:53:11 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, I am surprised to see the text in the latest rev of the doc change as suggested by Mallikarjun. I did not think there was a consensus on this subject. Just because I did not respond to Mallikarjun's last comment publicly should not be construed as agreement. Having the last word is hopefully not assumed to be consensus, otherwise a thread may never end. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Somesh Gupta > Sent: Friday, November 09, 2001 3:26 PM > To: ips > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > Mallikarjun, > > I never liked the SHOULD. It is not a design point. > If we really want to allow it, perhaps a negotiation > parameter is a better choice (which is only > marginally better). Error detection and recovery > have completely different design requirements than > the data path. > > So ideally we say MUST or nothing > or we negotiate it > > Also on your point A, it "MUST" be a MUST on > a single connection case except for when required > for error recovery (i.e. the very very rare - > case of digest error). > > Somesh > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Mallikarjun C. > > Sent: Friday, November 09, 2001 1:49 PM > > To: ips > > Subject: iSCSI: update on OOO CmdSNs/connection > > > > > > To those interested in this discussion: > > > > Julian and I had a phone conversation on the topic > > of OOO CmdSNs on a connection. An update follows. > > > > Julian agrees that there are valid error recovery > > scenarios where CmdSNs will legitimately end up OOO > > on a given connection. > > > > OTOH, I agree with two of Julian's scenarios that he > > pointed out right away - the "cleaning command" (command > > required to be sent after a retry copy to ensure flushing > > within 2^31 -1), and an immediate Logout posted with > > unacknowledged commands. Neither of this can be shipped > > OOO - since the former undoes the flushing intent, and > > the latter breaks the rule that nothing more follows a > > Logout on the connection (and troublesome in other ways, > > see below). > > > > In general, I share the concern with Julian that we > > have not closely scrutinized all possibilities. > > > > With that said, something along the following lines > > seemed reasonable - > > > > A)Initiator MUST send commands in increasing order of > > CmdSN on a connection if both the following are true - > > - operational ErrorRecoveryLevel is 0, > > - MaxConnections is negotiated to 1. > > B)In all the other cases, initiator SHOULD send commands > > in increasing order of CmdSN on a connection. It is > > strongly encouraged that commands with out-of-order > > CmdSNs be sent on a connection only if they are > > retransmitted commands due to digest error recovery > > and connection recovery. > > > > I also suggest the following upon further reflection- > > > > C)Add wording in section 2.2.2.1 to mandate that > > the cleaning command MUST be sent in-order after > > the retried command. > > D)Warn clearly that sending an immediate Logout command > > in the presence of other unacknowledged commands MAY > > create inadvertent discarding of certain commands (even > > if it is a recovery Logout), and MAY cause protocol > > errors leading to ungraceful shutdown of the connection. > > > > Hopefully A will bring the determinism that Somesh was > > looking for certain design points. B describes the more > > general n-connection session case. C & D are fixes for > > two identfied areas (so far) which will break. > > > > Comments? > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > From owner-ips@ece.cmu.edu Tue Nov 13 18:02:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20603 for ; Tue, 13 Nov 2001 18:02:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADMLwP29746 for ips-outgoing; Tue, 13 Nov 2001 17:21:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fADMLul29742 for ; Tue, 13 Nov 2001 17:21:56 -0500 (EST) Received: from deneb.dev.equallogic.com (deneb.dev.equallogic.com [172.16.1.99]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fADMLpQ01784 for ; Tue, 13 Nov 2001 17:21:51 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.2/8.11.2) with ESMTP id fADMLov12102 for ; Tue, 13 Nov 2001 17:21:51 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15345.40201.722000.377873@gargle.gargle.HOWL> Date: Tue, 13 Nov 2001 17:22:01 -0500 From: Paul Koning To: ips@ece.cmu.edu Subject: iSCSI: data and data sequences for Read X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I hope this isn't a dumb question, but between the -08 spec and the archives I'm puzzled about the details around data sequences in the case of Read operations. As far as I can tell, a read can result in one or more data sequences coming back. These are numbered with DataSN values that keep going up across sequence boundaries. Each sequence is limited by MaxBurstSize, but the total Read size (sum of all the sequences) is not bounded other than by SCSI. In the Write case, something analogous happens but there there's an R2T to control the flow of data sequences. As far as I can see, there is nothing analogous in the Read case. In other words, while MaxBurstSize limits the size of a data sequence, there is no mechanism limiting the number of bursts, or the rate at which you send back DatIn PDUs (other than TCP window control). Is that right or did I miss something? If it's right, what is the purpose of having the notion of a Data Sequence for DataIn, and what does MaxBurstSize do for you in that case? Thanks, paul From owner-ips@ece.cmu.edu Tue Nov 13 18:04:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20667 for ; Tue, 13 Nov 2001 18:04:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADMjWc01521 for ips-outgoing; Tue, 13 Nov 2001 17:45:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADMjUl01516 for ; Tue, 13 Nov 2001 17:45:30 -0500 (EST) Received: from hq-ex-c2.brocade.com (hq-ex-c2 [192.168.126.35]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id OAA24251; Tue, 13 Nov 2001 14:45:20 -0800 (PST) Received: by hq-bes-c2.brocade.com with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Nov 2001 14:45:19 -0800 Message-ID: From: Robert Snively To: "'ENDL_TX@computer.org'" , IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Date: Tue, 13 Nov 2001 14:45:14 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk I vote option 3. Ralph's reasoning is valid. From owner-ips@ece.cmu.edu Tue Nov 13 18:06:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20690 for ; Tue, 13 Nov 2001 18:06:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADMqRk02001 for ips-outgoing; Tue, 13 Nov 2001 17:52:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADMqQl01996 for ; Tue, 13 Nov 2001 17:52:26 -0500 (EST) Received: by ARIEL with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Nov 2001 14:52:20 -0800 Message-ID: From: Charles Monia To: IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Date: Tue, 13 Nov 2001 14:52:16 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Elizabeth: The iFCP spec explicitly lists the SOF/EOF codes accepted by the protocol and the actions to be taken if other codes are received. Charles > -----Original Message----- > From: Elizabeth Rodriguez [mailto:egrodriguez@lucent.com] > Sent: Tuesday, November 13, 2001 10:40 AM > To: Murali Rajagopal; Charles Monia; IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > > Charles, Franco > (and others interested in iFCP) > Murali's statement below does apply to the FCIP draft. > Do you feel the statement also applies to iFCP? > > Elizabeth > > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Monday, November 12, 2001 6:01 PM > To: Charles Monia; IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > > On the specific topic of supported SOF and EOF codes the ietf > documents > should be driven by the specification provided in the *most relevant * > document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI > should > be kept out of this. > > If we simply accept to adopt the SOF and EOF codes listed for BB-2 the > problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. > I don't see why we are making a big deal about this. > > -Murali > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Monia > Sent: Monday, November 12, 2001 3:33 PM > To: IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > Hi Folks: > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > The last time I checked, the FC-MI spec was not a "standards track" > document > (to use IETF terminology). If that's still the case, is FC-MI's > prohibition > of class 1 a sufficient basis for precluding class 1 support in the > encapsulation spec? > > Charles > > -----Original Message----- > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > Sent: Saturday, November 10, 2001 8:52 AM > > To: IPS Reflector > > Cc: Black_David@emc.com > > Subject: Re: FCencap: List ALL SOF/EOF codes > > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > > > Therefore, I am forced to agree with David. Class 1 MUST NOT > > be mentioned in the FC Encapsulation draft. If necessary, a > > note discussing interoperability and FC-MI can be added. > > > > Thanks. > > > > Ralph... > > > > Black_David@emc.com wrote: > > > > > FC-MI was going to prohibit Class 1 last time I checked. > Since the > > > I in FC-MI stands for "Interoperability", this seems like a > > reasonable > > > rationale for excluding Class 1 service. > > > > > > --David > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Tue Nov 13 18:06:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20712 for ; Tue, 13 Nov 2001 18:06:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADMJC629474 for ips-outgoing; Tue, 13 Nov 2001 17:19:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADMJAl29460 for ; Tue, 13 Nov 2001 17:19:10 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id OAA23561; Tue, 13 Nov 2001 14:19:01 -0800 (PST) From: "Murali Rajagopal" To: , "IPS Reflector" Subject: FCIP: RE: FCencap: List ALL SOF/EOF codes Date: Tue, 13 Nov 2001 14:20:09 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 Importance: Normal In-Reply-To: <3BF18ED4.A3495BAD@compuserve.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Ralph: Reference [4] in the current draft already exists for BB-2. In the past (see RFC 2625 Ref. showing pre-standard references) I had no difficulty in getting a RFC published with such references. Maybe Elizabeth can check to see if this is still the practice. -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ralph Weber Sent: Tuesday, November 13, 2001 1:21 PM To: IPS Reflector Subject: Re: FCencap: List ALL SOF/EOF codes I support option 3. Option 1 fails to reflect the reality of current T11 work on Class 4. Option 2 fails to consider the recent debate over iSCSI profiles. The IETF position on profiles (as stated by the ADs during last summer's debate) is that profiles must be stated as requirements in RFCs. Therefore, the T11 FC-MI carries the weight of a standard in the IETF and the FC-MI interoperability prohibition for Class 1 must be stated in the FC Encapsulation. (Either that or those folks wanting to write iSCSI profiles can resharpen their pens for round two.) Option 4 attempts to link the good works of option 3 with the bogus concept of profiles in option 2, which makes it every bit as bogus. Murali's option 5 guarantees that there will be no RFC on FC Encapsulation until 2003 since there will be no FC-BB-2 standard to reference until then. Thanks. Ralph... Elizabeth Rodriguez wrote: > > (Chair hat on) > > Let me jump back in here, and summarize as I see it: > > Codes from FC-BB cannot be taken intact, since there are invalid codes > in that specification. > Codes initially trimmed in January of this year, but Ralph correctly > points out, that was prior to the formation of the FC encapsulation > draft (e.g. discussion pertained directly to FCIP; did not take into > consideration iFCP) > > That said, we can modify the current FC encapsulation draft to include > other classes of service, if we so choose. > > We can choose to > 1) Keep the table as is. > Note: The current SOF/EOF codes defined in the FC encapsulation draft > match the currently defined FC-BB-2 subset. > > 2) Add Class 1 codes > Note: FC-MI (technical report, not a standard) does not support Class > 1. > Note2: Practicality of Class 1 over IP questioned. > > 3) Add Class 4 codes > Note: Class 4 work is ongoing, but SOF/EOF codes currently defined for > class 4 unchanged. > > 4) Add both class 1 and class 4 codes > > I would like to solicit input from everyone on what codes people feel > should be included in the FC encapsulation draft. > Note: Drafts following FC encapsulation may restrict classes of service > that draft supports. > > Thanks, > > Elizabeth > > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Monday, November 12, 2001 6:01 PM > To: Charles Monia; IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > On the specific topic of supported SOF and EOF codes the ietf documents > should be driven by the specification provided in the *most relevant * > document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI > should > be kept out of this. > > If we simply accept to adopt the SOF and EOF codes listed for BB-2 the > problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. > I don't see why we are making a big deal about this. > > -Murali > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Monia > Sent: Monday, November 12, 2001 3:33 PM > To: IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > Hi Folks: > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > The last time I checked, the FC-MI spec was not a "standards track" > document > (to use IETF terminology). If that's still the case, is FC-MI's > prohibition > of class 1 a sufficient basis for precluding class 1 support in the > encapsulation spec? > > Charles > > -----Original Message----- > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > Sent: Saturday, November 10, 2001 8:52 AM > > To: IPS Reflector > > Cc: Black_David@emc.com > > Subject: Re: FCencap: List ALL SOF/EOF codes > > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > > > Therefore, I am forced to agree with David. Class 1 MUST NOT > > be mentioned in the FC Encapsulation draft. If necessary, a > > note discussing interoperability and FC-MI can be added. > > > > Thanks. > > > > Ralph... > > > > Black_David@emc.com wrote: > > > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > > I in FC-MI stands for "Interoperability", this seems like a > > reasonable > > > rationale for excluding Class 1 service. > > > > > > --David > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > From owner-ips@ece.cmu.edu Tue Nov 13 18:10:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20853 for ; Tue, 13 Nov 2001 18:10:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADMY9Q00753 for ips-outgoing; Tue, 13 Nov 2001 17:34:09 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADMY8l00749 for ; Tue, 13 Nov 2001 17:34:08 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id <4W49APD5>; Tue, 13 Nov 2001 17:30:25 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293718B@CORPMX14> From: Black_David@emc.com To: muralir@lightsand.com, ips@ece.cmu.edu Subject: RE: FCIP: RE: FCencap: List ALL SOF/EOF codes Date: Tue, 13 Nov 2001 17:24:11 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk IMHO, you're both right. I have no problem with referencing a major stable version of an evolving T10 or T11 document (e.g., iSCSI will almost certainly have to do this for SAM-2). I also consider FC-MI to be authoritative due to both its interoperability focus and the expectation that it will be a de facto standard in the industry independent of its procedural situation in T11. I definitely want to see a reference to FC-MI, as I think it's significant that a T11 interoperability document prohibits Class 1 service. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Tuesday, November 13, 2001 5:20 PM > To: ENDL_TX@computer.org; IPS Reflector > Subject: FCIP: RE: FCencap: List ALL SOF/EOF codes > > > Ralph: > > Reference [4] in the current draft already exists for BB-2. > In the past > (see RFC 2625 Ref. showing pre-standard references) I had no > difficulty in > getting a RFC published with such references. Maybe Elizabeth > can check to > see if this is still the practice. > > -Murali > > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of Ralph > Weber > Sent: Tuesday, November 13, 2001 1:21 PM > To: IPS Reflector > Subject: Re: FCencap: List ALL SOF/EOF codes > > I support option 3. > > Option 1 fails to reflect the reality of current T11 work on > Class 4. > > Option 2 fails to consider the recent debate over iSCSI profiles. > The IETF position on profiles (as stated by the ADs during last > summer's debate) is that profiles must be stated as requirements > in RFCs. Therefore, the T11 FC-MI carries the weight of a standard > in the IETF and the FC-MI interoperability prohibition for Class 1 > must be stated in the FC Encapsulation. (Either that or those > folks wanting to write iSCSI profiles can resharpen their pens > for round two.) > > Option 4 attempts to link the good works of option 3 with > the bogus concept of profiles in option 2, which makes it > every bit as bogus. > > Murali's option 5 guarantees that there will be no RFC on > FC Encapsulation until 2003 since there will be no FC-BB-2 > standard to reference until then. > > Thanks. > > Ralph... > > Elizabeth Rodriguez wrote: > > > > > (Chair hat on) > > > > Let me jump back in here, and summarize as I see it: > > > > Codes from FC-BB cannot be taken intact, since there are > invalid codes > > in that specification. > > Codes initially trimmed in January of this year, but Ralph correctly > > points out, that was prior to the formation of the FC encapsulation > > draft (e.g. discussion pertained directly to FCIP; did not take into > > consideration iFCP) > > > > That said, we can modify the current FC encapsulation draft > to include > > other classes of service, if we so choose. > > > > We can choose to > > 1) Keep the table as is. > > Note: The current SOF/EOF codes defined in the FC > encapsulation draft > > match the currently defined FC-BB-2 subset. > > > > 2) Add Class 1 codes > > Note: FC-MI (technical report, not a standard) does not > support Class > > 1. > > Note2: Practicality of Class 1 over IP questioned. > > > > 3) Add Class 4 codes > > Note: Class 4 work is ongoing, but SOF/EOF codes currently > defined for > > class 4 unchanged. > > > > 4) Add both class 1 and class 4 codes > > > > I would like to solicit input from everyone on what codes > people feel > > should be included in the FC encapsulation draft. > > Note: Drafts following FC encapsulation may restrict > classes of service > > that draft supports. > > > > Thanks, > > > > Elizabeth > > > > -----Original Message----- > > From: Murali Rajagopal [mailto:muralir@lightsand.com] > > Sent: Monday, November 12, 2001 6:01 PM > > To: Charles Monia; IPS Reflector > > Subject: RE: FCencap: List ALL SOF/EOF codes > > > > On the specific topic of supported SOF and EOF codes the > ietf documents > > should be driven by the specification provided in the *most > relevant * > > document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI > > should > > be kept out of this. > > > > If we simply accept to adopt the SOF and EOF codes listed > for BB-2 the > > problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. > > I don't see why we are making a big deal about this. > > > > -Murali > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Charles Monia > > Sent: Monday, November 12, 2001 3:33 PM > > To: IPS Reflector > > Subject: RE: FCencap: List ALL SOF/EOF codes > > > > Hi Folks: > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > > prohibits Class 1 and I can find no letter ballot comments > > > asking that it be reinstated. > > > > The last time I checked, the FC-MI spec was not a "standards track" > > document > > (to use IETF terminology). If that's still the case, is FC-MI's > > prohibition > > of class 1 a sufficient basis for precluding class 1 support in the > > encapsulation spec? > > > > Charles > > > -----Original Message----- > > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > > Sent: Saturday, November 10, 2001 8:52 AM > > > To: IPS Reflector > > > Cc: Black_David@emc.com > > > Subject: Re: FCencap: List ALL SOF/EOF codes > > > > > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > > prohibits Class 1 and I can find no letter ballot comments > > > asking that it be reinstated. > > > > > > Therefore, I am forced to agree with David. Class 1 MUST NOT > > > be mentioned in the FC Encapsulation draft. If necessary, a > > > note discussing interoperability and FC-MI can be added. > > > > > > Thanks. > > > > > > Ralph... > > > > > > Black_David@emc.com wrote: > > > > > > > FC-MI was going to prohibit Class 1 last time I > checked. Since the > > > > I in FC-MI stands for "Interoperability", this seems like a > > > reasonable > > > > rationale for excluding Class 1 service. > > > > > > > > --David > > > > --------------------------------------------------- > > > > David L. Black, Senior Technologist > > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > > --------------------------------------------------- > > > > From owner-ips@ece.cmu.edu Tue Nov 13 18:12:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20919 for ; Tue, 13 Nov 2001 18:12:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fADMg8L01320 for ips-outgoing; Tue, 13 Nov 2001 17:42:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADMg6l01312 for ; Tue, 13 Nov 2001 17:42:07 -0500 (EST) Received: from muralipc ([192.168.2.58]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id OAA24035; Tue, 13 Nov 2001 14:41:53 -0800 (PST) From: "Murali Rajagopal" To: , Subject: RE: FCIP: RE: FCencap: List ALL SOF/EOF codes Date: Tue, 13 Nov 2001 14:43:01 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 Importance: Normal In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0293718B@CORPMX14> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Thanks David for the clarification. We can certainly reference FC-MI and FC-BB-2 as references for this purpose. -Murali P.S. I will bring in motion to add Class 4 to FC-BB-2 Codes in Austin T11 meeting -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Tuesday, November 13, 2001 2:24 PM To: muralir@lightsand.com; ips@ece.cmu.edu Subject: RE: FCIP: RE: FCencap: List ALL SOF/EOF codes IMHO, you're both right. I have no problem with referencing a major stable version of an evolving T10 or T11 document (e.g., iSCSI will almost certainly have to do this for SAM-2). I also consider FC-MI to be authoritative due to both its interoperability focus and the expectation that it will be a de facto standard in the industry independent of its procedural situation in T11. I definitely want to see a reference to FC-MI, as I think it's significant that a T11 interoperability document prohibits Class 1 service. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Tuesday, November 13, 2001 5:20 PM > To: ENDL_TX@computer.org; IPS Reflector > Subject: FCIP: RE: FCencap: List ALL SOF/EOF codes > > > Ralph: > > Reference [4] in the current draft already exists for BB-2. > In the past > (see RFC 2625 Ref. showing pre-standard references) I had no > difficulty in > getting a RFC published with such references. Maybe Elizabeth > can check to > see if this is still the practice. > > -Murali > > > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On > Behalf Of Ralph > Weber > Sent: Tuesday, November 13, 2001 1:21 PM > To: IPS Reflector > Subject: Re: FCencap: List ALL SOF/EOF codes > > I support option 3. > > Option 1 fails to reflect the reality of current T11 work on > Class 4. > > Option 2 fails to consider the recent debate over iSCSI profiles. > The IETF position on profiles (as stated by the ADs during last > summer's debate) is that profiles must be stated as requirements > in RFCs. Therefore, the T11 FC-MI carries the weight of a standard > in the IETF and the FC-MI interoperability prohibition for Class 1 > must be stated in the FC Encapsulation. (Either that or those > folks wanting to write iSCSI profiles can resharpen their pens > for round two.) > > Option 4 attempts to link the good works of option 3 with > the bogus concept of profiles in option 2, which makes it > every bit as bogus. > > Murali's option 5 guarantees that there will be no RFC on > FC Encapsulation until 2003 since there will be no FC-BB-2 > standard to reference until then. > > Thanks. > > Ralph... > > Elizabeth Rodriguez wrote: > > > > > (Chair hat on) > > > > Let me jump back in here, and summarize as I see it: > > > > Codes from FC-BB cannot be taken intact, since there are > invalid codes > > in that specification. > > Codes initially trimmed in January of this year, but Ralph correctly > > points out, that was prior to the formation of the FC encapsulation > > draft (e.g. discussion pertained directly to FCIP; did not take into > > consideration iFCP) > > > > That said, we can modify the current FC encapsulation draft > to include > > other classes of service, if we so choose. > > > > We can choose to > > 1) Keep the table as is. > > Note: The current SOF/EOF codes defined in the FC > encapsulation draft > > match the currently defined FC-BB-2 subset. > > > > 2) Add Class 1 codes > > Note: FC-MI (technical report, not a standard) does not > support Class > > 1. > > Note2: Practicality of Class 1 over IP questioned. > > > > 3) Add Class 4 codes > > Note: Class 4 work is ongoing, but SOF/EOF codes currently > defined for > > class 4 unchanged. > > > > 4) Add both class 1 and class 4 codes > > > > I would like to solicit input from everyone on what codes > people feel > > should be included in the FC encapsulation draft. > > Note: Drafts following FC encapsulation may restrict > classes of service > > that draft supports. > > > > Thanks, > > > > Elizabeth > > > > -----Original Message----- > > From: Murali Rajagopal [mailto:muralir@lightsand.com] > > Sent: Monday, November 12, 2001 6:01 PM > > To: Charles Monia; IPS Reflector > > Subject: RE: FCencap: List ALL SOF/EOF codes > > > > On the specific topic of supported SOF and EOF codes the > ietf documents > > should be driven by the specification provided in the *most > relevant * > > document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI > > should > > be kept out of this. > > > > If we simply accept to adopt the SOF and EOF codes listed > for BB-2 the > > problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. > > I don't see why we are making a big deal about this. > > > > -Murali > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Charles Monia > > Sent: Monday, November 12, 2001 3:33 PM > > To: IPS Reflector > > Subject: RE: FCencap: List ALL SOF/EOF codes > > > > Hi Folks: > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > > prohibits Class 1 and I can find no letter ballot comments > > > asking that it be reinstated. > > > > The last time I checked, the FC-MI spec was not a "standards track" > > document > > (to use IETF terminology). If that's still the case, is FC-MI's > > prohibition > > of class 1 a sufficient basis for precluding class 1 support in the > > encapsulation spec? > > > > Charles > > > -----Original Message----- > > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > > Sent: Saturday, November 10, 2001 8:52 AM > > > To: IPS Reflector > > > Cc: Black_David@emc.com > > > Subject: Re: FCencap: List ALL SOF/EOF codes > > > > > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > > prohibits Class 1 and I can find no letter ballot comments > > > asking that it be reinstated. > > > > > > Therefore, I am forced to agree with David. Class 1 MUST NOT > > > be mentioned in the FC Encapsulation draft. If necessary, a > > > note discussing interoperability and FC-MI can be added. > > > > > > Thanks. > > > > > > Ralph... > > > > > > Black_David@emc.com wrote: > > > > > > > FC-MI was going to prohibit Class 1 last time I > checked. Since the > > > > I in FC-MI stands for "Interoperability", this seems like a > > > reasonable > > > > rationale for excluding Class 1 service. > > > > > > > > --David > > > > --------------------------------------------------- > > > > David L. Black, Senior Technologist > > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > > --------------------------------------------------- > > > > From owner-ips@ece.cmu.edu Tue Nov 13 19:55:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA23329 for ; Tue, 13 Nov 2001 19:55:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE0DDh07551 for ips-outgoing; Tue, 13 Nov 2001 19:13:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imo-m05.mx.aol.com (imo-m05.mx.aol.com [64.12.136.8]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE0DBl07546 for ; Tue, 13 Nov 2001 19:13:11 -0500 (EST) Received: from VAHUJA@aol.com by imo-m05.mx.aol.com (mail_out_v31_r1.9.) id 3.152.3f0ffe5 (15887) for ; Tue, 13 Nov 2001 19:12:56 -0500 (EST) Received: from web38.aolmail.aol.com (web38.aolmail.aol.com [205.188.222.14]) by air-id08.mx.aol.com (v82.22) with ESMTP id MAILINID82-1113191256; Tue, 13 Nov 2001 19:12:56 -0500 Date: Tue, 13 Nov 2001 19:12:56 EST From: VAHUJA@aol.com Subject: SRP vs PKI for authentication To: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailer: Unknown (No Version) Message-ID: <152.3f0ffe5.29231108@aol.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit iSCSI draft 08 requires use of SRP for authentication, while for Fabric switch authentication, there are proposals in T11 that use PKI. The iSNS also allows PKI for authentication. I have also seen some IP concerns raised recently about SRP... So my question is - for iSCSI login-time authentication, are there compelling reasons for using SRP instead of PKI? From owner-ips@ece.cmu.edu Tue Nov 13 19:59:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA23383 for ; Tue, 13 Nov 2001 19:59:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE0AVN07369 for ips-outgoing; Tue, 13 Nov 2001 19:10:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from va2mc.ummailbox.net (va2mc.ummailbox.net [64.210.247.16]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE0ATl07365 for ; Tue, 13 Nov 2001 19:10:30 -0500 (EST) Received: from hqmailweb.COMMSTOR.Crossroads.com (HQMailWeb.Crossroads.com [63.237.99.250:25]) by va2mc.ummailbox.net with ESMTP id P1113-1910-6f6b01; Wed, 14 Nov 2001 00:10:26 GMT Received: from HQMAILNODE1.COMMSTOR.Crossroads.com ([10.5.1.42]) by hqmailweb.COMMSTOR.Crossroads.com with Microsoft SMTPSVC(5.0.2195.3779); Tue, 13 Nov 2001 18:10:03 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: iSCSI: security questions Disposition-Notification-To: "Lee Xing" Date: Tue, 13 Nov 2001 18:10:02 -0600 Message-ID: Thread-Topic: iSCSI: security questions Thread-Index: AcFsoLTomvBFvzOHQei9FVt267btaA== From: "Lee Xing" To: X-OriginalArrivalTime: 14 Nov 2001 00:10:03.0172 (UTC) FILETIME=[B5153E40:01C16CA0] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fAE0AUl07366 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hi, I got a few questions on iSCSI security. I would appreciate it if someone could help. Thank you. ================ Q1: iSCSI v.08, page 142 "The authentication method cannot assume an underlying IPSec protection, since IPSec is optional to use." IPSec is an option for IPv4, but it's mandatory for IPv6 (if I remember right). Should we make it more specific? Q2: iSCSI v.08 Chapter 10 (Security Consideration) mentions a few times of "...MUST implement...". Should we add something like "security is mandatory to implement, but not mandatory to use" in this chapter? This is stated explicitly in SEC-IPS v.04 draft, and also implied in Chapter 5 (Login Phase) of iSCSI v.08. Q3: SEC-IPS v.04, page 11 "Negotiation between Initiator and Target is used to determine which authentication algorithm to use (or whether to use one at all); the connection closes if either side requires authentication and no mutually acceptable algorithm can be agreed upon" The question is whether "none" is considered as an "acceptable algorithm". In other words, if initiator asks "AuthMethod=KRB5,SRP,none" during login, and target answers "AuthMethod=none", should the connection be closed, or should the initiator continue with LoginOperationalNegotiation stage? If latter is acceptable, should we reword the last sentence like "...and no mutually acceptable algorithm or "none" can be agreed upon"? Q4: SEC-IPS v.04, page32 "If IPsec protection is removed on a connection, it MUST be reinstated before iSCSI, iFCP or FCIP packets are sent." The question is do we have to check security every time before sending out iSCSI packets? Lee From owner-ips@ece.cmu.edu Tue Nov 13 21:22:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA24571 for ; Tue, 13 Nov 2001 21:22:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE1VQ212660 for ips-outgoing; Tue, 13 Nov 2001 20:31:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE1VPl12655 for ; Tue, 13 Nov 2001 20:31:25 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Nov 2001 20:31:20 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293718E@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: iFCP -06 comments Date: Tue, 13 Nov 2001 20:21:30 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk At the request of the iFCP authors, I've reviewed the -06 version of the iFCP draft. Here are my comments divided into Major, Minor and/or Editorial, and Formatting categories. ----- Major ----- 5.3.2/5.3.2.1/5.3.2.1.1 - These are written as descriptions of a possible implementation. That's fine, but it's necessary to be crisp about what the requirements are on an implementation that may not go about this in the fashion envisioned here. Please go over these sections, figure out the requirements on an arbitrary implementation and put in the requisite MUSTs and SHOULDs in addition to the current description of how things could be implemented. I believe the functional equivalent of the translation table and its entries MUST exist, so that's somewhere to start from. Other areas of the document have lesser versions of this problem, so please go over the entire document and check it for this. I also saw a number of places with lower case requirements words (e.g., "shall") that probably need to be upper case. 6.2.1 An N_PORT is identified by its network address consisting of: a) The N_PORT I/D assigned by the gateway to which the N_PORT is locally attached and b) The IP address of the gateway's iFCP Portal. Use of an IP address to identify a remote gateway needs to be reconsidered. Please add something to allow multiple iFCP implementations to exist at different TCP ports on the same IP address (or explain why this has to be prohibited). This is strongly related to the NAPT issues that the FCIP folks are in the midst of working through. 6.2.2.2 - This section carries a strong risk of over-specification. As work on TCP evolves, there's a strong risk of entries in this table conflicting with the then-applicable RFCs. The ground rule should be to only specify something here when it has serious consequences (i.e., there are potentially very bad effects from ignoring the SHOULD). I think I can accept this argument for the first 4 lines in the table, but I think the last three should be removed as I don't think they're crucial to iFCP per se (the discussion of them seems to amount to "good things to do in general", and there's a strong risk of further requirements changes/tweaks in this area). Keep in mind that I'm one of the co-authors of RFC 3168 (ECN). Also, I think keeping the "should"s and "should not"s lower case (so that they're advice to implementers rather than protocol requirements) is a good thing to do in this section. 6.2.2.3 - I think this Section needs to go away. 6.2.2.3.1 is essentially repeating information about TCP implementations in general that is already to be found elsewhere and should be removed. 6.2.2.3.2 should be moved to 6.2.3.2. I think the order of Sections 7 and 8 should be reversed, as the TCP connection control material is needed to understand how iFCP functions before discussing the interesting things it has to do to some ELSs. 9.2.1 - The last paragraph on what to do on loss of synchronization seems inadequate. It's current state appears to allow stale frame propagation by a compliant implementation. Was this the intended outcome? 10.4 b) - How can one be sure that the local gateway knows about all the remote gateways? I suspect this involves iSNS, and needs to be explained. 10.4 b) What happens when the broadcast frame exceeds the MTU size? This seems to result in a rather unreliable broadcast service, as all of the UDP datagrams could well be dropped, consistently and repeatedly for large enough broadcast frames. ---- Minor and/or Editorial ----- 4.4 a) It's not exactly correct to describe the Node WWN as being an identifier for the device. While that was the original intent, it isn't always implemented that way. In practice, I don't think Node WWNs are used for much, and that might be worth noting. 4.7.1 - It might be worth describing how Area ID and Port ID are assigned when an FC-AL loop is attached to a switch to give the reader a slightly better feel for this (Area ID is assigned to switch port and Port ID is the loop port identifier). 4.8 - Track state of things in T11 in determining whether its appropriate to add Class 4 and 6 to this description. From FCIP discussions, it sounds like at least Class 4 should be added. 4.9 - Might want to add a sentence to make it clear that these login processes occur at FC-2, and ULP (FC-4) protocol setup takes place over the established FC-2 N_Port to N_Port session in whatever manner the ULP desires Figure 7 - I think "IP network" would be a better term for what's below the line than "IP fabric". I understand what an "iFCP fabric" is, but I'm not at all sure about an "IP fabric". 5.2.1 - Might want to add a sentence indicating how an FC device discovers that Class 1 isn't supported (gets told by the iFCP gateway on Fabric Login). p. 20 All iFCP implementations MUST support operation in address translation mode. Support for address transparent mode is optional "optional" should be all upper-case. The implementation MUST NOT allow the mode to be changed after iFCP sessions have been established. So, the mode can be changed while a session is being established? I don't think so and suggest that the above wording be changed. One possibility would be to prohibit a mode change while any device is logged into it via FLOGI. 5.3.1 - Somewhere near the end of this section please say that FC frame CRCs have to be regenerated as a consequence of performing address translation. Both 5.3 and 5.3.1 contain some rationale text about the differences between address-transparent and address-translation mode and why one might select one or the other that might be better separated out from the description of how iFCP works in these modes. In particular, iSNS pops up without any introduction in 5.3.1 a) - this text really needs to come after a discussion of iSNS and its responsibilities for/interaction with iFCP. 5.3.1.1 - The whole discussion of domain ID assignment is rather imprecise. Please put in a "MUST" requirement for unique domain IDs, and indicate that iSNS can do this (with a specific description of how) in addition to manual assignment. 5.3.1.2 - Please upper-case "shall" in the first sentence here. Also applies to 5.3.2.3. 6.2.2.1 - For a), please indicate how the gateway knows which remote gateway to use and what it's address is. This involves translation table entries that were initialized by iSNS replies. 6.2.3 - Please be clear about what exactly is being terminated or aborted. I think the iFCP session (TCP connection) between gateways is what's involved, but there's enough FC mechanism discussion here to be unclear. 6.3 - I guess this reference to RFC1323 is ok, although it strikes me as superfluous. Please indicate that it's Appendix B of RFC 1323. 6.4.4 - State that the FC CRC MUST be recalculated due to the address translation. 7.3.1 - Two translation types are shown for the ABTX Exchange originator. I think this should be Type 1. 7.3.5 - Translation table is incomplete for FARP-REQ. 7.3.7 - Why is PLOGI in a subsection of 7.3 when it's not augmented? 7.3.8 - Three translation types are shown for the REC Exchange originator. Again, I think this should be type 1. Variants of this problem are also present in all of 7.3.9 through 7.3.15. 7.4 - Table 4 consists entirely of "n" and "M" entries, so delete the explanation of the unused "y" entry. 8.1 - For clarity indicate that the connection handle is needed to unbind connections (yes, I know this is discussed in the next section). 11.3.1 - Please remove the "MUST implement" for DES (it's ok to make DES OPTIONAL), and think about rewording the "As stated in" statements, as we are overriding some of the requirements in some of the reference RFCs. 11.3.1.1 - This is ok in a working version of the draft, but will vanish in the final version (ditto the subject to availability of an RFC language about AES earlier) because either we will make a normative reference to the AES RFCs or RFC-to-be, or we will delete requirements for AES. 11.3.2 If, however, the TCP session is maintained, then a Phase 2 delete message shall trigger a new Quick Mode exchange. Probably not a good idea. The issue here is that some hardware crypto accelerators only have room for a fixed number of phase 2 SAs and hence delete old ones to make room for new ones (ideally deleting the least recently used, but ...). Triggering a new Quick Mode immediately in response to any delete of an SA for an open TCP connection could thrash the SA state resources in such an accelerator. Triggering the new Quick Mode based on traffic sent over the SA should work better. 12.2 - Please delete d) as MPLS is *NOT* a QoS technology. In addition, the entire paragraph after d) appears to have very little content, and I'm not at all sure about the value of discussion SLAs here. The discussion of [00-603] is also questionable. Appendix B - Is there an external version of this material that could be referenced rather than including it here? ----- Formatting ----- There are a bunch of places where MS Word has inserted non-standard MS punctuation characters (mostly a u with a circumflex instead of a hyphen) Turn off Smart Quotes and Auto Correct and correct these. 6.3 has a formatting problem - probably an MS Word style that should not have been used. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Nov 13 22:39:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA27482 for ; Tue, 13 Nov 2001 22:39:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE2n3S17933 for ips-outgoing; Tue, 13 Nov 2001 21:49:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE2n2l17929 for ; Tue, 13 Nov 2001 21:49:02 -0500 (EST) Received: by ARIEL with Internet Mail Service (5.5.2653.19) id ; Tue, 13 Nov 2001 18:48:56 -0800 Message-ID: From: Charles Monia To: IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Date: Tue, 13 Nov 2001 18:48:55 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: Option 3 is agreeable to me. Charles > -----Original Message----- > From: Elizabeth Rodriguez [mailto:egrodriguez@lucent.com] > Sent: Tuesday, November 13, 2001 11:05 AM > To: Murali Rajagopal; Charles Monia; IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > > (Chair hat on) > > Let me jump back in here, and summarize as I see it: > > Codes from FC-BB cannot be taken intact, since there are invalid codes > in that specification. > Codes initially trimmed in January of this year, but Ralph correctly > points out, that was prior to the formation of the FC encapsulation > draft (e.g. discussion pertained directly to FCIP; did not take into > consideration iFCP) > > That said, we can modify the current FC encapsulation draft to include > other classes of service, if we so choose. > > We can choose to > 1) Keep the table as is. > Note: The current SOF/EOF codes defined in the FC encapsulation draft > match the currently defined FC-BB-2 subset. > > 2) Add Class 1 codes > Note: FC-MI (technical report, not a standard) does not support Class > 1. > Note2: Practicality of Class 1 over IP questioned. > > 3) Add Class 4 codes > Note: Class 4 work is ongoing, but SOF/EOF codes currently > defined for > class 4 unchanged. > > 4) Add both class 1 and class 4 codes > > I would like to solicit input from everyone on what codes people feel > should be included in the FC encapsulation draft. > Note: Drafts following FC encapsulation may restrict classes > of service > that draft supports. > > Thanks, > > Elizabeth > > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Monday, November 12, 2001 6:01 PM > To: Charles Monia; IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > > On the specific topic of supported SOF and EOF codes the ietf > documents > should be driven by the specification provided in the *most relevant * > document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI > should > be kept out of this. > > If we simply accept to adopt the SOF and EOF codes listed for BB-2 the > problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. > I don't see why we are making a big deal about this. > > -Murali > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Monia > Sent: Monday, November 12, 2001 3:33 PM > To: IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > Hi Folks: > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > The last time I checked, the FC-MI spec was not a "standards track" > document > (to use IETF terminology). If that's still the case, is FC-MI's > prohibition > of class 1 a sufficient basis for precluding class 1 support in the > encapsulation spec? > > Charles > > -----Original Message----- > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > Sent: Saturday, November 10, 2001 8:52 AM > > To: IPS Reflector > > Cc: Black_David@emc.com > > Subject: Re: FCencap: List ALL SOF/EOF codes > > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > > > Therefore, I am forced to agree with David. Class 1 MUST NOT > > be mentioned in the FC Encapsulation draft. If necessary, a > > note discussing interoperability and FC-MI can be added. > > > > Thanks. > > > > Ralph... > > > > Black_David@emc.com wrote: > > > > > FC-MI was going to prohibit Class 1 last time I checked. > Since the > > > I in FC-MI stands for "Interoperability", this seems like a > > reasonable > > > rationale for excluding Class 1 service. > > > > > > --David > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Tue Nov 13 22:45:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA27647 for ; Tue, 13 Nov 2001 22:45:13 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE2dxg17372 for ips-outgoing; Tue, 13 Nov 2001 21:39:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from eumenes.hosting.pacbell.net (eumenes.hosting.pacbell.net [216.100.98.24]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE2dql17362 for ; Tue, 13 Nov 2001 21:39:57 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by eumenes.hosting.pacbell.net id VAA18810; Tue, 13 Nov 2001 21:39:19 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: Subject: RE: iSCSI: Clarification (again) for Task Management Commands Date: Tue, 13 Nov 2001 18:36:16 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, This is probably based more on my ignorance than anything else. However I would appreciate any clarification that you could add. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Tuesday, November 13, 2001 9:59 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > Somesh, > > I assume you may want to reread 9.4. It works well on any number of > connections and it handles all commands for which there is a "delivery > uncertainty". For commands for which the delivery is certain - i.e, they > have an instantiated Task Block at both initiator and target there is no > uncertainty either and the implementation of "no more statuses" does not > raise any special issues wherever their statuses are. The initiator > portion of the state has to be marked as aborted and kept in place until > the status arrives. To clarify this implementation detail I have added > some words to the relevant part of the Task Management request: My reading of SAM-2 indicates that the following 2 things (applicable to the commands sent by the initiator which sent the Task Management Commands) "that a response for a command affected by a Task Management Function will not be returned after the response for the Task Management Function is returned" and the second one is that "for commands that have been delivered to the SCSI entity, it is not certain whether the response will be returned or not - except that the rule above must be valid" If my understanding is correct, then I believe there is a hole even in 9.4 - and I will take the time to create a scenario. If my understanding is not correct, then based on what the right mechanism is, I will take another look. > > For all these functions, if executed, the Task Management Function > Response MUST be returned using the Initiator Task Tag to identify the > operation for which it is responding. All those functions apply to the > referenced tasks regardless if they are proper SCSI tasks or tagged iSCSI > operations. Task management commands must be executed as if all the > commands having a CmdSN lower or equal to the task management CmdSN have > been received by the target (i.e., have to be executed as if received for > ordered delivery even when marked for immediate delivery). For all the > tasks covered by the task management response (i.e., with CmdSN > not higher > than the task management command CmdSN), additional responses MUST NOT be > delivered to the SCSI layer after the task management response. This > requirement implies that the initiator must keep around state until the > status is received from the target for an aborted task and the > target MUST > deliver to the initiator good status for an aborted task. > > Julo > > > > > "Somesh Gupta" > Sent by: owner-ips@ece.cmu.edu > 13-11-01 03:46 > Please respond to somesh_gupta > > > To: "IPS" > cc: > Subject: RE: iSCSI: Clarification (again) for Task > Management Commands > > > > Julian, > > Section 9.4 does (or so I can figure) > leave the sort of hole mentioned > below uncovered. On a multiple connection session, > the status for any of the commands effected by > the task management command could already be in > transit on any of the connections by the time the > task management command is received by the target, > and its reception is acknowledged - and then > received by the initiator. > > The barrier works well for a single connection > because the command sequence number for the task > management command makes sure that there is nothing > stuck in the "pipe". However, there could be status > "stuck" in the other pipes. > > The mechanism David indicated might be the only > way to be sure, which however might require a NOP > on each connection to ensure there is nothing > in the pipe. > > Somesh > > > -----Original Message----- > > From: Julian Satran [mailto:Julian_Satran@il.ibm.com] > > Sent: Saturday, November 10, 2001 7:26 AM > > To: somesh_gupta@silverbacksystems.com > > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > > > > Read 9.4 for one implementation - Julo > > > > > > > > > > "Somesh Gupta" > > Sent by: owner-ips@ece.cmu.edu > > 10-11-01 05:26 > > Please respond to somesh_gupta > > > > > > To: , > > cc: > > Subject: RE: iSCSI: Clarification (again) for Task > > Management Commands > > > > > > > > David, > > > > In iSCSI the multiple connection/session construct > > adds significant complexity in determining whether > > a response for a command (on a different connection > > than the one on which the task management command > > was sent) impacted by the task management command will > > be received or not - and when? > > > > On a single connection, or similar links, when the > > response for the task management command is > > received (or after a fixed additional time), no > > responses will be received for the commands aborted > > by the task management command. > > > > However, with multiple connections there is no > > such "flushing" event on connections other than the > > one on which the task management command was sent. > > > > I would hope that the protocol would address this > > situation - the current response seems to be be to > > put the task tag and some associated resources in > > a zombie state for an unspecified amount of time. > > > > Somesh > > > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Friday, November 09, 2001 6:53 PM > > > To: somesh_gupta@silverbacksystems.com; ips@ece.cmu.edu > > > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > > > > > > > Somesh, > > > > > > The language in question reflects fairly direct requirements > > > language to be found in SAM-2's description of SCSI Task Management. > > > FCP goes to serious lengths with FC sequence aborts to make sure > > > this behaves as required. > > > > > > For iSCSI, if responses to the aborted commands show up unexpectedly, > > > they have to be discarded. How the Initiator keeps track of that > > > is the Initiator's problem - keeping track of the CmdSN of the > > > Abort Task Set may be useful. > > > > > > --David > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > > > > > -----Original Message----- > > > > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > > > > Sent: Friday, November 09, 2001 4:55 PM > > > > To: IPS > > > > Subject: iSCSI: Clarification (again) for Task Management Commands > > > > > > > > > > > > Resend to add iSCSI tag. Sorry for missing it. > > > > > > > > On page 67 of the 8-92.txt draft (section 3.5.1), it > > > > says > > > > > > > > "For all the tasks covered by the task > > > > management response (i.e., with CmdSN not higher than the task > > > > management command CmdSN), additional responses MUST NOT be > delivered > > > > to the SCSI layer after the task management response." > > > > > > > > If there is a multiple connection session, > > > > a status for a command impacted by the task > > > > management command (say ABORT TASK SET) could > > > > be stuck in the pipe on one connection, while > > > > the ABORT TASK SET completes on another > > > > connection. > > > > > > > > How does the initiator iSCSI enforce the rule above? > > > > Seems to be the equivalent of sending the impacted > > > > commands on other connections in a zombie state, > > > > and not having a very good idea of how to get out. > > > > > > > > Similarly Section 9.4 provides additional rules, > > > > but seems to leave a hole open with regards to > > > > status already in flight on other connections. > > > > > > > > Any clarifications would be appreciated. > > > > > > > > Somesh > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Wed Nov 14 00:21:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA28802 for ; Wed, 14 Nov 2001 00:21:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE4MAX23731 for ips-outgoing; Tue, 13 Nov 2001 23:22:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from openrelay.msu.edu (openrelay.msu.edu [35.9.98.20]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE4M7l23721 for ; Tue, 13 Nov 2001 23:22:07 -0500 (EST) Received: from deanjose (cs6669177-165.houston.rr.com [66.69.177.165]) by openrelay.msu.edu (8.11.1/8.11.1) with SMTP id fAE4Abp22327; Tue, 13 Nov 2001 23:10:37 -0500 (EST) (envelope-from deanjose@msu.edu) Reply-To: From: "Joe Dean" To: "Charles Monia" , "IPS Reflector" Subject: RE: FCencap: List ALL SOF/EOF codes Date: Tue, 13 Nov 2001 22:24:24 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I say option 3 is the best route. --------------------------------------- Joe Dean Storage Systems Engineer Compaq Computer Corporation joe.dean@compaq.com deanjose@msu.edu -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Charles Monia Sent: Tuesday, November 13, 2001 9:49 PM To: IPS Reflector Subject: RE: FCencap: List ALL SOF/EOF codes Hi: Option 3 is agreeable to me. Charles > -----Original Message----- > From: Elizabeth Rodriguez [mailto:egrodriguez@lucent.com] > Sent: Tuesday, November 13, 2001 11:05 AM > To: Murali Rajagopal; Charles Monia; IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > > (Chair hat on) > > Let me jump back in here, and summarize as I see it: > > Codes from FC-BB cannot be taken intact, since there are invalid codes > in that specification. > Codes initially trimmed in January of this year, but Ralph correctly > points out, that was prior to the formation of the FC encapsulation > draft (e.g. discussion pertained directly to FCIP; did not take into > consideration iFCP) > > That said, we can modify the current FC encapsulation draft to include > other classes of service, if we so choose. > > We can choose to > 1) Keep the table as is. > Note: The current SOF/EOF codes defined in the FC encapsulation draft > match the currently defined FC-BB-2 subset. > > 2) Add Class 1 codes > Note: FC-MI (technical report, not a standard) does not support Class > 1. > Note2: Practicality of Class 1 over IP questioned. > > 3) Add Class 4 codes > Note: Class 4 work is ongoing, but SOF/EOF codes currently > defined for > class 4 unchanged. > > 4) Add both class 1 and class 4 codes > > I would like to solicit input from everyone on what codes people feel > should be included in the FC encapsulation draft. > Note: Drafts following FC encapsulation may restrict classes > of service > that draft supports. > > Thanks, > > Elizabeth > > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Monday, November 12, 2001 6:01 PM > To: Charles Monia; IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > > On the specific topic of supported SOF and EOF codes the ietf > documents > should be driven by the specification provided in the *most relevant * > document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI > should > be kept out of this. > > If we simply accept to adopt the SOF and EOF codes listed for BB-2 the > problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. > I don't see why we are making a big deal about this. > > -Murali > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Monia > Sent: Monday, November 12, 2001 3:33 PM > To: IPS Reflector > Subject: RE: FCencap: List ALL SOF/EOF codes > > Hi Folks: > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > The last time I checked, the FC-MI spec was not a "standards track" > document > (to use IETF terminology). If that's still the case, is FC-MI's > prohibition > of class 1 a sufficient basis for precluding class 1 support in the > encapsulation spec? > > Charles > > -----Original Message----- > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > Sent: Saturday, November 10, 2001 8:52 AM > > To: IPS Reflector > > Cc: Black_David@emc.com > > Subject: Re: FCencap: List ALL SOF/EOF codes > > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > prohibits Class 1 and I can find no letter ballot comments > > asking that it be reinstated. > > > > Therefore, I am forced to agree with David. Class 1 MUST NOT > > be mentioned in the FC Encapsulation draft. If necessary, a > > note discussing interoperability and FC-MI can be added. > > > > Thanks. > > > > Ralph... > > > > Black_David@emc.com wrote: > > > > > FC-MI was going to prohibit Class 1 last time I checked. > Since the > > > I in FC-MI stands for "Interoperability", this seems like a > > reasonable > > > rationale for excluding Class 1 service. > > > > > > --David > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > From owner-ips@ece.cmu.edu Wed Nov 14 02:11:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA15265 for ; Wed, 14 Nov 2001 02:11:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE6OSR00966 for ips-outgoing; Wed, 14 Nov 2001 01:24:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE6OQl00961 for ; Wed, 14 Nov 2001 01:24:26 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA41776 for ; Wed, 14 Nov 2001 07:24:17 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAE6OJg39944 for ; Wed, 14 Nov 2001 07:24:20 +0100 To: ips@ece.cmu.edu Importance: Low MIME-Version: 1.0 Subject: RE: Application for port-number (3260) X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 14 Nov 2001 08:24:17 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 14/11/2001 08:24:20, Serialize complete at 14/11/2001 08:24:20 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk ----- Forwarded by Julian Satran/Haifa/IBM on 14-11-01 08:23 ----- "IANA" 14-11-01 00:13 Please respond to "IANA" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Application for port-number (3260) Dear Julian, We have assigned the following user port number with you as the point of contact: iscsi-target 3260/tcp iSCSI port iscsi-target 3260/udp iSCSI port # Julian Satran Please notify the IANA if there is a change in contact information. We will make appropriate changes when your document is approved by the IESG. Thank you, Michelle S. Cotton IANA Administrator *************************************************************** Internet Assigned Numbers Authority (IANA) 4676 Admiralty Way, Suite 330 Marina del Rey, California 90292 Voice: (310) 823-9358 FAX: (310) 823-8649 email: iana@iana.org *************************************************************** From owner-ips@ece.cmu.edu Wed Nov 14 02:13:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA15278 for ; Wed, 14 Nov 2001 02:13:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE6d6101793 for ips-outgoing; Wed, 14 Nov 2001 01:39:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE6d4l01789 for ; Wed, 14 Nov 2001 01:39:04 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA102222 for ; Wed, 14 Nov 2001 07:38:57 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAE6d0F94398 for ; Wed, 14 Nov 2001 07:39:00 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: Clarification (again) for Task Management Commands X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 14 Nov 2001 08:38:58 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 14/11/2001 08:39:01, Serialize complete at 14/11/2001 08:39:01 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, 9.4 makes sure that even commands that dod not arrive yet at the target will not generate any additional responses. The words I added to task management are meant only to clarify how an implementation is supposed to handle status that might be still in flight - i.e. initiator must mark the task as aborted but keep it patiently (to avoid an ITT reuse) untill status arrives (or is certain not to arrive anymore). Julo "Somesh Gupta" Sent by: owner-ips@ece.cmu.edu 14-11-01 04:36 Please respond to somesh_gupta To: cc: Subject: RE: iSCSI: Clarification (again) for Task Management Commands Julian, This is probably based more on my ignorance than anything else. However I would appreciate any clarification that you could add. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Tuesday, November 13, 2001 9:59 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > Somesh, > > I assume you may want to reread 9.4. It works well on any number of > connections and it handles all commands for which there is a "delivery > uncertainty". For commands for which the delivery is certain - i.e, they > have an instantiated Task Block at both initiator and target there is no > uncertainty either and the implementation of "no more statuses" does not > raise any special issues wherever their statuses are. The initiator > portion of the state has to be marked as aborted and kept in place until > the status arrives. To clarify this implementation detail I have added > some words to the relevant part of the Task Management request: My reading of SAM-2 indicates that the following 2 things (applicable to the commands sent by the initiator which sent the Task Management Commands) "that a response for a command affected by a Task Management Function will not be returned after the response for the Task Management Function is returned" and the second one is that "for commands that have been delivered to the SCSI entity, it is not certain whether the response will be returned or not - except that the rule above must be valid" If my understanding is correct, then I believe there is a hole even in 9.4 - and I will take the time to create a scenario. If my understanding is not correct, then based on what the right mechanism is, I will take another look. > > For all these functions, if executed, the Task Management Function > Response MUST be returned using the Initiator Task Tag to identify the > operation for which it is responding. All those functions apply to the > referenced tasks regardless if they are proper SCSI tasks or tagged iSCSI > operations. Task management commands must be executed as if all the > commands having a CmdSN lower or equal to the task management CmdSN have > been received by the target (i.e., have to be executed as if received for > ordered delivery even when marked for immediate delivery). For all the > tasks covered by the task management response (i.e., with CmdSN > not higher > than the task management command CmdSN), additional responses MUST NOT be > delivered to the SCSI layer after the task management response. This > requirement implies that the initiator must keep around state until the > status is received from the target for an aborted task and the > target MUST > deliver to the initiator good status for an aborted task. > > Julo > > > > > "Somesh Gupta" > Sent by: owner-ips@ece.cmu.edu > 13-11-01 03:46 > Please respond to somesh_gupta > > > To: "IPS" > cc: > Subject: RE: iSCSI: Clarification (again) for Task > Management Commands > > > > Julian, > > Section 9.4 does (or so I can figure) > leave the sort of hole mentioned > below uncovered. On a multiple connection session, > the status for any of the commands effected by > the task management command could already be in > transit on any of the connections by the time the > task management command is received by the target, > and its reception is acknowledged - and then > received by the initiator. > > The barrier works well for a single connection > because the command sequence number for the task > management command makes sure that there is nothing > stuck in the "pipe". However, there could be status > "stuck" in the other pipes. > > The mechanism David indicated might be the only > way to be sure, which however might require a NOP > on each connection to ensure there is nothing > in the pipe. > > Somesh > > > -----Original Message----- > > From: Julian Satran [mailto:Julian_Satran@il.ibm.com] > > Sent: Saturday, November 10, 2001 7:26 AM > > To: somesh_gupta@silverbacksystems.com > > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > > > > Read 9.4 for one implementation - Julo > > > > > > > > > > "Somesh Gupta" > > Sent by: owner-ips@ece.cmu.edu > > 10-11-01 05:26 > > Please respond to somesh_gupta > > > > > > To: , > > cc: > > Subject: RE: iSCSI: Clarification (again) for Task > > Management Commands > > > > > > > > David, > > > > In iSCSI the multiple connection/session construct > > adds significant complexity in determining whether > > a response for a command (on a different connection > > than the one on which the task management command > > was sent) impacted by the task management command will > > be received or not - and when? > > > > On a single connection, or similar links, when the > > response for the task management command is > > received (or after a fixed additional time), no > > responses will be received for the commands aborted > > by the task management command. > > > > However, with multiple connections there is no > > such "flushing" event on connections other than the > > one on which the task management command was sent. > > > > I would hope that the protocol would address this > > situation - the current response seems to be be to > > put the task tag and some associated resources in > > a zombie state for an unspecified amount of time. > > > > Somesh > > > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Friday, November 09, 2001 6:53 PM > > > To: somesh_gupta@silverbacksystems.com; ips@ece.cmu.edu > > > Subject: RE: iSCSI: Clarification (again) for Task Management Commands > > > > > > > > > Somesh, > > > > > > The language in question reflects fairly direct requirements > > > language to be found in SAM-2's description of SCSI Task Management. > > > FCP goes to serious lengths with FC sequence aborts to make sure > > > this behaves as required. > > > > > > For iSCSI, if responses to the aborted commands show up unexpectedly, > > > they have to be discarded. How the Initiator keeps track of that > > > is the Initiator's problem - keeping track of the CmdSN of the > > > Abort Task Set may be useful. > > > > > > --David > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > --------------------------------------------------- > > > > > > > -----Original Message----- > > > > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > > > > Sent: Friday, November 09, 2001 4:55 PM > > > > To: IPS > > > > Subject: iSCSI: Clarification (again) for Task Management Commands > > > > > > > > > > > > Resend to add iSCSI tag. Sorry for missing it. > > > > > > > > On page 67 of the 8-92.txt draft (section 3.5.1), it > > > > says > > > > > > > > "For all the tasks covered by the task > > > > management response (i.e., with CmdSN not higher than the task > > > > management command CmdSN), additional responses MUST NOT be > delivered > > > > to the SCSI layer after the task management response." > > > > > > > > If there is a multiple connection session, > > > > a status for a command impacted by the task > > > > management command (say ABORT TASK SET) could > > > > be stuck in the pipe on one connection, while > > > > the ABORT TASK SET completes on another > > > > connection. > > > > > > > > How does the initiator iSCSI enforce the rule above? > > > > Seems to be the equivalent of sending the impacted > > > > commands on other connections in a zombie state, > > > > and not having a very good idea of how to get out. > > > > > > > > Similarly Section 9.4 provides additional rules, > > > > but seems to leave a hole open with regards to > > > > status already in flight on other connections. > > > > > > > > Any clarifications would be appreciated. > > > > > > > > Somesh > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Wed Nov 14 04:57:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA16940 for ; Wed, 14 Nov 2001 04:57:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAE96nD09658 for ips-outgoing; Wed, 14 Nov 2001 04:06:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE96ll09650 for ; Wed, 14 Nov 2001 04:06:47 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA58052; Wed, 14 Nov 2001 10:06:39 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAE96fF76200; Wed, 14 Nov 2001 10:06:42 +0100 To: "IANA" Cc: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: Application for port-number (3260) X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 14 Nov 2001 11:06:39 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 14/11/2001 11:06:43, Serialize complete at 14/11/2001 11:06:43 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear Michelle, Thanks. The UDP port is not needed. Regards, Julo "IANA" 14-11-01 00:13 Please respond to "IANA" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Application for port-number (3260) Dear Julian, We have assigned the following user port number with you as the point of contact: iscsi-target 3260/tcp iSCSI port iscsi-target 3260/udp iSCSI port # Julian Satran Please notify the IANA if there is a change in contact information. We will make appropriate changes when your document is approved by the IESG. Thank you, Michelle S. Cotton IANA Administrator *************************************************************** Internet Assigned Numbers Authority (IANA) 4676 Admiralty Way, Suite 330 Marina del Rey, California 90292 Voice: (310) 823-9358 FAX: (310) 823-8649 email: iana@iana.org *************************************************************** From owner-ips@ece.cmu.edu Wed Nov 14 06:44:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA17939 for ; Wed, 14 Nov 2001 06:44:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEAqoe26030 for ips-outgoing; Wed, 14 Nov 2001 05:52:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEAqnl26026 for ; Wed, 14 Nov 2001 05:52:49 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA13502; Wed, 14 Nov 2001 11:52:41 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAEAqeg61198; Wed, 14 Nov 2001 11:52:41 +0100 Importance: Normal Subject: Re: SRP vs PKI for authentication To: VAHUJA@aol.com Cc: X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Ofer Biran" Date: Wed, 14 Nov 2001 12:53:39 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 14/11/2001 12:52:43 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk iSCSI doesn't require the use of SRP - it's just the authentication method that MUST be implemented by each compliant iSCSI implementation s.t. interoperability is guaranteed. PKI (by SPKM-1 or SPKM-2) is defined and optional to implement and use in iSCSI. SRP was chosen in the Nashua May 2001 interim meeting, after long review of 6 authentication methods according to 6 criteria. The main issues with SPKM was complex administration of CAs/CRLs and lack of existing implementations. You can see the presentation for this review on http://www.haifa.il.ibm.com/satran/ips/iSCSI-Sec-review-Nashua.pdf . Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 VAHUJA@aol.com@ece.cmu.edu on 14/11/2001 02:12:56 Please respond to VAHUJA@aol.com Sent by: owner-ips@ece.cmu.edu To: cc: Subject: SRP vs PKI for authentication iSCSI draft 08 requires use of SRP for authentication, while for Fabric switch authentication, there are proposals in T11 that use PKI. The iSNS also allows PKI for authentication. I have also seen some IP concerns raised recently about SRP... So my question is - for iSCSI login-time authentication, are there compelling reasons for using SRP instead of PKI? From owner-ips@ece.cmu.edu Wed Nov 14 07:34:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA19670 for ; Wed, 14 Nov 2001 07:34:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEAX2e25032 for ips-outgoing; Wed, 14 Nov 2001 05:33:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEAWxl25027 for ; Wed, 14 Nov 2001 05:32:59 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA07788; Wed, 14 Nov 2001 11:32:51 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAEAWpF69160; Wed, 14 Nov 2001 11:32:51 +0100 Importance: Normal Subject: Re: iSCSI: security questions To: "Lee Xing" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Ofer Biran" Date: Wed, 14 Nov 2001 12:33:51 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 14/11/2001 12:32:53 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Lee, ================ Q1: iSCSI v.08, page 142 "The authentication method cannot assume an underlying IPSec protection, since IPSec is optional to use." IPSec is an option for IPv4, but it's mandatory for IPv6 (if I remember right). Should we make it more specific? + IPsec is mandatory to implement in IPV6 but not mandatory to use, the + policy can always be configured to plain processing. Q2: iSCSI v.08 Chapter 10 (Security Consideration) mentions a few times of "...MUST implement...". Should we add something like "security is mandatory to implement, but not mandatory to use" in this chapter? This is stated explicitly in SEC-IPS v.04 draft, and also implied in Chapter 5 (Login Phase) of iSCSI v.08. + I think that "MUST implement" is quite clear and standard RFC statement, + and you don't need "but optional to use" disclaimer each time it appears. Q3: SEC-IPS v.04, page 11 "Negotiation between Initiator and Target is used to determine which authentication algorithm to use (or whether to use one at all); the connection closes if either side requires authentication and no mutually acceptable algorithm can be agreed upon" The question is whether "none" is considered as an "acceptable algorithm". In other words, if initiator asks "AuthMethod=KRB5,SRP,none" during login, and target answers "AuthMethod=none", should the connection be closed, or should the initiator continue with LoginOperationalNegotiation stage? If latter is acceptable, should we reword the last sentence like "...and no mutually acceptable algorithm or "none" can be agreed upon"? + "if either side requires authentication" rules out your example, + because by suggesting "none" and choosing "none" no side required + authentication. Q4: SEC-IPS v.04, page32 "If IPsec protection is removed on a connection, it MUST be reinstated before iSCSI, iFCP or FCIP packets are sent." The question is do we have to check security every time before sending out iSCSI packets? + This statement is going to change as a result of the sync effort with + the security draft, at least it would become non-normative. Regard, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Wed Nov 14 09:45:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA25248 for ; Wed, 14 Nov 2001 09:45:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEDk3705639 for ips-outgoing; Wed, 14 Nov 2001 08:46:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEDk1l05630 for ; Wed, 14 Nov 2001 08:46:01 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA60806 for ; Wed, 14 Nov 2001 14:45:51 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAEDjog77750 for ; Wed, 14 Nov 2001 14:45:51 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: update on OOO CmdSNs/connection X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 14 Nov 2001 15:45:44 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 14/11/2001 15:45:52, Serialize complete at 14/11/2001 15:45:52 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, The reason we have put in the SHOULD is that it better reflects what is happening on recovery anyhow. The text also clearly outlines when the SHOULD becomes MUST and that is the case you where concerned about. Regards, Julo "Somesh Gupta" Sent by: owner-ips@ece.cmu.edu 13-11-01 22:53 Please respond to somesh_gupta To: "ips" cc: Subject: RE: iSCSI: update on OOO CmdSNs/connection Julian, I am surprised to see the text in the latest rev of the doc change as suggested by Mallikarjun. I did not think there was a consensus on this subject. Just because I did not respond to Mallikarjun's last comment publicly should not be construed as agreement. Having the last word is hopefully not assumed to be consensus, otherwise a thread may never end. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Somesh Gupta > Sent: Friday, November 09, 2001 3:26 PM > To: ips > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > Mallikarjun, > > I never liked the SHOULD. It is not a design point. > If we really want to allow it, perhaps a negotiation > parameter is a better choice (which is only > marginally better). Error detection and recovery > have completely different design requirements than > the data path. > > So ideally we say MUST or nothing > or we negotiate it > > Also on your point A, it "MUST" be a MUST on > a single connection case except for when required > for error recovery (i.e. the very very rare - > case of digest error). > > Somesh > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Mallikarjun C. > > Sent: Friday, November 09, 2001 1:49 PM > > To: ips > > Subject: iSCSI: update on OOO CmdSNs/connection > > > > > > To those interested in this discussion: > > > > Julian and I had a phone conversation on the topic > > of OOO CmdSNs on a connection. An update follows. > > > > Julian agrees that there are valid error recovery > > scenarios where CmdSNs will legitimately end up OOO > > on a given connection. > > > > OTOH, I agree with two of Julian's scenarios that he > > pointed out right away - the "cleaning command" (command > > required to be sent after a retry copy to ensure flushing > > within 2^31 -1), and an immediate Logout posted with > > unacknowledged commands. Neither of this can be shipped > > OOO - since the former undoes the flushing intent, and > > the latter breaks the rule that nothing more follows a > > Logout on the connection (and troublesome in other ways, > > see below). > > > > In general, I share the concern with Julian that we > > have not closely scrutinized all possibilities. > > > > With that said, something along the following lines > > seemed reasonable - > > > > A)Initiator MUST send commands in increasing order of > > CmdSN on a connection if both the following are true - > > - operational ErrorRecoveryLevel is 0, > > - MaxConnections is negotiated to 1. > > B)In all the other cases, initiator SHOULD send commands > > in increasing order of CmdSN on a connection. It is > > strongly encouraged that commands with out-of-order > > CmdSNs be sent on a connection only if they are > > retransmitted commands due to digest error recovery > > and connection recovery. > > > > I also suggest the following upon further reflection- > > > > C)Add wording in section 2.2.2.1 to mandate that > > the cleaning command MUST be sent in-order after > > the retried command. > > D)Warn clearly that sending an immediate Logout command > > in the presence of other unacknowledged commands MAY > > create inadvertent discarding of certain commands (even > > if it is a recovery Logout), and MAY cause protocol > > errors leading to ungraceful shutdown of the connection. > > > > Hopefully A will bring the determinism that Somesh was > > looking for certain design points. B describes the more > > general n-connection session case. C & D are fixes for > > two identfied areas (so far) which will break. > > > > Comments? > > -- > > Mallikarjun > > > > > > Mallikarjun Chadalapaka > > Networked Storage Architecture > > Network Storage Solutions Organization > > MS 5668 Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > From owner-ips@ece.cmu.edu Wed Nov 14 09:49:46 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA25502 for ; Wed, 14 Nov 2001 09:49:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEDkKQ05679 for ips-outgoing; Wed, 14 Nov 2001 08:46:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d06lmsgate-2.uk.ibm.com (d06lmsgate-2.uk.ibm.com [195.212.29.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEDkEl05659 for ; Wed, 14 Nov 2001 08:46:15 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d06lmsgate-2.uk.ibm.com (1.0.0) with ESMTP id NAA54552 for ; Wed, 14 Nov 2001 13:27:55 GMT Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAEDk0F75414 for ; Wed, 14 Nov 2001 14:46:01 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: data and data sequences for Read X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Wed, 14 Nov 2001 15:45:54 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 14/11/2001 15:46:03, Serialize complete at 14/11/2001 15:46:03 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Paul, Nothing limits the burst rate except the TCP window. The bursts for read accomplish several things: they might be good "turnaround" points for bi-directional commands (minor) they delimit data that can be (on request) positive ack-ed enabling resources to be released at target Julo Paul Koning Sent by: owner-ips@ece.cmu.edu 14-11-01 00:22 Please respond to Paul Koning To: ips@ece.cmu.edu cc: Subject: iSCSI: data and data sequences for Read I hope this isn't a dumb question, but between the -08 spec and the archives I'm puzzled about the details around data sequences in the case of Read operations. As far as I can tell, a read can result in one or more data sequences coming back. These are numbered with DataSN values that keep going up across sequence boundaries. Each sequence is limited by MaxBurstSize, but the total Read size (sum of all the sequences) is not bounded other than by SCSI. In the Write case, something analogous happens but there there's an R2T to control the flow of data sequences. As far as I can see, there is nothing analogous in the Read case. In other words, while MaxBurstSize limits the size of a data sequence, there is no mechanism limiting the number of bursts, or the rate at which you send back DatIn PDUs (other than TCP window control). Is that right or did I miss something? If it's right, what is the purpose of having the notion of a Data Sequence for DataIn, and what does MaxBurstSize do for you in that case? Thanks, paul From owner-ips@ece.cmu.edu Wed Nov 14 09:51:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA25555 for ; Wed, 14 Nov 2001 09:51:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEE8B707178 for ips-outgoing; Wed, 14 Nov 2001 09:08:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail3.microsoft.com (mail3.microsoft.com [131.107.3.123]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAE13Ml10829 for ; Tue, 13 Nov 2001 20:03:23 -0500 (EST) Received: from INET-VRS-03.redmond.corp.microsoft.com ([157.54.5.27]) by mail3.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966); Tue, 13 Nov 2001 17:02:58 -0800 Received: from 157.54.6.150 by INET-VRS-03.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Tue, 13 Nov 2001 17:02:58 -0800 Received: from red-imc-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-05.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966); Tue, 13 Nov 2001 17:02:58 -0800 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.2966); Tue, 13 Nov 2001 17:02:55 -0800 Received: from win-msg-03.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.198]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3588.0); Tue, 13 Nov 2001 17:02:14 -0800 X-MimeOLE: Produced By Microsoft Exchange V6.0.6063.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Subject: RE: iSCSI: data and data sequences for Read Date: Tue, 13 Nov 2001 17:02:14 -0800 Message-ID: Thread-Topic: iSCSI: data and data sequences for Read Thread-Index: AcFslxU7DLwRP70PS7+LtuHVhNLnMgAEGgEQ From: "Lakshmi Ramasubramanian" To: "Paul Koning" , X-OriginalArrivalTime: 14 Nov 2001 01:02:14.0979 (UTC) FILETIME=[FFC91130:01C16CA7] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fAE13Nl10830 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit MaxBurstSize applies both to the initiator and the target. When initiator sends data (WRITE) it limits the sequence to MaxBurstSize. Initiator does not require R2T for READ because when it issues read, it is assumed that it is ready to receive data. Also, the maximum read size it's going to request from target is MaxBurstSize. Is that right or am I missing something? -lakshmi -----Original Message----- From: Paul Koning [mailto:ni1d@arrl.net] Sent: Tuesday, November 13, 2001 2:22 PM To: ips@ece.cmu.edu Subject: iSCSI: data and data sequences for Read I hope this isn't a dumb question, but between the -08 spec and the archives I'm puzzled about the details around data sequences in the case of Read operations. As far as I can tell, a read can result in one or more data sequences coming back. These are numbered with DataSN values that keep going up across sequence boundaries. Each sequence is limited by MaxBurstSize, but the total Read size (sum of all the sequences) is not bounded other than by SCSI. In the Write case, something analogous happens but there there's an R2T to control the flow of data sequences. As far as I can see, there is nothing analogous in the Read case. In other words, while MaxBurstSize limits the size of a data sequence, there is no mechanism limiting the number of bursts, or the rate at which you send back DatIn PDUs (other than TCP window control). Is that right or did I miss something? If it's right, what is the purpose of having the notion of a Data Sequence for DataIn, and what does MaxBurstSize do for you in that case? Thanks, paul From owner-ips@ece.cmu.edu Wed Nov 14 09:57:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA25930 for ; Wed, 14 Nov 2001 09:57:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEE4Bx06894 for ips-outgoing; Wed, 14 Nov 2001 09:04:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from falcon.vixel.com (mail.vixel.com [207.115.190.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fADHqml06546 for ; Tue, 13 Nov 2001 12:52:48 -0500 (EST) Received: from vixel.com ([192.168.1.179]) by falcon.vixel.com (Netscape Messaging Server 4.15) with ESMTP id GMR2ZX00.0ID; Tue, 13 Nov 2001 09:52:45 -0800 Message-ID: <3BF15DD9.6FB966D7@vixel.com> Date: Tue, 13 Nov 2001 09:52:25 -0800 From: "Ken Hirata" Organization: Vixel Corporation X-Mailer: Mozilla 4.76 [en] (WinNT; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Black_David@emc.com CC: kzm@cisco.com, ips@ece.cmu.edu Subject: Re: FC Management MIB - proposed changes References: <277DD60FB639D511AC0400B0D068B71ECAD8FB@CORPMX14> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, One nit (or misunderstanding). In item 5 below, Zoned and Unzoned name services are simultaneously active. My understanding is that Zoned name services present a view to a device of all other devices with which it may communicate (same as your explanation). Unzoned name services are used by a Management application to retrieve information about all devices in the Fabric. Ken Black_David@emc.com wrote: > Folks, > > For simplicity, I would propose keeping the initial revision focused > on structural and format changes only - i.e., make no functional > changes that aren't required to bring the MIB into line with the > overall architecture of MIBs, use of SMIv2, and the like. Comments on > this, keyed to Keith's issue numbers: > > 1. The trap table doesn't match up with RFC 2573 - I presume > correcting this is part of the conversion to SMIv2. > > 3. Let's keep the replacement of RFC 2837 as (a) a proposal > and (b) Keith's recommendation to the WG as MIB Advisor, > but not formally adopt that approach until we have a version > of the MIB draft that would replace RFC 2837 available for > review by the WG. > > 5. I think there's a misunderstanding here. There is only one > name service in any FC fabric, period. The term "Simple Name > Service" is historical and refers to the current approach > to Fibre Channel fabric name service by contrast to the > originally-proposed X.500 directory-based approach (see the > original FC-GS) - the meaning of "Simple" should now be > obvious, as it's by comparison to X.500 ;-). > > Both the Zoned and Unzoned name services are simple name services. > The same objects can be used to represent both, as only one is > active at any time, and the communication interfaces/protocols > are identical. Client access to the name service is the same > in both cases; zoned vs. unzoned only affects server behavior > in terms of what is returned to a query. In addition the name > service objects are described to represent only entities > "known to this unit". So, in a zoned fabric, I would expect > the table in a switch to be completely populated, but the table > in an HBA to contain only the entries in that HBA's zone > because the HBA doesn't "know" about any others. > > The name server objects need to be retained - these are quite > important for fabric management visibility. Whether the fabric > is zoned or not and how the nameserver behaves can be determined > from the zone objects (i.e., for a switch, the nameserver objects > contain everything and the zone objects have to be consulted to > figure out what a query from a particular client to the nameserver > will return). > > 7. I would definitely take out Class 1 support, and reference FC-MI > (which prohibits Class 1 service) as an authority for doing so. > > 8. For the initial version, I would only carry forward the zone > objects existing in the current MIB and not define any new > functionality - that can be done in revisions after -00. > > 9. I'd prefer that the -00 version contain no additional functional > changes beyond those required to support the necessary > structure and format changes. Once we have that -00 baseline, > functional changes can be made from there. > > 2, 4, and 6 look fine to me, as does the new draft name. > > Comments? > > Thanks, > --David -- Kenneth Hirata Vixel Corporation Irvine, CA 92618 Phone: (949) 450-6100 Email: Ken.Hirata@Vixel.com From owner-ips@ece.cmu.edu Wed Nov 14 10:42:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA28917 for ; Wed, 14 Nov 2001 10:42:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEEOZm08394 for ips-outgoing; Wed, 14 Nov 2001 09:24:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zcars0m9.nortelnetworks.com (zcars0m9.nortelnetworks.com [47.129.242.157]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEEOXl08389 for ; Wed, 14 Nov 2001 09:24:34 -0500 (EST) Received: from zcars04f.ca.nortel.com (zcars04f.ca.nortel.com [47.129.242.57]) by zcars0m9.nortelnetworks.com (8.11.0/8.11.0) with ESMTP id fAEENiS13972 for ; Wed, 14 Nov 2001 09:23:44 -0500 (EST) Received: from zbl6c012.corpeast.baynetworks.com by zcars04f.ca.nortel.com; Wed, 14 Nov 2001 09:22:32 -0500 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c012.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id WMPW6T7W; Wed, 14 Nov 2001 09:21:37 -0500 Received: from cse-ns-laptop.nortelnetworks.com (cse-ns-laptop.us.nortel.com [47.16.69.109]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id VW7FB6GF; Wed, 14 Nov 2001 09:21:39 -0500 Message-Id: <5.1.0.14.2.20011114090528.034b61c8@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Wed, 14 Nov 2001 09:24:52 -0500 To: ENDL_TX@computer.org, IPS Reflector From: "Franco Travostino" Subject: Re: FCencap: List ALL SOF/EOF codes In-Reply-To: <3BF18ED4.A3495BAD@compuserve.com> References: <9410A0F67DFE7F4D998D4538236498360408F5@nc8220exchange.ral.lucent.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_176083975==_.ALT" X-Orig: Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_176083975==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed I vote for 3) as well. I still feel it would be useful to have a sentence in the introductory scope section (1.) to crisply define the FC scope (and discourage anyone else from falling into this rathole). Minimally, such sentence could be a forward pointer to the SOF table near the end, with supporting T11 citations (including FC-MI if this is the most authoritative doc we have got). thanks -franco At 04:21 PM 11/13/2001, Ralph Weber wrote: >I support option 3. > >Option 1 fails to reflect the reality of current T11 work on >Class 4. > >Option 2 fails to consider the recent debate over iSCSI profiles. >The IETF position on profiles (as stated by the ADs during last >summer's debate) is that profiles must be stated as requirements >in RFCs. Therefore, the T11 FC-MI carries the weight of a standard >in the IETF and the FC-MI interoperability prohibition for Class 1 >must be stated in the FC Encapsulation. (Either that or those >folks wanting to write iSCSI profiles can resharpen their pens >for round two.) > >Option 4 attempts to link the good works of option 3 with >the bogus concept of profiles in option 2, which makes it >every bit as bogus. > >Murali's option 5 guarantees that there will be no RFC on >FC Encapsulation until 2003 since there will be no FC-BB-2 >standard to reference until then. > >Thanks. > >Ralph... > >Elizabeth Rodriguez wrote: > > > > > (Chair hat on) > > > > Let me jump back in here, and summarize as I see it: > > > > Codes from FC-BB cannot be taken intact, since there are invalid codes > > in that specification. > > Codes initially trimmed in January of this year, but Ralph correctly > > points out, that was prior to the formation of the FC encapsulation > > draft (e.g. discussion pertained directly to FCIP; did not take into > > consideration iFCP) > > > > That said, we can modify the current FC encapsulation draft to include > > other classes of service, if we so choose. > > > > We can choose to > > 1) Keep the table as is. > > Note: The current SOF/EOF codes defined in the FC encapsulation draft > > match the currently defined FC-BB-2 subset. > > > > 2) Add Class 1 codes > > Note: FC-MI (technical report, not a standard) does not support Class > > 1. > > Note2: Practicality of Class 1 over IP questioned. > > > > 3) Add Class 4 codes > > Note: Class 4 work is ongoing, but SOF/EOF codes currently defined for > > class 4 unchanged. > > > > 4) Add both class 1 and class 4 codes > > > > I would like to solicit input from everyone on what codes people feel > > should be included in the FC encapsulation draft. > > Note: Drafts following FC encapsulation may restrict classes of service > > that draft supports. > > > > Thanks, > > > > Elizabeth > > > > -----Original Message----- > > From: Murali Rajagopal [mailto:muralir@lightsand.com] > > Sent: Monday, November 12, 2001 6:01 PM > > To: Charles Monia; IPS Reflector > > Subject: RE: FCencap: List ALL SOF/EOF codes > > > > On the specific topic of supported SOF and EOF codes the ietf documents > > should be driven by the specification provided in the *most relevant * > > document which in this case happens to be FC-BB-2 ant FC-MI. FC-MI > > should > > be kept out of this. > > > > If we simply accept to adopt the SOF and EOF codes listed for BB-2 the > > problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes. > > I don't see why we are making a big deal about this. > > > > -Murali > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Charles Monia > > Sent: Monday, November 12, 2001 3:33 PM > > To: IPS Reflector > > Subject: RE: FCencap: List ALL SOF/EOF codes > > > > Hi Folks: > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > > prohibits Class 1 and I can find no letter ballot comments > > > asking that it be reinstated. > > > > The last time I checked, the FC-MI spec was not a "standards track" > > document > > (to use IETF terminology). If that's still the case, is FC-MI's > > prohibition > > of class 1 a sufficient basis for precluding class 1 support in the > > encapsulation spec? > > > > Charles > > > -----Original Message----- > > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > > Sent: Saturday, November 10, 2001 8:52 AM > > > To: IPS Reflector > > > Cc: Black_David@emc.com > > > Subject: Re: FCencap: List ALL SOF/EOF codes > > > > > > > > > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001) > > > prohibits Class 1 and I can find no letter ballot comments > > > asking that it be reinstated. > > > > > > Therefore, I am forced to agree with David. Class 1 MUST NOT > > > be mentioned in the FC Encapsulation draft. If necessary, a > > > note discussing interoperability and FC-MI can be added. > > > > > > Thanks. > > > > > > Ralph... > > > > > > Black_David@emc.com wrote: > > > > > > > FC-MI was going to prohibit Class 1 last time I checked. Since the > > > > I in FC-MI stands for "Interoperability", this seems like a > > > reasonable > > > > rationale for excluding Class 1 service. > > > > > > > > --David > > > > --------------------------------------------------- > > > > David L. Black, Senior Technologist > > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > > --------------------------------------------------- > > > --=====================_176083975==_.ALT Content-Type: text/html; charset="us-ascii"
I vote for 3) as well. I still feel it would be useful to have a sentence in the introductory scope section (1.) to crisply define the FC scope  (and discourage anyone else from falling into this rathole). Minimally, such sentence could be a forward pointer to the SOF table near the end, with supporting T11 citations (including FC-MI if this is the most authoritative doc we have got).

thanks
-franco

At 04:21 PM 11/13/2001, Ralph Weber wrote:
I support option 3.

Option 1 fails to reflect the reality of current T11 work on
Class 4.

Option 2 fails to consider the recent debate over iSCSI profiles.
The IETF position on profiles (as stated by the ADs during last
summer's debate) is that profiles must be stated as requirements
in RFCs. Therefore, the T11 FC-MI carries the weight of a standard
in the IETF and the FC-MI interoperability prohibition for Class 1
must be stated in the FC Encapsulation. (Either that or those
folks wanting to write iSCSI profiles can resharpen their pens
for round two.)

Option 4 attempts to link the good works of option 3 with
the bogus concept of profiles in option 2, which makes it
every bit as bogus.

Murali's option 5 guarantees that there will be no RFC on
FC Encapsulation until 2003 since there will be no FC-BB-2
standard to reference until then.

Thanks.

Ralph...

Elizabeth Rodriguez wrote:

>
> (Chair hat on)
>
> Let me jump back in here, and summarize as I see it:
>
> Codes from FC-BB cannot be taken intact, since there are invalid codes
> in that specification.
> Codes initially trimmed in January of this year, but Ralph correctly
> points out, that was prior to the formation of the FC encapsulation
> draft (e.g. discussion pertained directly to FCIP; did not take into
> consideration iFCP)
>
> That said, we can modify the current FC encapsulation draft to include
> other classes of service, if we so choose.
>
> We can choose to
> 1) Keep the table as is.
> Note: The current SOF/EOF codes defined in the FC encapsulation draft
> match the currently defined FC-BB-2 subset.
>
> 2) Add Class 1 codes
> Note:  FC-MI (technical report, not a standard) does not support Class
> 1.
> Note2: Practicality of Class 1 over IP questioned.
>
> 3) Add Class 4 codes
> Note:  Class 4 work is ongoing, but SOF/EOF codes currently defined for
> class 4 unchanged.
>
> 4) Add both class 1 and class 4 codes
>
> I would like to solicit input from everyone on what codes people feel
> should be included in the FC encapsulation draft.
> Note:  Drafts following FC encapsulation may restrict classes of service
> that draft supports.
>
> Thanks,
>
> Elizabeth
>
> -----Original Message-----
> From: Murali Rajagopal [mailto:muralir@lightsand.com]
> Sent: Monday, November 12, 2001 6:01 PM
> To: Charles Monia; IPS Reflector
> Subject: RE: FCencap: List ALL SOF/EOF codes
>
> On the specific topic of supported SOF and EOF codes the ietf documents
> should be driven by the specification provided in the *most relevant *
> document which in this case happens to be FC-BB-2 ant FC-MI.  FC-MI
> should
> be kept out of this.
>
> If we simply accept to adopt the SOF and EOF codes listed for BB-2 the
> problem is solved. FYI, BB-2 only supports Class 2, 3, and F codes.
> I don't see why we are making a big deal about this.
>
> -Murali
>
> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Charles Monia
> Sent: Monday, November 12, 2001 3:33 PM
> To: IPS Reflector
> Subject: RE: FCencap: List ALL SOF/EOF codes
>
> Hi Folks:
>
> > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001)
> > prohibits Class 1 and I can find no letter ballot comments
> > asking that it be reinstated.
>
> The last time I checked, the FC-MI spec was not a "standards track"
> document
> (to use IETF terminology). If that's still the case, is FC-MI's
> prohibition
> of class 1 a sufficient basis for precluding class 1 support in the
> encapsulation spec?
>
> Charles
> > -----Original Message-----
> > From: Ralph Weber [mailto:ralphoweber@compuserve.com]
> > Sent: Saturday, November 10, 2001 8:52 AM
> > To: IPS Reflector
> > Cc: Black_David@emc.com
> > Subject: Re: FCencap: List ALL SOF/EOF codes
> >
> >
> > David's observation is correct. FC-MI rev 1.8 (28 Sept 2001)
> > prohibits Class 1 and I can find no letter ballot comments
> > asking that it be reinstated.
> >
> > Therefore, I am forced to agree with David. Class 1 MUST NOT
> > be mentioned in the FC Encapsulation draft. If necessary, a
> > note discussing interoperability and FC-MI can be added.
> >
> > Thanks.
> >
> > Ralph...
> >
> > Black_David@emc.com wrote:
> >
> > > FC-MI was going to prohibit Class 1 last time I checked.  Since the
> > > I in FC-MI stands for "Interoperability", this seems like a
> > reasonable
> > > rationale for excluding Class 1 service.
> > >
> > > --David
> > > ---------------------------------------------------
> > > David L. Black, Senior Technologist
> > > EMC Corporation, 42 South St., Hopkinton, MA  01748
> > > +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> > > black_david@emc.com       Mobile: +1 (978) 394-7754
> > > ---------------------------------------------------
> >
--=====================_176083975==_.ALT-- From owner-ips@ece.cmu.edu Wed Nov 14 12:15:08 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04671 for ; Wed, 14 Nov 2001 12:15:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEFItA12614 for ips-outgoing; Wed, 14 Nov 2001 10:18:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from va2mc.ummailbox.net (va2mc.ummailbox.net [64.210.247.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEFIrl12601 for ; Wed, 14 Nov 2001 10:18:53 -0500 (EST) Received: from hqmailweb.COMMSTOR.Crossroads.com (HQMailWeb.Crossroads.com [63.237.99.250:25]) by va2mc.ummailbox.net with ESMTP id C1114-1018-4c0904; Wed, 14 Nov 2001 15:18:51 GMT Received: from HQMAILNODE1.COMMSTOR.Crossroads.com ([10.5.1.42]) by hqmailweb.COMMSTOR.Crossroads.com with Microsoft SMTPSVC(5.0.2195.3779); Wed, 14 Nov 2001 09:18:43 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: iSCSI: security questions Disposition-Notification-To: "Lee Xing" Date: Wed, 14 Nov 2001 09:17:41 -0600 Message-ID: Thread-Topic: iSCSI: security questions Thread-Index: AcFtA9aYDuFziGRYSiWOK7hudj5IXgAF8ezA From: "Lee Xing" To: X-OriginalArrivalTime: 14 Nov 2001 15:18:43.0057 (UTC) FILETIME=[A5777E10:01C16D1F] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fAEFIrl12605 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Ofer, Thanks for the info. Please see my comments bellow. Regards, Lee Crossroads Systems, Inc. ================ Q3: SEC-IPS v.04, page 11 "Negotiation between Initiator and Target is used to determine which authentication algorithm to use (or whether to use one at all); the connection closes if either side requires authentication and no mutually acceptable algorithm can be agreed upon" The question is whether "none" is considered as an "acceptable algorithm". In other words, if initiator asks "AuthMethod=KRB5,SRP,none" during login, and target answers "AuthMethod=none", should the connection be closed, or should the initiator continue with LoginOperationalNegotiation stage? If latter is acceptable, should we reword the last sentence like "...and no mutually acceptable algorithm or "none" can be agreed upon"? + "if either side requires authentication" rules out your example, + because by suggesting "none" and choosing "none" no side required + authentication. # In iSCSI v.08, there are quite a few Login Phase # Examples which use "AuthMethod=KRB5,SRP,...,none". # I'm not sure which one (the Login Phase Examples, or # your comments) is more appropriate. From owner-ips@ece.cmu.edu Wed Nov 14 12:23:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05366 for ; Wed, 14 Nov 2001 12:23:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEGKGt17797 for ips-outgoing; Wed, 14 Nov 2001 11:20:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEGKDl17789 for ; Wed, 14 Nov 2001 11:20:13 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA63364; Wed, 14 Nov 2001 17:20:01 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAEGJxg72284; Wed, 14 Nov 2001 17:20:04 +0100 Importance: Normal Subject: RE: iSCSI: security questions To: "Lee Xing" Cc: X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Ofer Biran" Date: Wed, 14 Nov 2001 18:16:34 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 14/11/2001 18:20:05 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Lee, Sorry if I wasn't clear enough. All I was trying to say is that the statement "the connection closes if either side requires authentication and no mutually acceptable algorithm can be agreed upon" is OK since "requires authentication" for the initiator means that he doesn't offer "none", and "requires authentication" for the target means that he is not ready to accept "none". The example you gave is acceptable (as the iSCSI login examples) but doesn't pass the "if either side requires authentication" condition, so closing of connection is not implied by it. Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Lee Xing" @ece.cmu.edu on 14/11/2001 17:17:41 Please respond to "Lee Xing" Sent by: owner-ips@ece.cmu.edu To: cc: Subject: RE: iSCSI: security questions Ofer, Thanks for the info. Please see my comments bellow. Regards, Lee Crossroads Systems, Inc. ================ Q3: SEC-IPS v.04, page 11 "Negotiation between Initiator and Target is used to determine which authentication algorithm to use (or whether to use one at all); the connection closes if either side requires authentication and no mutually acceptable algorithm can be agreed upon" The question is whether "none" is considered as an "acceptable algorithm". In other words, if initiator asks "AuthMethod=KRB5,SRP,none" during login, and target answers "AuthMethod=none", should the connection be closed, or should the initiator continue with LoginOperationalNegotiation stage? If latter is acceptable, should we reword the last sentence like "...and no mutually acceptable algorithm or "none" can be agreed upon"? + "if either side requires authentication" rules out your example, + because by suggesting "none" and choosing "none" no side required + authentication. # In iSCSI v.08, there are quite a few Login Phase # Examples which use "AuthMethod=KRB5,SRP,...,none". # I'm not sure which one (the Login Phase Examples, or # your comments) is more appropriate. From owner-ips@ece.cmu.edu Wed Nov 14 13:50:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA10845 for ; Wed, 14 Nov 2001 13:50:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEHQZg23169 for ips-outgoing; Wed, 14 Nov 2001 12:26:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel2.hp.com (palrel2.hp.com [156.153.255.234]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEHQYl23164 for ; Wed, 14 Nov 2001 12:26:34 -0500 (EST) Received: from hpcuhe.cup.hp.com (hpcuhe.cup.hp.com [15.0.80.203]) by palrel2.hp.com (Postfix) with ESMTP id E91968E8; Wed, 14 Nov 2001 09:26:32 -0800 (PST) Received: from cup.hp.com (santoshr@hpindhhm.cup.hp.com [15.8.80.197]) by hpcuhe.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id JAA13372; Wed, 14 Nov 2001 09:26:27 -0800 (PST) Message-ID: <3BF2AB60.FF215AC@cup.hp.com> Date: Wed, 14 Nov 2001 09:35:28 -0800 From: Santosh Rao Organization: Hewlett Packard, Cupertino. X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.11.00 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: Lakshmi Ramasubramanian Cc: Paul Koning , ips@ece.cmu.edu Subject: Re: iSCSI: data and data sequences for Read References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Lakshmi Ramasubramanian wrote: > > MaxBurstSize applies both to the initiator and the target. When > initiator > sends data (WRITE) it limits the sequence to MaxBurstSize. I don't believe it is the initiator's responsibility to chunk outbound data sequences into MaxBurstSize chunks. It is the target's responsibility to ensure that it does not request > 'MaxBurstSize' bytes of data in any R2T that it issues. - Santosh > Initiator > does not > require R2T for READ because when it issues read, it is assumed that it > is ready > to receive data. Also, the maximum read size it's going to request from > target is > MaxBurstSize. > > Is that right or am I missing something? > > -lakshmi > > -----Original Message----- > From: Paul Koning [mailto:ni1d@arrl.net] > Sent: Tuesday, November 13, 2001 2:22 PM > To: ips@ece.cmu.edu > Subject: iSCSI: data and data sequences for Read > > I hope this isn't a dumb question, but between the -08 spec and the > archives I'm puzzled about the details around data sequences in the > case of Read operations. > > As far as I can tell, a read can result in one or more data sequences > coming back. These are numbered with DataSN values that keep going > up across sequence boundaries. Each sequence is limited by > MaxBurstSize, but the total Read size (sum of all the sequences) is > not bounded other than by SCSI. > > In the Write case, something analogous happens but there there's an > R2T to control the flow of data sequences. > > As far as I can see, there is nothing analogous in the Read case. In > other words, while MaxBurstSize limits the size of a data sequence, > there is no mechanism limiting the number of bursts, or the rate at > which you send back DatIn PDUs (other than TCP window control). > > Is that right or did I miss something? If it's right, what is the > purpose of having the notion of a Data Sequence for DataIn, and what > does MaxBurstSize do for you in that case? > > Thanks, > paul -- ################################## Santosh Rao Software Design Engineer, HP-UX iSCSI Driver Team, Hewlett Packard, Cupertino. email : santoshr@cup.hp.com Phone : 408-447-3751 ################################## From owner-ips@ece.cmu.edu Wed Nov 14 14:15:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12253 for ; Wed, 14 Nov 2001 14:15:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEHqMZ25097 for ips-outgoing; Wed, 14 Nov 2001 12:52:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from va2mc.ummailbox.net (va2mc.ummailbox.net [64.210.247.21]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEHqKl25092 for ; Wed, 14 Nov 2001 12:52:20 -0500 (EST) Received: from hqmailweb.COMMSTOR.Crossroads.com (HQMailWeb.Crossroads.com [63.237.99.250:25]) by va2mc.ummailbox.net with ESMTP id G1114-1252-6c3604; Wed, 14 Nov 2001 17:52:18 GMT Received: from HQMAILNODE1.COMMSTOR.Crossroads.com ([10.5.1.42]) by hqmailweb.COMMSTOR.Crossroads.com with Microsoft SMTPSVC(5.0.2195.3779); Wed, 14 Nov 2001 11:52:09 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: iSCSI: security questions Disposition-Notification-To: "Lee Xing" Date: Wed, 14 Nov 2001 11:51:45 -0600 Message-ID: Thread-Topic: iSCSI: security questions Thread-Index: AcFtKEDcNWze4gUwSqiUB8rbPfVeGwAAY31w From: "Lee Xing" To: X-OriginalArrivalTime: 14 Nov 2001 17:52:09.0571 (UTC) FILETIME=[14FA0B30:01C16D35] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fAEHqKl25093 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Ofer, Thanks for the info. Please see my comments below. Regards, Lee Xing Crossroads Systems, Inc. ============================= Lee, Sorry if I wasn't clear enough. All I was trying to say is that the statement "the connection closes if either side requires authentication and no mutually acceptable algorithm can be agreed upon" is OK since "requires authentication" for the initiator means that he doesn't offer "none", and "requires authentication" for the target means that he is not ready to accept "none". The example you gave is acceptable (as the iSCSI login examples) but doesn't pass the "if either side requires authentication" condition, so closing of connection is not implied by it. + Let's consider a Login Phase Example: + + I-> Login (CSG,NSG=0,1 T=1) + ... + AuthMethod=KRB5,SRP,none + + T-> Login-PR (CSG,NSG=0,1 T=1) + ... + AuthMethod=none + + does "CSG=0" mean that the initiator "requires + authentication"? If it does, is "none" in Login + AuthMethod list a legal value to have? If it is, + is "none" in Login-PR AuthMethod list a legal value + to have even though the target "requires authentication"? + If it is, should the connection closes, or should the + initiator continue with next Login Stage? If it + should continue with next Login Stage, then should + we reword the paragraph in SEC-IPS v.04? + Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 "Lee Xing" @ece.cmu.edu on 14/11/2001 17:17:41 Please respond to "Lee Xing" Sent by: owner-ips@ece.cmu.edu To: cc: Subject: RE: iSCSI: security questions Ofer, Thanks for the info. Please see my comments bellow. Regards, Lee Crossroads Systems, Inc. ================ Q3: SEC-IPS v.04, page 11 "Negotiation between Initiator and Target is used to determine which authentication algorithm to use (or whether to use one at all); the connection closes if either side requires authentication and no mutually acceptable algorithm can be agreed upon" The question is whether "none" is considered as an "acceptable algorithm". In other words, if initiator asks "AuthMethod=KRB5,SRP,none" during login, and target answers "AuthMethod=none", should the connection be closed, or should the initiator continue with LoginOperationalNegotiation stage? If latter is acceptable, should we reword the last sentence like "...and no mutually acceptable algorithm or "none" can be agreed upon"? + "if either side requires authentication" rules out your example, + because by suggesting "none" and choosing "none" no side required + authentication. # In iSCSI v.08, there are quite a few Login Phase # Examples which use "AuthMethod=KRB5,SRP,...,none". # I'm not sure which one (the Login Phase Examples, or # your comments) is more appropriate. From owner-ips@ece.cmu.edu Wed Nov 14 15:35:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15476 for ; Wed, 14 Nov 2001 15:35:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEJMZH02670 for ips-outgoing; Wed, 14 Nov 2001 14:22:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub.icann.org (mailhub.icann.org [192.0.34.33]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEH0sl21301 for ; Wed, 14 Nov 2001 12:00:55 -0500 (EST) Received: from tarim (tarim.icann.org [192.0.34.205]) by mailhub.icann.org (8.9.3/8.9.3) with SMTP id IAA24679; Wed, 14 Nov 2001 08:59:48 -0800 From: "IANA" To: "Julian Satran" Cc: Subject: RE: Application for port-number (3260) Date: Wed, 14 Nov 2001 08:54:00 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, We automatically assign both the tcp and udp ports. Thanks, Michelle S. Cotton IANA Administrator -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Wednesday, November 14, 2001 1:07 AM To: IANA Cc: ips@ece.cmu.edu Subject: RE: Application for port-number (3260) Dear Michelle, Thanks. The UDP port is not needed. Regards, Julo "IANA" 14-11-01 00:13 Please respond to "IANA" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Application for port-number (3260) Dear Julian, We have assigned the following user port number with you as the point of contact: iscsi-target 3260/tcp iSCSI port iscsi-target 3260/udp iSCSI port # Julian Satran Please notify the IANA if there is a change in contact information. We will make appropriate changes when your document is approved by the IESG. Thank you, Michelle S. Cotton IANA Administrator *************************************************************** Internet Assigned Numbers Authority (IANA) 4676 Admiralty Way, Suite 330 Marina del Rey, California 90292 Voice: (310) 823-9358 FAX: (310) 823-8649 email: iana@iana.org *************************************************************** From owner-ips@ece.cmu.edu Wed Nov 14 17:00:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA18713 for ; Wed, 14 Nov 2001 17:00:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEKxFD10787 for ips-outgoing; Wed, 14 Nov 2001 15:59:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1ab.compuserve.com (siaar1ab.compuserve.com [149.174.40.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEKxDl10781 for ; Wed, 14 Nov 2001 15:59:13 -0500 (EST) Received: (from mailgate@localhost) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id PAA19517 for ips@ece.cmu.edu; Wed, 14 Nov 2001 15:58:56 -0500 (EST) Received: from compuserve.com (dal-tgn-tkl-vty77.as.wcom.net [216.192.224.77]) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id PAA19444 for ; Wed, 14 Nov 2001 15:58:52 -0500 (EST) Message-ID: <3BF2DC13.F23D434@compuserve.com> Date: Wed, 14 Nov 2001 15:03:15 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: FCencap: Proposed changes for -04 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Since option 3 seems to be acceptable to all and since time is short, here are the changes I propose to make in draft-ietf-ips-fcencapsulation-04 Add words to the effect of: 1) [in Scope] T11 determinations of interoperability have been applied to restrict the applicability of this specification. 2) [in FC SOF and EOF] add SOFi4 and SOFn4 to table 1. 3) [in FC SOF and EOF] add a note that other SOF and EOF codes are defined in FC-BB but that they are not included here because T11 treats them as not interoperable (see FC-MI [FC-MI crossref]). If these changes fail to cover somebody's concerns, now would be a really good time to speak up. (Of course, there's always Last Call, to mention another good time.) Thanks. Ralph... From owner-ips@ece.cmu.edu Wed Nov 14 19:12:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA22334 for ; Wed, 14 Nov 2001 19:12:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEMvQQ20418 for ips-outgoing; Wed, 14 Nov 2001 17:57:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEMvPl20414 for ; Wed, 14 Nov 2001 17:57:25 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 14 Nov 2001 17:59:46 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937199@CORPMX14> From: Black_David@emc.com To: ENDL_TX@computer.org, ips@ece.cmu.edu Subject: RE: FCencap: Proposed changes for -04 Date: Wed, 14 Nov 2001 17:47:28 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk For 3), please explicitly say the Class 1 codes are an example of codes not included for this reason. Thanks, --David > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Wednesday, November 14, 2001 4:03 PM > To: IPS Reflector > Subject: FCencap: Proposed changes for -04 > > > Since option 3 seems to be acceptable to all and > since time is short, here are the changes I propose > to make in draft-ietf-ips-fcencapsulation-04 > > Add words to the effect of: > > 1) [in Scope] T11 determinations of interoperability > have been applied to restrict the applicability of > this specification. > > 2) [in FC SOF and EOF] add SOFi4 and SOFn4 to table 1. > > 3) [in FC SOF and EOF] add a note that other SOF and EOF > codes are defined in FC-BB but that they are not > included here because T11 treats them as not > interoperable (see FC-MI [FC-MI crossref]). > > If these changes fail to cover somebody's concerns, > now would be a really good time to speak up. (Of > course, there's always Last Call, to mention another > good time.) > > Thanks. > > Ralph... > > > > From owner-ips@ece.cmu.edu Wed Nov 14 19:12:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA22347 for ; Wed, 14 Nov 2001 19:12:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAEMqe720071 for ips-outgoing; Wed, 14 Nov 2001 17:52:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAEMqcl20060 for ; Wed, 14 Nov 2001 17:52:38 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id ; Wed, 14 Nov 2001 17:48:55 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937198@CORPMX14> From: Black_David@emc.com To: VAHUJA@aol.com, ips@ece.cmu.edu Subject: RE: SRP vs PKI for authentication Date: Wed, 14 Nov 2001 17:42:42 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ofer's provided part of the answer, in addition this is based in part on considerations for environments that are not prepared to invest heavily in security (so IPsec is not in use), but for which authentication is appropriate. SRP is considerably easier to set up and use than a PKI-based mechanism, and hence is the "MUST implement" to make good authentication easier to use in such environments in contrast to one of the SPKMs and the associated PKI and/or key generation and key management it requires. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: VAHUJA@aol.com [mailto:VAHUJA@aol.com] > Sent: Tuesday, November 13, 2001 7:13 PM > To: ips@ece.cmu.edu > Subject: SRP vs PKI for authentication > > > iSCSI draft 08 requires use of SRP for authentication, while > for Fabric switch authentication, there are proposals in T11 > that use PKI. The iSNS also allows PKI for authentication. I > have also seen some IP concerns raised recently about SRP... > > So my question is - for iSCSI login-time authentication, are > there compelling reasons for using SRP instead of PKI? > From owner-ips@ece.cmu.edu Thu Nov 15 02:33:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA14974 for ; Thu, 15 Nov 2001 02:33:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAF6TPG18132 for ips-outgoing; Thu, 15 Nov 2001 01:29:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAF6TMl18128 for ; Thu, 15 Nov 2001 01:29:23 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA257520 for ; Thu, 15 Nov 2001 07:29:16 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAF6TJE89428 for ; Thu, 15 Nov 2001 07:29:19 +0100 Subject: Re: iSCSI: SCSI timeout handling change To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 Message-ID: From: "Julian Satran" Date: Thu, 15 Nov 2001 08:28:51 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 15/11/2001 08:29:20 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, A very good initiative (considering the low timeouts some OSs have and the fact the expected behavior of the SCSI stack is abort). According to our phone conversation here is a summary of the changes: Ordered sending returns being a MUST for all cases except recovery. For OOO implementers may use prefetching or the mechanism already in place - multiple connections. The wording in 2.2.2.1 is: On any given connection, the iSCSI initiator MUST send the commands in increasing order of CmdSN except for retransmitted commands due to digest error recovery and connection recovery. Abort task - section 3.5.1 reads 1.1.1 Function The Task Management functions provide an initiator with a way to explicitly control the execution of one or more Tasks (SCSI and iSCSI tasks). The Task Management functions are (for a more detailed description of SCSI task management see [SAM2]): 1 ABORT TASK - aborts the task identified by the Referenced Task Tag field. 2 ABORT TASK SET - aborts all Tasks issued by this initiator on the Logical Unit. 3 CLEAR ACA - clears the Auto Contingent Allegiance condition. 4 CLEAR TASK SET - Aborts all Tasks (from all initiators) for the Logical Unit. 5 LOGICAL UNIT RESET 6 TARGET WARM RESET 7 TARGET COLD RESET 8 TASK REASSIGN - reassign connection allegiance for the task identified by the Initiator Task Tag field on this connection, thus resuming the iSCSI exchanges for the task For all these functions, if executed, the Task Management Function Response MUST be returned using the Initiator Task Tag to identify the operation for which it is responding. All those functions apply to the referenced tasks regardless if they are proper SCSI tasks or tagged iSCSI operations. Task management commands must be executed as if all the commands having a CmdSN lower or equal to the task management CmdSN have been received by the target (i.e., have to be executed as if received for ordered delivery even when marked for immediate delivery). For all the tasks covered by the task management response (i.e., with CmdSN not higher than the task management command CmdSN), additional responses MUST NOT be delivered to the SCSI layer after the task management response. This requirement implies that the initiator must keep around state until the status is received from the target for all aborted tasks and the target MUST deliver to the initiator good status for all aborted task for which no status was delivered yet. The task management response MAY be issued by the target immediately after marking all tasks to be aborted. ABORT TASK MUST be issued on the same connection to which the task to be aborted is allegiant at the time the Task Management Request is issued if the connection is still active (it is not undergoing an implicit or explicit logout). If the connection is being implicitly or explicitly logged out (i.e., no other request will be issued on the failing connection and no other response will be received on the failing connection) then an ABORT TASK function request may be issued on another connection. This Task Management request will then both establish a new allegiance for the command to be aborted, and abort it as well (i.e., the task to be aborted will not have to be retried or reassigned, and its status if issued but not acknowledged will be reissued). For the ABORT TASK function, the target MUST NOT deliver additional responses after sending the task management response. In case both responses were delivered, whether the initiator should deliver task responses before delivering the task management response or not while an ABORT TASK is executing is a matter of implementation. This requirement implies that the initiator must keep around state until the status is received from the target for an aborted task and the target MUST deliver to the initiator good status for an aborted task if no status was delivered yet. The task management response MUST be issued after the command status (if any) was issued. For the LOGICAL UNIT RESET function, the target MUST behave as dictated by the Logical Unit Reset function in [SAM2]. The TARGET RESET function (WARM and COLD) implementation is OPTIONAL and when implemented should act as described below. Target Reset MAY be also subject to SCSI access controls for the requesting initiator. When not implemented or when authorization fails at target, Target Reset functions should end as if the function was executed successfully and the response qualifier will detail what was executed. For the TARGET WARM RESET and TARGET COLD RESET functions, the target cancels all pending operations and are both equivalent to the Target Reset function specified by [SAM2]. They can both affect many other initiators. In addition, for the TARGET COLD RESET the target then MUST terminate all of its TCP connections to all initiators (all sessions are terminated). For the TASK REASSIGN function, the target should reassign the connection allegiance to this new connection (and thus resume iSCSI exchanges for the task). TASK REASSIGN MUST be received by the target ONLY after the connection on which the command was previously executing has been successfully logged-out. For additional usage semantics, see section 8.1. TASK REASSIGN MUST be issued as an immediate command. Section 8.6 reads 1.1 SCSI Timeouts An iSCSI initiator MAY attempt to plug a command sequence gap on the target end (in the absence of an acknowledgement of the command by way of ExpCmdSN) before the ULP timeout by retrying the unacknowledged command, as described in section 8.1. On a ULP timeout for a command that carried a CmdSN of n, if the ExpCmdSN is still less than (n+1) on ULP timeout, the iSCSI initiator MUST abort the command using the Abort Task task management function request. In this process, the target may see the abort request while missing the original command itself due to one of the following reasons: - the original command was dropped due to digest error, or - the connection the original command sent on was successfully logged out (on logout unacknowledged commands issued on the connection being logged out are discarded) If the abort request is received and the original command is missing, targets MUST consider the original command with that RefCmdSN to be received and issue a task management response with the response code "Task specified in the Referenced Task Tag field was not in task set" and any state referring to the aborted task (if any) at the initiator can be discarded. If the original command exists, as with any abort the initiator expects a concluding status (that will not be delivered to SCSI) and the target MUST supply a status at abort time if it was not delivered earlier. The task management response is issued after the status. Julo "Mallikarjun C." To: ips Subject: iSCSI: SCSI timeout handling change Sent by: owner-ips@ece. cmu.edu 13-11-01 21:21 Please respond to cbm All: Currently, if a command is not acknowledged by the ULP timeout, iSCSI mandates the initiators to tear up the session. The rationale behind this is that if the initiator could not get the command through (in possibly multiple retries) even by the ULP timeout, there's a serious problem with the session. But there are some drawbacks to this approach - - tearing up a session due to a NIC failure is disruptive to potentially several other active tasks on other NICs. - this puts those initiator implementations not wanting to do within-connection recovery (i.e. no retries) at a disadvantage, since one digest error would cause potentially several active I/Os to be terminated. - (albeit not very serious, ) this behavior is different from today's storage stacks' expectations - of being able to selectively abort one I/O on a timeout (with no command retransmissions). To address these issues, and also to simplify the current Task Management request PDU, I propose the following changes to handling SCSI timeouts - Following changes to section 3.5: - Abort Task MUST always be sent immediate. - Abort Task task management function request MUST be sent with its CmdSN equal to the CmdSN of the task to be aborted, and the Referenced Task Tag initialized to the ITT of the task to be aborted. - Consequent to the above, drop the RefCmdSN field in the Task Management command payload that is currently only used by the Abort Task function. Following changes to section 8.6: Propose the following text to replace the current - An iSCSI initiator MAY attempt to plug a command sequence gap on the target end (in the absence of an acknowledgement of the command by way of ExpCmdSN) before the ULP timeout by retrying the unacknowledged command, as described in section 8.1. On a ULP timeout for a command that carried a CmdSN of n, if the ExpCmdSN is still less than (n+1) on ULP timeout, the iSCSI initiator MUST abort the command using the Abort Task task management function request. In this process, the target may see the abort request before the original command itself due to one of the three reasons - - the original command was dropped due to digest error, or - the Abort Task request was shipped out-of-order on the same connection, or - the connection the original command sent on was successfully logged out. If the abort request is received prior to the original command, targets MUST consider the original command with that CmdSN to be received and discard the original command if and when received - i.e. treating it as a duplicate CmdSN. Initiators desirous of maintaining command ordering while maintaining the same session MUST NOT issue Abort Task on an unacknowledged command because of this reason. Following changes to section 2.2.2.1: - The above approach exposes the possibility that some stale (aborted from target's perspective) commands could be stuck in the TCP connection long enough for the CmdSN wrap - similar to the issue we dealt with for command retries. So, aborting unacknowledged commands should require the same flushing actions described for command retries. [ I almost would prefer at this point to require flushing all connections every 2^31 -1 commands starting from InitCmdSN, than enumerating these cases individually...] Comments? -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Thu Nov 15 13:30:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA10238 for ; Thu, 15 Nov 2001 13:30:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAFHWvU25115 for ips-outgoing; Thu, 15 Nov 2001 12:32:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAFHWql25099 for ; Thu, 15 Nov 2001 12:32:52 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA09432; Thu, 15 Nov 2001 18:32:41 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAFHWiI37576; Thu, 15 Nov 2001 18:32:45 +0100 Importance: Normal Subject: RE: iSCSI: security questions To: "Lee Xing" Cc: X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Ofer Biran" Date: Thu, 15 Nov 2001 19:33:37 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 15/11/2001 19:32:46 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Lee, + Let's consider a Login Phase Example: + + I-> Login (CSG,NSG=0,1 T=1) + ... + AuthMethod=KRB5,SRP,none + + T-> Login-PR (CSG,NSG=0,1 T=1) + ... + AuthMethod=none + + does "CSG=0" mean that the initiator "requires + authentication"? If it does, is "none" in Login + AuthMethod list a legal value to have? If it is, + is "none" in Login-PR AuthMethod list a legal value + to have even though the target "requires authentication"? + If it is, should the connection closes, or should the + initiator continue with next Login Stage? If it + should continue with next Login Stage, then should + we reword the paragraph in SEC-IPS v.04? "CSG=0" means that the initiator starts the login phase in the SecurityNegotiation stage. "AuthMethod=KRB5,SRP,none" means that it doesn't require authentication - since he offers also the "none" option. And indeed it also sets "NSG=1", s.t. if the target chooses "none" (and agrees to the stage transition by "NSG=1 T=1") - the stage transition can occur immediately on the next initiator Login command. Regards, Ofer Ofer Biran Storage and Systems Technology IBM Research Lab in Haifa biran@il.ibm.com 972-4-8296253 From owner-ips@ece.cmu.edu Thu Nov 15 15:17:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA16974 for ; Thu, 15 Nov 2001 15:17:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAFIxb902120 for ips-outgoing; Thu, 15 Nov 2001 13:59:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lysimachus.hosting.pacbell.net (lysimachus.hosting.pacbell.net [216.100.98.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAFIxZl02115 for ; Thu, 15 Nov 2001 13:59:35 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by lysimachus.hosting.pacbell.net id NAA26729; Thu, 15 Nov 2001 13:58:03 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: Subject: RE: iSCSI: update on OOO CmdSNs/connection Date: Thu, 15 Nov 2001 10:54:53 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, As was mentioned previously, there is a signficant different in how you design for error recovery corner cases, and how you design for the main path. The SHOULD is neither. I would prefer one of the following 1. Make it a MUST or not mention it at all. or 2. Make it a negotiable parameter. The way it is, it does not provide any benefit to the initiator (it is a MUST in a very important case), and for the target it becomes a nebulous design choice. Regards, Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Wednesday, November 14, 2001 5:46 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > Somesh, > > The reason we have put in the SHOULD is that it better reflects what is > happening on recovery anyhow. > The text also clearly outlines when the SHOULD becomes MUST and that is > the case you where concerned about. > > Regards, > Julo > > > > > > "Somesh Gupta" > Sent by: owner-ips@ece.cmu.edu > 13-11-01 22:53 > Please respond to somesh_gupta > > > To: "ips" > cc: > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > > Julian, > > I am surprised to see the text in the latest rev > of the doc change as suggested by Mallikarjun. > I did not think there was a consensus on this > subject. > > Just because I did not respond to Mallikarjun's > last comment publicly should not be construed > as agreement. Having the last word is hopefully > not assumed to be consensus, otherwise a thread > may never end. > > Somesh > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Somesh Gupta > > Sent: Friday, November 09, 2001 3:26 PM > > To: ips > > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > > > > Mallikarjun, > > > > I never liked the SHOULD. It is not a design point. > > If we really want to allow it, perhaps a negotiation > > parameter is a better choice (which is only > > marginally better). Error detection and recovery > > have completely different design requirements than > > the data path. > > > > So ideally we say MUST or nothing > > or we negotiate it > > > > Also on your point A, it "MUST" be a MUST on > > a single connection case except for when required > > for error recovery (i.e. the very very rare - > > case of digest error). > > > > Somesh > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Mallikarjun C. > > > Sent: Friday, November 09, 2001 1:49 PM > > > To: ips > > > Subject: iSCSI: update on OOO CmdSNs/connection > > > > > > > > > To those interested in this discussion: > > > > > > Julian and I had a phone conversation on the topic > > > of OOO CmdSNs on a connection. An update follows. > > > > > > Julian agrees that there are valid error recovery > > > scenarios where CmdSNs will legitimately end up OOO > > > on a given connection. > > > > > > OTOH, I agree with two of Julian's scenarios that he > > > pointed out right away - the "cleaning command" (command > > > required to be sent after a retry copy to ensure flushing > > > within 2^31 -1), and an immediate Logout posted with > > > unacknowledged commands. Neither of this can be shipped > > > OOO - since the former undoes the flushing intent, and > > > the latter breaks the rule that nothing more follows a > > > Logout on the connection (and troublesome in other ways, > > > see below). > > > > > > In general, I share the concern with Julian that we > > > have not closely scrutinized all possibilities. > > > > > > With that said, something along the following lines > > > seemed reasonable - > > > > > > A)Initiator MUST send commands in increasing order of > > > CmdSN on a connection if both the following are true - > > > - operational ErrorRecoveryLevel is 0, > > > - MaxConnections is negotiated to 1. > > > B)In all the other cases, initiator SHOULD send commands > > > in increasing order of CmdSN on a connection. It is > > > strongly encouraged that commands with out-of-order > > > CmdSNs be sent on a connection only if they are > > > retransmitted commands due to digest error recovery > > > and connection recovery. > > > > > > I also suggest the following upon further reflection- > > > > > > C)Add wording in section 2.2.2.1 to mandate that > > > the cleaning command MUST be sent in-order after > > > the retried command. > > > D)Warn clearly that sending an immediate Logout command > > > in the presence of other unacknowledged commands MAY > > > create inadvertent discarding of certain commands (even > > > if it is a recovery Logout), and MAY cause protocol > > > errors leading to ungraceful shutdown of the connection. > > > > > > Hopefully A will bring the determinism that Somesh was > > > looking for certain design points. B describes the more > > > general n-connection session case. C & D are fixes for > > > two identfied areas (so far) which will break. > > > > > > Comments? > > > -- > > > Mallikarjun > > > > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 15 15:23:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA17399 for ; Thu, 15 Nov 2001 15:23:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAFJEoK03375 for ips-outgoing; Thu, 15 Nov 2001 14:14:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAFJEnl03369 for ; Thu, 15 Nov 2001 14:14:49 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id UAA259414 for ; Thu, 15 Nov 2001 20:14:42 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAFJEkI32810 for ; Thu, 15 Nov 2001 20:14:47 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI: update on OOO CmdSNs/connection X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 15 Nov 2001 21:14:44 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 15/11/2001 21:14:48, Serialize complete at 15/11/2001 21:14:48 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, Read further - yes it is back a MUST (it does not make sense to make it a blanket SHOULD). Julo "Somesh Gupta" Sent by: owner-ips@ece.cmu.edu 15-11-01 20:54 Please respond to somesh_gupta To: cc: Subject: RE: iSCSI: update on OOO CmdSNs/connection Julian, As was mentioned previously, there is a signficant different in how you design for error recovery corner cases, and how you design for the main path. The SHOULD is neither. I would prefer one of the following 1. Make it a MUST or not mention it at all. or 2. Make it a negotiable parameter. The way it is, it does not provide any benefit to the initiator (it is a MUST in a very important case), and for the target it becomes a nebulous design choice. Regards, Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Wednesday, November 14, 2001 5:46 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > Somesh, > > The reason we have put in the SHOULD is that it better reflects what is > happening on recovery anyhow. > The text also clearly outlines when the SHOULD becomes MUST and that is > the case you where concerned about. > > Regards, > Julo > > > > > > "Somesh Gupta" > Sent by: owner-ips@ece.cmu.edu > 13-11-01 22:53 > Please respond to somesh_gupta > > > To: "ips" > cc: > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > > Julian, > > I am surprised to see the text in the latest rev > of the doc change as suggested by Mallikarjun. > I did not think there was a consensus on this > subject. > > Just because I did not respond to Mallikarjun's > last comment publicly should not be construed > as agreement. Having the last word is hopefully > not assumed to be consensus, otherwise a thread > may never end. > > Somesh > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Somesh Gupta > > Sent: Friday, November 09, 2001 3:26 PM > > To: ips > > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > > > > Mallikarjun, > > > > I never liked the SHOULD. It is not a design point. > > If we really want to allow it, perhaps a negotiation > > parameter is a better choice (which is only > > marginally better). Error detection and recovery > > have completely different design requirements than > > the data path. > > > > So ideally we say MUST or nothing > > or we negotiate it > > > > Also on your point A, it "MUST" be a MUST on > > a single connection case except for when required > > for error recovery (i.e. the very very rare - > > case of digest error). > > > > Somesh > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Mallikarjun C. > > > Sent: Friday, November 09, 2001 1:49 PM > > > To: ips > > > Subject: iSCSI: update on OOO CmdSNs/connection > > > > > > > > > To those interested in this discussion: > > > > > > Julian and I had a phone conversation on the topic > > > of OOO CmdSNs on a connection. An update follows. > > > > > > Julian agrees that there are valid error recovery > > > scenarios where CmdSNs will legitimately end up OOO > > > on a given connection. > > > > > > OTOH, I agree with two of Julian's scenarios that he > > > pointed out right away - the "cleaning command" (command > > > required to be sent after a retry copy to ensure flushing > > > within 2^31 -1), and an immediate Logout posted with > > > unacknowledged commands. Neither of this can be shipped > > > OOO - since the former undoes the flushing intent, and > > > the latter breaks the rule that nothing more follows a > > > Logout on the connection (and troublesome in other ways, > > > see below). > > > > > > In general, I share the concern with Julian that we > > > have not closely scrutinized all possibilities. > > > > > > With that said, something along the following lines > > > seemed reasonable - > > > > > > A)Initiator MUST send commands in increasing order of > > > CmdSN on a connection if both the following are true - > > > - operational ErrorRecoveryLevel is 0, > > > - MaxConnections is negotiated to 1. > > > B)In all the other cases, initiator SHOULD send commands > > > in increasing order of CmdSN on a connection. It is > > > strongly encouraged that commands with out-of-order > > > CmdSNs be sent on a connection only if they are > > > retransmitted commands due to digest error recovery > > > and connection recovery. > > > > > > I also suggest the following upon further reflection- > > > > > > C)Add wording in section 2.2.2.1 to mandate that > > > the cleaning command MUST be sent in-order after > > > the retried command. > > > D)Warn clearly that sending an immediate Logout command > > > in the presence of other unacknowledged commands MAY > > > create inadvertent discarding of certain commands (even > > > if it is a recovery Logout), and MAY cause protocol > > > errors leading to ungraceful shutdown of the connection. > > > > > > Hopefully A will bring the determinism that Somesh was > > > looking for certain design points. B describes the more > > > general n-connection session case. C & D are fixes for > > > two identfied areas (so far) which will break. > > > > > > Comments? > > > -- > > > Mallikarjun > > > > > > > > > Mallikarjun Chadalapaka > > > Networked Storage Architecture > > > Network Storage Solutions Organization > > > MS 5668 Hewlett-Packard, Roseville. > > > cbm@rose.hp.com > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 15 18:30:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA27517 for ; Thu, 15 Nov 2001 18:30:54 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAFMNIs18593 for ips-outgoing; Thu, 15 Nov 2001 17:23:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from erie.mcdata.com (mcgate.mcdata.com [144.49.1.5]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAFKerl10627 for ; Thu, 15 Nov 2001 15:40:57 -0500 (EST) Received: by erie.mcdata.com with Internet Mail Service (5.5.2653.19) id ; Thu, 15 Nov 2001 13:40:43 -0700 Message-ID: <1B8A58D6C376FE4DB329408E27013842447E49@grizzly.mcdata.com> From: Robert Grant To: "'ips@ece.cmu.edu'" Subject: iSCSI: Representing iSCSI devices on FC fabrics Date: Thu, 15 Nov 2001 13:40:40 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello all, I have a question on the representation of iSCSI devices into Fibre Channel fabrics for an iSCSI-to-FC "gateway" device and would like to solicit people's thoughts on how best to do this. A gateway device will allow iSCSI devices and FCP devices to access each other, but in order to do this a consistent representation of the devices is needed. I haven't been able to reconcile the iSCSI and FCP standards using what's currently in the iSCSI standard, and wanted to see if there was any support to expanding the iSCSI standard to address this (a standard solution is, of course, much more preferred to every gateway vendor doing it in their own proprietary way). In particular, how would an iSCSI device map onto Fibre Channel's World Wide Name (WWN)? Would every device have its own WWN, or could many iSCSI devices use a single WWN? There have been some discussions (for example, there was even discussion of including a WWN field in the iSCSI Login for a Gateway to proxy with in http://www.pdl.cmu.edu/mailinglists/ips/mail/msg01616.html), but what is the current view? A first approach might be that many iSCSI devices could use a single WWN. This can work well for FC-AL devices "directly attached " to the IP network or for small FC fabrics - and where the predominant interconnect and management of that interconnect is the IP network. This approach views the FC fabric as flat (or at least perhaps that FC zoning is "turned off"). As the FC fabric gets bigger, though, this first approach can create two layers of management - one must first configure the FC network and then configure the IP network (since the individual iSCSI devices sharing a single WWN can only be zoned as a group). The two layers are first "this group of iSCSI devices can access this zone" on the FC side and then "this iSCSI device can access this FC device in this zone" on the iSCSI side. If there was a clean integration with FC zoning (and associated management of the FC zoning), this may be avoided. A further complication is that, as the FC fabric gets even bigger, a single iSCSI device could end up with multiple entry points (i.e. paths through multiple gateways) into a single FC fabric. Is there any common way to represent iSCSI devices (for instance, with respect to WWNs) that allows the unique identification of that iSCSI device - even though there are multiple entrypoints onto the FC fabric? The case of multiple gateways (possibly from different vendors) is the clearest example of the need for a standard. Thank you for your time and I look forward to all comments/suggestions. Regards, Rob Rob Grant McDATA Corporation From owner-ips@ece.cmu.edu Thu Nov 15 20:33:11 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01160 for ; Thu, 15 Nov 2001 20:33:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAG0a2m27949 for ips-outgoing; Thu, 15 Nov 2001 19:36:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail1.aarohicommunications.com (cpe-66-87-94-158.ca.sprintbbd.net [66.87.94.158]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAFNh8l24230 for ; Thu, 15 Nov 2001 18:43:08 -0500 (EST) Received: from LINUX15 (cpe-66-87-94-158.ca.sprintbbd.net [66.87.94.158]) by mail1.aarohicommunications.com (Mirapoint) with ESMTP id AAJ00864 (AUTH shaileshm); Thu, 15 Nov 2001 15:43:48 -0800 (PST) From: "Shailesh Manjrekar" To: "'John Hufferd'" , "'Julian Satran'" Cc: Subject: RE: iSCSI minimum PDU length Date: Thu, 15 Nov 2001 15:44:08 -0800 Message-ID: <000b01c16e2f$6e8e43a0$ea01a8c0@aarohicommunications.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2627 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Have we reached a consensus whether the minimum PDU size would be 512 or 1024. Also for command PDU's ( 48bytes + header ) does it mean that we would end up using a MTU size ( 1500 bytes ) TCP segment for this. This would add segment processing overheads at offload engine for just 48bytes + header. Please clarify, Regards, Shailesh. Aarohi Communications. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] On Behalf Of John Hufferd Sent: Thursday, October 25, 2001 2:36 AM To: Julian Satran Cc: ips@ece.cmu.edu Subject: Re: iSCSI minimum PDU length If you are only talking about Data PDUs, then at least I understand. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com ---------------------- Forwarded by John Hufferd/San Jose/IBM on 10/25/2001 02:34 AM --------------------------- John Hufferd/San Jose/IBM@IBMUS@ece.cmu.edu on 10/25/2001 02:12:07 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI minimum PDU length Julian, Perhaps I do not understand what you are getting at here, but since there will be implementations with only R2T type Sessions, then there should be a lot of PDUs that just carry the Command (48 bytes + digest) it seems that a 1024 is kind of big. What am I missing? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/24/2001 10:34:38 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: iSCSI minimum PDU length With no votes against we have settled (again) on single PDU length (per connection, per direction) for all types of PDUs. But I think we erred on the low side by suggesting 64 as a minimum. It is low (it was mentioned) and bad for text request/response. How about settling for 1024? Julo From owner-ips@ece.cmu.edu Fri Nov 16 00:19:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA06504 for ; Fri, 16 Nov 2001 00:19:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAG4cuk13220 for ips-outgoing; Thu, 15 Nov 2001 23:38:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lysimachus.hosting.pacbell.net (lysimachus.hosting.pacbell.net [216.100.98.17]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAG4cql13210 for ; Thu, 15 Nov 2001 23:38:52 -0500 (EST) Received: from somesh (sdsl-64-139-0-182.dsl.sca.megapath.net [64.139.0.182]) by lysimachus.hosting.pacbell.net id XAA19183; Thu, 15 Nov 2001 23:38:44 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Reply-To: From: "Somesh Gupta" To: Subject: RE: iSCSI: update on OOO CmdSNs/connection Date: Thu, 15 Nov 2001 20:35:35 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit You change it faster than I can read :-) Instead of Read further you should have said, Read the latest. Somesh > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Julian Satran > Sent: Thursday, November 15, 2001 11:15 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > Somesh, > > Read further - yes it is back a MUST (it does not make sense to make it a > blanket SHOULD). > > Julo > > > > > "Somesh Gupta" > Sent by: owner-ips@ece.cmu.edu > 15-11-01 20:54 > Please respond to somesh_gupta > > > To: > cc: > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > > Julian, > > As was mentioned previously, there is a signficant > different in how you design for error recovery > corner cases, and how you design for the main path. > > The SHOULD is neither. I would prefer one of the following > > 1. Make it a MUST or not mention it at all. > or > 2. Make it a negotiable parameter. > > The way it is, it does not provide any benefit to > the initiator (it is a MUST in a very important case), > and for the target it becomes a nebulous design choice. > > Regards, > Somesh > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Julian Satran > > Sent: Wednesday, November 14, 2001 5:46 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > > > > Somesh, > > > > The reason we have put in the SHOULD is that it better reflects what is > > happening on recovery anyhow. > > The text also clearly outlines when the SHOULD becomes MUST and that is > > the case you where concerned about. > > > > Regards, > > Julo > > > > > > > > > > > > "Somesh Gupta" > > Sent by: owner-ips@ece.cmu.edu > > 13-11-01 22:53 > > Please respond to somesh_gupta > > > > > > To: "ips" > > cc: > > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > > > > > > Julian, > > > > I am surprised to see the text in the latest rev > > of the doc change as suggested by Mallikarjun. > > I did not think there was a consensus on this > > subject. > > > > Just because I did not respond to Mallikarjun's > > last comment publicly should not be construed > > as agreement. Having the last word is hopefully > > not assumed to be consensus, otherwise a thread > > may never end. > > > > Somesh > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Somesh Gupta > > > Sent: Friday, November 09, 2001 3:26 PM > > > To: ips > > > Subject: RE: iSCSI: update on OOO CmdSNs/connection > > > > > > > > > Mallikarjun, > > > > > > I never liked the SHOULD. It is not a design point. > > > If we really want to allow it, perhaps a negotiation > > > parameter is a better choice (which is only > > > marginally better). Error detection and recovery > > > have completely different design requirements than > > > the data path. > > > > > > So ideally we say MUST or nothing > > > or we negotiate it > > > > > > Also on your point A, it "MUST" be a MUST on > > > a single connection case except for when required > > > for error recovery (i.e. the very very rare - > > > case of digest error). > > > > > > Somesh > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > Mallikarjun C. > > > > Sent: Friday, November 09, 2001 1:49 PM > > > > To: ips > > > > Subject: iSCSI: update on OOO CmdSNs/connection > > > > > > > > > > > > To those interested in this discussion: > > > > > > > > Julian and I had a phone conversation on the topic > > > > of OOO CmdSNs on a connection. An update follows. > > > > > > > > Julian agrees that there are valid error recovery > > > > scenarios where CmdSNs will legitimately end up OOO > > > > on a given connection. > > > > > > > > OTOH, I agree with two of Julian's scenarios that he > > > > pointed out right away - the "cleaning command" (command > > > > required to be sent after a retry copy to ensure flushing > > > > within 2^31 -1), and an immediate Logout posted with > > > > unacknowledged commands. Neither of this can be shipped > > > > OOO - since the former undoes the flushing intent, and > > > > the latter breaks the rule that nothing more follows a > > > > Logout on the connection (and troublesome in other ways, > > > > see below). > > > > > > > > In general, I share the concern with Julian that we > > > > have not closely scrutinized all possibilities. > > > > > > > > With that said, something along the following lines > > > > seemed reasonable - > > > > > > > > A)Initiator MUST send commands in increasing order of > > > > CmdSN on a connection if both the following are true - > > > > - operational ErrorRecoveryLevel is 0, > > > > - MaxConnections is negotiated to 1. > > > > B)In all the other cases, initiator SHOULD send commands > > > > in increasing order of CmdSN on a connection. It is > > > > strongly encouraged that commands with out-of-order > > > > CmdSNs be sent on a connection only if they are > > > > retransmitted commands due to digest error recovery > > > > and connection recovery. > > > > > > > > I also suggest the following upon further reflection- > > > > > > > > C)Add wording in section 2.2.2.1 to mandate that > > > > the cleaning command MUST be sent in-order after > > > > the retried command. > > > > D)Warn clearly that sending an immediate Logout command > > > > in the presence of other unacknowledged commands MAY > > > > create inadvertent discarding of certain commands (even > > > > if it is a recovery Logout), and MAY cause protocol > > > > errors leading to ungraceful shutdown of the connection. > > > > > > > > Hopefully A will bring the determinism that Somesh was > > > > looking for certain design points. B describes the more > > > > general n-connection session case. C & D are fixes for > > > > two identfied areas (so far) which will break. > > > > > > > > Comments? > > > > -- > > > > Mallikarjun > > > > > > > > > > > > Mallikarjun Chadalapaka > > > > Networked Storage Architecture > > > > Network Storage Solutions Organization > > > > MS 5668 Hewlett-Packard, Roseville. > > > > cbm@rose.hp.com > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Fri Nov 16 03:12:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA24020 for ; Fri, 16 Nov 2001 03:11:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAG78rg21550 for ips-outgoing; Fri, 16 Nov 2001 02:08:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailnpd.hcltech.com ([203.197.145.76]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAG78jl21539 for ; Fri, 16 Nov 2001 02:08:46 -0500 (EST) Received: from npd.hcltech.com (tskariah-pc.hcltech.com [192.168.100.72]) by mailnpd.hcltech.com (8.11.0/8.11.0) with ESMTP id fAG75ae20865; Fri, 16 Nov 2001 12:35:41 +0530 Message-ID: <3BF4BC35.365326B4@npd.hcltech.com> Date: Fri, 16 Nov 2001 12:41:49 +0530 From: Thanu Skariah X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.2-2 i686) X-Accept-Language: en MIME-Version: 1.0 To: Robert Grant CC: "'ips@ece.cmu.edu'" Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics References: <1B8A58D6C376FE4DB329408E27013842447E49@grizzly.mcdata.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Robert, iSCSI allows different naming formats, of which one format is the EUI format (See the example in sec 2.2 .7 and the naming draft - http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name-disc-02.txt ) The EUI representation is of the form eui . . Each FC device's WWName can be used to form the corresponding iSCSI name for the device. This is what we are doing on a linux based software FCP/iSCSI gateway that we are implementing, and this is why : (From the naming and discovery draft ): BeginQuote " Type "eui." (IEEE EUI format) The IEEE iSCSI name might be used when a manufacturer is already basing unique identifiers on World-Wide Names as defined in the SCSI SPC-2 specification. It may also be used by a gateway representing a Fibre Channel or SCSI device that is already adequately identified using a world-wide name. " End Quote Thanks, Thanu Robert Grant wrote: > Hello all, > > I have a question on the representation of iSCSI > devices into Fibre Channel fabrics for an iSCSI-to-FC "gateway" device and > would like to solicit people's thoughts on how best to do this. A gateway > device will allow iSCSI devices and FCP devices to access each other, but in > order to do this a consistent representation of the devices is needed. I > haven't been able to reconcile the iSCSI and FCP standards using what's > currently in the iSCSI standard, and wanted to see if there was any support > to expanding the iSCSI standard to address this (a standard solution is, of > course, much more preferred to every gateway vendor doing it in their own > proprietary way). In particular, how would an iSCSI device map onto Fibre > Channel's World Wide Name (WWN)? Would every device have its own WWN, or > could many iSCSI devices use a single WWN? There have been some discussions > (for example, there was even discussion of including a WWN field in the > iSCSI Login for a Gateway to proxy with in > http://www.pdl.cmu.edu/mailinglists/ips/mail/msg01616.html), but what is the > current view? > > A first approach might be that many iSCSI devices > could use a single WWN. This can work well for FC-AL devices "directly > attached " to the IP network or for small FC fabrics - and where the > predominant interconnect and management of that interconnect is the IP > network. > > This approach views the FC fabric as flat (or at > least perhaps that FC zoning is "turned off"). As the FC fabric gets bigger, > though, this first approach can create two layers of management - one must > first configure the FC network and then configure the IP network (since the > individual iSCSI devices sharing a single WWN can only be zoned as a group). > The two layers are first "this group of iSCSI devices can access this zone" > on the FC side and then "this iSCSI device can access this FC device in this > zone" on the iSCSI side. If there was a clean integration with FC zoning > (and associated management of the FC zoning), this may be avoided. > > A further complication is that, as the FC fabric > gets even bigger, a single iSCSI device could end up with multiple entry > points (i.e. paths through multiple gateways) into a single FC fabric. Is > there any common way to represent iSCSI devices (for instance, with respect > to WWNs) that allows the unique identification of that iSCSI device - even > though there are multiple entrypoints onto the FC fabric? The case of > multiple gateways (possibly from different vendors) is the clearest example > of the need for a standard. > > Thank you for your time and I look forward to all > comments/suggestions. > > Regards, > Rob > > Rob Grant > McDATA Corporation From owner-ips@ece.cmu.edu Fri Nov 16 04:47:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA25200 for ; Fri, 16 Nov 2001 04:47:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAG85pt24653 for ips-outgoing; Fri, 16 Nov 2001 03:05:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAG85nl24646 for ; Fri, 16 Nov 2001 03:05:49 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA13112 for ; Fri, 16 Nov 2001 09:05:42 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAG85kC84958 for ; Fri, 16 Nov 2001 09:05:46 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: RE: iSCSI minimum PDU length X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 16 Nov 2001 10:05:43 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 16/11/2001 10:05:48, Serialize complete at 16/11/2001 10:05:48 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Shailesh, With a high level of enthusiasm the consensus was on 512 (2 voices for 512 against 1 voice for 1024). I am not sure about what you are asking in the second part of the message. The 512 refers to the data-segment of an iSCSI PDU and states the minimum that MUST be supported. If you are asking about TCP it says nothing about TCP (TCP is a stream of bytes). In practical terms you would like to have TCP support packets that are at least 560 (or 564 if CRCs are enabled - 512+48). Julo "Shailesh Manjrekar" 16-11-01 01:44 Please respond to "Shailesh Manjrekar" To: John Hufferd/San Jose/IBM@IBMUS, Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: iSCSI minimum PDU length Julian, Have we reached a consensus whether the minimum PDU size would be 512 or 1024. Also for command PDU's ( 48bytes + header ) does it mean that we would end up using a MTU size ( 1500 bytes ) TCP segment for this. This would add segment processing overheads at offload engine for just 48bytes + header. Please clarify, Regards, Shailesh. Aarohi Communications. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] On Behalf Of John Hufferd Sent: Thursday, October 25, 2001 2:36 AM To: Julian Satran Cc: ips@ece.cmu.edu Subject: Re: iSCSI minimum PDU length If you are only talking about Data PDUs, then at least I understand. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com ---------------------- Forwarded by John Hufferd/San Jose/IBM on 10/25/2001 02:34 AM --------------------------- John Hufferd/San Jose/IBM@IBMUS@ece.cmu.edu on 10/25/2001 02:12:07 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI minimum PDU length Julian, Perhaps I do not understand what you are getting at here, but since there will be implementations with only R2T type Sessions, then there should be a lot of PDUs that just carry the Command (48 bytes + digest) it seems that a 1024 is kind of big. What am I missing? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 10/24/2001 10:34:38 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: iSCSI minimum PDU length With no votes against we have settled (again) on single PDU length (per connection, per direction) for all types of PDUs. But I think we erred on the low side by suggesting 64 as a minimum. It is low (it was mentioned) and bad for text request/response. How about settling for 1024? Julo From owner-ips@ece.cmu.edu Fri Nov 16 11:50:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA10006 for ; Fri, 16 Nov 2001 11:50:21 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGG24704662 for ips-outgoing; Fri, 16 Nov 2001 11:02:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cisco.com (cypher.cisco.com [171.69.11.18]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGG22l04654 for ; Fri, 16 Nov 2001 11:02:02 -0500 (EST) Received: (from kzm@localhost) by cisco.com (8.8.8/2.6/Cisco List Logging/8.8.8) id IAA20993; Fri, 16 Nov 2001 08:01:49 -0800 (PST) From: Keith McCloghrie Message-Id: <200111161601.IAA20993@cisco.com> Subject: Re: FC Management MIB - proposed changes To: Black_David@emc.com Date: Fri, 16 Nov 2001 08:01:49 -0800 (PST) Cc: kzm@cisco.com, ips@ece.cmu.edu In-Reply-To: <277DD60FB639D511AC0400B0D068B71E02937160@CORPMX14> from "Black_David@emc.com" at Nov 12, 2001 01:07:44 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit > - The thing that's going away is the trap registration table. > The existing traps become notifications. This is what > I was expecting. They will if they are still relevant. At least some of them will no longer be relevant (e.g., connUnitSensorStatusChange, because the Sensor objects will be removed). Keith. From owner-ips@ece.cmu.edu Fri Nov 16 11:50:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA10044 for ; Fri, 16 Nov 2001 11:50:30 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGFXaN02508 for ips-outgoing; Fri, 16 Nov 2001 10:33:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ztxmail03.ztx.compaq.com (ztxmail03.ztx.compaq.com [161.114.1.207]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGFXYl02498 for ; Fri, 16 Nov 2001 10:33:34 -0500 (EST) Received: by ztxmail03.ztx.compaq.com (Postfix, from userid 12345) id 3E9A035AA; Fri, 16 Nov 2001 09:33:28 -0600 (CST) Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail03.ztx.compaq.com (Postfix) with ESMTP id 391FA34CD for ; Fri, 16 Nov 2001 09:33:28 -0600 (CST) Received: from cceexc17.americas.cpqcorp.net ([16.110.250.84]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Fri, 16 Nov 2001 09:33:28 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics Date: Fri, 16 Nov 2001 09:33:27 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C6019E20CB@cceexc17.americas.cpqcorp.net> Thread-Topic: iSCSI: Representing iSCSI devices on FC fabrics Thread-Index: AcFubjRZK3uPCtasQFeyQaMzz6nVRgAQ+tTw From: "Elliott, Robert" To: X-OriginalArrivalTime: 16 Nov 2001 15:33:28.0043 (UTC) FILETIME=[09C923B0:01C16EB4] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fAGFXYl02499 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Beware than a FC WWN is not the same as an EUI-64. They're both based on the IEEE OUI/company_id, but the FC WWN uses the 4 most significant bits for a "Name Address Authority" field and has 4 fewer bits available in the vendor specified portion. EUI-64 = (MSB) { 20-bit company_id, 44-bit vendor specified } (LSB) FC WWN = (MSB) { 4-bit NAA field, 20-bit company_id, 40-bit vendor specified } (LSB) (for the IEEE Registered NAA type) --- Rob Elliott, Compaq Server Storage Robert.Elliott@compaq.com > -----Original Message----- > From: Thanu Skariah [mailto:tskariah@npd.hcltech.com] > Sent: Friday, November 16, 2001 1:12 AM > To: Robert Grant > Cc: 'ips@ece.cmu.edu' > Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics > > > Robert, > > > iSCSI allows different naming formats, of which one > format is the EUI format (See the example in sec 2.2 .7 and > the naming draft - > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name- > disc-02.txt ) > > The EUI representation is of the form eui . . Each > FC device's WWName can be used to form the corresponding iSCSI > name for the device. This is what we are doing on a linux > based software FCP/iSCSI gateway that we are implementing, > and this is why : > > (From the naming and discovery draft ): > > BeginQuote " > > Type "eui." (IEEE EUI format) > > The IEEE iSCSI name might be used when a manufacturer is already > basing unique identifiers on World-Wide Names as defined > in the SCSI SPC-2 specification. > > It may also be used by a gateway representing a Fibre Channel or > SCSI device that is already adequately identified using a > world-wide name. > > " End Quote > > > Thanks, > Thanu From owner-ips@ece.cmu.edu Fri Nov 16 12:32:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA12784 for ; Fri, 16 Nov 2001 12:32:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGGW4507000 for ips-outgoing; Fri, 16 Nov 2001 11:32:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGGW3l06996 for ; Fri, 16 Nov 2001 11:32:03 -0500 (EST) Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.99.140.22]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA37304; Fri, 16 Nov 2001 11:29:22 -0500 Received: from d03nm041.boulder.ibm.com (d03nm041.boulder.ibm.com [9.99.140.41]) by westrelay01.boulder.ibm.com (8.11.1m3/NCO v5.00) with ESMTP id fAGGW0331604; Fri, 16 Nov 2001 09:32:01 -0700 Importance: Normal Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics To: "Elliott, Robert" Cc: From: "Jim Hafner" Date: Fri, 16 Nov 2001 07:51:51 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM041/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/16/2001 08:32:00 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Rob, You make a good point. But I think it should also be mentioned that going the route indicated in the first reply may NOT guarantee WWUniqueness (what if two vendors do the same thing!). Jim Hafner "Elliott, Robert" @ece.cmu.edu on 11/16/2001 07:33:27 am Sent by: owner-ips@ece.cmu.edu To: cc: Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics Beware than a FC WWN is not the same as an EUI-64. They're both based on the IEEE OUI/company_id, but the FC WWN uses the 4 most significant bits for a "Name Address Authority" field and has 4 fewer bits available in the vendor specified portion. EUI-64 = (MSB) { 20-bit company_id, 44-bit vendor specified } (LSB) FC WWN = (MSB) { 4-bit NAA field, 20-bit company_id, 40-bit vendor specified } (LSB) (for the IEEE Registered NAA type) --- Rob Elliott, Compaq Server Storage Robert.Elliott@compaq.com > -----Original Message----- > From: Thanu Skariah [mailto:tskariah@npd.hcltech.com] > Sent: Friday, November 16, 2001 1:12 AM > To: Robert Grant > Cc: 'ips@ece.cmu.edu' > Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics > > > Robert, > > > iSCSI allows different naming formats, of which one > format is the EUI format (See the example in sec 2.2 .7 and > the naming draft - > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name- > disc-02.txt ) > > The EUI representation is of the form eui . . Each > FC device's WWName can be used to form the corresponding iSCSI > name for the device. This is what we are doing on a linux > based software FCP/iSCSI gateway that we are implementing, > and this is why : > > (From the naming and discovery draft ): > > BeginQuote " > > Type "eui." (IEEE EUI format) > > The IEEE iSCSI name might be used when a manufacturer is already > basing unique identifiers on World-Wide Names as defined > in the SCSI SPC-2 specification. > > It may also be used by a gateway representing a Fibre Channel or > SCSI device that is already adequately identified using a > world-wide name. > > " End Quote > > > Thanks, > Thanu From owner-ips@ece.cmu.edu Fri Nov 16 12:55:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14138 for ; Fri, 16 Nov 2001 12:55:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGGTvH06801 for ips-outgoing; Fri, 16 Nov 2001 11:29:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.brocade.com (asbestos.brocade.com [63.121.140.244]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGGTtl06797 for ; Fri, 16 Nov 2001 11:29:55 -0500 (EST) Received: from hq-ex-c2.brocade.com (hq-ex-c2 [192.168.126.35]) by mail.brocade.com (8.8.8+Sun/8.8.8) with ESMTP id IAA28689; Fri, 16 Nov 2001 08:29:40 -0800 (PST) Received: by hq-ex-c2.brocade.com with Internet Mail Service (5.5.2653.19) id ; Fri, 16 Nov 2001 08:29:39 -0800 Message-ID: From: Robert Snively To: "'Thanu Skariah'" , Robert Grant Cc: "'ips@ece.cmu.edu'" Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics Date: Fri, 16 Nov 2001 08:29:38 -0800 X-Mailer: Internet Mail Service (5.5.2653.19) Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, Robert Grant has a legitimate concern. The iqn.* format and the eui.* format do not correctly map the name space used by FCP or SCSI disk drives, which use either a Fibre Channel name format (specified in FC-FS from www.t11.org, clause 14) or a SCSI format (specified in SPC-2 from www.t10.org, clause 8.4.4). The Fibre Channel name format applies to targets, initiators, ports, and logical units. The SCSI format only applies to ports and logical units. Neither of these map to the eui.* format. However, a specialized FC name format is now defined to map from an eui.* format, which should be useful for at least some FCP/iSCSI bridges. You can of course map the iqn.* format very successfully by simply including the appropriate FC WWN as part of the text string, but that is done at a level outside the scope of iSCSI. If you want to make a STANDARD way to do this (instead of a user administered way such as iqn.*), then two new formats could be defined: fc.* = where * equals the FC-FS clause 14 defined 64-bit value. and scsi.* = where * equals the multiple variable length values specified in SPC-2, clause 8.4.4 Fortunately, at the logical unit level (which is the only level that any true SCSI bigot cares about anyway :-) ), the identifier (usually an FC WWN) is mapped through the INQUIRY command independent of the transport mechanism. There are several useful mappings that can be used to instantiate iSCSI LUs in FC or vice versa. Bob Snively e-mail: rsnively@brocade.com Brocade Communications Systems phone: 408 487 8135 1745 Technology Drive San Jose, CA 95110 > -----Original Message----- > From: Thanu Skariah [mailto:tskariah@npd.hcltech.com] > Sent: Thursday, November 15, 2001 11:12 PM > To: Robert Grant > Cc: 'ips@ece.cmu.edu' > Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics > > > Robert, > > > iSCSI allows different naming formats, of which one > format is the EUI > format (See the example > in sec 2.2 .7 and the naming draft - > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name- > disc-02.txt ) > > The EUI representation is of the form eui . . Each > FC device's > WWName > can be used to form the corresponding iSCSI name for the > device. This is what > we are > doing on a linux based software FCP/iSCSI gateway that we are > implementing, and > this > is why : > > (From the naming and discovery draft ): > > BeginQuote " > > Type "eui." (IEEE EUI format) > > The IEEE iSCSI name might be used when a manufacturer is already > basing unique identifiers on World-Wide Names as defined > in the SCSI > SPC-2 specification. > > It may also be used by a gateway representing a Fibre Channel or > SCSI device that is already adequately identified using a > world-wide > name. > > " End Quote > > > Thanks, > Thanu > > > > Robert Grant wrote: > > > Hello all, > > > > I have a question on the > representation of iSCSI > > devices into Fibre Channel fabrics for an iSCSI-to-FC > "gateway" device and > > would like to solicit people's thoughts on how best to do > this. A gateway > > device will allow iSCSI devices and FCP devices to access > each other, but in > > order to do this a consistent representation of the devices > is needed. I > > haven't been able to reconcile the iSCSI and FCP standards > using what's > > currently in the iSCSI standard, and wanted to see if there > was any support > > to expanding the iSCSI standard to address this (a standard > solution is, of > > course, much more preferred to every gateway vendor doing > it in their own > > proprietary way). In particular, how would an iSCSI device > map onto Fibre > > Channel's World Wide Name (WWN)? Would every device have > its own WWN, or > > could many iSCSI devices use a single WWN? There have been > some discussions > > (for example, there was even discussion of including a WWN > field in the > > iSCSI Login for a Gateway to proxy with in > > > http://www.pdl.cmu.edu/mailinglists/ips/mail/msg01616.html), > but what is the > > current view? > > > > A first approach might be that many > iSCSI devices > > could use a single WWN. This can work well for FC-AL > devices "directly > > attached " to the IP network or for small FC fabrics - and where the > > predominant interconnect and management of that > interconnect is the IP > > network. > > > > This approach views the FC fabric > as flat (or at > > least perhaps that FC zoning is "turned off"). As the FC > fabric gets bigger, > > though, this first approach can create two layers of > management - one must > > first configure the FC network and then configure the IP > network (since the > > individual iSCSI devices sharing a single WWN can only be > zoned as a group). > > The two layers are first "this group of iSCSI devices can > access this zone" > > on the FC side and then "this iSCSI device can access this > FC device in this > > zone" on the iSCSI side. If there was a clean integration > with FC zoning > > (and associated management of the FC zoning), this may be avoided. > > > > A further complication is that, as > the FC fabric > > gets even bigger, a single iSCSI device could end up with > multiple entry > > points (i.e. paths through multiple gateways) into a single > FC fabric. Is > > there any common way to represent iSCSI devices (for > instance, with respect > > to WWNs) that allows the unique identification of that > iSCSI device - even > > though there are multiple entrypoints onto the FC fabric? > The case of > > multiple gateways (possibly from different vendors) is the > clearest example > > of the need for a standard. > > > > Thank you for your time and I look > forward to all > > comments/suggestions. > > > > Regards, > > Rob > > > > Rob Grant > > McDATA Corporation > > From owner-ips@ece.cmu.edu Fri Nov 16 14:52:55 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22223 for ; Fri, 16 Nov 2001 14:52:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGIogQ17346 for ips-outgoing; Fri, 16 Nov 2001 13:50:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGIodl17335 for ; Fri, 16 Nov 2001 13:50:39 -0500 (EST) Received: from MURALIRLAPTOP ([192.168.2.38]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id KAA18966 for ; Fri, 16 Nov 2001 10:50:32 -0800 (PST) From: "Murali Rajagopal" To: Subject: FCIP Teleconf Minutes of 11/14 Date: Fri, 16 Nov 2001 10:49:32 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Minutes of FCIP Telecon 11/14/01 Minutes taken by Larry Lamers. Attendees: Ralph Weber Jim Nelson Milan Merhar Anil Rijhsinghani Raj Bhagwat Dave Peterson Bob Snively Larry Lamers Andy Helland Steve Wilson Neil Wanamaker Murali Rajagopal Kha Sin Teow NAPTs solution - Reviewed the modified short frame for Larry's education. Comments on draft - Bob's comment page 23 - regarding handling of a second short frame - ignore. Add a note to the table the end echoing the frame does absolve the orignating end from checking for a match. Dave - modify paragraph regarding wwn uniqueness - add parantheticall FCIP LEP and DE to paragraph immediately preceding clause 8. Anil - Do we add a timeout value recommendation for short frame - 90 seconds minimum. Model discussion: where does the green boundary exist? page 3 issue with right side pairing - need another FCIP entity in lower right or remove blue line unresolved cases on page 3 page 2 has tentative agreement but needs additional detail to be really usefull need for two models of page 2 source and destination Regards, ========================================== Lawrence J. Lamers email: ljlamers@ieee.org Cell Phone: (408) 234-0071 Home Phone: (408) 226-0343 From owner-ips@ece.cmu.edu Fri Nov 16 14:53:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22301 for ; Fri, 16 Nov 2001 14:53:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGJIWA19565 for ips-outgoing; Fri, 16 Nov 2001 14:18:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ztxmail04.ztx.compaq.com (ztxmail04.ztx.compaq.com [161.114.1.208]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGJIUl19561 for ; Fri, 16 Nov 2001 14:18:31 -0500 (EST) Received: by ztxmail04.ztx.compaq.com (Postfix, from userid 12345) id 3B5ABE74; Fri, 16 Nov 2001 13:18:25 -0600 (CST) Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 1913BC69 for ; Fri, 16 Nov 2001 13:18:25 -0600 (CST) Received: from cceexc17.americas.cpqcorp.net ([16.110.250.84]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Fri, 16 Nov 2001 13:18:24 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics Date: Fri, 16 Nov 2001 13:18:24 -0600 Message-ID: <78AF3C342AEAEF4BA33B35A8A15668C6014D6C9C@cceexc17.americas.cpqcorp.net> Thread-Topic: iSCSI: Representing iSCSI devices on FC fabrics Thread-Index: AcFubjRZK3uPCtasQFeyQaMzz6nVRgAQ+tTwAAhLSeA= From: "Elliott, Robert" To: X-OriginalArrivalTime: 16 Nov 2001 19:18:24.0737 (UTC) FILETIME=[76713510:01C16ED3] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fAGJIVl19562 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Correction - the company_id is 24 bits, not 20 bits. EUI-64 = (MSB) { 24-bit company_id, 40-bit vendor specified } (LSB) FC WWN = (MSB) { 4-bit NAA field, 24-bit company_id, 36-bit vendor specified } (LSB) (for the IEEE Registered NAA type) > -----Original Message----- > From: Elliott, Robert > Sent: Friday, November 16, 2001 9:33 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics > > > Beware than a FC WWN is not the same as an EUI-64. They're both based > on the IEEE OUI/company_id, but the FC WWN uses the 4 most significant > bits for a "Name Address Authority" field and has 4 fewer > bits available > in the vendor specified portion. > > EUI-64 = (MSB) { 20-bit company_id, > 44-bit vendor specified } (LSB) > > FC WWN = (MSB) { 4-bit NAA field, > 20-bit company_id, > 40-bit vendor specified } (LSB) > (for the IEEE Registered NAA type) > > --- > Rob Elliott, Compaq Server Storage > Robert.Elliott@compaq.com From owner-ips@ece.cmu.edu Fri Nov 16 14:54:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22339 for ; Fri, 16 Nov 2001 14:54:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGIogV17347 for ips-outgoing; Fri, 16 Nov 2001 13:50:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGIodl17334 for ; Fri, 16 Nov 2001 13:50:39 -0500 (EST) Received: from MURALIRLAPTOP ([192.168.2.38]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id KAA18962 for ; Fri, 16 Nov 2001 10:50:32 -0800 (PST) From: "Murali Rajagopal" To: Subject: FCIP Conference Call Minutes - 11/12/01 Date: Fri, 16 Nov 2001 10:49:31 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit FCIP Conference Call Minutes: Meeting Date: 11/12/01 - 1PM-3PM PST Minutes by: Venkat Rangan, Rhapsody Networks Next conference call: 11/14/01 - 1PM-3PM PST Next meeting owner: Larry Lamers, San Valley Attendees: Venkat Rangan, Rhapsody Networks Ralph Weber, ENDL Murali Rajagopal, Lightsand Andy Hellend, Lightsand Elizabeth Rodriguez, Lucent Bill Creek, Lucent Milan Merhar, Pirus Anil Rijhsinghani, McData Kha Sin Teow, Brocade Neil Wanamaker, Akara Bob Snively, Brocade Jim Nelson, Vixel Don Fraser, Compaq Agenda: 1. NAPT Solution aka Setup Protocol Murali and Bob have had discussions that have tied up all the loose ends with the NAPT proposal version 8 as well as the contents from the "FCIP_Setup_Protocol.pdf" distributed by Murali on 11/2. The gist of it is that the receiver information will be useful and provided. Policy can state whether a zero-receiver information is acceptable or not. The sender provides what he thinks is the Receiver's WWN and the receiver can either accept it or provide a "restart-short-frame". Details to be provided in a new document from Bob, with all the authors studying it and being prepared for the Wednesday's call, to conclude. The objective is to get the new proposal studied and accepted by the team, so Ralph can integrate that into a FCIP Draft 6b. It will not be integrated into a NAPTs version 9. Draft 6b will be sent to the Internet Draft office by Wednesday evening. The authors discussed many details of the proposal. Can the FCIP Short-Frame be longer than the shortest FC frame? We don't have an answer yet. Short Frame has an FCIP Identifier of the source of the link. If the Echo short frame comes back with mis-matched information, the connection is blown away. The NULL identifier behavior is described, and how a device treats NULL identifiers is an FC Entity defined policy. The new proposal is designed to handle cases where the receiver cannot accept certain conditions on multi-connection links such as treating Class-F and priority based connections, as well as informing the sender an appropriate set of parameters so a future connection may succeed. 2. Model Discussions Andy added some items to the model after discussions with Larry. This is more of a FC-BB issue, but seems relevant for FCIP. The model is pretty much the same as before, i.e., the Irvine Meeting. A model will be posted out for discussions by Wednesday, for a discussion by the FCIP Author's group after the NAPTs additions to Draft 6b will be discussed. Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com From owner-ips@ece.cmu.edu Fri Nov 16 17:32:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA27568 for ; Fri, 16 Nov 2001 17:32:55 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGLLor29113 for ips-outgoing; Fri, 16 Nov 2001 16:21:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail1.aarohicommunications.com (cpe-66-87-94-158.ca.sprintbbd.net [66.87.94.158]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGHc5l11890 for ; Fri, 16 Nov 2001 12:38:06 -0500 (EST) Received: from LINUX9 (linux9.aarohicommunications.com [192.168.1.219]) by mail1.aarohicommunications.com (Mirapoint) with ESMTP id AAJ01115 (AUTH sureshtk); Fri, 16 Nov 2001 09:39:04 -0800 (PST) From: "Suresh Tanjore" To: Subject: Question On R2T Behaviour? Date: Fri, 16 Nov 2001 09:39:17 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi all, On page 75 of iSCSi 0.8 revision " The data length of the DATA PDU MUST *not exceed* the desired data transer length specified in the R2T. If the R2T is answered with a sequence of data PDUs the buffer offset and Length must be within the range of those specfied by R2T, the last PDU SHOULD have the F bit set to 1. The data-out PDU ordering is goverened by DataPDUInOrder." My question is if the data length of the DATA PDU is less than the desired data transfer length and the F bit is set, what should be the behaviour of the target. For the sake of this discussion let us assume DataPDUInOrder to be set to Yes. 1. It looks intutive that target may send another R2T with different sequence number and collect the data initially that the host was not able to send for the previous R2T request. Or 2. It is an error that need to be detected by target that the host was not able to fully satisfy the R2T request sent by the target in the first place. Please let me know what is the right behaviour and should this be documented in the specification if it is not already documented. If it is documented may be i missed reading this, please point to me where it is defined. Thanks sureshtk From owner-ips@ece.cmu.edu Fri Nov 16 17:35:54 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA27704 for ; Fri, 16 Nov 2001 17:35:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGLugY01810 for ips-outgoing; Fri, 16 Nov 2001 16:56:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGLubl01785 for ; Fri, 16 Nov 2001 16:56:37 -0500 (EST) Received: by ARIEL with Internet Mail Service (5.5.2653.19) id ; Fri, 16 Nov 2001 13:56:31 -0800 Message-ID: From: Kevin Gibbons To: ips@ece.cmu.edu Subject: iSNS: open source v1.2 posted at Source Forge Date: Fri, 16 Nov 2001 13:56:30 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The iSNS Version 1.2 open source implementation has been posted at Source Forge. This version implements version 05 of the iSNS Specification. The supported platforms are Linux and NT/Win2000. A client implementation optimized for iSCSI targets and initiators is also posted. The location is: http://sourceforge.net/projects/linuxisns/ An alternative site is http://www.nishansystems.com/iSNS/index.html NOTE: the version currently posted at the CMU IPS site is not the most recent version. -- Kevin From owner-ips@ece.cmu.edu Fri Nov 16 18:29:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA29114 for ; Fri, 16 Nov 2001 18:29:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGMVkV04547 for ips-outgoing; Fri, 16 Nov 2001 17:31:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGMVjl04543 for ; Fri, 16 Nov 2001 17:31:45 -0500 (EST) Received: from xatlrelay2.atl.hp.com (xatlrelay2.atl.hp.com [15.45.89.191]) by atlrel1.hp.com (Postfix) with ESMTP id D597BC57 for ; Fri, 16 Nov 2001 17:31:44 -0500 (EST) Received: from xatlbh1.atl.hp.com (xatlbh1.atl.hp.com [15.45.89.186]) by xatlrelay2.atl.hp.com (Postfix) with ESMTP id 42A941F535 for ; Fri, 16 Nov 2001 17:31:44 -0500 (EST) Received: by xatlbh1.atl.hp.com with Internet Mail Service (5.5.2653.19) id ; Fri, 16 Nov 2001 17:31:42 -0500 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87805CCF225@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "'ips@ece.cmu.edu'" Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics Date: Fri, 16 Nov 2001 17:31:39 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This use of the eui. naming format will ONLY be acceptable if the iSCSI gateway device is the only iSCSI gateway device in the world that is allowed to "represent" these FC devices - can your implementation guarantee that? Another iSCSI gateway on the same FC fabric (another of your gateways?) might also use that naming convention (as Jim H points out) and that would violate the requirements of iSCSI naming. The eui. naming format was intended for use when the iSCSI device "owns" the EUI number (as those FC devices "own" their WWN). This gateway usage is not what was intended. Marjorie Krueger Networked Storage Architecture Networked Storage Solutions Org. Hewlett-Packard tel: +1 916 785 2656 fax: +1 916 785 0391 email: marjorie_krueger@hp.com > -----Original Message----- > From: Thanu Skariah [mailto:tskariah@npd.hcltech.com] > Sent: Thursday, November 15, 2001 11:12 PM > To: Robert Grant > Cc: 'ips@ece.cmu.edu' > Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics > > > Robert, > > > iSCSI allows different naming formats, of which one > format is the EUI > format (See the example > in sec 2.2 .7 and the naming draft - > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name- > disc-02.txt ) > > The EUI representation is of the form eui . . Each > FC device's > WWName > can be used to form the corresponding iSCSI name for the > device. This is what > we are > doing on a linux based software FCP/iSCSI gateway that we are > implementing, and > this > is why : > > (From the naming and discovery draft ): > > BeginQuote " > > Type "eui." (IEEE EUI format) > > The IEEE iSCSI name might be used when a manufacturer is already > basing unique identifiers on World-Wide Names as defined > in the SCSI > SPC-2 specification. > > It may also be used by a gateway representing a Fibre Channel or > SCSI device that is already adequately identified using a > world-wide > name. > > " End Quote > > > Thanks, > Thanu > > > > Robert Grant wrote: > > > Hello all, > > > > I have a question on the > representation of iSCSI > > devices into Fibre Channel fabrics for an iSCSI-to-FC > "gateway" device and > > would like to solicit people's thoughts on how best to do > this. A gateway > > device will allow iSCSI devices and FCP devices to access > each other, but in > > order to do this a consistent representation of the devices > is needed. I > > haven't been able to reconcile the iSCSI and FCP standards > using what's > > currently in the iSCSI standard, and wanted to see if there > was any support > > to expanding the iSCSI standard to address this (a standard > solution is, of > > course, much more preferred to every gateway vendor doing > it in their own > > proprietary way). In particular, how would an iSCSI device > map onto Fibre > > Channel's World Wide Name (WWN)? Would every device have > its own WWN, or > > could many iSCSI devices use a single WWN? There have been > some discussions > > (for example, there was even discussion of including a WWN > field in the > > iSCSI Login for a Gateway to proxy with in > > http://www.pdl.cmu.edu/mailinglists/ips/mail/msg01616.html), but what is the > current view? > > A first approach might be that many iSCSI devices > could use a single WWN. This can work well for FC-AL devices "directly > attached " to the IP network or for small FC fabrics - and where the > predominant interconnect and management of that interconnect is the IP > network. > > This approach views the FC fabric as flat (or at > least perhaps that FC zoning is "turned off"). As the FC fabric gets bigger, > though, this first approach can create two layers of management - one must > first configure the FC network and then configure the IP network (since the > individual iSCSI devices sharing a single WWN can only be zoned as a group). > The two layers are first "this group of iSCSI devices can access this zone" > on the FC side and then "this iSCSI device can access this FC device in this > zone" on the iSCSI side. If there was a clean integration with FC zoning > (and associated management of the FC zoning), this may be avoided. > > A further complication is that, as the FC fabric > gets even bigger, a single iSCSI device could end up with multiple entry > points (i.e. paths through multiple gateways) into a single FC fabric. Is > there any common way to represent iSCSI devices (for instance, with respect > to WWNs) that allows the unique identification of that iSCSI device - even > though there are multiple entrypoints onto the FC fabric? The case of > multiple gateways (possibly from different vendors) is the clearest example > of the need for a standard. > > Thank you for your time and I look forward to all > comments/suggestions. > > Regards, > Rob > > Rob Grant > McDATA Corporation From owner-ips@ece.cmu.edu Fri Nov 16 19:31:14 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA00418 for ; Fri, 16 Nov 2001 19:31:14 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGNJ2T08223 for ips-outgoing; Fri, 16 Nov 2001 18:19:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lucy.trebia.com ([65.192.191.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAGMxNl06331 for ; Fri, 16 Nov 2001 17:59:23 -0500 (EST) Received: from trebiaachadda (140.186.40.85 [140.186.40.85]) by lucy.trebia.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id WYQ2FX3A; Fri, 16 Nov 2001 17:58:20 -0500 Message-ID: <005001c16ef2$41fec080$9a7fa8c0@trebiaachadda> From: "Anshul Chadda" To: "Suresh Tanjore" , References: Subject: Re: Question On R2T Behaviour? Date: Fri, 16 Nov 2001 17:58:39 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi Suresh: I believe that the target can take step 1. But then the problem might still prevail. Why?? If the initiator wasn't able to send Data for the first R2T, there is no guarantee that it will be able to send the same data (that it couldn't send before) for the second R2T. So, the target cannot try sending R2T asking for the same data infinitely. I don't think it is an error if the initiator is not able to send the whole data, involved in an R2T, as long as it sets the F bit to 1 in the final Data-Out PDU (mentioned in Step 2). I would assume that the target has an option to take care of this behavior by sending SCSI Response PDU with U-bit set and the Residual Count set to a data value that it was not able to receive.This step can be taken by the target as a conservative one and can serve as a fallback solution in case step 1 doesn't work. Anshul > 1. It looks intutive that target may send another R2T with different > sequence number and collect the data initially that the host > was not able to send for the previous R2T request. > > Or > > 2. It is an error that need to be detected by target that the host was not > able to fully satisfy the R2T request sent by the target in the first > place. > From owner-ips@ece.cmu.edu Fri Nov 16 19:33:32 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA00449 for ; Fri, 16 Nov 2001 19:33:32 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAGNiHR10483 for ips-outgoing; Fri, 16 Nov 2001 18:44:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web21006.mail.yahoo.com (web21006.mail.yahoo.com [216.136.227.60]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAGNiEl10476 for ; Fri, 16 Nov 2001 18:44:15 -0500 (EST) Message-ID: <20011116234413.7095.qmail@web21006.mail.yahoo.com> Received: from [63.107.133.89] by web21006.mail.yahoo.com via HTTP; Fri, 16 Nov 2001 15:44:13 PST Date: Fri, 16 Nov 2001 15:44:13 -0800 (PST) From: Subbu Subramaniam Subject: questions on iSCSI draft To: ips@ece.cmu.edu Cc: subbu_ario@yahoo.com MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, I am reading through the iSCSI draft, and a I have a few questions. I would appreciate if someone clarifies: 1. The default value of FirstBurstSize (amount of data that can be included immediate with a commdn) is 128 units, and that of DataPDULength (amount of data in a DATA-only PDU) is 16 units. If a target can accept 128 bytes of data immediately with a command, do we expect that the same target will only accept less number of units in a later PDU? Why? Should the FirstBurstSize not be less than DataPDULength and not vice versa? (asme argument holds for initiator, since both can specify this option). 2. If a target has sent more than one R2T PDUs, and DataSequenceInOrder is "no", then can the initiator interleave data between the two sequences? It seems that the target should be able to distinguish between the PDUs belonging to the two sequences using the "Buffer Offset" field, and the spec does not prohibit the initiator from sending two sequences simultaneously. Thanks, -Subbu __________________________________________________ Do You Yahoo!? Find the one for you at Yahoo! Personals http://personals.yahoo.com From owner-ips@ece.cmu.edu Fri Nov 16 22:47:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA04416 for ; Fri, 16 Nov 2001 22:47:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAH2keM21243 for ips-outgoing; Fri, 16 Nov 2001 21:46:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imo-r04.mx.aol.com (imo-r04.mx.aol.com [152.163.225.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAH2kcl21237 for ; Fri, 16 Nov 2001 21:46:38 -0500 (EST) Received: from VAHUJA@aol.com by imo-r04.mx.aol.com (mail_out_v31_r1.9.) id f.70.132d7d9c (4332); Fri, 16 Nov 2001 21:45:17 -0500 (EST) From: VAHUJA@aol.com Message-ID: <70.132d7d9c.2927293d@aol.com> Date: Fri, 16 Nov 2001 21:45:17 EST Subject: Re: SRP vs PKI for authentication To: Black_David@emc.com CC: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: AOL 5.0 for Windows sub 138 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, I understand your and Ofer's views. But lets view from another point. For the same storage network, we may potentially have PKI as the choice for authentication for one part, and SRP for another. There are ways to simplify built-in PKI for storage networks as being applied for FC security (I expect you dont need an RA or a CA, and the trust hierarchy is left to the customer). So for those networks that want to ONLY authenticate entities in storage networks, we may potentially have two schemes that "MUST" be implemented. From owner-ips@ece.cmu.edu Sat Nov 17 10:06:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA25419 for ; Sat, 17 Nov 2001 10:06:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAHDZh204876 for ips-outgoing; Sat, 17 Nov 2001 08:35:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dcmtechdom.dcmtech.co.in ([203.190.136.130]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAHDZbl04866 for ; Sat, 17 Nov 2001 08:35:39 -0500 (EST) Received: by DCMTECHDOM with Internet Mail Service (5.5.2653.19) id ; Sat, 17 Nov 2001 19:08:18 +0530 Message-ID: <7FADCB99FC82D41199F9000629A85D1A0293A651@DCMTECHDOM> From: Nitin Dhingra To: "'ips@ece.cmu.edu'" Subject: iSCSI: Function 4 Of Task Mgmt Command Date: Sat, 17 Nov 2001 19:08:16 +0530 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk How does the initiator decide to choose Task Management Function 4 i.e. Clear Task Set - Aborts all Tasks (from all initiators) for the Logical Unit? From owner-ips@ece.cmu.edu Sat Nov 17 12:13:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA27624 for ; Sat, 17 Nov 2001 12:13:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAHGCrV13211 for ips-outgoing; Sat, 17 Nov 2001 11:12:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAHGCpl13206 for ; Sat, 17 Nov 2001 11:12:51 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA35106 for ; Sat, 17 Nov 2001 17:12:44 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAHGCoo131232 for ; Sat, 17 Nov 2001 17:12:50 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: Question On R2T Behaviour? X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 17 Nov 2001 18:12:50 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 17/11/2001 18:12:53, Serialize complete at 17/11/2001 18:12:53 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Suresh, The only statement that can be made is that the target is the master of the data transfers. For the case you describe any of the behaviors you describe is legal. The behavior is defined by SSCSI (SAM). The results are implementation dependent. The only think that we might say is that the results "should match expectation" and "the residual counts should match transfers". In practical terms you will find many targets that will terminate the transfer with an error status if an R2T request is not properly fulfilled. Julo "Suresh Tanjore" Sent by: owner-ips@ece.cmu.edu 16-11-01 19:39 Please respond to "Suresh Tanjore" To: cc: Subject: Question On R2T Behaviour? Hi all, On page 75 of iSCSi 0.8 revision " The data length of the DATA PDU MUST *not exceed* the desired data transer length specified in the R2T. If the R2T is answered with a sequence of data PDUs the buffer offset and Length must be within the range of those specfied by R2T, the last PDU SHOULD have the F bit set to 1. The data-out PDU ordering is goverened by DataPDUInOrder." My question is if the data length of the DATA PDU is less than the desired data transfer length and the F bit is set, what should be the behaviour of the target. For the sake of this discussion let us assume DataPDUInOrder to be set to Yes. 1. It looks intutive that target may send another R2T with different sequence number and collect the data initially that the host was not able to send for the previous R2T request. Or 2. It is an error that need to be detected by target that the host was not able to fully satisfy the R2T request sent by the target in the first place. Please let me know what is the right behaviour and should this be documented in the specification if it is not already documented. If it is documented may be i missed reading this, please point to me where it is defined. Thanks sureshtk From owner-ips@ece.cmu.edu Sun Nov 18 11:07:20 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA25904 for ; Sun, 18 Nov 2001 11:07:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAIF35B05697 for ips-outgoing; Sun, 18 Nov 2001 10:03:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAIF34l05693 for ; Sun, 18 Nov 2001 10:03:04 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA49410 for ; Sun, 18 Nov 2001 16:02:57 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAIF34l66056 for ; Sun, 18 Nov 2001 16:03:04 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: iSCSI editorial changes X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sun, 18 Nov 2001 17:03:05 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 18/11/2001 17:03:07, Serialize complete at 18/11/2001 17:03:07 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues: iSCSI-09 is out. The main editorial changes have been postponed for 10 - that will be out soon, but will probably not make the Salt Lake City cutoff date. The changes will involve a new tool (we might get finally the appendixes numbered), an overview of the commands - in line with chapters 2-6 and the command details moved to a later chapter and perhaps "recovering" some appendixes to the main body. Julo From owner-ips@ece.cmu.edu Sun Nov 18 11:11:02 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA25955 for ; Sun, 18 Nov 2001 11:11:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAIEuCm05359 for ips-outgoing; Sun, 18 Nov 2001 09:56:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAIEu9l05350 for ; Sun, 18 Nov 2001 09:56:09 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id PAA53912; Sun, 18 Nov 2001 15:56:01 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAIEu4l92872; Sun, 18 Nov 2001 15:56:04 +0100 To: internet-drafts@ietf.org Cc: bassoon@YOGI.PDL.CMU.EDU, ips@ece.cmu.edu MIME-Version: 1.0 Subject: new iSCSI draft 09.txt X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sun, 18 Nov 2001 16:56:05 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 18/11/2001 16:56:10, Serialize complete at 18/11/2001 16:56:10 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk On behalf of a group of authors from several organizations as part of the IPS working group I submit a revision of an IETF-draft for immediate publication. It specifies iSCSI - a SCSI Over TCP protocol and the file name is "draft-ietf-ips-iSCSI-09.txt". It completely replaces "draft-ietf-ips-iSCSI-08.txt". The draft can be found at: http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iSCSI-09.txt There is also a pdf version that shows the changes from the previous version at: http://www.haifa.il.ibm.com/satran/ips/draft-ietf-ips-iSCSI-09.pdf Julian Satran - IBM Research Laboratory at Haifa From owner-ips@ece.cmu.edu Sun Nov 18 19:24:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA00156 for ; Sun, 18 Nov 2001 19:24:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAINLgQ02521 for ips-outgoing; Sun, 18 Nov 2001 18:21:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAINLel02514 for ; Sun, 18 Nov 2001 18:21:41 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA92530 for ; Sun, 18 Nov 2001 18:19:00 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAINLd4228232 for ; Sun, 18 Nov 2001 16:21:39 -0700 X-Priority: 1 (High) Importance: Normal Subject: iSCSI:new iSCSI draft 09.txt, some text omitted To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Sun, 18 Nov 2001 15:20:51 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/18/2001 04:21:39 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I think you perhaps overlooked, for the new draft, the wordage presented by Andre Asselin in the area of 2.2.7 and 3.12.8 shown below. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/05/2001 10:21:55 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: spec revs to make TargetName reqd on every connection thanks - julo Andre Asselin@IBMUS 05-11-01 22:55 To: Julian Satran/Haifa/IBM@IBMIL cc: John Hufferd/San Jose/IBM@IBMUS, Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: spec revs to make TargetName reqd on every connection Julian, Attached is an attempt to pull together all the required spec updates to make InitiatorName required on every login, TargetName required on every normal login, and clarify related text. Does this look good to everyone? Any places I missed? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Appendix D, TargetName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). To This key must be provided by the initiator of the TCP connection to the remote endpoint in the first login request if the initiator is not establishing a discovery session. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). Appendix D, InitiatorName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The Initiator key enables the initiator to identify itself to the remote endpoint. To This key must be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The InitiatorName key enables the initiator to identify itself to the remote endpoint. The current version of the 6th paragraph in chapter 5 reads: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. A suggested rewrite would be (building on the text suggested by Bob Russell): All initial Login requests MUST include the InitiatorName key=value pair. If the initial Login request is also a leading Login (TSID=0) and the new session is to be a discovery session, then the initial Login request MUST also include the SessionType=discovery key=value pair. If the initial Login request is a leading Login and the new session is to be a normal session, then the initial Login request MUST also include the TargetName key=value pair and MAY also include the SessionType=normal key=value pair. All initial Login requests that are not also a leading Login (TSID != 0) MUST include the TargetName key=value pair. Also, this text appears in 2.2.7: The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session. The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. A suggested rewrite would be: The initiator must present its iSCSI Initiator Name in the first login request. If the initiator is not establishing a discovery session (see 2.4), it also must present the iSCSI Target Name to which it wishes to connect in the first login request. The key "SessionType=discovery" is sent by the initiator on the Initial Login request to indicate a discovery session. See chapter 5 for a more detailed description of the Login process. Section 3.12.8 currently reads: The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies the session with that initiator. Suggested rewrite (melding current text w/John's rewrite): The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity). From owner-ips@ece.cmu.edu Mon Nov 19 02:06:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA18708 for ; Mon, 19 Nov 2001 02:06:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJ69pL25120 for ips-outgoing; Mon, 19 Nov 2001 01:09:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJ69ml25109 for ; Mon, 19 Nov 2001 01:09:48 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA62012 for ; Mon, 19 Nov 2001 07:09:42 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAJ69mm47156 for ; Mon, 19 Nov 2001 07:09:48 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 19 Nov 2001 08:09:50 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 19/11/2001 08:09:51, Serialize complete at 19/11/2001 08:09:51 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, The change was already made when Andre asked for it. I've checked it and it did not disappear. The wording may be different but the names are required at each login. I've checked it now and it appears everywhere. If I omitted to specify it somewhere please let me know. Regards, Julo John Hufferd/San Jose/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 19-11-01 01:20 Please respond to John Hufferd To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI:new iSCSI draft 09.txt, some text omitted Julian, I think you perhaps overlooked, for the new draft, the wordage presented by Andre Asselin in the area of 2.2.7 and 3.12.8 shown below. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/05/2001 10:21:55 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: spec revs to make TargetName reqd on every connection thanks - julo Andre Asselin@IBMUS 05-11-01 22:55 To: Julian Satran/Haifa/IBM@IBMIL cc: John Hufferd/San Jose/IBM@IBMUS, Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: spec revs to make TargetName reqd on every connection Julian, Attached is an attempt to pull together all the required spec updates to make InitiatorName required on every login, TargetName required on every normal login, and clarify related text. Does this look good to everyone? Any places I missed? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Appendix D, TargetName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). To This key must be provided by the initiator of the TCP connection to the remote endpoint in the first login request if the initiator is not establishing a discovery session. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). Appendix D, InitiatorName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The Initiator key enables the initiator to identify itself to the remote endpoint. To This key must be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The InitiatorName key enables the initiator to identify itself to the remote endpoint. The current version of the 6th paragraph in chapter 5 reads: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. A suggested rewrite would be (building on the text suggested by Bob Russell): All initial Login requests MUST include the InitiatorName key=value pair. If the initial Login request is also a leading Login (TSID=0) and the new session is to be a discovery session, then the initial Login request MUST also include the SessionType=discovery key=value pair. If the initial Login request is a leading Login and the new session is to be a normal session, then the initial Login request MUST also include the TargetName key=value pair and MAY also include the SessionType=normal key=value pair. All initial Login requests that are not also a leading Login (TSID != 0) MUST include the TargetName key=value pair. Also, this text appears in 2.2.7: The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session. The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. A suggested rewrite would be: The initiator must present its iSCSI Initiator Name in the first login request. If the initiator is not establishing a discovery session (see 2.4), it also must present the iSCSI Target Name to which it wishes to connect in the first login request. The key "SessionType=discovery" is sent by the initiator on the Initial Login request to indicate a discovery session. See chapter 5 for a more detailed description of the Login process. Section 3.12.8 currently reads: The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies the session with that initiator. Suggested rewrite (melding current text w/John's rewrite): The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity). From owner-ips@ece.cmu.edu Mon Nov 19 05:16:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA24222 for ; Mon, 19 Nov 2001 05:16:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJ93Vn04173 for ips-outgoing; Mon, 19 Nov 2001 04:03:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJ93Sl04163 for ; Mon, 19 Nov 2001 04:03:28 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id EAA11108 for ; Mon, 19 Nov 2001 04:00:47 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAJ93Q4136132 for ; Mon, 19 Nov 2001 02:03:26 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Mon, 19 Nov 2001 01:02:19 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/19/2001 02:03:26 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, With regard to point 2.2.7, I now see the wordage is probably OK, I think I was thrown off by the lack of change bars, at that point, in your PDF version. However, the 3.12.8 Item (which is now at 3.12.7) we thought needed to have the last sentence changed from: "...this uniquely identifies the session with that initiator." to: "...this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity)." We spent a lot of time discussing this, and found there was a lot of confusion. So we wanted to unambiguously state the value of TSID did not have scope beyond the Specific Target, which might exist amoung others, in the same Network Entity. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/18/2001 10:09:50 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted John, The change was already made when Andre asked for it. I've checked it and it did not disappear. The wording may be different but the names are required at each login. I've checked it now and it appears everywhere. If I omitted to specify it somewhere please let me know. Regards, Julo John Hufferd/San Jose/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 19-11-01 01:20 Please respond to John Hufferd To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI:new iSCSI draft 09.txt, some text omitted Julian, I think you perhaps overlooked, for the new draft, the wordage presented by Andre Asselin in the area of 2.2.7 and 3.12.8 shown below. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/05/2001 10:21:55 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: spec revs to make TargetName reqd on every connection thanks - julo Andre Asselin@IBMUS 05-11-01 22:55 To: Julian Satran/Haifa/IBM@IBMIL cc: John Hufferd/San Jose/IBM@IBMUS, Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: spec revs to make TargetName reqd on every connection Julian, Attached is an attempt to pull together all the required spec updates to make InitiatorName required on every login, TargetName required on every normal login, and clarify related text. Does this look good to everyone? Any places I missed? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Appendix D, TargetName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). To This key must be provided by the initiator of the TCP connection to the remote endpoint in the first login request if the initiator is not establishing a discovery session. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). Appendix D, InitiatorName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The Initiator key enables the initiator to identify itself to the remote endpoint. To This key must be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The InitiatorName key enables the initiator to identify itself to the remote endpoint. The current version of the 6th paragraph in chapter 5 reads: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. A suggested rewrite would be (building on the text suggested by Bob Russell): All initial Login requests MUST include the InitiatorName key=value pair. If the initial Login request is also a leading Login (TSID=0) and the new session is to be a discovery session, then the initial Login request MUST also include the SessionType=discovery key=value pair. If the initial Login request is a leading Login and the new session is to be a normal session, then the initial Login request MUST also include the TargetName key=value pair and MAY also include the SessionType=normal key=value pair. All initial Login requests that are not also a leading Login (TSID != 0) MUST include the TargetName key=value pair. Also, this text appears in 2.2.7: The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session. The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. A suggested rewrite would be: The initiator must present its iSCSI Initiator Name in the first login request. If the initiator is not establishing a discovery session (see 2.4), it also must present the iSCSI Target Name to which it wishes to connect in the first login request. The key "SessionType=discovery" is sent by the initiator on the Initial Login request to indicate a discovery session. See chapter 5 for a more detailed description of the Login process. Section 3.12.8 currently reads: The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies the session with that initiator. Suggested rewrite (melding current text w/John's rewrite): The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity). From owner-ips@ece.cmu.edu Mon Nov 19 06:40:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA25522 for ; Mon, 19 Nov 2001 06:40:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJA9TL18318 for ips-outgoing; Mon, 19 Nov 2001 05:09:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJA9Ql18313 for ; Mon, 19 Nov 2001 05:09:27 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA37528 for ; Mon, 19 Nov 2001 11:09:19 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAJA1Sm65536 for ; Mon, 19 Nov 2001 11:01:28 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 19 Nov 2001 12:01:30 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 19/11/2001 12:01:31, Serialize complete at 19/11/2001 12:01:31 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, I'll add it to the to-do list provided that we have a think called "iSCSI Target Network Entity"... Regards, Julo John Hufferd@IBMUS 19-11-01 11:02 To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted Julian, With regard to point 2.2.7, I now see the wordage is probably OK, I think I was thrown off by the lack of change bars, at that point, in your PDF version. However, the 3.12.8 Item (which is now at 3.12.7) we thought needed to have the last sentence changed from: "...this uniquely identifies the session with that initiator." to: "...this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity)." We spent a lot of time discussing this, and found there was a lot of confusion. So we wanted to unambiguously state the value of TSID did not have scope beyond the Specific Target, which might exist amoung others, in the same Network Entity. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted John, The change was already made when Andre asked for it. I've checked it and it did not disappear. The wording may be different but the names are required at each login. I've checked it now and it appears everywhere. If I omitted to specify it somewhere please let me know. Regards, Julo John Hufferd/San Jose/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 19-11-01 01:20 Please respond to John Hufferd To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI:new iSCSI draft 09.txt, some text omitted Julian, I think you perhaps overlooked, for the new draft, the wordage presented by Andre Asselin in the area of 2.2.7 and 3.12.8 shown below. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/05/2001 10:21:55 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: spec revs to make TargetName reqd on every connection thanks - julo Andre Asselin@IBMUS 05-11-01 22:55 To: Julian Satran/Haifa/IBM@IBMIL cc: John Hufferd/San Jose/IBM@IBMUS, Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: spec revs to make TargetName reqd on every connection Julian, Attached is an attempt to pull together all the required spec updates to make InitiatorName required on every login, TargetName required on every normal login, and clarify related text. Does this look good to everyone? Any places I missed? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Appendix D, TargetName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). To This key must be provided by the initiator of the TCP connection to the remote endpoint in the first login request if the initiator is not establishing a discovery session. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). Appendix D, InitiatorName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The Initiator key enables the initiator to identify itself to the remote endpoint. To This key must be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The InitiatorName key enables the initiator to identify itself to the remote endpoint. The current version of the 6th paragraph in chapter 5 reads: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. A suggested rewrite would be (building on the text suggested by Bob Russell): All initial Login requests MUST include the InitiatorName key=value pair. If the initial Login request is also a leading Login (TSID=0) and the new session is to be a discovery session, then the initial Login request MUST also include the SessionType=discovery key=value pair. If the initial Login request is a leading Login and the new session is to be a normal session, then the initial Login request MUST also include the TargetName key=value pair and MAY also include the SessionType=normal key=value pair. All initial Login requests that are not also a leading Login (TSID != 0) MUST include the TargetName key=value pair. Also, this text appears in 2.2.7: The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session. The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. A suggested rewrite would be: The initiator must present its iSCSI Initiator Name in the first login request. If the initiator is not establishing a discovery session (see 2.4), it also must present the iSCSI Target Name to which it wishes to connect in the first login request. The key "SessionType=discovery" is sent by the initiator on the Initial Login request to indicate a discovery session. See chapter 5 for a more detailed description of the Login process. Section 3.12.8 currently reads: The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies the session with that initiator. Suggested rewrite (melding current text w/John's rewrite): The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity). From owner-ips@ece.cmu.edu Mon Nov 19 10:21:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03867 for ; Mon, 19 Nov 2001 10:21:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJE1XN01001 for ips-outgoing; Mon, 19 Nov 2001 09:01:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailnpd.hcltech.com ([203.197.145.76]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAJE1Ql00985 for ; Mon, 19 Nov 2001 09:01:28 -0500 (EST) Received: from npd.hcltech.com (tskariah-pc.hcltech.com [192.168.100.72]) by mailnpd.hcltech.com (8.11.0/8.11.0) with ESMTP id fAJDuve23450; Mon, 19 Nov 2001 19:27:06 +0530 Message-ID: <3BF9112C.D094F063@npd.hcltech.com> Date: Mon, 19 Nov 2001 19:33:24 +0530 From: Thanu Skariah X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.2-2 i686) X-Accept-Language: en MIME-Version: 1.0 To: "Elliott, Robert" CC: ips@ece.cmu.edu Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics References: <78AF3C342AEAEF4BA33B35A8A15668C6014D6C9C@cceexc17.americas.cpqcorp.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi, There is one document on T11 which deals with the mapping required below (though not for all NAA types). Check out ftp://209.26.30.221/t11/pub/fc/fs/01-531v2.pdf Thanks, Thanu "Elliott, Robert" wrote: > Correction - the company_id is 24 bits, not 20 bits. > > EUI-64 = (MSB) { 24-bit company_id, > 40-bit vendor specified } (LSB) > > FC WWN = (MSB) { 4-bit NAA field, > 24-bit company_id, > 36-bit vendor specified } (LSB) > (for the IEEE Registered NAA type) > > > -----Original Message----- > > From: Elliott, Robert > > Sent: Friday, November 16, 2001 9:33 AM > > To: ips@ece.cmu.edu > > Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics > > > > > > Beware than a FC WWN is not the same as an EUI-64. They're both based > > on the IEEE OUI/company_id, but the FC WWN uses the 4 most significant > > bits for a "Name Address Authority" field and has 4 fewer > > bits available > > in the vendor specified portion. > > > > EUI-64 = (MSB) { 20-bit company_id, > > 44-bit vendor specified } (LSB) > > > > FC WWN = (MSB) { 4-bit NAA field, > > 20-bit company_id, > > 40-bit vendor specified } (LSB) > > (for the IEEE Registered NAA type) > > > > --- > > Rob Elliott, Compaq Server Storage > > Robert.Elliott@compaq.com > From owner-ips@ece.cmu.edu Mon Nov 19 12:05:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09985 for ; Mon, 19 Nov 2001 12:05:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJFu8M09977 for ips-outgoing; Mon, 19 Nov 2001 10:56:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJFu6l09973 for ; Mon, 19 Nov 2001 10:56:06 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA54838 for ; Mon, 19 Nov 2001 16:55:58 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAJFu6m87242 for ; Mon, 19 Nov 2001 16:56:06 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: iSCSI errata X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Mon, 19 Nov 2001 17:56:06 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 19/11/2001 17:56:09, Serialize complete at 19/11/2001 17:56:09 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, The txt and pdf files or 09 contained by error two leftovers from the time the digests were part of the security stage negotiation- namely the second bullet of the first list on page 120 (5.2) and the text paragraph that follows the list (the bullet starts with "-PDU Integrity" and the paragraph with "If security"). Those have to be removed (have been already on the copies on my site. Sorry for the inconvenience. Julo From owner-ips@ece.cmu.edu Mon Nov 19 13:43:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16338 for ; Mon, 19 Nov 2001 13:43:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJHorD19557 for ips-outgoing; Mon, 19 Nov 2001 12:50:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJHoql19552 for ; Mon, 19 Nov 2001 12:50:52 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 19 Nov 2001 12:53:15 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E029371BF@CORPMX14> From: Black_David@emc.com To: nitin.dhingra@dcmtech.co.in, ips@ece.cmu.edu Subject: RE: iSCSI: Function 4 Of Task Mgmt Command Date: Mon, 19 Nov 2001 12:40:42 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk By means not specified here. Clear Task Set is a defined SCSI task management function - see Section 6 of SAM-2 - and hence this is a SCSI level decision. SCSI has had to deal with ill-behaved devices that have occasionally needed hard whacks to get them to behave (e.g., start paying attention to the Initiator again). Clear Task Set is among the "hard whacks" available for this purpose. iSCSI is required to provide the means for Clear Task Set to work if a SCSI Initiator chooses to use it. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Nitin Dhingra [mailto:nitin.dhingra@dcmtech.co.in] > Sent: Saturday, November 17, 2001 8:38 AM > To: 'ips@ece.cmu.edu' > Subject: iSCSI: Function 4 Of Task Mgmt Command > > > How does the initiator decide to choose Task Management Function 4 > i.e. Clear Task Set - Aborts all Tasks (from all initiators) > for the Logical > Unit? > > From owner-ips@ece.cmu.edu Mon Nov 19 13:43:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16371 for ; Mon, 19 Nov 2001 13:43:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJHjT819139 for ips-outgoing; Mon, 19 Nov 2001 12:45:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJHjRl19135 for ; Mon, 19 Nov 2001 12:45:27 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 19 Nov 2001 12:47:45 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E029371BE@CORPMX14> From: Black_David@emc.com To: VAHUJA@aol.com Cc: ips@ece.cmu.edu Subject: RE: SRP vs PKI for authentication Date: Mon, 19 Nov 2001 12:35:12 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Such is life. The alternative of allowing implementers to choose to implement one of SRP or PKI as the "MUST" mechanism results in an implementation that chooses SRP and doesn't implement PKI failing to interoperate with an implementation that chooses PKI and doesn't implement SRP. This is not acceptable, hence there has to be at least one "MUST" mechanism that everyone has to implement. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: VAHUJA@aol.com [mailto:VAHUJA@aol.com] > Sent: Friday, November 16, 2001 9:45 PM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: Re: SRP vs PKI for authentication > > > David, > I understand your and Ofer's views. But lets view from > another point. For the > same storage network, we may potentially have PKI as the choice for > authentication for one part, and SRP for another. There are > ways to simplify > built-in PKI for storage networks as being applied for FC > security (I expect > you dont need an RA or a CA, and the trust hierarchy is left to the > customer). So for those networks that want to ONLY > authenticate entities in > storage networks, we may potentially have two schemes that "MUST" be > implemented. > From owner-ips@ece.cmu.edu Mon Nov 19 14:48:31 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA20424 for ; Mon, 19 Nov 2001 14:48:31 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJIiGE24057 for ips-outgoing; Mon, 19 Nov 2001 13:44:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJIiEl24048 for ; Mon, 19 Nov 2001 13:44:14 -0500 (EST) Received: by ariel.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Sun, 18 Nov 2001 20:56:30 -0800 Message-ID: From: Charles Monia To: "David Black (E-mail)" Cc: "Ips (E-mail)" Subject: RE:iFCP: iFCP -06 comments Date: Sun, 18 Nov 2001 20:56:24 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi David: Thanks for reviewing the specification. The following are responses to selected comments. We (the co-authors) are in agreement with all others and will change the spec accordingly. > 9.2.1 - The last paragraph on what to do on loss of > synchronization seems > inadequate. It's current state appears to allow stale frame > propagation > by a compliant implementation. Was this the intended outcome? > The issue is what to do if a gateway that was enforcing stale frame detection enters the unsynchronized state. The original intent was to make the gateway response discretionary. i.e.. Either continue operation but suspend enforcement or terminate operation. For consistency, we propose that an implementation enforcing the detection of stale frames should always do the latter (terminate operation). Otherwise, of course, stale frame detection would not have been enabled to begin with. > 10.4 b) What happens when the broadcast frame exceeds the MTU > size? This > seems to result in a rather unreliable broadcast service, as > all of the UDP > datagrams could well be dropped, consistently and repeatedly for large > enough broadcast frames. We will rewrite the section to: a) Specify the use of source fragmentation if the datagram exceeds the PMTU. b) Require that the receiving broadcast server be capable of accepting the maximum FC frame size. As a point of information, the expected frame sizes are small. In FC, factors limiting broadcast frame size are the fabric and N_PORT MTUs. Fibre Channel requires an N_PORT MTU to be at least 256 bytes. Therefore, for a broadcast frame to be receivable by all N_PORTs, the frame size should not exceed that limit. > 7.3.7 - Why is PLOGI in a subsection of 7.3 when it's not augmented? > iFCP defines an augmented ELS as any ELS requiring gateway intervention, whether or not there's supplemental information to be processed. > 7.3.8 - Three translation types are shown for the REC > Exchange originator. > Again, I think this should be type 1. Variants of this > problem are also > present in all of 7.3.9 through 7.3.15. > The issue relates to the manner in which the following ELSs are augmented: 7.3.10, 7.3.11 -- Read Exchange Status Block (RES) and RES accept 7.3.12 -- Read Link Error Status (RLS) 7.3.13 -- Read Sequence Status Block (RSS) 7.13.14 -- Reinstate Recovery Qualifier (RRQ) 7.13.15 -- Request Sequence Initiative (RSI) (REC has since been deleted from FC-FS and will be removed from the iFCP spec.) As we interpret FC-FS, RES, RLS, and RSS contain N_PORT identifiers in the payload which may not necessarily correspond to those of the frame source or destination. In the case of RES, for example, FC-FS states: "The RES ELS Request Sequence requests an N_Port to return the Exchange Status Block as defined in 17.8.1 for the RX_ID or OX_ID originated by the S_ID specified in the Payload of this Request Sequence..." To us, it appears that nothing in the spec requires the S_ID to correspond to that of the requesting or responding N_PORTs. We will request clarification on this point from T11. For RSI (Request Sequence Initiative) and RRQ (Reinstate Recovery Qualifier), the parameter in question is the S_ID of the exchange originator, which may correspond to either the ELS originator or recipient, hence we believe a translation type of 1 (frame originator) or 2 (frame recipient) is correct. > 12.2 - Please delete d) as MPLS is *NOT* a QoS technology. After a discussion of over-provisioning, IntServ and DiffServ, we feel MPLS is appropriate since it touches on traffic engineering. For instance, one may load a MPLS forwarding equivalence class (FEC) with QoS class significance, in addition to other considerations such as protection and diversity for the given path. The complementarity and compatibility of MPLS with, say, DiffServ is explored in draft-ietf-mpls-diff-ext-09.txt (wherein the PHB bits are copied to the EXP bits of the MPLS shim header). > In addition, > the entire paragraph after d) appears to have very little content, and > I'm not at all sure about the value of discussion SLAs here. While we are willing to remove this, we believe there is value in taking the reader through a QoS use case for iFCP. If possible, we'd like to salvage this material and would appreciate suggestions on enhancing or presenting it more appropriately. Charles Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, November 13, 2001 5:22 PM > To: ips@ece.cmu.edu > Subject: iFCP -06 comments > > > At the request of the iFCP authors, I've reviewed the -06 version of > the iFCP draft. Here are my comments divided into Major, Minor and/or > Editorial, and Formatting categories. > > ----- Major ----- > > 5.3.2/5.3.2.1/5.3.2.1.1 - These are written as descriptions > of a possible > implementation. That's fine, but it's necessary to be crisp > about what > the requirements are on an implementation that may not go > about this in > the fashion envisioned here. Please go over these sections, > figure out > the requirements on an arbitrary implementation and put in > the requisite > MUSTs and SHOULDs in addition to the current description of how things > could be implemented. I believe the functional equivalent of the > translation > table and its entries MUST exist, so that's somewhere to start from. > Other areas of the document have lesser versions of this > problem, so please > go over the entire document and check it for this. I also > saw a number > of places with lower case requirements words (e.g., "shall") > that probably > need to be upper case. > > 6.2.1 > An N_PORT is identified by its network address consisting of: > > a) The N_PORT I/D assigned by the gateway to which > the N_PORT is > locally attached and > > b) The IP address of the gateway's iFCP Portal. > > Use of an IP address to identify a remote gateway needs to be > reconsidered. > Please add something to allow multiple iFCP implementations > to exist at > different TCP ports on the same IP address (or explain why this has to > be prohibited). This is strongly related to the NAPT issues that the > FCIP folks are in the midst of working through. > > 6.2.2.2 - This section carries a strong risk of over-specification. > As work on TCP evolves, there's a strong risk of entries in this > table conflicting with the then-applicable RFCs. The ground > rule should > be to only specify something here when it has serious consequences > (i.e., there are potentially very bad effects from ignoring > the SHOULD). > I think I can accept this argument for the first 4 lines in the table, > but I think the last three should be removed as I don't think > they're crucial to iFCP per se (the discussion of them seems to > amount to "good things to do in general", and there's a strong risk > of further requirements changes/tweaks in this area). Keep in mind > that I'm one of the co-authors of RFC 3168 (ECN). Also, I think > keeping the "should"s and "should not"s lower case (so that they're > advice to implementers rather than protocol requirements) is a good > thing to do in this section. > > 6.2.2.3 - I think this Section needs to go away. 6.2.2.3.1 > is essentially > repeating information about TCP implementations in general > that is already > to be found elsewhere and should be removed. 6.2.2.3.2 > should be moved > to 6.2.3.2. > > I think the order of Sections 7 and 8 should be reversed, as the TCP > connection control material is needed to understand how iFCP functions > before discussing the interesting things it has to do to some ELSs. > > 9.2.1 - The last paragraph on what to do on loss of > synchronization seems > inadequate. It's current state appears to allow stale frame > propagation > by a compliant implementation. Was this the intended outcome? > > 10.4 b) - How can one be sure that the local gateway knows > about all the > remote gateways? I suspect this involves iSNS, and needs to > be explained. > > 10.4 b) What happens when the broadcast frame exceeds the MTU > size? This > seems to result in a rather unreliable broadcast service, as > all of the UDP > datagrams could well be dropped, consistently and repeatedly for large > enough broadcast frames. > > > ---- Minor and/or Editorial ----- > > 4.4 a) It's not exactly correct to describe the Node WWN > as being an identifier for the device. While that > was the original intent, it isn't always implemented > that way. In practice, I don't think Node WWNs are > used for much, and that might be worth noting. > > 4.7.1 - It might be worth describing how Area ID and Port ID > are assigned when an FC-AL loop is attached to a switch > to give the reader a slightly better feel for this > (Area ID is assigned to switch port and Port ID is the > loop port identifier). > > 4.8 - Track state of things in T11 in determining whether > its appropriate to add Class 4 and 6 to this description. > From FCIP discussions, it sounds like at least Class 4 > should be added. > > 4.9 - Might want to add a sentence to make it clear that these > login processes occur at FC-2, and ULP (FC-4) protocol > setup takes place over the established FC-2 N_Port to N_Port > session in whatever manner the ULP desires > > Figure 7 - I think "IP network" would be a better term for what's > below the line than "IP fabric". I understand what an > "iFCP fabric" is, but I'm not at all sure about an "IP fabric". > > 5.2.1 - Might want to add a sentence indicating how an FC device > discovers that Class 1 isn't supported (gets told by the > iFCP gateway on Fabric Login). > > p. 20 > All iFCP implementations MUST support operation in address > translation mode. Support for address transparent > mode is optional > > "optional" should be all upper-case. > > The implementation MUST NOT allow the mode to be > changed after iFCP sessions have been established. > > So, the mode can be changed while a session is being > established? I don't > think so and suggest that the above wording be changed. One > possibility > would be to prohibit a mode change while any device is logged into it > via FLOGI. > > 5.3.1 - Somewhere near the end of this section please say that FC > frame CRCs have to be regenerated as a consequence of performing > address translation. > > Both 5.3 and 5.3.1 contain some rationale text about the > differences between > address-transparent and address-translation mode and why one > might select > one or the other that might be better separated out from the > description > of how iFCP works in these modes. In particular, iSNS pops > up without any > introduction in 5.3.1 a) - this text really needs to come > after a discussion > of iSNS and its responsibilities for/interaction with iFCP. > > > 5.3.1.1 - The whole discussion of domain ID assignment is > rather imprecise. > Please put in a "MUST" requirement for unique domain IDs, and > indicate that > iSNS can do this (with a specific description of how) in > addition to manual > assignment. > > 5.3.1.2 - Please upper-case "shall" in the first sentence here. Also > applies > to 5.3.2.3. > > 6.2.2.1 - For a), please indicate how the gateway knows which remote > gateway to use and what it's address is. This involves translation > table entries that were initialized by iSNS replies. > > 6.2.3 - Please be clear about what exactly is being > terminated or aborted. > I think the iFCP session (TCP connection) between gateways is > what's involved, but there's enough FC mechanism discussion here > to be unclear. > > 6.3 - I guess this reference to RFC1323 is ok, although it strikes me > as superfluous. Please indicate that it's Appendix B of RFC 1323. > > 6.4.4 - State that the FC CRC MUST be recalculated due to the > address translation. > > 7.3.1 - Two translation types are shown for the ABTX Exchange > originator. > I think this should be Type 1. > > 7.3.5 - Translation table is incomplete for FARP-REQ. > > 7.3.7 - Why is PLOGI in a subsection of 7.3 when it's not augmented? > > 7.3.8 - Three translation types are shown for the REC > Exchange originator. > Again, I think this should be type 1. Variants of this > problem are also > present in all of 7.3.9 through 7.3.15. > > 7.4 - Table 4 consists entirely of "n" and "M" entries, so delete the > explanation of the unused "y" entry. > > 8.1 - For clarity indicate that the connection handle is > needed to unbind > connections (yes, I know this is discussed in the next section). > > 11.3.1 - Please remove the "MUST implement" for DES (it's ok > to make DES > OPTIONAL), and think about rewording the "As stated in" statements, as > we are overriding some of the requirements in some of the > reference RFCs. > > 11.3.1.1 - This is ok in a working version of the draft, but > will vanish > in the final version (ditto the subject to availability of an RFC > language about AES earlier) because either we will make a normative > reference to the AES RFCs or RFC-to-be, or we will delete requirements > for AES. > > 11.3.2 > If, however, the TCP session is > maintained, then a Phase 2 delete message shall > trigger a new Quick > Mode exchange. > > Probably not a good idea. The issue here is that some hardware crypto > accelerators only have room for a fixed number of phase 2 SAs > and hence > delete old ones to make room for new ones (ideally deleting the least > recently used, but ...). Triggering a new Quick Mode immediately in > response to any delete of an SA for an open TCP connection could > thrash the SA state resources in such an accelerator. Triggering > the new Quick Mode based on traffic sent over the SA should > work better. > > 12.2 - Please delete d) as MPLS is *NOT* a QoS technology. > In addition, > the entire paragraph after d) appears to have very little content, and > I'm not at all sure about the value of discussion SLAs here. The > discussion of [00-603] is also questionable. > > Appendix B - Is there an external version of this material that could > be referenced rather than including it here? > > ----- Formatting ----- > > There are a bunch of places where MS Word > has inserted non-standard MS punctuation characters > (mostly a u with a circumflex instead of a hyphen) > Turn off Smart Quotes and Auto Correct and correct these. > > 6.3 has a formatting problem - probably an MS Word style > that should not have been used. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ece.cmu.edu Mon Nov 19 14:55:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA20930 for ; Mon, 19 Nov 2001 14:55:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJIkTi24273 for ips-outgoing; Mon, 19 Nov 2001 13:46:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJIkPl24254 for ; Mon, 19 Nov 2001 13:46:25 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA26420 for ; Mon, 19 Nov 2001 13:43:42 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAJIkIS198208 for ; Mon, 19 Nov 2001 11:46:18 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Mon, 19 Nov 2001 10:45:11 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/19/2001 11:46:19 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk The Network Enity is called out in the figure in section 2.5 . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/19/2001 02:01:30 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted John, I'll add it to the to-do list provided that we have a think called "iSCSI Target Network Entity"... Regards, Julo John Hufferd@IBMUS 19-11-01 11:02 To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted Julian, With regard to point 2.2.7, I now see the wordage is probably OK, I think I was thrown off by the lack of change bars, at that point, in your PDF version. However, the 3.12.8 Item (which is now at 3.12.7) we thought needed to have the last sentence changed from: "...this uniquely identifies the session with that initiator." to: "...this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity)." We spent a lot of time discussing this, and found there was a lot of confusion. So we wanted to unambiguously state the value of TSID did not have scope beyond the Specific Target, which might exist amoung others, in the same Network Entity. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI:new iSCSI draft 09.txt, some text omitted John, The change was already made when Andre asked for it. I've checked it and it did not disappear. The wording may be different but the names are required at each login. I've checked it now and it appears everywhere. If I omitted to specify it somewhere please let me know. Regards, Julo John Hufferd/San Jose/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 19-11-01 01:20 Please respond to John Hufferd To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: iSCSI:new iSCSI draft 09.txt, some text omitted Julian, I think you perhaps overlooked, for the new draft, the wordage presented by Andre Asselin in the area of 2.2.7 and 3.12.8 shown below. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/05/2001 10:21:55 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: spec revs to make TargetName reqd on every connection thanks - julo Andre Asselin@IBMUS 05-11-01 22:55 To: Julian Satran/Haifa/IBM@IBMIL cc: John Hufferd/San Jose/IBM@IBMUS, Jim Hafner/Almaden/IBM@IBMUS, ips@ece.cmu.edu From: Andre Asselin/Raleigh/IBM@IBMUS Subject: spec revs to make TargetName reqd on every connection Julian, Attached is an attempt to pull together all the required spec updates to make InitiatorName required on every login, TargetName required on every normal login, and clarify related text. Does this look good to everyone? Any places I missed? Andre Asselin IBM ServeRAID Software Development Research Triangle Park, NC Appendix D, TargetName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint before the end of the login phase. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). To This key must be provided by the initiator of the TCP connection to the remote endpoint in the first login request if the initiator is not establishing a discovery session. The iSCSI Target Name specifies the worldwide unique name of the target. The TargetName key may also be returned by the "SendTargets" text request (and that is its only use when issued by a target). Appendix D, InitiatorName option: Remove "LO" from Use. Change This key MUST be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The Initiator key enables the initiator to identify itself to the remote endpoint. To This key must be provided by the initiator of the TCP connection to the remote endpoint at the first Login of login phase for every connection. The InitiatorName key enables the initiator to identify itself to the remote endpoint. The current version of the 6th paragraph in chapter 5 reads: The initial Login request of the first connection of a session (leading login) MUST include the InitiatorName key=value pair. The leading Login request MAY also include the SessionType key=value pair in which case if the SessionType is not "discovery" then the leading Login Request MUST also include the key=value pair TargetName. A suggested rewrite would be (building on the text suggested by Bob Russell): All initial Login requests MUST include the InitiatorName key=value pair. If the initial Login request is also a leading Login (TSID=0) and the new session is to be a discovery session, then the initial Login request MUST also include the SessionType=discovery key=value pair. If the initial Login request is a leading Login and the new session is to be a normal session, then the initial Login request MUST also include the TargetName key=value pair and MAY also include the SessionType=normal key=value pair. All initial Login requests that are not also a leading Login (TSID != 0) MUST include the TargetName key=value pair. Also, this text appears in 2.2.7: The initiator MUST present both its iSCSI Initiator Name and the iSCSI Target Name to which it wishes to connect in the first login request of a new session. The only exception is if a discovery session (see 2.4) is to be established; the iSCSI Initiator Name is still required, but the iSCSI Target Name may be ignored. The key "SessionType=discovery" is sent by the initiator at login to indicate a discovery session. A suggested rewrite would be: The initiator must present its iSCSI Initiator Name in the first login request. If the initiator is not establishing a discovery session (see 2.4), it also must present the iSCSI Target Name to which it wishes to connect in the first login request. The key "SessionType=discovery" is sent by the initiator on the Initial Login request to indicate a discovery session. See chapter 5 for a more detailed description of the Login process. Section 3.12.8 currently reads: The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies the session with that initiator. Suggested rewrite (melding current text w/John's rewrite): The TSID is the target assigned component of the session identifier (SSID). Together with the ISID provided by the initiator, this uniquely identifies a session from that specific target to that specific initiator. That is, the TSID is a unique value within the scope of a specific target (not necessarily unique within the iSCSI Target Network Entity). From owner-ips@ece.cmu.edu Mon Nov 19 22:15:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA10773 for ; Mon, 19 Nov 2001 22:15:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAJMA1311720 for ips-outgoing; Mon, 19 Nov 2001 17:10:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from med.corp.rhapsodynetworks.com (64-160-62-201.rhapsodynetworks.com [64.160.62.201] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAJLrxl10389 for ; Mon, 19 Nov 2001 16:54:05 -0500 (EST) Received: by med.corp.rhapsodynetworks.com with Internet Mail Service (5.5.2653.19) id ; Mon, 19 Nov 2001 13:53:41 -0800 Message-ID: <45BEF1D68145D51186C100B0D0AABE6543E0DF@med.corp.rhapsodynetworks.com> From: Venkat Rangan To: "'Black_David@emc.com'" , Venkat Rangan , ips@ece.cmu.edu Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Date: Mon, 19 Nov 2001 13:53:31 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, > The existing text in the IPS security draft on pre-shared keys and IKE > authentication modes is fine in the absence of the recent WWN short > frame addition. That addition changes the security properties of FCIP > by introducing an inband authentication/authorization for which it > is necessary to provide security. Doing so without using an inband > authentication protocol impacts group pre-shared keys and other things. The main intent of WWN short frame is merely a mechanism to identify an FCIP entity peer and its properties not an inband authentication/authorization mechanism. The assumption here is that as long as IPSec performs a source IP address validation against IKE address identifiers, a receiving FCIP entity is guaranteed that the short frame was sent by the source referred to in the source IP address. If it makes through the authentication (ESP) process, the WWN can be processed for FCIP link identification. Even when an external IPSec gateway is in use, that gateway has already established the necessary IKE Phase 1 so any Phase 2 SAs over which the reveived short frames arrive are also secured from a spoof. If group pre-shared keys are in use, and one member of the group impersonates another member and sends an attack FCIP Short Frame, that would create/add that connection at the receiving FCIP entity. However, this attack is probably no different from a man-in-the-middle attack even in the absence of the FCIP Short Frame, when the group pre-shared key has been compromised and an attacker has gained it. For FCIP implementations (unlike iSCSI) we do not expect large groups being handed out "generic" group pre-shared keys. > In addition, that assumption makes IKE and ESP cryptographic > integrity at least "SHOULD use" for FCIP, and I can't promise that > the Security ADs will settle for "SHOULD use" as opposed to "MUST > use". The reason for this change from the "MAY use" that applied > prior to the introduction of the WWN short frame is that the > authentication/authorization performed by that short frame is a > class of function that is far more important, expected, and widely > used than cryptographic integrity - the assumption uses cryptographic > integrity to secure a mandatory authentication mechanism and hence > increases the requirement for cryptographic integrity. If a receiver does not trust the sender of an FCIP Short Frame, it can choose to use IKE and ESP cryptographic integrity. If it believes that another IPSec gateway has established secure connections to the peer, it processes the FCIP Short Frame, much the same way as any other FCIP encapsulated frame. > And as things currently stand, the "administrative establishment" of > that association will need to be done not only at the sender of the > short frame, but also at the recipient. When IKE is in use, both > establishment of the association and the check at the receiver (IKE > identity for IPsec SA and WWN in short frame that arrived on that SA > are associated) will need to be "MUST"s. At the recipient, one could state that a validation of the IKE address identifier (the IP address of the source FCIP entity) SHOULD be performed against the set of IP addresses that are allowed to send the WWN specified in the short frame. If this makes it impractical to use group pre-shared keys, that deployment can always use non-group pre-shared keys or use public key certs. Regards, Venkat Rangan Rhapsody Networks Inc. http://www.rhapsodynetworks.com -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Tuesday, November 13, 2001 9:55 AM To: vrangan@rhapsodynetworks.com; ips@ece.cmu.edu Subject: RE: FCIP: NAPTs Solution Proposal (issue from Irvine, CA Interim meeting) Venkat, Let me try to summarize and explain without quoting your entire message: -- Pre-shared keys The existing text in the IPS security draft on pre-shared keys and IKE authentication modes is fine in the absence of the recent WWN short frame addition. That addition changes the security properties of FCIP by introducing an inband authentication/authorization for which it is necessary to provide security. Doing so without using an inband authentication protocol impacts group pre-shared keys and other things. -- WWN short frame and administration > "The usage of SLPv2 by FCIP is described in [64]. FCIP Entities assume > that once the IKE identity of a peer is established, the FCIP Entity > Name carried in FCIP Short Frame is also implicity accepted as the > authenticated peer. Any such association between the IKE identity and > the FCIP Entitiy Name is administratively established." > Do you see any further clarification required in this area? > Also, is there > any conflict with the FCIP Short Frame proposal (the NAPTs) solution? Work is definitely required here, because that short frame is serving an authentication/authorization purpose and hence the means need to be provided to adequately secure it. The assumption in the second sentence above isn't sufficient because it opens up nasty attacks including the denial of service ones I described earlier. In addition, that assumption makes IKE and ESP cryptographic integrity at least "SHOULD use" for FCIP, and I can't promise that the Security ADs will settle for "SHOULD use" as opposed to "MUST use". The reason for this change from the "MAY use" that applied prior to the introduction of the WWN short frame is that the authentication/authorization performed by that short frame is a class of function that is far more important, expected, and widely used than cryptographic integrity - the assumption uses cryptographic integrity to secure a mandatory authentication mechanism and hence increases the requirement for cryptographic integrity. And as things currently stand, the "administrative establishment" of that association will need to be done not only at the sender of the short frame, but also at the recipient. When IKE is in use, both establishment of the association and the check at the receiver (IKE identity for IPsec SA and WWN in short frame that arrived on that SA are associated) will need to be "MUST"s. Group pre-shared keys make these sort of checks difficult to specify and use properly - the fastest way to resolve this is to make group pre-shared keys "MUST NOT use" for FCIP. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Nov 20 12:44:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA23011 for ; Tue, 20 Nov 2001 12:44:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAKGk6R02279 for ips-outgoing; Tue, 20 Nov 2001 11:46:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAKGk5l02274 for ; Tue, 20 Nov 2001 11:46:05 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id ; Tue, 20 Nov 2001 11:42:15 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E029371CF@CORPMX14> From: Black_David@emc.com To: vrangan@rhapsodynetworks.com, ips@ece.cmu.edu Subject: FCIP: WWN short frame and IPsec Date: Tue, 20 Nov 2001 11:35:52 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk [Subject line changed to match topic under discussion.] Venkat, > > The existing text in the IPS security draft on pre-shared keys and IKE > > authentication modes is fine in the absence of the recent WWN short > > frame addition. That addition changes the security properties of FCIP > > by introducing an inband authentication/authorization for which it > > is necessary to provide security. Doing so without using an inband > > authentication protocol impacts group pre-shared keys and other things. > > The main intent of WWN short frame is merely a mechanism to identify an FCIP > entity peer and its properties not an inband authentication/authorization > mechanism. While that may have been the intent, the WWN is serving as inband authentication/authorization (e.g., to allow a new TCP connection to join the set of connections that make up an FCIP inter-switch link - see explanation in my previous message about how not securing this makes denial of service attacks possible). > The assumption here is that as long as IPSec performs a source IP address > validation against IKE address identifiers, a receiving FCIP entity is > guaranteed that the short frame was sent by the source referred to in the > source IP address. If it makes through the authentication (ESP) process, > the WWN can be processed for FCIP link identification. Even when an external > IPSec gateway is in use, that gateway has already established the necessary > IKE Phase 1 so any Phase 2 SAs over which the received short frames arrive > are also secured from a spoof. In other words, IPsec is being used to secure the short WWN frame, which is always present. That has at least two consequences: (1) IPsec has to be at least "SHOULD use", not "MAY use", and I can't promise that this won't turn into "MUST use". (2) The identity presented in the IKE authentication for the Phase 1 SA MUST be checked against the WWN in the short frame to ensure that the WWN is presented by an IKE identity that is authorized to do so. It's not acceptable to assume that these two would match if they were checked. > If group pre-shared keys are in use, and one member of the group impersonates > another member and sends an attack FCIP Short Frame, that would create/add > that connection at the receiving FCIP entity. However, this attack is probably > no different from a man-in-the-middle attack even in the absence of the > FCIP Short Frame, when the group pre-shared key has been compromised and an > attacker has gained it. For FCIP implementations (unlike iSCSI) we do not > expect large groups being handed out "generic" group pre-shared keys. It's not that simple. One of the things that concerns me here is that improved FC security (e.g., the SLAP work mentioned by Bob) will not be able to check the WWN in the short frame on TCP connections other than the first. The fastest way to resolve this issue is to make group preshared keys "SHOULD NOT use" or "MUST NOT use" (both of which are stronger than "we do not expect"). Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Nov 20 12:49:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA23264 for ; Tue, 20 Nov 2001 12:49:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAKGTiE00955 for ips-outgoing; Tue, 20 Nov 2001 11:29:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAKGThl00951 for ; Tue, 20 Nov 2001 11:29:43 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 20 Nov 2001 11:29:37 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E029371CE@CORPMX14> From: Black_David@emc.com To: cmonia@NishanSystems.com Cc: ips@ece.cmu.edu Subject: RE: iFCP: iFCP -06 comments Date: Tue, 20 Nov 2001 11:19:20 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, > > 9.2.1 - The last paragraph on what to do on loss of synchronization seems > > inadequate. Its current state appears to allow stale frame propagation > > by a compliant implementation. Was this the intended outcome? > > > The issue is what to do if a gateway that was enforcing stale frame > detection enters the unsynchronized state. The original intent was to make > the gateway response discretionary. i.e.. Either continue operation but > suspend enforcement or terminate operation. > > For consistency, we propose that an implementation enforcing the detection > of stale frames should always do the latter (terminate operation). > Otherwise, of course, stale frame detection would not have been enabled to > begin with. Ok, also please make sure that clear guidance and requirements are provided for when to use stale frame detection - I think this is at least a SHOULD, and we'll need to make this consistent among FCIP and iFCP to the extent possible. > > 10.4 b) What happens when the broadcast frame exceeds the MTU size? This > > seems to result in a rather unreliable broadcast service, as all of the UDP > > datagrams could well be dropped, consistently and repeatedly for large > > enough broadcast frames. > > We will rewrite the section to: > > a) Specify the use of source fragmentation if the datagram exceeds the PMTU. Details will need to be supplied, as the large FC frame is being fragmented into UDP datagrams, and hence the UDP datagrams each need to contain information about how to reassemble them into an FC frame (unlike TCP where one can simply follow the bytestream delivered by TCP). Also, please add a discussion about the different levels of reliability assumed/provided by Fibre Channel vs. UDP - FC expects that all frames will be delivered in the absence of unusual/extraordinary circumstances, whereas UDP is best effort, and the network is free to drop UDP datagrams if delivering them would be seriously inconvenient (e.g., router or switch ran out of buffer space). The latter discussion is needed to check whether UDP is an acceptable implementation of FC broadcast. > b) Require that the receiving broadcast server be capable of accepting the > maximum FC frame size. Fine. > > 7.3.7 - Why is PLOGI in a subsection of 7.3 when it's not augmented? > > > iFCP defines an augmented ELS as any ELS requiring gateway intervention, > whether or not there's supplemental information to be processed. This strikes me as excessive consistency and a poor definition of augmentation. PLOGI is already discussed in the material on iFCP session setup in Section 5. A note at the top of Section 7.3 pointing back to that discussion of special handling of PLOGI is sufficient and may allow the PLOGI frame diagram to be deleted (if it's not deleted, moving it to Section 5 would be a good idea). > > 7.3.8 - Three translation types are shown for the REC Exchange originator. > > Again, I think this should be type 1. Variants of this problem are also > > present in all of 7.3.9 through 7.3.15. > > > The issue relates to the manner in which the following ELSs > are augmented: > > 7.3.10, 7.3.11 -- Read Exchange Status Block (RES) and RES accept > 7.3.12 -- Read Link Error Status (RLS) > 7.3.13 -- Read Sequence Status Block (RSS) > 7.13.14 -- Reinstate Recovery Qualifier (RRQ) > 7.13.15 -- Request Sequence Initiative (RSI) > > (REC has since been deleted from FC-FS and will be removed from the iFCP > spec.) > > As we interpret FC-FS, RES, RLS, and RSS contain N_PORT identifiers in the > payload which may not necessarily correspond to those of the frame source or > destination. In the case of RES, for example, FC-FS states: > > "The RES ELS Request Sequence requests an N_Port to return the Exchange > Status Block as defined in 17.8.1 for the RX_ID or OX_ID originated by the > S_ID specified in the Payload of this Request Sequence..." > > To us, it appears that nothing in the spec requires the S_ID to correspond > to that of the requesting or responding N_PORTs. We will request > clarification on this point from T11. It might also be useful to ask Bob Snively directly, as the most important ULP for these ELSs is FCP-2. > For RSI (Request Sequence Initiative) and RRQ (Reinstate Recovery > Qualifier), the parameter in question is the S_ID of the exchange > originator, which may correspond to either the ELS originator or recipient, > hence we believe a translation type of 1 (frame originator) or 2 (frame > recipient) is correct. In any case, it is necessary to spell out the rules/procedures that the gateway originating the augmented ELS MUST follow in order to determine which type of translation to use. The current text leaves it to the gateway implementer's discretion which is almost certain to result in incorrect choices being made. > > 12.2 - Please delete d) as MPLS is *NOT* a QoS technology. > > After a discussion of over-provisioning, IntServ and DiffServ, we feel MPLS > is appropriate since it touches on traffic engineering. For instance, one > may load a MPLS forwarding equivalence class (FEC) with QoS class > significance, in addition to other considerations such as protection and > diversity for the given path. The complementarity and compatibility of MPLS > with, say, DiffServ is explored in draft-ietf-mpls-diff-ext-09.txt (wherein > the PHB bits are copied to the EXP bits of the MPLS shim header). The text starting with "For instance" above is fine. Insert that discussion, remove the existing MPLS RFC reference and replace it with a reference to that draft (which has been approved by the IESG to become an RFC). > > In addition, > > the entire paragraph after d) appears to have very little content, and > > I'm not at all sure about the value of discussion SLAs here. > > While we are willing to remove this, we believe there is value in taking the > reader through a QoS use case for iFCP. > > If possible, we'd like to salvage this material and would appreciate > suggestions on enhancing or presenting it more appropriately. I offer the following guidance ... - Discussing the QoS requirements of iFCP and how existing IETF mechanisms can be used to meet them is fine and encouraged. Explaining how and why Fibre Channel traffic may need low jitter, high bandwidth, etc. from an IP network is relevant. - Discussion of SLAs and related contractual arrangements is not a good idea. The IETF considers an SLA to be a contractual vehicle that will not be specified in any RFC. The terms closest to what you're trying to convey are Service Level Specification and Traffic Conditioning Specification -- see Section 2 of draft-ietf-diffserv-new-terms-06.txt . I'm also interested in the outcome/resolution of the 6.2.1 issue: > > Use of an IP address to identify a remote gateway needs to be reconsidered. > > Please add something to allow multiple iFCP implementations to exist at > > different TCP ports on the same IP address (or explain why this has to > > be prohibited). This is strongly related to the NAPT issues that the > > FCIP folks are in the midst of working through. Please inform the list of how the authors propose to resolve this. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Nov 20 17:34:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA09454 for ; Tue, 20 Nov 2001 17:34:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAKLWFp25458 for ips-outgoing; Tue, 20 Nov 2001 16:32:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAKLWDl25451 for ; Tue, 20 Nov 2001 16:32:13 -0500 (EST) Received: from xparelay1.corp.hp.com (unknown [15.58.136.173]) by palrel10.hp.com (Postfix) with ESMTP id 459421F6B2; Tue, 20 Nov 2001 13:32:07 -0800 (PST) Received: from xpabh3.corp.hp.com (xpabh3.corp.hp.com [15.58.136.223]) by xparelay1.corp.hp.com (Postfix) with ESMTP id F264B1F50A; Tue, 20 Nov 2001 13:32:06 -0800 (PST) Received: by xpabh3.corp.hp.com with Internet Mail Service (5.5.2653.19) id ; Tue, 20 Nov 2001 13:32:06 -0800 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87805CCF231@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "'Thanu Skariah'" Cc: "Ips Reflector (E-mail)" Subject: RE: iSCSI: Representing iSCSI devices on FC fabrics Date: Tue, 20 Nov 2001 13:32:04 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Are you suggesting that two gateways should *not* present the same > iSCSI name for a given FC node ? I thought they should present the same > iSCSI name and different iSCSI addresses (since it maps the path to > the target) for any given FC node. I was attempting to very explicitly state that two iSCSI gateways should !NOT! present the *same* iSCSI name for a given FC node!! I see how you may have formed this conclusion and this area needs to be clarified in the Naming and Discovery document. It's the gateway device that really needs to be the named iSCSI node, because it's not as simple as a "proxy passthru". The gateway device itself contains all kinds of iSCSI configuration and state that (I'm assuming) is not shared with another iSCSI gateway that might present the same name in the EUI naming scheme you outlined. The "iSCSI Node" that must have the unique name is the iSCSI gateway, not the FC device it is proxying for. For instance, one gateway will (should) have an access control list for iSCSI authentication of initiators or targets. A separate gateway using the naming scheme you outlined will not share this information, so for all practical purposes, it is *not* the same iSCSI device, even though it may represent the same FC devices as another gateway. In addition (and more importantly), there are rules the iSCSI target/initiator nodes must follow in order to assure there is "no parallel nexus" for SCSI (ISID assignment and target portal group representation - see section 2.5.2, 2.5.3 of iSCSI draft 09). In order to correctly follow those rules, separate gateways exporting duplicate iSCSI names for FC devices would have to share real-time state information between boxes. While that may be part of your design, I'm sure it's not part of competing gateway vendors designs to share this information cross-vendor :-) Marjorie Krueger Networked Storage Architecture Networked Storage Solutions Org. Hewlett-Packard tel: +1 916 785 2656 fax: +1 916 785 0391 email: marjorie_krueger@hp.com > -----Original Message----- > From: Thanu Skariah [mailto:tskariah@npd.hcltech.com] > Sent: Tuesday, November 20, 2001 5:57 AM > To: KRUEGER,MARJORIE (HP-Roseville,ex1) > Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics > > > Hi, > > Are you suggesting that two gateways should *not* present the same > iSCSI name for a given FC node ? I thought they should present the same > iSCSI name and different iSCSI addresses (since it maps the path to > the target) for any given FC node. > > The iSCSI spec only seems to mandate that the node name is permanent > and if every gateway were to use the same format (eui in this case), the > same end > device (node) is always represented the same way. But yes, nothing forces a > gateway to represent it using this format. > > > Also, consider the following from the iSCSI Naming and > Discovery draft > Sec2.1 : > > 5. iSCSI names must support integration with existing unique naming > schemes. > > Again, check the portion in my original mail that quotes the same > draft. It talks of a gateway "representing" an FC device, not owning. > I thought this would be sufficient rules for a gateway naming scheme. > > If it isn't we may need to incorporate text along the lines suggested > by Robert Snively into the naming draft. > > Thanks, > Thanu > > > "KRUEGER,MARJORIE (HP-Roseville,ex1)" wrote: > > > > This use of the eui. naming format will ONLY be acceptable > if the iSCSI > > gateway device is the only iSCSI gateway device in the > world that is allowed > > to "represent" these FC devices - can your implementation > guarantee that? > > Another iSCSI gateway on the same FC fabric (another of > your gateways?) > > might also use that naming convention (as Jim H points out) > and that would > > violate the requirements of iSCSI naming. > > > > The eui. naming format was intended for use when the iSCSI > device "owns" the > > EUI number (as those FC devices "own" their WWN). This > gateway usage is not > > what was intended. > > > > Marjorie Krueger > > Networked Storage Architecture > > Networked Storage Solutions Org. > > Hewlett-Packard > > tel: +1 916 785 2656 > > fax: +1 916 785 0391 > > email: marjorie_krueger@hp.com > > > > > -----Original Message----- > > > From: Thanu Skariah [mailto:tskariah@npd.hcltech.com] > > > Sent: Thursday, November 15, 2001 11:12 PM > > > To: Robert Grant > > > Cc: 'ips@ece.cmu.edu' > > > Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics > > > > > > > > > Robert, > > > > > > > > > iSCSI allows different naming formats, of which one > > > format is the EUI > > > format (See the example > > > in sec 2.2 .7 and the naming draft - > > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name- > > > disc-02.txt ) > > > > > > The EUI representation is of the form eui . . Each > > > FC device's > > > WWName > > > can be used to form the corresponding iSCSI name for the > > > device. This is what > > > we are > > > doing on a linux based software FCP/iSCSI gateway that we are > > > implementing, and > > > this > > > is why : > > > > > > (From the naming and discovery draft ): > > > > > > BeginQuote " > > > > > > Type "eui." (IEEE EUI format) > > > > > > The IEEE iSCSI name might be used when a manufacturer > is already > > > basing unique identifiers on World-Wide Names as defined > > > in the SCSI > > > SPC-2 specification. > > > > > > It may also be used by a gateway representing a Fibre > Channel or > > > SCSI device that is already adequately identified using a > > > world-wide > > > name. > > > > > > " End Quote > > > > > > > > > Thanks, > > > Thanu > > > > > > > > > > > > Robert Grant wrote: > > > > > > > Hello all, > > > > > > > > I have a question on the > > > representation of iSCSI > > > > devices into Fibre Channel fabrics for an iSCSI-to-FC > > > "gateway" device and > > > > would like to solicit people's thoughts on how best to do > > > this. A gateway > > > > device will allow iSCSI devices and FCP devices to access > > > each other, but in > > > > order to do this a consistent representation of the devices > > > is needed. I > > > > haven't been able to reconcile the iSCSI and FCP standards > > > using what's > > > > currently in the iSCSI standard, and wanted to see if there > > > was any support > > > > to expanding the iSCSI standard to address this (a standard > > > solution is, of > > > > course, much more preferred to every gateway vendor doing > > > it in their own > > > > proprietary way). In particular, how would an iSCSI device > > > map onto Fibre > > > > Channel's World Wide Name (WWN)? Would every device have > > > its own WWN, or > > > > could many iSCSI devices use a single WWN? There have been > > > some discussions > > > > (for example, there was even discussion of including a WWN > > > field in the > > > > iSCSI Login for a Gateway to proxy with in > > > > > > > http://www.pdl.cmu.edu/mailinglists/ips/mail/msg01616.html), > but what is the > > > current view? > > > > > > A first approach might be that > many iSCSI devices > > > could use a single WWN. This can work well for FC-AL > devices "directly > > > attached " to the IP network or for small FC fabrics - > and where the > > > predominant interconnect and management of that > interconnect is the IP > > > network. > > > > > > This approach views the FC fabric > as flat (or at > > > least perhaps that FC zoning is "turned off"). As the FC > fabric gets > > bigger, > > > though, this first approach can create two layers of > management - one must > > > first configure the FC network and then configure the IP > network (since > > the > > > individual iSCSI devices sharing a single WWN can only be > zoned as a > > group). > > > The two layers are first "this group of iSCSI devices can > access this > > zone" > > > on the FC side and then "this iSCSI device can access > this FC device in > > this > > > zone" on the iSCSI side. If there was a clean integration > with FC zoning > > > (and associated management of the FC zoning), this may be avoided. > > > > > > A further complication is that, > as the FC fabric > > > gets even bigger, a single iSCSI device could end up with > multiple entry > > > points (i.e. paths through multiple gateways) into a > single FC fabric. Is > > > there any common way to represent iSCSI devices (for > instance, with > > respect > > > to WWNs) that allows the unique identification of that > iSCSI device - even > > > though there are multiple entrypoints onto the FC fabric? > The case of > > > multiple gateways (possibly from different vendors) is > the clearest > > example > > > of the need for a standard. > > > > > > Thank you for your time and I > look forward to all > > > comments/suggestions. > > > > > > Regards, > > > Rob > > > > > > Rob Grant > > > McDATA Corporation > From owner-ips@ece.cmu.edu Tue Nov 20 17:34:15 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA09466 for ; Tue, 20 Nov 2001 17:34:15 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAKL16k22797 for ips-outgoing; Tue, 20 Nov 2001 16:01:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12805.mail.yahoo.com (web12805.mail.yahoo.com [216.136.174.40]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAKL14l22791 for ; Tue, 20 Nov 2001 16:01:04 -0500 (EST) Message-ID: <20011120210103.47775.qmail@web12805.mail.yahoo.com> Received: from [207.219.118.250] by web12805.mail.yahoo.com via HTTP; Tue, 20 Nov 2001 13:01:03 PST Date: Tue, 20 Nov 2001 13:01:03 -0800 (PST) From: Luben Tuikov Subject: iSCSI v8 CRC32C To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello, I've been trying to get CRC32C working. I read a few papers, derived the equations myself, etc, etc, but still cannot get CRC32C to work... My basic framework is: (not precomputing a table -- just force): poly= 1edc6f41; crc = all 1's; while more bits in message do crc shift left, adding the next message bit at right; if carry then crc = crc XOR poly; end if; end while; crc = NOT crc; (e.g. complement the register) What is wrong with this implementation? Thanks, -l ===== -- __________________________________________________ Do You Yahoo!? Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. http://geocities.yahoo.com/ps/info1 From owner-ips@ece.cmu.edu Tue Nov 20 18:56:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11267 for ; Tue, 20 Nov 2001 18:56:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAKMmj001779 for ips-outgoing; Tue, 20 Nov 2001 17:48:45 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAKMmil01775 for ; Tue, 20 Nov 2001 17:48:44 -0500 (EST) Received: by ariel.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Tue, 20 Nov 2001 14:48:38 -0800 Message-ID: From: Charles Monia To: "'Black_David@emc.com'" Cc: ips@ece.cmu.edu Subject: RE: iFCP: iFCP -06 comments Date: Tue, 20 Nov 2001 14:48:37 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David: > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, November 20, 2001 8:19 AM > To: cmonia@NishanSystems.com > Cc: ips@ece.cmu.edu > Subject: RE: iFCP: iFCP -06 comments > > > > > 10.4 b) What happens when the broadcast frame exceeds the > MTU size? > This > > > seems to result in a rather unreliable broadcast service, > as all of the > UDP > > > datagrams could well be dropped, consistently and > repeatedly for large > > > enough broadcast frames. > > > > We will rewrite the section to: > > > > a) Specify the use of source fragmentation if the datagram > exceeds the > PMTU. > > Details will need to be supplied, as the large FC frame is > being fragmented > into UDP datagrams, and hence the UDP datagrams each need to contain > information about how to reassemble them into an FC frame We had intended to use IP datagram fragmentation (this should have been mentioned explicitly, I guess). What more do we need to specify in the way of a fragmentation and reassembly mechanism? Charles From owner-ips@ece.cmu.edu Tue Nov 20 20:05:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13282 for ; Tue, 20 Nov 2001 20:05:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAL0aIS08993 for ips-outgoing; Tue, 20 Nov 2001 19:36:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAL0aHl08988 for ; Tue, 20 Nov 2001 19:36:17 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id ; Tue, 20 Nov 2001 19:32:27 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E029371DC@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: Salt Lake City meeting Date: Tue, 20 Nov 2001 19:26:03 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk We're starting to put together the agenda for the IPS WG meetings in Salt Lake City, currently scheduled for Monday and Tuesday mornings. The goals of these meetings are (in order) to: (1) Close technical issues in the main protocol drafts to make them ready for WG Last Call. (2) Close technical issues in other drafts that have to accompany the main protocol drafts (e.g. FC encap, iSNS, IPS security). (3) Close technical issues in any other draft that we can quickly declare victory on (i.e., WG Last Call prior to the Minneapolis IETF meeting is possible). Drafts not falling into one of the above categories should expect at most limited time in the meeting (e.g., 5 min update). Draft authors - please make sure that your Technical Coordinator (iSCSI - John Hufferd, FCIP - Murali Rajagopal, iFCP - Franco Travostino) is aware of the state of your draft and the time you think you'll need. For drafts not falling cleanly into one of those three technical areas, please send requests for agenda time directly to Elizabeth and me. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Nov 20 20:07:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA13339 for ; Tue, 20 Nov 2001 20:07:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAL060e07163 for ips-outgoing; Tue, 20 Nov 2001 19:06:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [128.221.11.32]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAL05wl07159 for ; Tue, 20 Nov 2001 19:05:58 -0500 (EST) Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 20 Nov 2001 19:05:53 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E029371DA@CORPMX14> From: Black_David@emc.com To: cmonia@NishanSystems.com Cc: ips@ece.cmu.edu Subject: RE: iFCP: iFCP -06 comments Date: Tue, 20 Nov 2001 18:55:44 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > > > > 10.4 b) What happens when the broadcast frame exceeds the MTU size? [... snip ...] > We had intended to use IP datagram fragmentation Oh ... I'm not sure that's a real good idea. Using IP fragmentation is one of the few ways to make UDP datagram delivery even less reliable than it already is. This puts FC broadcast at the mercy of code that doesn't fragment correctly at both iFCP and network nodes (infrequently used code paths are subject to bit rot) and systems in the network that can't or don't want to deal with IP fragments (e.g., firewalls and NAT/NAPT boxes) and hence drop rather than fragment. For source fragmentation, discovering the MTU to fragment to may also be a problem area. While I'm not going to put my foot down yet, I would advise thinking about using TCP to deliver FC broadcast frames over iFCP. If FC broadcasts are infrequent and not performance critical, perhaps iSNS could provide the FC broadcast forwarding service? This would also clean up any issues around whether an iFCP gateway knows about all the other gateways in the fabric, as the iSNS server always knows this. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Nov 20 21:15:29 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA14547 for ; Tue, 20 Nov 2001 21:15:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAL1Ccf11174 for ips-outgoing; Tue, 20 Nov 2001 20:12:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar2aa.compuserve.com (siaar2aa.compuserve.com [149.174.40.137]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAL1CZl11168 for ; Tue, 20 Nov 2001 20:12:35 -0500 (EST) Received: (from mailgate@localhost) by siaar2aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id UAA19404 for ips@ece.cmu.edu; Tue, 20 Nov 2001 20:12:29 -0500 (EST) Received: from compuserve.com (chi-tgn-gvn-vty44.as.wcom.net [216.192.148.44]) by siaar2aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id UAA19394 for ; Tue, 20 Nov 2001 20:12:27 -0500 (EST) Message-ID: <3BFB0088.17DC9A83@compuserve.com> Date: Tue, 20 Nov 2001 19:16:56 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win98; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: Re: Salt Lake City meeting References: <277DD60FB639D511AC0400B0D068B71E029371DC@CORPMX14> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit David, It is my opinion the the FC Encapsulation draft has zero open technical issues. In defense of this position, I will remind you that the only change in the latest revision is addition of statements about what the FC Encapsulation does not do. That seems to suggest that the draft's discussion of what the FC Encapsulation does do is very solid. Thanks. Ralph... Black_David@emc.com wrote: > > We're starting to put together the agenda for the IPS WG > meetings in Salt Lake City, currently scheduled for Monday > and Tuesday mornings. The goals of these meetings are > (in order) to: > > (1) Close technical issues in the main protocol drafts to make > them ready for WG Last Call. > (2) Close technical issues in other drafts that have to > accompany the main protocol drafts (e.g. FC encap, > iSNS, IPS security). > (3) Close technical issues in any other draft that we can > quickly declare victory on (i.e., WG Last Call prior > to the Minneapolis IETF meeting is possible). > > Drafts not falling into one of the above categories should > expect at most limited time in the meeting (e.g., 5 min update). > > Draft authors - please make sure that your Technical Coordinator > (iSCSI - John Hufferd, FCIP - Murali Rajagopal, iFCP - Franco > Travostino) is aware of the state of your draft and the time > you think you'll need. For drafts not falling cleanly into > one of those three technical areas, please send requests for > agenda time directly to Elizabeth and me. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ece.cmu.edu Tue Nov 20 22:38:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA17293 for ; Tue, 20 Nov 2001 22:38:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAL2M0o15288 for ips-outgoing; Tue, 20 Nov 2001 21:22:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAL2Lwl15283 for ; Tue, 20 Nov 2001 21:21:58 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 41BB72F50; Tue, 20 Nov 2001 19:21:57 -0700 (MST) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by msgrel1.cos.agilent.com (Postfix) with SMTP id BF661492; Tue, 20 Nov 2001 19:21:56 -0700 (MST) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Tue, 20 Nov 2001 19:21:56 -0700 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Tue, 20 Nov 2001 19:21:56 -0700 Message-ID: <01A7DAF31F93D511AEE300D0B706ED920CF6F4@axcs13.cos.agilent.com> From: vince_cavanna@agilent.com To: ltuikov@yahoo.com Cc: ips@ece.cmu.edu, vince_cavanna@agilent.com, pat_thaler@agilent.com Subject: RE: iSCSI v8 CRC32C Date: Tue, 20 Nov 2001 19:21:49 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C17233.46C26C90" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C17233.46C26C90 Content-Type: text/plain; charset="ISO-8859-1" Hi Luben, Looks like you are not taking into account the fact that iSCSi assumes that the least significant BIT within a byte is transmitted first (whereas the most significant BYTE is transmitted first)! Recall that the bit that is transmitted first represents the coefficient of the highest order term of the message polynomial. The reason for this strange behavior is the desire to be consistent with Ethernet. It seems this detail of ethernet CRC was not realized by many but fortunately has been captured in a parameter called "bit reflection flag" in a popular CRC tool that many are using. See also the attached email that I sent to the reflector in August wherein I objected, though not strongly, to copying Ethernet to such a degree. I subsequently yielded when it was claimed that Ethernet implementors managed to cope with the potential source of confusion and the benefits of being consistent outweighed the problems. Eventually we will learn to cope too :-). If you want a Mathematica worksheet showing the polynomial math behind the calculations as well as the required bit/byte mappings ask me and I will dig it up - after the Holidays. I was going to publish it to the reflector but don't have the time to put it in a more acceptable format. Vince -----Original Message----- From: Luben Tuikov [mailto:ltuikov@yahoo.com] Sent: Tuesday, November 20, 2001 1:01 PM To: ips@ece.cmu.edu Subject: iSCSI v8 CRC32C Hello, I've been trying to get CRC32C working. I read a few papers, derived the equations myself, etc, etc, but still cannot get CRC32C to work... My basic framework is: (not precomputing a table -- just force): poly= 1edc6f41; crc = all 1's; while more bits in message do crc shift left, adding the next message bit at right; if carry then crc = crc XOR poly; end if; end while; crc = NOT crc; (e.g. complement the register) What is wrong with this implementation? Thanks, -l ===== -- __________________________________________________ Do You Yahoo!? Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. http://geocities.yahoo.com/ps/info1 ------_=_NextPart_000_01C17233.46C26C90 Content-Type: message/rfc822 Content-Description: RE: iSCSI CRC: A CRC-checking example Message-ID: From: vince_cavanna@agilent.com To: Julian_Satran@il.ibm.com, mbakke@cisco.com Cc: vince_cavanna@agilent.com, Pat.Thaler@d12relay01.de.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI CRC: A CRC-checking example Date: Wed, 22 Aug 2001 10:42:50 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Julian and Mark and others, After I took into account the bit "reflection" and the byte order (Big Endian on Bytes and Little Endian on bits) I did finally obtain, using polynomial math, the (corrected) results you show in the CRC examples. The results have also been confirmed independently by an implementor here at Agilent. I am the one who originally suggested to Julian that we specify the CRC algorithm the same way as in ethernet even though for iSCSI it is not really important to initialize the CRC register to all 1s and to complement the CRC before transmission since there are other means to detect extra or missing PDU bytes. However I had not realzied until recently that conformance with the ethernet algorithm implies bit reflection. I had not been aware that in ethernet the bits are sent out on the serial link with the least significant bit first AND that the corresponding message polynomial is formed from the bits in the sequence that they appear on the serial link; thus the need for bit "reflection". Now that I understand the need for bit reflection (taken into account in the rocksoft parameterized CRC generator by setting the in and out reflection flag parameters to TRUE) I am not sure I agree that we want it in iSCSI. The penalty for full conformity with ethernet seems too great. If people feel strongly that we must keep the bit reflection I think that to make the existing documentation clear and unambiguous we would need to explicitly show the mapping of bits in the iSCSI PDU to coefficients of the message polynomial that represents the iSCSI PDU. We would also need to show the mapping of the CRC bits to the coefficients of its representative polynomial. If you don't agree I will elaborate further later this week to try to convince you. My objective is to be able to easily and unambiguously describe the polynomial math behind the algorithm; right now, with the bit reflection and without the explicit mapping it is awkward. Vince ------_=_NextPart_000_01C17233.46C26C90-- From owner-ips@ece.cmu.edu Wed Nov 21 07:35:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA11062 for ; Wed, 21 Nov 2001 07:35:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALBnCg29609 for ips-outgoing; Wed, 21 Nov 2001 06:49:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1ab.compuserve.com (siaar1ab.compuserve.com [149.174.40.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALBnAl29603 for ; Wed, 21 Nov 2001 06:49:10 -0500 (EST) Received: (from mailgate@localhost) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id GAA07382 for ips@ece.cmu.edu; Wed, 21 Nov 2001 06:48:59 -0500 (EST) Received: from compuserve.com (chi-tgn-gvu-vty137.as.wcom.net [216.192.152.137]) by siaar1ab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id GAA07358 for ; Wed, 21 Nov 2001 06:48:48 -0500 (EST) Message-ID: <3BFB95AD.B6001B3D@compuserve.com> Date: Wed, 21 Nov 2001 05:53:17 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win98; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: FCIP & FC Encap @ Internet Drafts Office Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The following have been sent to the Internet Drafts Office for publication: draft-ietf-ips-fcovertcpip-07.txt draft-ietf-ips-fcovertcpip-07.pdf draft-ietf-ips-fcencapsulation-04.txt draft-ietf-ips-fcencapsulation-04.pdf For those of you who prefer to open their Christmas presents on the 22nd of December and those who want the change bars in a PDF file but cannot find them at the Internet Drafts Office, the files are available now as: draft-ietf-ips-fcovertcpip-07.txt --> ftp://ftp.t11.org/t11/pub/fc/bb-2/01-600v0.txt draft-ietf-ips-fcovertcpip-07.pdf --> ftp://ftp.t11.org/t11/pub/fc/bb-2/01-600v0.pdf draft-ietf-ips-fcencapsulation-04.txt --> ftp://ftp.t11.org/t11/pub/fc/bb-2/01-524v1.txt draft-ietf-ips-fcencapsulation-04.pdf --> ftp://ftp.t11.org/t11/pub/fc/bb-2/01-524v1.pdf See you in Salt Lake. Ralph... From owner-ips@ece.cmu.edu Wed Nov 21 10:05:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA18428 for ; Wed, 21 Nov 2001 10:05:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALDvUS06891 for ips-outgoing; Wed, 21 Nov 2001 08:57:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailnpd.hcltech.com ([203.197.145.76]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fALDvMl06879 for ; Wed, 21 Nov 2001 08:57:23 -0500 (EST) Received: from npd.hcltech.com (tskariah-pc.hcltech.com [192.168.100.72]) by mailnpd.hcltech.com (8.11.0/8.11.0) with ESMTP id fALDroe00423; Wed, 21 Nov 2001 19:23:51 +0530 Message-ID: <3BFBB37A.B646F8E3@npd.hcltech.com> Date: Wed, 21 Nov 2001 19:30:26 +0530 From: Thanu Skariah X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.2-2 i686) X-Accept-Language: en MIME-Version: 1.0 To: "KRUEGER,MARJORIE (HP-Roseville,ex1)" CC: "Ips Reflector (E-mail)" Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics References: <6BD67FFB937FD411A04F00D0B74FE87805CCF231@xrose06.rose.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I see the disconnect now. We have designed the gateway to represent each FC node as an iSCSI end point, not the gateway. This way multiple gateways only represent multiple paths to the same iSCSI node. Hope this answers the other points you have raised below. Thanks, Thanu "KRUEGER,MARJORIE (HP-Roseville,ex1)" wrote: > > > Are you suggesting that two gateways should *not* present the same > > iSCSI name for a given FC node ? I thought they should present the same > > iSCSI name and different iSCSI addresses (since it maps the path to > > the target) for any given FC node. > > I was attempting to very explicitly state that two iSCSI gateways should > !NOT! present the *same* iSCSI name for a given FC node!! > > I see how you may have formed this conclusion and this area needs to be > clarified in the Naming and Discovery document. > > It's the gateway device that really needs to be the named iSCSI node, > because it's not as simple as a "proxy passthru". The gateway device itself > contains all kinds of iSCSI configuration and state that (I'm assuming) is > not shared with another iSCSI gateway that might present the same name in > the EUI naming scheme you outlined. The "iSCSI Node" that must have the > unique name is the iSCSI gateway, not the FC device it is proxying for. For > instance, one gateway will (should) have an access control list for iSCSI > authentication of initiators or targets. A separate gateway using the > naming scheme you outlined will not share this information, so for all > practical purposes, it is *not* the same iSCSI device, even though it may > represent the same FC devices as another gateway. > > In addition (and more importantly), there are rules the iSCSI > target/initiator nodes must follow in order to assure there is "no parallel > nexus" for SCSI (ISID assignment and target portal group representation - > see section 2.5.2, 2.5.3 of iSCSI draft 09). In order to correctly follow > those rules, separate gateways exporting duplicate iSCSI names for FC > devices would have to share real-time state information between boxes. > While that may be part of your design, I'm sure it's not part of competing > gateway vendors designs to share this information cross-vendor :-) > > Marjorie Krueger > Networked Storage Architecture > Networked Storage Solutions Org. > Hewlett-Packard > tel: +1 916 785 2656 > fax: +1 916 785 0391 > email: marjorie_krueger@hp.com > > > -----Original Message----- > > From: Thanu Skariah [mailto:tskariah@npd.hcltech.com] > > Sent: Tuesday, November 20, 2001 5:57 AM > > To: KRUEGER,MARJORIE (HP-Roseville,ex1) > > Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics > > > > > > Hi, > > > > Are you suggesting that two gateways should *not* present the > same > > iSCSI name for a given FC node ? I thought they should present the same > > iSCSI name and different iSCSI addresses (since it maps the path to > > the target) for any given FC node. > > > > The iSCSI spec only seems to mandate that the node name is permanent > > and if every gateway were to use the same format (eui in this case), the > > same end > > device (node) is always represented the same way. But yes, nothing forces > a > > gateway to represent it using this format. > > > > > > Also, consider the following from the iSCSI Naming and > > Discovery draft > > Sec2.1 : > > > > 5. iSCSI names must support integration with existing unique naming > > schemes. > > > > Again, check the portion in my original mail that quotes the same > > draft. It talks of a gateway "representing" an FC device, not owning. > > I thought this would be sufficient rules for a gateway naming scheme. > > > > If it isn't we may need to incorporate text along the lines suggested > > by Robert Snively into the naming draft. > > > > Thanks, > > Thanu > > > > > > "KRUEGER,MARJORIE (HP-Roseville,ex1)" wrote: > > > > > > This use of the eui. naming format will ONLY be acceptable > > if the iSCSI > > > gateway device is the only iSCSI gateway device in the > > world that is allowed > > > to "represent" these FC devices - can your implementation > > guarantee that? > > > Another iSCSI gateway on the same FC fabric (another of > > your gateways?) > > > might also use that naming convention (as Jim H points out) > > and that would > > > violate the requirements of iSCSI naming. > > > > > > The eui. naming format was intended for use when the iSCSI > > device "owns" the > > > EUI number (as those FC devices "own" their WWN). This > > gateway usage is not > > > what was intended. > > > > > > Marjorie Krueger > > > Networked Storage Architecture > > > Networked Storage Solutions Org. > > > Hewlett-Packard > > > tel: +1 916 785 2656 > > > fax: +1 916 785 0391 > > > email: marjorie_krueger@hp.com > > > > > > > -----Original Message----- > > > > From: Thanu Skariah [mailto:tskariah@npd.hcltech.com] > > > > Sent: Thursday, November 15, 2001 11:12 PM > > > > To: Robert Grant > > > > Cc: 'ips@ece.cmu.edu' > > > > Subject: Re: iSCSI: Representing iSCSI devices on FC fabrics > > > > > > > > > > > > Robert, > > > > > > > > > > > > iSCSI allows different naming formats, of which one > > > > format is the EUI > > > > format (See the example > > > > in sec 2.2 .7 and the naming draft - > > > > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name- > > > > disc-02.txt ) > > > > > > > > The EUI representation is of the form eui . . Each > > > > FC device's > > > > WWName > > > > can be used to form the corresponding iSCSI name for the > > > > device. This is what > > > > we are > > > > doing on a linux based software FCP/iSCSI gateway that we are > > > > implementing, and > > > > this > > > > is why : > > > > > > > > (From the naming and discovery draft ): > > > > > > > > BeginQuote " > > > > > > > > Type "eui." (IEEE EUI format) > > > > > > > > The IEEE iSCSI name might be used when a manufacturer > > is already > > > > basing unique identifiers on World-Wide Names as defined > > > > in the SCSI > > > > SPC-2 specification. > > > > > > > > It may also be used by a gateway representing a Fibre > > Channel or > > > > SCSI device that is already adequately identified using a > > > > world-wide > > > > name. > > > > > > > > " End Quote > > > > > > > > > > > > Thanks, > > > > Thanu > > > > > > > > > > > > > > > > Robert Grant wrote: > > > > > > > > > Hello all, > > > > > > > > > > I have a question on the > > > > representation of iSCSI > > > > > devices into Fibre Channel fabrics for an iSCSI-to-FC > > > > "gateway" device and > > > > > would like to solicit people's thoughts on how best to do > > > > this. A gateway > > > > > device will allow iSCSI devices and FCP devices to access > > > > each other, but in > > > > > order to do this a consistent representation of the devices > > > > is needed. I > > > > > haven't been able to reconcile the iSCSI and FCP standards > > > > using what's > > > > > currently in the iSCSI standard, and wanted to see if there > > > > was any support > > > > > to expanding the iSCSI standard to address this (a standard > > > > solution is, of > > > > > course, much more preferred to every gateway vendor doing > > > > it in their own > > > > > proprietary way). In particular, how would an iSCSI device > > > > map onto Fibre > > > > > Channel's World Wide Name (WWN)? Would every device have > > > > its own WWN, or > > > > > could many iSCSI devices use a single WWN? There have been > > > > some discussions > > > > > (for example, there was even discussion of including a WWN > > > > field in the > > > > > iSCSI Login for a Gateway to proxy with in > > > > > > > > > > http://www.pdl.cmu.edu/mailinglists/ips/mail/msg01616.html), > > but what is the > > > > current view? > > > > > > > > A first approach might be that > > many iSCSI devices > > > > could use a single WWN. This can work well for FC-AL > > devices "directly > > > > attached " to the IP network or for small FC fabrics - > > and where the > > > > predominant interconnect and management of that > > interconnect is the IP > > > > network. > > > > > > > > This approach views the FC fabric > > as flat (or at > > > > least perhaps that FC zoning is "turned off"). As the FC > > fabric gets > > > bigger, > > > > though, this first approach can create two layers of > > management - one must > > > > first configure the FC network and then configure the IP > > network (since > > > the > > > > individual iSCSI devices sharing a single WWN can only be > > zoned as a > > > group). > > > > The two layers are first "this group of iSCSI devices can > > access this > > > zone" > > > > on the FC side and then "this iSCSI device can access > > this FC device in > > > this > > > > zone" on the iSCSI side. If there was a clean integration > > with FC zoning > > > > (and associated management of the FC zoning), this may be avoided. > > > > > > > > A further complication is that, > > as the FC fabric > > > > gets even bigger, a single iSCSI device could end up with > > multiple entry > > > > points (i.e. paths through multiple gateways) into a > > single FC fabric. Is > > > > there any common way to represent iSCSI devices (for > > instance, with > > > respect > > > > to WWNs) that allows the unique identification of that > > iSCSI device - even > > > > though there are multiple entrypoints onto the FC fabric? > > The case of > > > > multiple gateways (possibly from different vendors) is > > the clearest > > > example > > > > of the need for a standard. > > > > > > > > Thank you for your time and I > > look forward to all > > > > comments/suggestions. > > > > > > > > Regards, > > > > Rob > > > > > > > > Rob Grant > > > > McDATA Corporation > > From owner-ips@ece.cmu.edu Wed Nov 21 11:33:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA23332 for ; Wed, 21 Nov 2001 11:33:49 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALFexR13881 for ips-outgoing; Wed, 21 Nov 2001 10:40:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from svldns02.veritas.com (bay-bridge.veritas.com [143.127.3.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALFevl13877 for ; Wed, 21 Nov 2001 10:40:57 -0500 (EST) Received: from megami.veritas.com (megami.veritas.com [10.182.128.180]) by svldns02.veritas.com (8.11.6/8.11.6) with SMTP id fALFd4305453 for ; Wed, 21 Nov 2001 07:39:04 -0800 (PST) Received: from rahul([10.212.108.206]) (1512 bytes) by megami.veritas.com via sendmail with P:smtp/R:smart_host/T:smtp (sender: ) id for ; Wed, 21 Nov 2001 07:40:50 -0800 (PST) (Smail-3.2.0.101 1997-Dec-17 #15 built 2001-Aug-30) Message-ID: <002601c172a1$cb0b2190$ce6cd40a@rahul> From: "Rahul Bhagwat" To: Subject: Selectively exposing Porgal Groups to Initiators Date: Wed, 21 Nov 2001 21:02:39 +0530 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi, 1. Is it required that TargetAddresses of an iSCSI target advertised to a directory service include portal group tag ? 2. Is it mandatory for an Initiator to use "SendTargets" to discover TargetAddress for an iSCSI target (even if if has a set of addresses either statically configured or found through a directory service)? 3. Is it okay to restrict access to an Initiator (based on it's iSCSI name) to only a subset of total Target Portal Groups supported by the iSCSI target? In this scenario, an Initiator may find out the TargetAddresses for an iSCSI target using a directory service, and try to connect a normal operational session to any of these addresses without using SendTargets. The iSCSI target can return an error code for login as "Initiator not Authorized" which in fact is not completely true. Initiator is authorized to only use a subset of portal groups. Regards, Rahul From owner-ips@ece.cmu.edu Wed Nov 21 11:39:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA23775 for ; Wed, 21 Nov 2001 11:39:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALFStE12992 for ips-outgoing; Wed, 21 Nov 2001 10:28:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALBw5l00059 for ; Wed, 21 Nov 2001 06:58:10 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA09580; Wed, 21 Nov 2001 06:58:04 -0500 (EST) Message-Id: <200111211158.GAA09580@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-name-disc-03.txt Date: Wed, 21 Nov 2001 06:58:04 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Naming and Discovery Author(s) : M. Bakke et al. Filename : draft-ietf-ips-iscsi-name-disc-03.txt Pages : 22 Date : 20-Nov-01 This document describes iSCSI [7] naming and discovery details. This document complements the iSCSI Protocol draft. Flexibility is the key guiding principle behind this document. That is, an effort has been made to satisfy the needs of both small isolated environments, as well as large environments requiring secure/scalable solutions A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name-disc-03.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-name-disc-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-name-disc-03.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011120151401.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-name-disc-03.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-name-disc-03.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011120151401.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Wed Nov 21 11:41:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA23944 for ; Wed, 21 Nov 2001 11:41:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALFSoQ12978 for ips-outgoing; Wed, 21 Nov 2001 10:28:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALBvsl00045 for ; Wed, 21 Nov 2001 06:57:54 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA09564; Wed, 21 Nov 2001 06:57:52 -0500 (EST) Message-Id: <200111211157.GAA09564@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-fcovertcpip-07.txt Date: Wed, 21 Nov 2001 06:57:52 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Fibre Channel Over TCP/IP (FCIP) Author(s) : M. Rajagopal, R. Bhagwat et al. Filename : draft-ietf-ips-fcovertcpip-07.txt Pages : 56 Date : 20-Nov-01 Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the interconnection of islands of Fibre Channel storage area networks over IP-based networks to form a unified storage area network in a single Fibre Channel fabric. FCIP relies on IP-based network services to provide the connectivity between the storage area network islands over local area networks, metropolitan area networks, or wide area networks. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-fcovertcpip-07.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-fcovertcpip-07.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-fcovertcpip-07.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011120151349.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-fcovertcpip-07.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-fcovertcpip-07.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011120151349.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Wed Nov 21 15:06:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03193 for ; Wed, 21 Nov 2001 15:06:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALJCAh01524 for ips-outgoing; Wed, 21 Nov 2001 14:12:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel6.hp.com (atlrel6.hp.com [192.151.27.8]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALJC9l01519 for ; Wed, 21 Nov 2001 14:12:09 -0500 (EST) Received: from xatlrelay2.atl.hp.com (unknown [15.45.89.191]) by atlrel6.hp.com (Postfix) with ESMTP id B542D1F8D2; Wed, 21 Nov 2001 14:11:53 -0500 (EST) Received: from xatlbh3.atl.hp.com (xatlbh3.atl.hp.com [15.45.89.188]) by xatlrelay2.atl.hp.com (Postfix) with ESMTP id 1F4411F52E; Wed, 21 Nov 2001 14:11:53 -0500 (EST) Received: by xatlbh3.atl.hp.com with Internet Mail Service (5.5.2653.19) id ; Wed, 21 Nov 2001 14:11:52 -0500 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87805CCF23B@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "'Rahul Bhagwat'" , ips@ece.cmu.edu Subject: RE: Selectively exposing Porgal Groups to Initiators Date: Wed, 21 Nov 2001 14:11:45 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > 1. Is it required that TargetAddresses of an iSCSI target advertised to a > directory service include portal group tag ? Yes, information is necessary to communicate to the initiator which target addresses can be used to form a multi-connection session. > 2. Is it mandatory for an Initiator to use "SendTargets" to discover > TargetAddress for an iSCSI target (even if if has a set of addresses > either statically configured or found through a directory service)? To my knowledge no iSCSI document has declared it mandatory, but it's recommended to ensure the initiator has current addressing information for this target. > 3. Is it okay to restrict access to an Initiator (based on it's iSCSI name) > to only a subset of total Target Portal Groups supported by the iSCSI > target? Yes > > In this scenario, an Initiator may find out the TargetAddresses for an iSCSI > target using a directory service, and try to connect a normal operational > session to any of these addresses without using SendTargets. The > iSCSI target can return an error code for login as "Initiator not Authorized" > which in fact is not > completely true. Initiator is authorized to only use a subset > of portal groups. Correct, although "not completely true" is subjective. Another perspective is that this initiator is not authorized to use this *target port*, which is completely true. Marjorie From owner-ips@ece.cmu.edu Wed Nov 21 15:08:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03243 for ; Wed, 21 Nov 2001 15:08:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALJRJp02557 for ips-outgoing; Wed, 21 Nov 2001 14:27:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from penguin-ext.wise.edt.ericsson.se (penguin-ext.wise.edt.ericsson.se [194.237.142.110]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALHjql25514 for ; Wed, 21 Nov 2001 12:45:53 -0500 (EST) Received: from ms.uab.ericsson.se. (ms.uab.ericsson.se [134.138.201.16]) by penguin.wise.edt.ericsson.se (8.11.0/8.11.0/WIREfire-1.3) with ESMTP id fALHjnC07264 for ; Wed, 21 Nov 2001 18:45:49 +0100 (MET) Received: from uabs31c015 (uabs31c015 [134.138.218.15]) by ms.uab.ericsson.se. (8.11.1/8.11.1/uab-3.6) with ESMTP id fALHjm328248; Wed, 21 Nov 2001 18:45:48 +0100 (MET) Received: from uab.ericsson.se by uabs31c015 (8.9.3+Sun/client-1.3uab2) id SAA08370; Wed, 21 Nov 2001 18:45:44 +0100 (MET) Message-Id: <200111211745.SAA08370@uabs31c015> Date: Wed, 21 Nov 2001 06:58:04 -0500 From: Internet-Drafts@ietf.org Reply-To: =?iso-8859-1?Q?Mats_Bergstr=F6m?= Subject: I-D ACTION:draft-ietf-ips-iscsi-name-disc-03.txt To: Mats Bergstrom cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: MULTIPART/mixed; BOUNDARY="-2133890903-33463914-1006364748=:15031" Content-Transfer-Encoding: BINARY Sender: owner-ips@ece.cmu.edu Precedence: bulk ---2133890903-33463914-1006364748=:15031 Content-Type: TEXT/plain; charset=us-ascii A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Naming and Discovery Author(s) : M. Bakke et al. Filename : draft-ietf-ips-iscsi-name-disc-03.txt Pages : 22 Date : 20-Nov-01 This document describes iSCSI [7] naming and discovery details. This document complements the iSCSI Protocol draft. Flexibility is the key guiding principle behind this document. That is, an effort has been made to satisfy the needs of both small isolated environments, as well as large environments requiring secure/scalable solutions A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name-disc-03.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-name-disc-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-name-disc-03.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ---2133890903-33463914-1006364748=:15031 Content-Type: APPLICATION/octet-stream; BOUNDARY=OtherAccess Content-Transfer-Encoding: BASE64 LS1PdGhlckFjY2Vzcw0KQ29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFs LWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwtc2VydmVyIjsNCglzZXJ2ZXI9 Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRleHQvcGxh aW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTEyMDE1MTQwMS5JLURAaWV0Zi5vcmc+ DQoNCkVOQ09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFm dC1pZXRmLWlwcy1pc2NzaS1uYW1lLWRpc2MtMDMudHh0DQoNCi0tT3RoZXJB Y2Nlc3MNCkNvbnRlbnQtVHlwZTogTWVzc2FnZS9FeHRlcm5hbC1ib2R5Ow0K CW5hbWU9ImRyYWZ0LWlldGYtaXBzLWlzY3NpLW5hbWUtZGlzYy0wMy50eHQi Ow0KCXNpdGU9ImZ0cC5pZXRmLm9yZyI7DQoJYWNjZXNzLXR5cGU9ImFub24t ZnRwIjsNCglkaXJlY3Rvcnk9ImludGVybmV0LWRyYWZ0cyINCg0KQ29udGVu dC1UeXBlOiB0ZXh0L3BsYWluDQpDb250ZW50LUlEOgk8MjAwMTExMjAxNTE0 MDEuSS1EQGlldGYub3JnPg0KDQotLU90aGVyQWNjZXNzLS0NCg== ---2133890903-33463914-1006364748=:15031-- From owner-ips@ece.cmu.edu Wed Nov 21 15:09:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03326 for ; Wed, 21 Nov 2001 15:09:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALJRCR02549 for ips-outgoing; Wed, 21 Nov 2001 14:27:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from penguin-ext.wise.edt.ericsson.se (penguin-ext.wise.edt.ericsson.se [194.237.142.110]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALHjil25500 for ; Wed, 21 Nov 2001 12:45:44 -0500 (EST) Received: from ms.uab.ericsson.se. (ms.uab.ericsson.se [134.138.201.16]) by penguin.wise.edt.ericsson.se (8.11.0/8.11.0/WIREfire-1.3) with ESMTP id fALHjeC07220 for ; Wed, 21 Nov 2001 18:45:40 +0100 (MET) Received: from uabs31c015 (uabs31c015 [134.138.218.15]) by ms.uab.ericsson.se. (8.11.1/8.11.1/uab-3.6) with ESMTP id fALHjd328235; Wed, 21 Nov 2001 18:45:39 +0100 (MET) Received: from uab.ericsson.se by uabs31c015 (8.9.3+Sun/client-1.3uab2) id SAA08365; Wed, 21 Nov 2001 18:45:35 +0100 (MET) Message-Id: <200111211745.SAA08365@uabs31c015> Date: Wed, 21 Nov 2001 06:57:52 -0500 From: Internet-Drafts@ietf.org Reply-To: =?iso-8859-1?Q?Mats_Bergstr=F6m?= Subject: I-D ACTION:draft-ietf-ips-fcovertcpip-07.txt To: Mats Bergstrom cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: MULTIPART/mixed; BOUNDARY="-2133890903-1903590565-1006364739=:15031" Content-Transfer-Encoding: BINARY Sender: owner-ips@ece.cmu.edu Precedence: bulk ---2133890903-1903590565-1006364739=:15031 Content-Type: TEXT/plain; charset=us-ascii A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Fibre Channel Over TCP/IP (FCIP) Author(s) : M. Rajagopal, R. Bhagwat et al. Filename : draft-ietf-ips-fcovertcpip-07.txt Pages : 56 Date : 20-Nov-01 Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the interconnection of islands of Fibre Channel storage area networks over IP-based networks to form a unified storage area network in a single Fibre Channel fabric. FCIP relies on IP-based network services to provide the connectivity between the storage area network islands over local area networks, metropolitan area networks, or wide area networks. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-fcovertcpip-07.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-fcovertcpip-07.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-fcovertcpip-07.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ---2133890903-1903590565-1006364739=:15031 Content-Type: APPLICATION/octet-stream; BOUNDARY=OtherAccess Content-Transfer-Encoding: BASE64 LS1PdGhlckFjY2Vzcw0KQ29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFs LWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwtc2VydmVyIjsNCglzZXJ2ZXI9 Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRleHQvcGxh aW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTEyMDE1MTM0OS5JLURAaWV0Zi5vcmc+ DQoNCkVOQ09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFm dC1pZXRmLWlwcy1mY292ZXJ0Y3BpcC0wNy50eHQNCg0KLS1PdGhlckFjY2Vz cw0KQ29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJbmFt ZT0iZHJhZnQtaWV0Zi1pcHMtZmNvdmVydGNwaXAtMDcudHh0IjsNCglzaXRl PSJmdHAuaWV0Zi5vcmciOw0KCWFjY2Vzcy10eXBlPSJhbm9uLWZ0cCI7DQoJ ZGlyZWN0b3J5PSJpbnRlcm5ldC1kcmFmdHMiDQoNCkNvbnRlbnQtVHlwZTog dGV4dC9wbGFpbg0KQ29udGVudC1JRDoJPDIwMDExMTIwMTUxMzQ5LkktREBp ZXRmLm9yZz4NCg0KLS1PdGhlckFjY2Vzcy0tDQo= ---2133890903-1903590565-1006364739=:15031-- From owner-ips@ece.cmu.edu Wed Nov 21 15:09:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03337 for ; Wed, 21 Nov 2001 15:09:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALJQw202519 for ips-outgoing; Wed, 21 Nov 2001 14:26:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from albatross-ext.wise.edt.ericsson.se (albatross-ext.wise.edt.ericsson.se [194.237.142.116]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALHfol25229 for ; Wed, 21 Nov 2001 12:41:51 -0500 (EST) Received: from ms.uab.ericsson.se. (ms.uab.ericsson.se [134.138.201.16]) by albatross.wise.edt.ericsson.se (8.11.0/8.11.0/WIREfire-1.3) with ESMTP id fALHfif27917 for ; Wed, 21 Nov 2001 18:41:44 +0100 (MET) Received: from uabs31c015 (uabs31c015 [134.138.218.15]) by ms.uab.ericsson.se. (8.11.1/8.11.1/uab-3.6) with ESMTP id fALHfi327698; Wed, 21 Nov 2001 18:41:44 +0100 (MET) Received: from uab.ericsson.se by uabs31c015 (8.9.3+Sun/client-1.3uab2) id SAA08284; Wed, 21 Nov 2001 18:41:38 +0100 (MET) Message-Id: <200111211741.SAA08284@uabs31c015> Date: Wed, 21 Nov 2001 06:57:52 -0500 From: Internet-Drafts@ietf.org Reply-To: =?iso-8859-1?Q?Mats_Bergstr=F6m?= Subject: I-D ACTION:draft-ietf-ips-fcovertcpip-07.txt To: Mats Bergstrom cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: MULTIPART/mixed; BOUNDARY="-2133890903-684387517-1006364503=:15031" Content-Transfer-Encoding: BINARY Sender: owner-ips@ece.cmu.edu Precedence: bulk ---2133890903-684387517-1006364503=:15031 Content-Type: TEXT/plain; charset=us-ascii A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Fibre Channel Over TCP/IP (FCIP) Author(s) : M. Rajagopal, R. Bhagwat et al. Filename : draft-ietf-ips-fcovertcpip-07.txt Pages : 56 Date : 20-Nov-01 Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the interconnection of islands of Fibre Channel storage area networks over IP-based networks to form a unified storage area network in a single Fibre Channel fabric. FCIP relies on IP-based network services to provide the connectivity between the storage area network islands over local area networks, metropolitan area networks, or wide area networks. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-fcovertcpip-07.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-fcovertcpip-07.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-fcovertcpip-07.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ---2133890903-684387517-1006364503=:15031 Content-Type: APPLICATION/octet-stream; BOUNDARY=OtherAccess Content-Transfer-Encoding: BASE64 LS1PdGhlckFjY2Vzcw0KQ29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFs LWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwtc2VydmVyIjsNCglzZXJ2ZXI9 Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRleHQvcGxh aW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTEyMDE1MTM0OS5JLURAaWV0Zi5vcmc+ DQoNCkVOQ09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFm dC1pZXRmLWlwcy1mY292ZXJ0Y3BpcC0wNy50eHQNCg0KLS1PdGhlckFjY2Vz cw0KQ29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJbmFt ZT0iZHJhZnQtaWV0Zi1pcHMtZmNvdmVydGNwaXAtMDcudHh0IjsNCglzaXRl PSJmdHAuaWV0Zi5vcmciOw0KCWFjY2Vzcy10eXBlPSJhbm9uLWZ0cCI7DQoJ ZGlyZWN0b3J5PSJpbnRlcm5ldC1kcmFmdHMiDQoNCkNvbnRlbnQtVHlwZTog dGV4dC9wbGFpbg0KQ29udGVudC1JRDoJPDIwMDExMTIwMTUxMzQ5LkktREBp ZXRmLm9yZz4NCg0KLS1PdGhlckFjY2Vzcy0tDQo= ---2133890903-684387517-1006364503=:15031-- From owner-ips@ece.cmu.edu Wed Nov 21 15:10:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03447 for ; Wed, 21 Nov 2001 15:10:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALJRNf02568 for ips-outgoing; Wed, 21 Nov 2001 14:27:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from penguin-ext.wise.edt.ericsson.se (penguin-ext.wise.edt.ericsson.se [194.237.142.110]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALHkIl25540 for ; Wed, 21 Nov 2001 12:46:19 -0500 (EST) Received: from ms.uab.ericsson.se. (ms.uab.ericsson.se [134.138.201.16]) by penguin.wise.edt.ericsson.se (8.11.0/8.11.0/WIREfire-1.3) with ESMTP id fALHkEC07444 for ; Wed, 21 Nov 2001 18:46:14 +0100 (MET) Received: from uabs31c015 (uabs31c015 [134.138.218.15]) by ms.uab.ericsson.se. (8.11.1/8.11.1/uab-3.6) with ESMTP id fALHkE328287; Wed, 21 Nov 2001 18:46:14 +0100 (MET) Received: from uab.ericsson.se by uabs31c015 (8.9.3+Sun/client-1.3uab2) id SAA08453; Wed, 21 Nov 2001 18:46:10 +0100 (MET) Message-Id: <200111211746.SAA08453@uabs31c015> Date: Wed, 21 Nov 2001 06:58:04 -0500 From: Internet-Drafts@ietf.org Reply-To: =?iso-8859-1?Q?Mats_Bergstr=F6m?= Subject: I-D ACTION:draft-ietf-ips-iscsi-name-disc-03.txt To: Mats Bergstrom cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: MULTIPART/mixed; BOUNDARY="-2133890903-1254324197-1006364774=:15031" Content-Transfer-Encoding: BINARY Sender: owner-ips@ece.cmu.edu Precedence: bulk ---2133890903-1254324197-1006364774=:15031 Content-Type: TEXT/plain; charset=us-ascii A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Naming and Discovery Author(s) : M. Bakke et al. Filename : draft-ietf-ips-iscsi-name-disc-03.txt Pages : 22 Date : 20-Nov-01 This document describes iSCSI [7] naming and discovery details. This document complements the iSCSI Protocol draft. Flexibility is the key guiding principle behind this document. That is, an effort has been made to satisfy the needs of both small isolated environments, as well as large environments requiring secure/scalable solutions A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-name-disc-03.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-name-disc-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-name-disc-03.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ---2133890903-1254324197-1006364774=:15031 Content-Type: APPLICATION/octet-stream; BOUNDARY=OtherAccess Content-Transfer-Encoding: BASE64 LS1PdGhlckFjY2Vzcw0KQ29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFs LWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwtc2VydmVyIjsNCglzZXJ2ZXI9 Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRleHQvcGxh aW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTEyMDE1MTQwMS5JLURAaWV0Zi5vcmc+ DQoNCkVOQ09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFm dC1pZXRmLWlwcy1pc2NzaS1uYW1lLWRpc2MtMDMudHh0DQoNCi0tT3RoZXJB Y2Nlc3MNCkNvbnRlbnQtVHlwZTogTWVzc2FnZS9FeHRlcm5hbC1ib2R5Ow0K CW5hbWU9ImRyYWZ0LWlldGYtaXBzLWlzY3NpLW5hbWUtZGlzYy0wMy50eHQi Ow0KCXNpdGU9ImZ0cC5pZXRmLm9yZyI7DQoJYWNjZXNzLXR5cGU9ImFub24t ZnRwIjsNCglkaXJlY3Rvcnk9ImludGVybmV0LWRyYWZ0cyINCg0KQ29udGVu dC1UeXBlOiB0ZXh0L3BsYWluDQpDb250ZW50LUlEOgk8MjAwMTExMjAxNTE0 MDEuSS1EQGlldGYub3JnPg0KDQotLU90aGVyQWNjZXNzLS0NCg== ---2133890903-1254324197-1006364774=:15031-- From owner-ips@ece.cmu.edu Wed Nov 21 15:10:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03459 for ; Wed, 21 Nov 2001 15:10:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALJC8701513 for ips-outgoing; Wed, 21 Nov 2001 14:12:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from albatross-ext.wise.edt.ericsson.se (albatross-ext.wise.edt.ericsson.se [194.237.142.116]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALHfol25229 for ; Wed, 21 Nov 2001 12:41:51 -0500 (EST) Received: from ms.uab.ericsson.se. (ms.uab.ericsson.se [134.138.201.16]) by albatross.wise.edt.ericsson.se (8.11.0/8.11.0/WIREfire-1.3) with ESMTP id fALHfif27917 for ; Wed, 21 Nov 2001 18:41:44 +0100 (MET) Received: from uabs31c015 (uabs31c015 [134.138.218.15]) by ms.uab.ericsson.se. (8.11.1/8.11.1/uab-3.6) with ESMTP id fALHfi327698; Wed, 21 Nov 2001 18:41:44 +0100 (MET) Received: from uab.ericsson.se by uabs31c015 (8.9.3+Sun/client-1.3uab2) id SAA08284; Wed, 21 Nov 2001 18:41:38 +0100 (MET) Message-Id: <200111211741.SAA08284@uabs31c015> Date: Wed, 21 Nov 2001 06:57:52 -0500 From: Internet-Drafts@ietf.org Reply-To: =?iso-8859-1?Q?Mats_Bergstr=F6m?= Subject: I-D ACTION:draft-ietf-ips-fcovertcpip-07.txt To: Mats Bergstrom cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: MULTIPART/mixed; BOUNDARY="-2133890903-684387517-1006364503=:15031" Content-Transfer-Encoding: BINARY Sender: owner-ips@ece.cmu.edu Precedence: bulk ---2133890903-684387517-1006364503=:15031 Content-Type: TEXT/plain; charset=us-ascii A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Fibre Channel Over TCP/IP (FCIP) Author(s) : M. Rajagopal, R. Bhagwat et al. Filename : draft-ietf-ips-fcovertcpip-07.txt Pages : 56 Date : 20-Nov-01 Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the interconnection of islands of Fibre Channel storage area networks over IP-based networks to form a unified storage area network in a single Fibre Channel fabric. FCIP relies on IP-based network services to provide the connectivity between the storage area network islands over local area networks, metropolitan area networks, or wide area networks. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-fcovertcpip-07.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-fcovertcpip-07.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-fcovertcpip-07.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ---2133890903-684387517-1006364503=:15031 Content-Type: APPLICATION/octet-stream; BOUNDARY=OtherAccess Content-Transfer-Encoding: BASE64 LS1PdGhlckFjY2Vzcw0KQ29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFs LWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwtc2VydmVyIjsNCglzZXJ2ZXI9 Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRleHQvcGxh aW4NCkNvbnRlbnQtSUQ6CTwyMDAxMTEyMDE1MTM0OS5JLURAaWV0Zi5vcmc+ DQoNCkVOQ09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFm dC1pZXRmLWlwcy1mY292ZXJ0Y3BpcC0wNy50eHQNCg0KLS1PdGhlckFjY2Vz cw0KQ29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJbmFt ZT0iZHJhZnQtaWV0Zi1pcHMtZmNvdmVydGNwaXAtMDcudHh0IjsNCglzaXRl PSJmdHAuaWV0Zi5vcmciOw0KCWFjY2Vzcy10eXBlPSJhbm9uLWZ0cCI7DQoJ ZGlyZWN0b3J5PSJpbnRlcm5ldC1kcmFmdHMiDQoNCkNvbnRlbnQtVHlwZTog dGV4dC9wbGFpbg0KQ29udGVudC1JRDoJPDIwMDExMTIwMTUxMzQ5LkktREBp ZXRmLm9yZz4NCg0KLS1PdGhlckFjY2Vzcy0tDQo= ---2133890903-684387517-1006364503=:15031-- From owner-ips@ece.cmu.edu Wed Nov 21 16:35:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08188 for ; Wed, 21 Nov 2001 16:35:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALKhQa07883 for ips-outgoing; Wed, 21 Nov 2001 15:43:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALKhOl07877 for ; Wed, 21 Nov 2001 15:43:25 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA23936; Wed, 21 Nov 2001 15:40:41 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fALKhL1264114; Wed, 21 Nov 2001 13:43:21 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: Selectively exposing Porgal Groups to Initiators To: "'Rahul Bhagwat'" Cc: ips@ece.cmu.edu, "KRUEGER,MARJORIE (HP-Roseville,ex1)" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Wed, 21 Nov 2001 12:42:33 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/21/2001 01:43:20 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Rahul, Marjorie's answers are correct, but you might want to be very careful with your terms. Your point three talks about limiting Portal Groups to specific Initiators. I am concerned about your use of the words iSCSI Target, which could apply to a couple of different things. In a Target Network Entity there can be more then one iSCSI Target Node (SCSI Device). Each Target Node can have more then one iSCSI (SCSI) Target Port connected to it. Part of the name of this iSCSI (SCSI) Target Port includes the Portal Group Tag. So if, in your point 3, the term iSCSI Target meant iSCSI Target Node, then you would be able to set up different iSCSI (SCSI) Target Ports (each with different Portal Group Tags) that can access the resources at the same iSCSI Target Node. The ACL would then probably be applied at the iSCSI (SCSI) Target Port. To be sure we are on the same page here, please reference the charts that have been placed at: http://www.haifa.il.ibm.com/satran/ips/iSCSIConfigurationExamples.pdf (Note: I use the term iSCSI (SCSI) Port to represent the concept of "SCSI Port" within the context of iSCSI.) . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "KRUEGER,MARJORIE (HP-Roseville,ex1)" @ece.cmu.edu on 11/21/2001 11:11:45 AM Sent by: owner-ips@ece.cmu.edu To: "'Rahul Bhagwat'" , ips@ece.cmu.edu cc: Subject: RE: Selectively exposing Porgal Groups to Initiators > 1. Is it required that TargetAddresses of an iSCSI target advertised to a > directory service include portal group tag ? Yes, information is necessary to communicate to the initiator which target addresses can be used to form a multi-connection session. > 2. Is it mandatory for an Initiator to use "SendTargets" to discover > TargetAddress for an iSCSI target (even if if has a set of addresses > either statically configured or found through a directory service)? To my knowledge no iSCSI document has declared it mandatory, but it's recommended to ensure the initiator has current addressing information for this target. > 3. Is it okay to restrict access to an Initiator (based on it's iSCSI name) > to only a subset of total Target Portal Groups supported by the iSCSI > target? Yes > > In this scenario, an Initiator may find out the TargetAddresses for an iSCSI > target using a directory service, and try to connect a normal operational > session to any of these addresses without using SendTargets. The > iSCSI target can return an error code for login as "Initiator not Authorized" > which in fact is not > completely true. Initiator is authorized to only use a subset > of portal groups. Correct, although "not completely true" is subjective. Another perspective is that this initiator is not authorized to use this *target port*, which is completely true. Marjorie From owner-ips@ece.cmu.edu Wed Nov 21 16:39:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08511 for ; Wed, 21 Nov 2001 16:39:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALKeYa07670 for ips-outgoing; Wed, 21 Nov 2001 15:40:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sandmail.sandburst.com (sandmail.sandburst.com [216.57.132.42]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALKeUl07665 for ; Wed, 21 Nov 2001 15:40:30 -0500 (EST) To: rdma@yahoogroups.com, ips@ece.cmu.edu Cc: chase@cs.duke.edu, jpink@microsoft.com, allyn@cisco.com, csapuntz@cisco.com, jim_wendt@hp.com, jim.williams@emulex.com, JeffMogul@acm.org Subject: New version of TCU ULP Framing draft Date: Wed, 21 Nov 2001 15:38:11 -0500 From: Stephen Bailey Message-Id: <20011121204013.1506E4E7A@sandmail.sandburst.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk I have submitted the next version of the TCP ULP Framing protocol draft, draft-ietf-tsvwg-tcp-ulp-frame-01.txt to the Internet Drafts area. Until it appears, a copy can be found at: http://www.cs.uchicago.edu/~steph/draft-ietf-tsvwg-tcp-ulp-frame-01.txt Steph From owner-ips@ece.cmu.edu Wed Nov 21 18:18:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA12965 for ; Wed, 21 Nov 2001 18:18:11 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALMC3C13711 for ips-outgoing; Wed, 21 Nov 2001 17:12:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fALMBxl13698 for ; Wed, 21 Nov 2001 17:11:59 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA10610 for ; Wed, 21 Nov 2001 17:09:16 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fALMBt1241090 for ; Wed, 21 Nov 2001 15:11:55 -0700 X-Priority: 1 (High) Importance: Normal Subject: iSCSI: 09 Draft TTT To: "Julian Satran" Cc: "ips" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Wed, 21 Nov 2001 14:11:08 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/21/2001 03:11:55 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, (minor editing items) I have found two places that still call the TTT as Target Task Tag instead of Target Transfer Tag. The first one is on page 89 third paragraph under 3.11.3 and the second one is on page 112 at the end of the 2nd paragraph above 3.18.1. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Wed Nov 21 19:44:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA14847 for ; Wed, 21 Nov 2001 19:44:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fALNg4i19132 for ips-outgoing; Wed, 21 Nov 2001 18:42:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12801.mail.yahoo.com (web12801.mail.yahoo.com [216.136.174.36]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fALNSOl18228 for ; Wed, 21 Nov 2001 18:28:25 -0500 (EST) Message-ID: <20011121232823.14517.qmail@web12801.mail.yahoo.com> Received: from [207.219.118.250] by web12801.mail.yahoo.com via HTTP; Wed, 21 Nov 2001 15:28:23 PST Date: Wed, 21 Nov 2001 15:28:23 -0800 (PST) From: Luben Tuikov Subject: CRC32C magic value -> shouldn't it be 0 To: iscsi MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello there, I've been implementing CRC32C for several days now and I was NEVER able to get the results of the test cases of iSCSI draft v8. I got to a point where I worked out the math myself and the results I get are consistent. I have two implementations, one simple shifting and another with table lookup. They both yield the same results. The problem is that any legal transformations, reorderings, etc, as specified by the link, TCP, Ethernet, etc. protocols are transparent to the sender and the receiver. E.g. data sent is the same data received (assuming there was no noise or data corruption -- i.e. illegal transformations). If this is observed then I get ``magic'' value of 0 rather than 0x1c2d19ed. Here is my reasoning: In the context of pp. 150-151 of iSCSI draft v8: M(x) has been built MSB(lsb), i.e. the coefficient to x^(n-1), a_(n-1) is the LS bit of the MS Byte, ... so on to a_0 which is the MS bit of the LS byte. Let M'(x) = x^32*M(x). Then R(x) is computed as usual and then R(x) = ~R(x) as per the draft, i.e. complement the coefficeints to yield r_31, ..., r_0, which are appended in that order to the message. Call this message M''(x). Now M''(x) looks like this: M''(x) = a_(n+31)*x^(n+31) + ... + r_31*x^31 + ... + r_0. Now, this M''(x) is sent verbatim, and then received verbatim on the other end of the wire -- as mentioned above -- assuming there was no noise (all other link, Ethernet, etc. transformations are transparent to us). Now that the message has been received: Then R(x) is computed as usual and then R(x) = ~R(x) (note the algorithm symmetricity). At this point I get 0 and not 0x1c2d19ed. The reason being is that M''(x) has a remainder ~0, since the remainder was complemented before attaching it to M'(x) to form M''(x). But since we always complement, we get R(x) = ~~R(x) (once before sending and once after receiving) so the remainder is 0 as it should be. Any comments, pointers, etc. would be greatly appreciated. -l ===== -- __________________________________________________ Do You Yahoo!? Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. http://geocities.yahoo.com/ps/info1 From owner-ips@ece.cmu.edu Thu Nov 22 02:20:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA09686 for ; Thu, 22 Nov 2001 02:20:46 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAM6aOn11322 for ips-outgoing; Thu, 22 Nov 2001 01:36:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAM6aMl11317 for ; Thu, 22 Nov 2001 01:36:22 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA36984 for ; Thu, 22 Nov 2001 07:36:16 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fAM6aKr128250 for ; Thu, 22 Nov 2001 07:36:25 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: 09 Draft TTT X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 22 Nov 2001 08:36:20 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 22/11/2001 08:36:29, Serialize complete at 22/11/2001 08:36:29 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk thanks - Happy Thanksgiving, Julo John Hufferd@IBMUS 22-11-01 00:10 To: Julian Satran/Haifa/IBM@IBMIL@IBMDE cc: From: John Hufferd/San Jose/IBM@IBMUS Subject: iSCSI: 09 Draft TTT Julian, (minor editing items) I have found two places that still call the TTT as Target Task Tag instead of Target Transfer Tag. The first one is on page 89 third paragraph under 3.11.3 and the second one is on page 112 at the end of the 2nd paragraph above 3.18.1. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Thu Nov 22 05:16:50 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA12030 for ; Thu, 22 Nov 2001 05:16:50 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAM9Ltx01276 for ips-outgoing; Thu, 22 Nov 2001 04:21:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dcmtechdom.dcmtech.co.in ([203.190.136.130]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAM9Lnl01270 for ; Thu, 22 Nov 2001 04:21:51 -0500 (EST) Received: by DCMTECHDOM with Internet Mail Service (5.5.2653.19) id ; Thu, 22 Nov 2001 14:54:34 +0530 Message-ID: <7FADCB99FC82D41199F9000629A85D1A0293A683@DCMTECHDOM> From: Nitin Dhingra To: "'Julian Satran'" , ips@ece.cmu.edu Subject: iScsi: InitialR2T Date: Thu, 22 Nov 2001 14:54:25 +0530 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Under The Section "InitialR2T" the last sentence says "Note that only the first outgoing data burst (immediate data and/or or separate PDUs) can be sent unsolicited by an R2T." My question : If you are sending unsolicited data why do you send an R2T for that? - Nitin From owner-ips@ece.cmu.edu Thu Nov 22 05:18:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA12084 for ; Thu, 22 Nov 2001 05:18:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAM9aTS01993 for ips-outgoing; Thu, 22 Nov 2001 04:36:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mayfair.vxindia.veritas.com (bay-bridge.veritas.com [143.127.3.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAM9aQl01986 for ; Thu, 22 Nov 2001 04:36:26 -0500 (EST) Received: from RAHULB (rahulb.vxindia.veritas.com [10.212.80.59]) by mayfair.vxindia.veritas.com (8.9.3/8.9.3) with SMTP id PAA22658; Thu, 22 Nov 2001 15:06:01 +0530 (IST) Message-ID: <00ea01c17339$30677060$3b50d40a@vxindia.veritas.com> From: "Rahul Bhagwat" To: "Nitin Dhingra" , "'Julian Satran'" , References: <7FADCB99FC82D41199F9000629A85D1A0293A683@DCMTECHDOM> Subject: Re: iScsi: InitialR2T Date: Thu, 22 Nov 2001 15:06:38 +0530 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi, The use of InitialR2T=no means unsolicited data. The way the specs look at unsolicited data is "solicited data with an implied R2T". Regards, Rahul ----- Original Message ----- From: "Nitin Dhingra" To: "'Julian Satran'" ; Sent: Thursday, November 22, 2001 2:54 PM Subject: iScsi: InitialR2T > > Under The Section "InitialR2T" the last sentence says > > "Note that only the first outgoing data burst (immediate data and/or or > separate PDUs) can be sent unsolicited by an R2T." > > My question : If you are sending unsolicited data why do you send an R2T for > that? > > - Nitin > From owner-ips@ece.cmu.edu Thu Nov 22 05:59:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA12514 for ; Thu, 22 Nov 2001 05:59:03 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAMA8xd03616 for ips-outgoing; Thu, 22 Nov 2001 05:08:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dcmtechdom.dcmtech.co.in ([203.190.136.130]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAMA8tl03608 for ; Thu, 22 Nov 2001 05:08:56 -0500 (EST) Received: by DCMTECHDOM with Internet Mail Service (5.5.2653.19) id ; Thu, 22 Nov 2001 15:41:41 +0530 Message-ID: <7FADCB99FC82D41199F9000629A85D1A0293A684@DCMTECHDOM> From: Nitin Dhingra To: "'Rahul Bhagwat'" , "'Julian Satran'" , ips@ece.cmu.edu Subject: RE: iScsi: InitialR2T Date: Thu, 22 Nov 2001 15:41:39 +0530 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Unsolicited data means "solicited data with an implied R2T" very fine I have no objections I'll put my question in a different manner: If you are sending unsolicited data ( means implied R2T's no R2T's to be sent from the target ) Then why do you need to send an R2T for the first outgoing data if InitialR2T is "no"? Let me know if somehow I have misinterpreted the last sentence of InitialR2T. - nitin -----Original Message----- From: Rahul Bhagwat [mailto:rahulb@veritas.com] Sent: Thursday, November 22, 2001 1:37 AM To: Nitin Dhingra; 'Julian Satran'; ips@ece.cmu.edu Subject: Re: iScsi: InitialR2T Hi, The use of InitialR2T=no means unsolicited data. The way the specs look at unsolicited data is "solicited data with an implied R2T". Regards, Rahul ----- Original Message ----- From: "Nitin Dhingra" To: "'Julian Satran'" ; Sent: Thursday, November 22, 2001 2:54 PM Subject: iScsi: InitialR2T > > Under The Section "InitialR2T" the last sentence says > > "Note that only the first outgoing data burst (immediate data and/or or > separate PDUs) can be sent unsolicited by an R2T." > > My question : If you are sending unsolicited data why do you send an R2T for > that? > > - Nitin > From owner-ips@ece.cmu.edu Thu Nov 22 06:34:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA13003 for ; Thu, 22 Nov 2001 06:34:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAMAcW605143 for ips-outgoing; Thu, 22 Nov 2001 05:38:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from bramg1.net.external.hp.com (bramg1.net.external.hp.com [192.6.126.73]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAMAcUl05139 for ; Thu, 22 Nov 2001 05:38:30 -0500 (EST) Received: from fowey.BR.ITC.HP.COM (fowey.br.itc.hp.com [15.145.8.186]) by bramg1.net.external.hp.com (Postfix) with SMTP id 0AD0568; Thu, 22 Nov 2001 11:38:25 +0100 (MET) Received: from 15.145.8.186 by fowey.BR.ITC.HP.COM (InterScan E-Mail VirusWall NT); Thu, 22 Nov 2001 10:38:24 -0000 (GMT Standard Time) Received: by fowey.br.itc.hp.com with Internet Mail Service (5.5.2653.19) id ; Thu, 22 Nov 2001 10:38:24 -0000 Message-ID: <0B9A57FF1D57D411B47500D0B73E5CC104FF1CD2@dickens.bri.hp.com> From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" To: "'Nitin Dhingra'" , "'Julian Satran'" , ips@ece.cmu.edu Subject: RE: iScsi: InitialR2T Date: Thu, 22 Nov 2001 10:38:10 -0000 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Nitin, Perhaps I can explain: Under normal (default) circumstances after sending a SCSI Command PDU (with W=1) the initiator must wait for the target to issue an R2T before it can send any Data-out PDUs. This first R2T is what is termed the InitialR2T. However, as you have encountered there is a text parameter that can negotiated (InitialR2T) which (if both sides agree) allow the initiator to send a burst of data no longer than the FirstBurstSize. This first burst can be in one or more Data-out PDUs (depending on the value of MaxRecvPDULength). After the first burst (denoted by the F bit being set in the last Data-PDU of the first burst), the initiator MUST wait for an R2T before sending the next burst of data. This next burst is then solicited. To be able to make use of the unsolicited feature, both sides must agree to it by each side sending InitialR2T=no. If either party sends InitialR2T=yes then I'm afraid all data has to be sent solicited. I'd agreed that the sentence is a little misleading so perhaps it should read: "Note that only the first outgoing data burst (immediate data and/or or separate PDUs) can be sent unsolicited (i.e. not requiring an explicit R2T)." Julian, please could you make the above change. Cheers Matthew Burbridge Senior Development Engineer NIS-Bristol Hewlett Packard Telnet: 312 7010 E-mail: matthewb@bri.hp.com -----Original Message----- From: Nitin Dhingra [mailto:nitin.dhingra@dcmtech.co.in] Sent: Thursday, November 22, 2001 10:12 AM To: 'Rahul Bhagwat'; 'Julian Satran'; ips@ece.cmu.edu Subject: RE: iScsi: InitialR2T Unsolicited data means "solicited data with an implied R2T" very fine I have no objections I'll put my question in a different manner: If you are sending unsolicited data ( means implied R2T's no R2T's to be sent from the target ) Then why do you need to send an R2T for the first outgoing data if InitialR2T is "no"? Let me know if somehow I have misinterpreted the last sentence of InitialR2T. - nitin -----Original Message----- From: Rahul Bhagwat [mailto:rahulb@veritas.com] Sent: Thursday, November 22, 2001 1:37 AM To: Nitin Dhingra; 'Julian Satran'; ips@ece.cmu.edu Subject: Re: iScsi: InitialR2T Hi, The use of InitialR2T=no means unsolicited data. The way the specs look at unsolicited data is "solicited data with an implied R2T". Regards, Rahul ----- Original Message ----- From: "Nitin Dhingra" To: "'Julian Satran'" ; Sent: Thursday, November 22, 2001 2:54 PM Subject: iScsi: InitialR2T > > Under The Section "InitialR2T" the last sentence says > > "Note that only the first outgoing data burst (immediate data and/or or > separate PDUs) can be sent unsolicited by an R2T." > > My question : If you are sending unsolicited data why do you send an R2T for > that? > > - Nitin > From owner-ips@ece.cmu.edu Thu Nov 22 06:35:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA13018 for ; Thu, 22 Nov 2001 06:35:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAMB5H706442 for ips-outgoing; Thu, 22 Nov 2001 06:05:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailnpd.hcltech.com ([203.197.145.76]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAMB5El06437 for ; Thu, 22 Nov 2001 06:05:14 -0500 (EST) Received: from priya-pc (priya-pc.hcltech.com [192.168.100.88]) by mailnpd.hcltech.com (8.11.0/8.11.0) with SMTP id fAMAwSe30249; Thu, 22 Nov 2001 16:28:30 +0530 Content-Type: text/plain; charset="iso-8859-1" From: Priya Viswambharan To: Nitin Dhingra , "'Julian Satran'" , ips@ece.cmu.edu Subject: Re: iScsi: InitialR2T Date: Thu, 22 Nov 2001 16:30:19 +0530 X-Mailer: KMail [version 1.2] References: <7FADCB99FC82D41199F9000629A85D1A0293A683@DCMTECHDOM> In-Reply-To: <7FADCB99FC82D41199F9000629A85D1A0293A683@DCMTECHDOM> MIME-Version: 1.0 Message-Id: <0111221630190T.20183@priya-pc> Content-Transfer-Encoding: 8bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hi Nitin, The way to interpret "Note that only the first outgoing data burst (immediate data and/or or separate PDUs) can be sent unsolicited by an R2T." is that only the first outgoing data burst can be sent without having been solicited by an R2T. Thanks Priya On Thursday 22 November 2001 02:54 pm, Nitin Dhingra wrote: > Under The Section "InitialR2T" the last sentence says > > "Note that only the first outgoing data burst (immediate > data and/or or separate PDUs) can be sent unsolicited by > an R2T." > > My question : If you are sending unsolicited data why do > you send an R2T for that? > > - Nitin From owner-ips@ece.cmu.edu Thu Nov 22 11:05:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17829 for ; Thu, 22 Nov 2001 11:05:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAMFBnW19125 for ips-outgoing; Thu, 22 Nov 2001 10:11:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12801.mail.yahoo.com (web12801.mail.yahoo.com [216.136.174.36]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAM82wl15820 for ; Thu, 22 Nov 2001 03:02:58 -0500 (EST) Message-ID: <20011122080257.73684.qmail@web12801.mail.yahoo.com> Received: from [24.42.134.59] by web12801.mail.yahoo.com via HTTP; Thu, 22 Nov 2001 00:02:57 PST Date: Thu, 22 Nov 2001 00:02:57 -0800 (PST) From: Luben Tuikov Subject: Cont'd: CRC32C magic value -> shouldn't it be 0 To: iscsi In-Reply-To: <20011121232823.14517.qmail@web12801.mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk (Cont'd from my previous message which is attached at the end.) Furthermore I can XOR any value to R(x), recall that in aritmetic modulo 2 XOR is both addition and subtraction, to get that SAME value on the other end when ``unscrambling'' the message. (This result is detailed here below.) This means that, just before sending the message over, M''(x), I can XOR _any_ 32 bit unsigned value to the last 32 bits of M''(x), in order to get that SAME value at the receiver's end... So, I XOR-ed 0x1c2d19ed _just before sending over the wire_ just to get that same value over at the receiver's end after CRC checking. Reason being: M''(x) has no remainder* when divided by G(x) modulo 2, so M''(x) + A(x), where A(x) is a polynomial of degree 31 or less (that's important), will yield a remainder A when divided by G(x) which has degree 32 (A(x) not divisible by G(x) modulo 2, since A(x) < G(x) in context arithmetic modulo 2). * Actually it has, 32 1's, since we had to complement R(x) before appending it to M'(x) (as per the draft), but for sake of simplicity lets ignore this for now. All this means that if I receive a message from your implementations, and XOR 0x1c2d19ed just after I get it from the wire, and then do CRC checking I will get remainder 0. So why XOR it in the first place? Please also note that complementing R(x) gives no apparent edge as it is complemented again by the symmetricity of the algorithm. Please comment on this matter. -l --- Luben Tuikov wrote: > Hello there, > > I've been implementing CRC32C for several days now and I was NEVER > able to get the results of the test cases of iSCSI draft v8. > > I got to a point where I worked out the math myself and the results I > get are consistent. > > I have two implementations, one simple shifting and another with > table > lookup. They both yield the same results. > > The problem is that any legal transformations, reorderings, etc, as > specified by the link, TCP, Ethernet, etc. protocols are transparent > to > the sender and the receiver. E.g. data sent is the same data received > (assuming there was no noise or data corruption -- i.e. illegal > transformations). > > If this is observed then I get ``magic'' value of 0 rather than > 0x1c2d19ed. Here is my reasoning: > > In the context of pp. 150-151 of iSCSI draft v8: > > M(x) has been built MSB(lsb), i.e. the coefficient to x^(n-1), > a_(n-1) > is the LS bit of the MS Byte, ... so on to a_0 which is the MS bit of > the LS byte. > > Let M'(x) = x^32*M(x). > > Then R(x) is computed as usual and then R(x) = ~R(x) as per the > draft, > i.e. complement the coefficeints to yield r_31, ..., r_0, which are > appended in that order to the message. Call this message M''(x). > > Now M''(x) looks like this: > M''(x) = a_(n+31)*x^(n+31) + ... + r_31*x^31 + ... + r_0. > > Now, this M''(x) is sent verbatim, and then received verbatim on the > other end of the wire -- as mentioned above -- assuming there was no > noise (all other link, Ethernet, etc. transformations are transparent > to us). > > Now that the message has been received: > Then R(x) is computed as usual and then R(x) = ~R(x) (note the > algorithm symmetricity). At this point I get 0 and not 0x1c2d19ed. > The > reason being is that M''(x) has a remainder ~0, since the remainder > was > complemented before attaching it to M'(x) to form M''(x). But since > we > always complement, we get R(x) = ~~R(x) (once before sending and once > after receiving) so the remainder is 0 as it should be. > > Any comments, pointers, etc. would be greatly appreciated. > > -l > > > > ===== > -- > > __________________________________________________ > Do You Yahoo!? > Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. > http://geocities.yahoo.com/ps/info1 ===== -- __________________________________________________ Do You Yahoo!? Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. http://geocities.yahoo.com/ps/info1 From owner-ips@ece.cmu.edu Fri Nov 23 05:24:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA16254 for ; Fri, 23 Nov 2001 05:24:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAN9UHi14439 for ips-outgoing; Fri, 23 Nov 2001 04:30:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAN9UGl14435 for ; Fri, 23 Nov 2001 04:30:16 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id EAA86756 for ; Fri, 23 Nov 2001 04:27:33 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAN9UDT216724 for ; Fri, 23 Nov 2001 02:30:13 -0700 X-Priority: 1 (High) Importance: Normal Subject: iSCSI:Clear Task Set To: "Julian Satran" Cc: "ips" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Fri, 23 Nov 2001 01:29:21 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/23/2001 02:30:13 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, and List (using v 0.9) In point 9.4, just before 9.5 the Table entry associated with Clear Task Set applies to: "All tasks associated with the specified LUN and initiator. For all other initiators all tasks at LUN with no regard to order." Perhaps we mean LU here, but I know that the iSCSI layer does not have information about LU, only about the LU Number (LUN) in the command. We can not tell, at the iSCSI layer, if the LU represented by a LUN on Session 1, has any relationship to any LUN on any other session. This is because each initiator may have their own numbering for LUs. Therefore, do we just pass the Clear Task Set to the SCSI layer and hope for the best, or does the iSCSI layer also suppose to apply the Clear Task Set to all the sessions that it has coming into the iSCSI (SCSI) Target Port? If the latter, again how will that work when the iSCSI layer has no idea what LU an Initiator's LUN will map to? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Fri Nov 23 11:52:12 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA19350 for ; Fri, 23 Nov 2001 11:52:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fANFuWM14873 for ips-outgoing; Fri, 23 Nov 2001 10:56:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fANFuVl14869 for ; Fri, 23 Nov 2001 10:56:31 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA69426 for ; Fri, 23 Nov 2001 16:56:23 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v4.98) with ESMTP id fANFuZV167904 for ; Fri, 23 Nov 2001 16:56:35 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI:Clear Task Set X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Fri, 23 Nov 2001 17:56:36 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 23/11/2001 17:56:39, Serialize complete at 23/11/2001 17:56:39 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, The LUN is just a mistake there - in all three instances it should be LU. The definitions are in accordance with SAM . There are two task management modes - tasks sets-per-initiator at each LU or common for all initiators. The mode is a SCSI issue controlled by a field in the Control-Mode page. Clear task set MAY clear all the tasks in the task set - even if common to all initiators if that is the way the task set is managed. That is also the difference between clear-task-set and abort task set. Julo John Hufferd@IBMUS 23-11-01 11:29 To: Julian Satran/Haifa/IBM@IBMIL@IBMDE cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: iSCSI:Clear Task Set Julian, and List (using v 0.9) In point 9.4, just before 9.5 the Table entry associated with Clear Task Set applies to: "All tasks associated with the specified LUN and initiator. For all other initiators all tasks at LUN with no regard to order." Perhaps we mean LU here, but I know that the iSCSI layer does not have information about LU, only about the LU Number (LUN) in the command. We can not tell, at the iSCSI layer, if the LU represented by a LUN on Session 1, has any relationship to any LUN on any other session. This is because each initiator may have their own numbering for LUs. Therefore, do we just pass the Clear Task Set to the SCSI layer and hope for the best, or does the iSCSI layer also suppose to apply the Clear Task Set to all the sessions that it has coming into the iSCSI (SCSI) Target Port? If the latter, again how will that work when the iSCSI layer has no idea what LU an Initiator's LUN will map to? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Fri Nov 23 12:45:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA21237 for ; Fri, 23 Nov 2001 12:45:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fANGuQ118017 for ips-outgoing; Fri, 23 Nov 2001 11:56:26 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fANGuNl18010 for ; Fri, 23 Nov 2001 11:56:24 -0500 (EST) Received: from deneb.dev.equallogic.com (localhost.localdomain [127.0.0.1]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fANGuGG18800; Fri, 23 Nov 2001 11:56:17 -0500 Received: from PKONING.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.6/8.11.2) with ESMTP id fANGuFI31126; Fri, 23 Nov 2001 11:56:16 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15358.32709.807000.493652@gargle.gargle.HOWL> Date: Fri, 23 Nov 2001 11:56:37 -0500 From: Paul Koning To: ltuikov@yahoo.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI v8 CRC32C References: <20011120210103.47775.qmail@web12805.mail.yahoo.com> X-Mailer: VM 6.95 under 21.4 (patch 3) "Academic Rigor" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Excerpt of message (sent 20 November 2001) by Luben Tuikov: > Hello, > > I've been trying to get CRC32C working. I read a few papers, > derived the equations myself, etc, etc, but still cannot get > CRC32C to work... A good paper to read is the classic Ethernet spec (V1, not 802.3). In particular, see appendix C. > My basic framework is: (not precomputing a table -- just force): > > poly= 1edc6f41; > crc = all 1's; > while more bits in message do > crc shift left, adding the next message bit at right; > if carry then > crc = crc XOR poly; That looks wrong; the Ethernet spec as I read it seems to say you need "if (carry xor messagebit)" > end if; > end while; > crc = NOT crc; (e.g. complement the register) Re your later note: if you get a 0 remainder, you're missing something. Probably the fact that the initial all 1 value is mathematically equivalent to complementing the initial 32 bits of the message. paul From owner-ips@ece.cmu.edu Fri Nov 23 16:13:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA23511 for ; Fri, 23 Nov 2001 16:13:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fANKSqr29329 for ips-outgoing; Fri, 23 Nov 2001 15:28:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12806.mail.yahoo.com (web12806.mail.yahoo.com [216.136.174.41]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fANKSol29324 for ; Fri, 23 Nov 2001 15:28:51 -0500 (EST) Message-ID: <20011123202850.70352.qmail@web12806.mail.yahoo.com> Received: from [207.219.118.250] by web12806.mail.yahoo.com via HTTP; Fri, 23 Nov 2001 12:28:50 PST Date: Fri, 23 Nov 2001 12:28:50 -0800 (PST) From: Luben Tuikov Subject: Re: iSCSI v8 CRC32C To: Paul Koning Cc: ips@ece.cmu.edu In-Reply-To: <15358.32709.807000.493652@gargle.gargle.HOWL> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Paul, thank you for replying. --- Paul Koning wrote: > > A good paper to read is the classic Ethernet spec (V1, > not 802.3). In particular, see appendix C. Yes, no doubt it is. But I think that the iSCSI draft gives enough information and specs as to the implementation of CRC32C. > That looks wrong; the Ethernet spec as I read it seems to > say you need "if (carry xor messagebit)" The algorithm presented is a classical algorithm on computing the CRC (i.e. the remainder). If Ethernet uses a different if-condition then they must be implementing a slightly different algorithm. > Re your later note: if you get a 0 remainder, you're > missing something. On the contrary -- means that there was no noise or I struck luck to get undetectable error (1 in 10^-40). > Probably the fact that the initial all 1 > value is mathematically equivalent to complementing the > initial 32 bits of the message. As the draft points out, setting the register to all 1's, is just like complementing the first 32 bits of the message to 1. E.g. imagine your message always starting with 32 1's. This is so that runs of 0's at the beginning of the message are caught. -- Luben Tuikov, Senior Software Engineer, Splentec Ltd. Bus: 905-707-1954x112, 9-5 EST. Fax: 905-707-1974. ===== -- __________________________________________________ Do You Yahoo!? Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. http://geocities.yahoo.com/ps/info1 From owner-ips@ece.cmu.edu Fri Nov 23 17:13:57 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA24219 for ; Fri, 23 Nov 2001 17:13:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fANLVHl02486 for ips-outgoing; Fri, 23 Nov 2001 16:31:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fANLVFl02482 for ; Fri, 23 Nov 2001 16:31:15 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA17000 for ; Fri, 23 Nov 2001 16:28:33 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fANLVEL169746 for ; Fri, 23 Nov 2001 14:31:14 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI:Clear Task Set To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Fri, 23 Nov 2001 13:30:04 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/23/2001 02:31:13 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, and list, The question now becomes, if we have all that carefully thought out processing that is defined in 9.4 in how to handle the other Tasks/Commands that are "In Flight", and not yet given to SCSI, how does that apply to the other Sessions with other initiators? That is, at the iSCSI Target layer we do not have the LU Number to LU mapping on any Session with any Initiator, so how do we cause the careful processing, which is defined in 9.4, to occur on the other sessions that may have and association to the subject LU? Especially since all we know is a LU Number on the Clearing Session. Perhaps we do not care about the 9.4 processing on the other Session and Other Initiators and just let SCSI layer do its thing, and at the iSCSI layer we pay no attention to the other Sessions. Do you think this is correct? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/23/2001 07:56:36 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI:Clear Task Set John, The LUN is just a mistake there - in all three instances it should be LU. The definitions are in accordance with SAM . There are two task management modes - tasks sets-per-initiator at each LU or common for all initiators. The mode is a SCSI issue controlled by a field in the Control-Mode page. Clear task set MAY clear all the tasks in the task set - even if common to all initiators if that is the way the task set is managed. That is also the difference between clear-task-set and abort task set. Julo John Hufferd@IBMUS 23-11-01 11:29 To: Julian Satran/Haifa/IBM@IBMIL@IBMDE cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: iSCSI:Clear Task Set Julian, and List (using v 0.9) In point 9.4, just before 9.5 the Table entry associated with Clear Task Set applies to: "All tasks associated with the specified LUN and initiator. For all other initiators all tasks at LUN with no regard to order." Perhaps we mean LU here, but I know that the iSCSI layer does not have information about LU, only about the LU Number (LUN) in the command. We can not tell, at the iSCSI layer, if the LU represented by a LUN on Session 1, has any relationship to any LUN on any other session. This is because each initiator may have their own numbering for LUs. Therefore, do we just pass the Clear Task Set to the SCSI layer and hope for the best, or does the iSCSI layer also suppose to apply the Clear Task Set to all the sessions that it has coming into the iSCSI (SCSI) Target Port? If the latter, again how will that work when the iSCSI layer has no idea what LU an Initiator's LUN will map to? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Fri Nov 23 17:14:56 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA24231 for ; Fri, 23 Nov 2001 17:14:56 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fANM62f04291 for ips-outgoing; Fri, 23 Nov 2001 17:06:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antipater.hosting.pacbell.net (antipater.hosting.pacbell.net [216.100.99.13]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fANM60l04287 for ; Fri, 23 Nov 2001 17:06:01 -0500 (EST) Received: from gwendal.corp.silverbacksystems.com (sdsl-66-80-0-42.dsl.sca.megapath.net [66.80.0.42]) by antipater.hosting.pacbell.net id RAA09952; Fri, 23 Nov 2001 17:05:44 -0500 (EST) [ConcentricHost SMTP Relay 1.7] Content-Type: text/plain; charset="iso-8859-1" From: Gwendal Grignou To: "ips" Subject: Task management questions Date: Fri, 23 Nov 2001 14:03:02 -0800 X-Mailer: KMail [version 1.2] MIME-Version: 1.0 Message-Id: <01112314030202.01156@gwendal.corp.silverbacksystems.com> Content-Transfer-Encoding: 8bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit I have a question about the section 3.5.1 page 69. The paragrah starting with "ABORT TASK MUST",the last sentene says "The task management response MUST be issued after the command status (if any) was issued." : it means the initiator will receive, on the same connection, a status for the aborted task and AFTER, a response for the task management function. In the paragraph before (starting with "For all these functions,",), it is specified that the target should send a status for all aborted tasks, but the task management response can happen at any time after the task management request has been received by the target "The task management response MAY be issued by the target immediately after marking all tasks to be aborted." Why such a difference between "ABORT TASK" function and other functions ? About, "LOGICAL UNIT RESET", it is stated "the target MUST behave as dictated by the Logical Unit Reset function in [SAM2]." However, in SAM2, taks should be aborted as described in section 5.6 of SAM2, which says , in section 5.6.2, about initiator aborting their own task "When an initiator causes its own task(s) to be aborted, no notification that the task(s) have been aborted shall be returned to the initiator other than the completion response for the command or task management function action that caused the task(s) to be aborted and notification(s) associated with related effects of the action (e.g., a target reset unit attention condition)." Does it means the iSCSI target may not send a status for the aborted tasks in case of a "LOGICAL UNIT RESET" ? (and subsequently for TARGET RESET [WARM or COLD] which triggers LOGICAL UNIT RESET for each LUs [SAM2, section 5.8.6]). It does not match with the requirement in the paragraph concerning all the task management functions in general. Can you elaborate more on those requirements ? Thanks a lot, Gwendal. From owner-ips@ece.cmu.edu Fri Nov 23 17:55:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA24487 for ; Fri, 23 Nov 2001 17:55:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fANMIni04948 for ips-outgoing; Fri, 23 Nov 2001 17:18:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fANMIml04942 for ; Fri, 23 Nov 2001 17:18:48 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA35194 for ; Fri, 23 Nov 2001 17:16:05 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fANMIkL221140 for ; Fri, 23 Nov 2001 15:18:46 -0700 X-Priority: 1 (High) Importance: Normal Subject: iSCSI: Abort Task Set To: "Julian Satran" , ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Fri, 23 Nov 2001 14:17:55 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/23/2001 03:18:46 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, and List, The discription of the Abort Task Set Function in 3.5.1 says it: "aborts all Tasks issued by this initiator on the Logical Unit." The question is, what do we mean by initiator, in this case. Do we mean iSCSI Initiator Node, or do we mean iSCSI (SCSI) Initiator Port. Since we could have several different Session between the iSCSI Initiator Node and the SCSI Target Port, but only one between the SCSI Initiator Port and an SCSI Target Port. So if we mean the latter, perhaps we should change the wordage to: "aborts all Tasks issued via this Session on the Logical Unit." . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Sat Nov 24 02:49:23 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA16642 for ; Sat, 24 Nov 2001 02:49:23 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAO76Rs01154 for ips-outgoing; Sat, 24 Nov 2001 02:06:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAO76Ol01146 for ; Sat, 24 Nov 2001 02:06:24 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA48388 for ; Sat, 24 Nov 2001 08:06:18 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAO76TX37858 for ; Sat, 24 Nov 2001 08:06:29 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI:Clear Task Set X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 24 Nov 2001 09:06:29 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 24/11/2001 09:06:34, Serialize complete at 24/11/2001 09:06:34 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, That looks more like in T10 territory. T10 defines differently Abort Task Set and Clear Task Set. We could either: decide not to implement clear task set (T10 allows that but "per target" not "per transport") enable clear task set - in which case we have to say something about the relative order of the task management request with regard to the task comming from other initiators - and that is what I attempted to say in 9.4 Julo John Hufferd/San Jose/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 23-11-01 23:30 To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: iSCSI:Clear Task Set Julian, and list, The question now becomes, if we have all that carefully thought out processing that is defined in 9.4 in how to handle the other Tasks/Commands that are "In Flight", and not yet given to SCSI, how does that apply to the other Sessions with other initiators? That is, at the iSCSI Target layer we do not have the LU Number to LU mapping on any Session with any Initiator, so how do we cause the careful processing, which is defined in 9.4, to occur on the other sessions that may have and association to the subject LU? Especially since all we know is a LU Number on the Clearing Session. Perhaps we do not care about the 9.4 processing on the other Session and Other Initiators and just let SCSI layer do its thing, and at the iSCSI layer we pay no attention to the other Sessions. Do you think this is correct? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/23/2001 07:56:36 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI:Clear Task Set John, The LUN is just a mistake there - in all three instances it should be LU. The definitions are in accordance with SAM . There are two task management modes - tasks sets-per-initiator at each LU or common for all initiators. The mode is a SCSI issue controlled by a field in the Control-Mode page. Clear task set MAY clear all the tasks in the task set - even if common to all initiators if that is the way the task set is managed. That is also the difference between clear-task-set and abort task set. Julo John Hufferd@IBMUS 23-11-01 11:29 To: Julian Satran/Haifa/IBM@IBMIL@IBMDE cc: ips@ece.cmu.edu From: John Hufferd/San Jose/IBM@IBMUS Subject: iSCSI:Clear Task Set Julian, and List (using v 0.9) In point 9.4, just before 9.5 the Table entry associated with Clear Task Set applies to: "All tasks associated with the specified LUN and initiator. For all other initiators all tasks at LUN with no regard to order." Perhaps we mean LU here, but I know that the iSCSI layer does not have information about LU, only about the LU Number (LUN) in the command. We can not tell, at the iSCSI layer, if the LU represented by a LUN on Session 1, has any relationship to any LUN on any other session. This is because each initiator may have their own numbering for LUs. Therefore, do we just pass the Clear Task Set to the SCSI layer and hope for the best, or does the iSCSI layer also suppose to apply the Clear Task Set to all the sessions that it has coming into the iSCSI (SCSI) Target Port? If the latter, again how will that work when the iSCSI layer has no idea what LU an Initiator's LUN will map to? . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Sat Nov 24 02:54:45 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA16678 for ; Sat, 24 Nov 2001 02:54:45 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAO7BqU01435 for ips-outgoing; Sat, 24 Nov 2001 02:11:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAO7Bol01429 for ; Sat, 24 Nov 2001 02:11:50 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA18364 for ; Sat, 24 Nov 2001 08:11:44 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAO7Bsw120244 for ; Sat, 24 Nov 2001 08:11:54 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: Abort Task Set X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sat, 24 Nov 2001 09:11:55 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 24/11/2001 09:11:59, Serialize complete at 24/11/2001 09:11:59 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, The meaning is SCSI initiator - i.e. SCSI initiator port. That is defined as the default meaning at the begining of the document. T10 defines the command meaning. Julo John Hufferd@IBMUS 24-11-01 00:17 To: Julian Satran/Haifa/IBM@IBMIL@IBMDE, ips@ece.cmu.edu cc: From: John Hufferd/San Jose/IBM@IBMUS Subject: iSCSI: Abort Task Set Julian, and List, The discription of the Abort Task Set Function in 3.5.1 says it: "aborts all Tasks issued by this initiator on the Logical Unit." The question is, what do we mean by initiator, in this case. Do we mean iSCSI Initiator Node, or do we mean iSCSI (SCSI) Initiator Port. Since we could have several different Session between the iSCSI Initiator Node and the SCSI Target Port, but only one between the SCSI Initiator Port and an SCSI Target Port. So if we mean the latter, perhaps we should change the wordage to: "aborts all Tasks issued via this Session on the Logical Unit." . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Sat Nov 24 04:16:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA18677 for ; Sat, 24 Nov 2001 04:16:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAO8AxX04135 for ips-outgoing; Sat, 24 Nov 2001 03:10:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12802.mail.yahoo.com (web12802.mail.yahoo.com [216.136.174.37]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAO8Avl04130 for ; Sat, 24 Nov 2001 03:10:57 -0500 (EST) Message-ID: <20011124081056.98754.qmail@web12802.mail.yahoo.com> Received: from [24.42.134.59] by web12802.mail.yahoo.com via HTTP; Sat, 24 Nov 2001 00:10:56 PST Date: Sat, 24 Nov 2001 00:10:56 -0800 (PST) From: Luben Tuikov Subject: CRC32C code -- please check To: iscsi MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Here is some C code at the end, the draft quoted and my notes alongside it. Please tell me where and what is wrong with the interpretation, logic, etc. > Padding bytes, when present, in a segment covered by a > CRC, should be set to 0 and are included in the CRC. The > CRC should be calculated as follows: > > - data are assumed to be in the numbering order that appears > in the draft - start with byte 0 bit 0 continue byte 1 bit > 0 etc. (Big Endian on bytes / Little Endian on bits) Ok. The code presented satisfies this requirement. I swap the bits in the bytes on the fly. Please note that we DO NOT have to swap the bits in the last 4 bytes representing the CRC. This is clearly specified in the draft. Please also note that the draft doesn't specify how the data is to be transmitted, it only specifies how to build M(x), i.e. MSB(lsb). > - the CRC register is initialized with all 1s (equivalent to > complementing the first 32 bits of the message) Ok. The code also satisfies this requirement. In fact this has no significance to the algorithm and R(x), if the _same_ algorithm is used in both the sender and the receiver. As the draft says, this just complements the first 32 bits of the message, e.g. imagine the first 32 bits of the message were already 1's. > - the n PDU bits are considered coefficients of a polynomial > M(x) of order n-1, with bit 0 of byte 0 being x^(n-1) Ok. > - the polynomial is multiplied by x^32 and divided by G(x)- the > generator polynomial - producing a remainder R(x) of degree <= > 31 > - the coefficients of R(x) are considered a 32 bit sequence So, here we get M'(x) = x^32 * M(x). Getting R(x) is trivial. So we have: M'(x) = Q(x)*G(x) + R(x) = Q(x)*G(x) - R(x), (1) since plus is same as minus in modulo 2 arithmetic. > - the bit sequence is complemented and the result is the CRC Now, from boolean logic we know that b XOR 1 = ~b, then, ~R(x) = R(x) XOR I(x) = R(x) + I(x) (2) where I(x) is a polynomial of the same degree as R(x) and whose coefficients are all equal to 1. So, to get M''(x) which we send we have: M''(x) = M'(x) + ~R(x) (draft) = Q(x)*G(x) - R(x) + R(x) + I(x) (using (1) and (2)) = Q(x)*G(x) + I(x). or succinctly, M''(x) = Q(x)*G(x) + I(x), (3) and this is the message which we send. On the other end we get M''(x), assuming there was no noise, and all link/TCP/Ethernet transformations are transparent for the sender and the receiver. > - after the last bit of the original segment the CRC bits are > transmitted with x^31 first followed by x^30 etc. ( whenever > examples are given the value to be specified in examples > follows the same rules of representation as the rest of this > document) Ok. > - a receiver of a "good" segment (data or header) built using > the generator 0x11edc6f41 will end-up having in the CRC > register the value 0x1c2d19ed (this a register value and not a > word as outlined in this draft) Ok, so the receiver got M''(x) assuming no noise. M''(x) = Q(x)*G(x) + I(x). (this is (3)) Now we work the algorithm again: ..., find the remainder: Clearly, from (3) the remainder of M''(x) modulo G(x) is I(x). Then we complement the value as per the draft: ~I(x) = 0. And we end up with remainder 0. Now let A(x) be the polynomial represented by 0x1c2d19ed. Then, we can add A(x) to M''(x) to get: M'''(x) = Q(x)*G(x) + I(x) + A(x) (from (3)) = Q(x)*G(x) + ~A(x). (from (2)) Now we send M'''(x), we receive M'''(x), and we work the algorithm again. Clearly the remainder is ~A(x) and performing the final complement we get: ~~A(x) = A(x), which is 0x1c2d19ed. So then I can just go ahead and send M'''(x), since iSCSI is expecting to get A(x) as a remainder. The C code: ---------cut-here-------------- /* crc32c.c -- Implement CRC32C: 32 bit CRC of the polynomial represented by 0x11EDC6F41. Copyright (C) 2001 Splentec Ltd. */ #include #include #include #define SAMPLE_SIZE 28 #define BIT(v, bit) ((v) & (((typeof ((v))) 1) << (bit))) const uint32_t Poly = 0x1edc6f41UL; const uint32_t Magic = 0x1c2d19edUL; uint32_t table[256]; /* 28 bytes + 4 bytes for the CRC */ uint8_t zero[SAMPLE_SIZE+4]; uint8_t ones[SAMPLE_SIZE+4]; uint8_t incs[SAMPLE_SIZE+4]; uint8_t decs[SAMPLE_SIZE+4]; int test_packet(uint8_t *packet, size_t size, uint32_t (*crc_func)(uint8_t *, size_t)); int calc_table(uint32_t *table, uint32_t poly); uint32_t crc(uint8_t *p, size_t size); uint32_t crc_force(uint8_t *p, size_t size); static inline uint8_t mirror8(uint8_t v); /* -------------------------------------------------- */ int main() { uint32_t i; printf("Making the samples...\n"); for (i = 0; i < SAMPLE_SIZE; i++) { ones[i] = 0xFF; incs[i] = i; decs[i] = 0x1f - i; } /* -------------------------------------------------- */ printf("Generating the table...\n"); calc_table(table, Poly); /* for (i = 0; i < 256; i++) printf("0x%08xU%s", table[i], (i % 4) == 3 ? ",\n" : ", "); */ /* -------------------------------------------------- */ printf("Testing...with table\n"); test_packet(zero, SAMPLE_SIZE+4, crc); test_packet(ones, SAMPLE_SIZE+4, crc); test_packet(incs, SAMPLE_SIZE+4, crc); test_packet(decs, SAMPLE_SIZE+4, crc); /* -------------------------------------------------- */ printf("Testing...without table\n"); test_packet(zero, SAMPLE_SIZE+4, crc_force); test_packet(ones, SAMPLE_SIZE+4, crc_force); test_packet(incs, SAMPLE_SIZE+4, crc_force); test_packet(decs, SAMPLE_SIZE+4, crc_force); return 0; } /* end main() */ /* -------------------------------------------------- */ /* test an augmented packet */ int test_packet(uint8_t *packet, size_t size, uint32_t (*crc_func)(uint8_t *, size_t)) { uint32_t r; r = crc_func(packet, size); /* Now here we can form M'''(x) by doing the following: r ^= Magic; */ *((uint32_t *)(packet + SAMPLE_SIZE)) = htonl(r); printf("packet %p has CRC %#x -> sending -> ", packet, r); /* The packet is sent and received on the other end. Then we process it: */ r = crc_func(packet, size); printf("packet has CRC %#x\n", r); /* zero out the CRC space for later testing with no table lookup */ packet[SAMPLE_SIZE] = packet[SAMPLE_SIZE+1] = packet[SAMPLE_SIZE+2] = packet[SAMPLE_SIZE+3] = 0; return 0; } /* end test_packet() */ /* -------------------------------------------------- */ int calc_table(uint32_t *table, uint32_t poly) { register uint32_t crc, i; for (i = 0; i < 256; i++) { crc = i << 24; crc = BIT(crc, 31) ? ((crc<<1)^poly) : (crc<<1); /* 0 */ crc = BIT(crc, 31) ? ((crc<<1)^poly) : (crc<<1); /* 1 */ crc = BIT(crc, 31) ? ((crc<<1)^poly) : (crc<<1); /* 2 */ crc = BIT(crc, 31) ? ((crc<<1)^poly) : (crc<<1); /* 3 */ crc = BIT(crc, 31) ? ((crc<<1)^poly) : (crc<<1); /* 4 */ crc = BIT(crc, 31) ? ((crc<<1)^poly) : (crc<<1); /* 5 */ crc = BIT(crc, 31) ? ((crc<<1)^poly) : (crc<<1); /* 6 */ crc = BIT(crc, 31) ? ((crc<<1)^poly) : (crc<<1); /* 7 */ table[i] = crc; } return 0; } /* end calc_table() */ /* -------------------------------------------------- */ /* p points to an augmented message, i.e. 32 bits have been added at the end and set to 0. The last 4 bytes are never mirrored since the draft is clear on that: mirror only when building the polynomial, and send the CRC 31 bit first, then 30, etc. */ uint32_t crc(uint8_t *p, size_t size) { uint32_t r = ~((uint32_t) 0UL); uint8_t mp; for ( ; size-- > 0; p++) { /* do not mirror CRC code! (last 4 bytes) */ mp = (size > 4) ? mirror8(*p) : *p; r = ((r << 8) | mp) ^ table[r >> 24]; } return ~r; } /* end crc() */ /* -------------------------------------------------- */ /* p points to an augmented message, i.e. 32 bits have been added at the end and set to 0. The last 4 bytes are never mirrored since the draft is clear on that: mirror only when building the polynomial, and send the CRC 31 bit first, then 30, etc. */ uint32_t crc_force(uint8_t *p, size_t size) { uint32_t r = ~((uint32_t) 0UL), t; uint8_t mp; int i; for ( ; size-- > 0; p++) { /* do not mirror CRC code (last 4 bytes) */ mp = (size > 4) ? mirror8(*p) : *p; for (i = 7; i >= 0; i--) { t = r; r = (r << 1) | (BIT(mp, i) >> i); if (BIT(t, 31)) r ^= Poly; } } return ~r; } /* end crc_force() */ /* -------------------------------------------------- */ static inline uint8_t mirror8(uint8_t v) { return ((0x80 & v) >> 7) | ((0x40 & v) >> 5) | ((0x20 & v) >> 3) | ((0x10 & v) >> 1) | ((8 & v) << 1) | ((4 & v) << 3) | ((2 & v) << 5) | ((1 & v) << 7); } /* end mirror8() */ /* -------------------------------------------------- */ ---------------------cut-here---------------------- Thanks in advance, Luben ===== -- __________________________________________________ Do You Yahoo!? Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. http://geocities.yahoo.com/ps/info1 From owner-ips@ece.cmu.edu Sat Nov 24 09:58:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA20836 for ; Sat, 24 Nov 2001 09:58:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAODtgl00824 for ips-outgoing; Sat, 24 Nov 2001 08:55:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lucy.trebia.com ([65.192.191.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAODtdl00819 for ; Sat, 24 Nov 2001 08:55:39 -0500 (EST) Received: by lucy with Internet Mail Service (5.5.2653.19) id ; Sat, 24 Nov 2001 08:54:10 -0500 Message-ID: <63616B261CEFD411834D00D0B7A86CF757607D@lucy> From: Sudhir Srinivasan To: "'ENDL_TX@computer.org'" , IPS Reflector Subject: FCIP: v0.7 editorial suggestion Date: Sat, 24 Nov 2001 08:54:10 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, An editorial suggestion: The pFlags definition is: |----------------Bit--------------------| | | | 31 30 29 28 27 26 25 24 | +-----------------------------+----+----+ | Reserved | Ch | SF | +-----------------------------+----+----+ Fig. 8 pFlags Field Bits with the "Ch" bit to the left of the "SF" bit. I suggest the first two columns in Table 1 be swapped to reflect this. When looking at Figure 9, it was slightly confusing at first to map the pFlags value shown in the Short Frame header back to Table 1. Regards, Sudhir From owner-ips@ece.cmu.edu Sat Nov 24 16:25:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA24902 for ; Sat, 24 Nov 2001 16:25:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAOJrUd18257 for ips-outgoing; Sat, 24 Nov 2001 14:53:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAOJrRl18251 for ; Sat, 24 Nov 2001 14:53:27 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA12014 for ; Sat, 24 Nov 2001 14:50:34 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAOJrEV184096; Sat, 24 Nov 2001 12:53:14 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: Abort Task Set To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Sat, 24 Nov 2001 11:52:05 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/24/2001 12:53:14 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, That is fine with me, except we did NOT define "SCSI initiator" to be "SCSI initiator port". The places in the Document that I have see the words SCSI initiator is either Standalone (in areas that could mean SCSI initiator port), or as (pre)qualifiers to "port" or "device", as in SCSI initiator port, or SCSI initiator device. Because "SCSI Initiator" is NOT defined, and could mean port or device. I think we need to be more percicse -- and in 3.5.1 we should say SCSI Initiator Port, not just the word "inititor". . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Julian Satran" @ece.cmu.edu on 11/23/2001 11:11:55 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: Abort Task Set John, The meaning is SCSI initiator - i.e. SCSI initiator port. That is defined as the default meaning at the begining of the document. T10 defines the command meaning. Julo John Hufferd@IBMUS 24-11-01 00:17 To: Julian Satran/Haifa/IBM@IBMIL@IBMDE, ips@ece.cmu.edu cc: From: John Hufferd/San Jose/IBM@IBMUS Subject: iSCSI: Abort Task Set Julian, and List, The discription of the Abort Task Set Function in 3.5.1 says it: "aborts all Tasks issued by this initiator on the Logical Unit." The question is, what do we mean by initiator, in this case. Do we mean iSCSI Initiator Node, or do we mean iSCSI (SCSI) Initiator Port. Since we could have several different Session between the iSCSI Initiator Node and the SCSI Target Port, but only one between the SCSI Initiator Port and an SCSI Target Port. So if we mean the latter, perhaps we should change the wordage to: "aborts all Tasks issued via this Session on the Logical Unit." . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Sat Nov 24 19:19:36 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26529 for ; Sat, 24 Nov 2001 19:19:36 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAONY5529168 for ips-outgoing; Sat, 24 Nov 2001 18:34:05 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAONY3l29164 for ; Sat, 24 Nov 2001 18:34:03 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA23350; Sat, 24 Nov 2001 18:31:18 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAONXoV162848; Sat, 24 Nov 2001 16:33:50 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: Task management questions To: "ips" Cc: Gwendal Grignou X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Sat, 24 Nov 2001 15:32:40 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/24/2001 04:34:00 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I think Gwendal has a point. This and section 9.4 are very hard sections to read and understand. Perhaps we did not mean all that is said here. In addition too, or in support of Gwendal, I also found that the paragraph that begins "For all these Functions..." in 3.5.1 needs to be Harmonized with the "Abort Task Must..." Paragraph since they are clearly at odds, especially since the Abort Task is one of the functions addressed in the previous paragraph. But I simply do not understand how the following statement: "...the target MUST deliver to the initiator good status for all aborted task for which no status was delivered yet. The task management response MAY be issued by the target immediately after marking all tasks to be aborted." can be made to work with section 9.4 which goes to great pain to "silently discard" commands that are covered by the Abort but have not yet been executed. (See the last item b on page 156.) I think silently discard means do not return Status. After we sort out the above, we still have to deal with the differences right within 9.4. That is the first item d on page 156 states: "Note: Any entries that had been marked as a candidate for cleanup have now been delivered by the target to its SCSI layer. The SCSI layer will have to determine if they are to be aborted." and the last item b states: "remove the oldest barrier list item and evaluate all queued entries that have a CmdSN field less than ExpCmdSN, removing and silently discarding each queued command that meets the criteria for cleanup candidacy (as specified by the task management function)" This means DO NOT SEND THEM TO SCSI ! Therefore, they had not been delivered by the target to its SCSI layer as stated above. So we have one place saying that SCSI layer will determine, and the other place finding a method not to send them to SCSI. I understand what is being attempted here (to meet the requirements of SAM-2 (1157-D) section 5.5.2 that states "... no notification that the task(s) have been aborted shall be returned to the initiator...." however, our documentation needs some clean-up to make it consistent. This is NOT a change to the protocol, just a clean up of our documentation, but the way it is now it is hard to perform the protocol and follow the documentation in this specific area. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Gwendal Grignou @ece.cmu.edu on 11/23/2001 02:03:02 PM Sent by: owner-ips@ece.cmu.edu To: "ips" cc: Subject: Task management questions I have a question about the section 3.5.1 page 69. The paragrah starting with "ABORT TASK MUST",the last sentene says "The task management response MUST be issued after the command status (if any) was issued." : it means the initiator will receive, on the same connection, a status for the aborted task and AFTER, a response for the task management function. In the paragraph before (starting with "For all these functions,",), it is specified that the target should send a status for all aborted tasks, but the task management response can happen at any time after the task management request has been received by the target "The task management response MAY be issued by the target immediately after marking all tasks to be aborted." Why such a difference between "ABORT TASK" function and other functions ? About, "LOGICAL UNIT RESET", it is stated "the target MUST behave as dictated by the Logical Unit Reset function in [SAM2]." However, in SAM2, taks should be aborted as described in section 5.6 of SAM2, which says , in section 5.6.2, about initiator aborting their own task "When an initiator causes its own task(s) to be aborted, no notification that the task(s) have been aborted shall be returned to the initiator other than the completion response for the command or task management function action that caused the task(s) to be aborted and notification(s) associated with related effects of the action (e.g., a target reset unit attention condition)." Does it means the iSCSI target may not send a status for the aborted tasks in case of a "LOGICAL UNIT RESET" ? (and subsequently for TARGET RESET [WARM or COLD] which triggers LOGICAL UNIT RESET for each LUs [SAM2, section 5.8.6]). It does not match with the requirement in the paragraph concerning all the task management functions in general. Can you elaborate more on those requirements ? Thanks a lot, Gwendal. From owner-ips@ece.cmu.edu Sun Nov 25 11:39:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA22114 for ; Sun, 25 Nov 2001 11:39:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAPFZDq25766 for ips-outgoing; Sun, 25 Nov 2001 10:35:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAPFZ5l25755; Sun, 25 Nov 2001 10:35:06 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA267458; Sun, 25 Nov 2001 16:34:48 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAPFYjk46772; Sun, 25 Nov 2001 16:34:54 +0100 To: "John Hufferd" Cc: Gwendal Grignou , "ips" , owner-ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: Task management questions X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Sun, 25 Nov 2001 17:34:49 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 25/11/2001 17:35:07, Serialize complete at 25/11/2001 17:35:07 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, For some reason Gwendal's input did not reach me (is he subscribed to ips?). As for reading - the text is meant as a guide for the implementer and is not for casual reading. It was already updated with input from two implementers (IBM and Matthew Burbridge from HP). The barrier mechanism in 9.4 is the iSCSI barrier meant to discard commands that have not been delivered yet to SCSI (and thus avoid having them start execution). It has to be read carefully as it involves two mechanism - an initiation and actions to be done when ExpCmdSN changes. I will add some wording to 3.5.1 to the effect that it refers to commands that have reached SCSI (where past the iSCSI barrier). The action mandated is meant to enable ITT recycling (making sure that a task gets status even if this status will not be delivered to SCSI at initiator). Regards, Julo John Hufferd/San Jose/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 25-11-01 01:32 To: "ips" cc: Gwendal Grignou Subject: Re: Task management questions I think Gwendal has a point. This and section 9.4 are very hard sections to read and understand. Perhaps we did not mean all that is said here. In addition too, or in support of Gwendal, I also found that the paragraph that begins "For all these Functions..." in 3.5.1 needs to be Harmonized with the "Abort Task Must..." Paragraph since they are clearly at odds, especially since the Abort Task is one of the functions addressed in the previous paragraph. But I simply do not understand how the following statement: "...the target MUST deliver to the initiator good status for all aborted task for which no status was delivered yet. The task management response MAY be issued by the target immediately after marking all tasks to be aborted." can be made to work with section 9.4 which goes to great pain to "silently discard" commands that are covered by the Abort but have not yet been executed. (See the last item b on page 156.) I think silently discard means do not return Status. After we sort out the above, we still have to deal with the differences right within 9.4. That is the first item d on page 156 states: "Note: Any entries that had been marked as a candidate for cleanup have now been delivered by the target to its SCSI layer. The SCSI layer will have to determine if they are to be aborted." and the last item b states: "remove the oldest barrier list item and evaluate all queued entries that have a CmdSN field less than ExpCmdSN, removing and silently discarding each queued command that meets the criteria for cleanup candidacy (as specified by the task management function)" This means DO NOT SEND THEM TO SCSI ! Therefore, they had not been delivered by the target to its SCSI layer as stated above. So we have one place saying that SCSI layer will determine, and the other place finding a method not to send them to SCSI. I understand what is being attempted here (to meet the requirements of SAM-2 (1157-D) section 5.5.2 that states "... no notification that the task(s) have been aborted shall be returned to the initiator...." however, our documentation needs some clean-up to make it consistent. This is NOT a change to the protocol, just a clean up of our documentation, but the way it is now it is hard to perform the protocol and follow the documentation in this specific area. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com Gwendal Grignou @ece.cmu.edu on 11/23/2001 02:03:02 PM Sent by: owner-ips@ece.cmu.edu To: "ips" cc: Subject: Task management questions I have a question about the section 3.5.1 page 69. The paragrah starting with "ABORT TASK MUST",the last sentene says "The task management response MUST be issued after the command status (if any) was issued." : it means the initiator will receive, on the same connection, a status for the aborted task and AFTER, a response for the task management function. In the paragraph before (starting with "For all these functions,",), it is specified that the target should send a status for all aborted tasks, but the task management response can happen at any time after the task management request has been received by the target "The task management response MAY be issued by the target immediately after marking all tasks to be aborted." Why such a difference between "ABORT TASK" function and other functions ? About, "LOGICAL UNIT RESET", it is stated "the target MUST behave as dictated by the Logical Unit Reset function in [SAM2]." However, in SAM2, taks should be aborted as described in section 5.6 of SAM2, which says , in section 5.6.2, about initiator aborting their own task "When an initiator causes its own task(s) to be aborted, no notification that the task(s) have been aborted shall be returned to the initiator other than the completion response for the command or task management function action that caused the task(s) to be aborted and notification(s) associated with related effects of the action (e.g., a target reset unit attention condition)." Does it means the iSCSI target may not send a status for the aborted tasks in case of a "LOGICAL UNIT RESET" ? (and subsequently for TARGET RESET [WARM or COLD] which triggers LOGICAL UNIT RESET for each LUs [SAM2, section 5.8.6]). It does not match with the requirement in the paragraph concerning all the task management functions in general. Can you elaborate more on those requirements ? Thanks a lot, Gwendal. From owner-ips@ece.cmu.edu Sun Nov 25 12:05:17 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA22523 for ; Sun, 25 Nov 2001 12:05:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAPGHHY28022 for ips-outgoing; Sun, 25 Nov 2001 11:17:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar2aa.compuserve.com (siaar2aa.compuserve.com [149.174.40.137]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAPCaKl17233 for ; Sun, 25 Nov 2001 07:36:20 -0500 (EST) Received: (from mailgate@localhost) by siaar2aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id HAA00288 for ips@ece.cmu.edu; Sun, 25 Nov 2001 07:36:14 -0500 (EST) Received: from compuserve.com (dal-tgn-tkd-vty5.as.wcom.net [206.175.233.5]) by siaar2aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id HAA00233; Sun, 25 Nov 2001 07:36:07 -0500 (EST) Message-ID: <3C00E6FA.17BF1FAC@compuserve.com> Date: Sun, 25 Nov 2001 06:41:31 -0600 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector CC: Sudhir Srinivasan Subject: Re: FCIP: v0.7 editorial suggestion References: <63616B261CEFD411834D00D0B7A86CF757607D@lucy> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I believe that the logic and conceptual structure of Table 1 will be less clear if the columns are swapped. For better or for worse, the positions of the bits in Figure 8 represents the order in which they were defined not the logic and purpose of the bits. So the figure and the table are disjoint, as would be the case for any design that evolved over time. Finally, I find it impossible to believe that anyone capable of understanding NAPTs could become confused by Figure 8 and Table 1. Thanks. Ralph... Sudhir Srinivasan wrote: > Ralph, > > An editorial suggestion: > > The pFlags definition is: > |----------------Bit--------------------| > | | > | 31 30 29 28 27 26 25 24 | > +-----------------------------+----+----+ > | Reserved | Ch | SF | > +-----------------------------+----+----+ > Fig. 8 pFlags Field Bits > > with the "Ch" bit to the left of the "SF" bit. > > I suggest the first two columns in Table 1 be swapped > to reflect this. When looking at Figure 9, it was > slightly confusing at first to map the pFlags value > shown in the Short Frame header back to Table 1. > > Regards, > Sudhir From owner-ips@ece.cmu.edu Mon Nov 26 11:31:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08984 for ; Mon, 26 Nov 2001 11:31:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAQFCFB23321 for ips-outgoing; Mon, 26 Nov 2001 10:12:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cygnus.equallogic.com (cygnus.equallogic.com [65.170.102.10]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAQFCDl23316 for ; Mon, 26 Nov 2001 10:12:13 -0500 (EST) Received: from deneb.dev.equallogic.com (localhost.localdomain [127.0.0.1]) by cygnus.equallogic.com (8.11.6/8.11.6) with ESMTP id fAQFBh505576; Mon, 26 Nov 2001 10:11:43 -0500 Received: from pkoning.dev.equallogic.com.equallogic.com (localhost.localdomain [127.0.0.1]) by deneb.dev.equallogic.com (8.11.6/8.11.2) with SMTP id fAQFBh330889; Mon, 26 Nov 2001 10:11:43 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15362.23498.246432.983877@pkoning.dev.equallogic.com> Date: Mon, 26 Nov 2001 10:12:10 -0500 (EST) From: Paul Koning To: ltuikov@yahoo.com Cc: ips@ece.cmu.edu Subject: Re: CRC32C code -- please check References: <20011124081056.98754.qmail@web12802.mail.yahoo.com> X-Mailer: VM 6.75 under 21.1 (patch 11) "Carlsbad Caverns" XEmacs Lucid Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit >>>>> "Luben" == Luben Tuikov writes: Luben> Here is some C code at the end, the draft quoted and my notes Luben> alongside it. ... Luben> So, here we get M'(x) = x^32 * M(x). Getting R(x) is trivial. Luben> So we have: Luben> M'(x) = Q(x)*G(x) + R(x) = Q(x)*G(x) - R(x), (1) Luben> since plus is same as minus in modulo 2 arithmetic. >> - the bit sequence is complemented and the result is the Luben> CRC Luben> Now, from boolean logic we know that b XOR 1 = ~b, then, Luben> ~R(x) = R(x) XOR I(x) = R(x) + I(x) (2) Luben> where I(x) is a polynomial of the same degree as R(x) and Luben> whose coefficients are all equal to 1. So, to get M''(x) which Luben> we send we have: Luben> M''(x) = M'(x) + ~R(x) (draft) = Q(x)*G(x) - R(x) + R(x) + Luben> I(x) (using (1) and (2)) = Q(x)*G(x) + I(x). Luben> or succinctly, Luben> M''(x) = Q(x)*G(x) + I(x), (3) Luben> and this is the message which we send. Not quite. Your M' is taken from M by complementing the first 32 bits and appending 32 zeroes. That's what you feed into the CRC calculation -- but what you transmit has the original (uncomplemented) first 32 bits. Luben> On the other end we Luben> get M''(x), assuming there was no noise, and all Luben> link/TCP/Ethernet transformations are transparent for the Luben> sender and the receiver. ... Luben> Ok, so the receiver got M''(x) assuming no noise. Luben> M''(x) = Q(x)*G(x) + I(x). (this is (3)) Luben> Now we work the algorithm again: ..., find the remainder: Luben> Clearly, from (3) the remainder of M''(x) modulo G(x) is I(x). Luben> Then we complement the value as per the draft: Luben> ~I(x) = 0. Luben> And we end up with remainder 0. I think I see the problem. In CRC generation, you process the message (M). In CRC checking, there are two approaches: 1. You process M and compare the 4 bytes that follow (the CRC) with what you computed. This is fine, if you know ahead of time where M ends. 2. If you don't know where M ends ahead of time (as in the Ethernet MAC layer) or you simply prefer to process the whole received message, you run everything including the received CRC through the CRC algorithm. Note that the string processed in this case is 4 bytes longer than in case (1), because it includes the CRC. That's what you described, but I believe you missed a step. The CRC algorithm says: take the message (M'' in this case) and multiply by x^32, then divide by the CRC polynomial. You missed the multiply. In other words, you should have: (M''(x)*x^32) mod G(x) rather than simply M''(x) mod G(x) and that is *not* zero. I haven't done the math but I expect that you will find the answer to be the non-zero constant that appears in the spec. Ethernet works the same way, except for a change in the polynomial. You suggested earlier that Ethernet is using a non-standard algorithm because it comes up with a non-zero remainder at the receive end. That is not the case. You should not need any fudge factors to make the answer come out according to the spec if you get the algorithm right... paul From owner-ips@ece.cmu.edu Mon Nov 26 12:23:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14153 for ; Mon, 26 Nov 2001 12:23:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAQGL4m29055 for ips-outgoing; Mon, 26 Nov 2001 11:21:04 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAQBEGl08912 for ; Mon, 26 Nov 2001 06:14:17 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA18964; Mon, 26 Nov 2001 06:14:08 -0500 (EST) Message-Id: <200111261114.GAA18964@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-security-05.txt Date: Mon, 26 Nov 2001 06:14:08 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Securing iSCSI, iFCP and FCIP Author(s) : B. Aboba et al. Filename : draft-ietf-ips-security-05.txt Pages : 56 Date : 21-Nov-01 This document discusses how iSCSI, iFCP, and FCIP utilize IPsec to provide security. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-security-05.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-security-05.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-security-05.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011121141004.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-security-05.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-security-05.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011121141004.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Mon Nov 26 12:26:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14363 for ; Mon, 26 Nov 2001 12:26:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAQGLIG29083 for ips-outgoing; Mon, 26 Nov 2001 11:21:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAQBELl08930 for ; Mon, 26 Nov 2001 06:14:21 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA18950; Mon, 26 Nov 2001 06:14:01 -0500 (EST) Message-Id: <200111261114.GAA18950@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-fcencapsulation-04.txt Date: Mon, 26 Nov 2001 06:14:01 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : FC Frame Encapsulation Author(s) : R. Weber, M. Rajagopal, F. Travostino, V. Chau, M. O'Donnell, C. Monia, M. Merhar Filename : draft-ietf-ips-fcencapsulation-04.txt Pages : 15 Date : 21-Nov-01 This is the ips (IP Storage) working group draft describing the common encapsulation format and a procedure for the measurement and calculation of frame transit time through the IP network. This specification is intended for use by any IETF protocol that encapsulates Fibre Channel (FC) frames. This draft describes a frame header containing information mandated for encapsulating, transmitting, de-encapsulating, and calculating the transit times of FC frames. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-fcencapsulation-04.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-fcencapsulation-04.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-fcencapsulation-04.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011121140950.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-fcencapsulation-04.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-fcencapsulation-04.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011121140950.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Mon Nov 26 12:27:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14441 for ; Mon, 26 Nov 2001 12:27:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAQGMBI29158 for ips-outgoing; Mon, 26 Nov 2001 11:22:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.bld.us.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAQGM9l29153 for ; Mon, 26 Nov 2001 11:22:09 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e33.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id KAA17216 for ; Mon, 26 Nov 2001 10:19:24 -0600 Received: from d03nm041.boulder.ibm.com (d03nm041.boulder.ibm.com [9.99.140.41]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAQGM6R187996 for ; Mon, 26 Nov 2001 09:22:07 -0700 Importance: Normal Subject: Re: iSCSI: Abort Task Set To: "John Hufferd" Cc: ips@ece.cmu.edu From: "Jim Hafner" Date: Mon, 26 Nov 2001 08:22:05 -0800 Message-ID: X-MIMETrack: Serialize by Router on D03NM041/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/26/2001 08:22:06 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk John, You've found another place where use of "initiator" is vague. In this case, however, the meaning is SCSI Initiator Port (as that is the meaning of "initiator" in SAM-2, where this wording is probably crypted). Since the SCSI Initiator Port is the endpoint of the session, perhaps your suggested rewording would be a good clarification. Jim Hafner John Hufferd/San Jose/IBM@IBMUS@ece.cmu.edu on 11/23/2001 02:17:55 pm Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: iSCSI: Abort Task Set Julian, and List, The discription of the Abort Task Set Function in 3.5.1 says it: "aborts all Tasks issued by this initiator on the Logical Unit." The question is, what do we mean by initiator, in this case. Do we mean iSCSI Initiator Node, or do we mean iSCSI (SCSI) Initiator Port. Since we could have several different Session between the iSCSI Initiator Node and the SCSI Target Port, but only one between the SCSI Initiator Port and an SCSI Target Port. So if we mean the latter, perhaps we should change the wordage to: "aborts all Tasks issued via this Session on the Logical Unit." . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Mon Nov 26 12:32:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14937 for ; Mon, 26 Nov 2001 12:32:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAQGLD429070 for ips-outgoing; Mon, 26 Nov 2001 11:21:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAQBEIl08917 for ; Mon, 26 Nov 2001 06:14:18 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA18976; Mon, 26 Nov 2001 06:14:14 -0500 (EST) Message-Id: <200111261114.GAA18976@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-09.txt Date: Mon, 26 Nov 2001 06:14:13 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Author(s) : J. Satran et al. Filename : draft-ietf-ips-iscsi-09.txt Pages : 230 Date : 21-Nov-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. This memo describes a transport protocol for SCSI that operates on top of TCP. The iSCSI protocol aims to be fully compliant with the requirements laid out in the SCSI Architecture Model - 2 [SAM2] document. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-09.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-09.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-09.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011121141017.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-09.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-09.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011121141017.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Mon Nov 26 15:08:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA26568 for ; Mon, 26 Nov 2001 15:08:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAQIorS12770 for ips-outgoing; Mon, 26 Nov 2001 13:50:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magnemite.MaXXan.com (makesans.com [63.203.52.243]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAQIopl12763 for ; Mon, 26 Nov 2001 13:50:51 -0500 (EST) content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: FCIP V07 suggestion X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Date: Mon, 26 Nov 2001 10:47:12 -0800 Message-ID: <281144B165617E4FAF6D85689871D1B7068B5B@magnemite.MaXXan.com> Thread-Topic: FCIP V07 suggestion Thread-Index: AcF2qrnGYHO0390IEdWUbQCw0PIHqQ== From: "Chong Peng" To: Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fAQIoql12764 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hi, In the FCIP draft V07 () page 28, it says: "After the TCP connect request has been accepted, the FCIP Entity SHALL wait for the FCIP Short Frame sent by the originator of the TCP connect request as the first bytes received through the Encapsulated Frame Receiver Portal on the accepted connection." I think this text is confusing. The "FCIP Short Frame" cannot be received through the "Encapsulated Frame Receiver Portal" because this Portal does not exist when the "FCIP Short Frame" is to be received: (1) According to the draft (page 13), Portals (including "Encapsulated Frame Receiver Portal") belong to the FCIP_DE. This implies (to me) that Portals cannot exist without their corresponding FCIP_DE. (2) Again according to the draft (page 26 and page 29), FCIP_DE is created only after "FCIP Short Frame" are exchanged between the originator and recipient of the TCP connection. It might make sense for the aforementioned text in the draft be changed to: "After the TCP connect request has been accepted, the FCIP Entity SHALL wait for the FCIP Short Frame sent by the originator of the TCP connect request as the first bytes received on the accepted connection." Chong Peng From owner-ips@ece.cmu.edu Mon Nov 26 15:45:43 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29675 for ; Mon, 26 Nov 2001 15:45:43 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAQJWke16069 for ips-outgoing; Mon, 26 Nov 2001 14:32:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAQJWhl16056 for ; Mon, 26 Nov 2001 14:32:43 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel12.hp.com (Postfix) with ESMTP id 1E9131F9DF for ; Mon, 26 Nov 2001 11:32:36 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id LAA15791 for ; Mon, 26 Nov 2001 11:34:05 -0800 (PST) Message-Id: <200111261934.LAA15791@core.rose.hp.com> Date: Mon, 26 Nov 2001 11:45:19 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: iSCSI: unsolicited data Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, I don't recall the earlier discussion being conclusive on this topic, so let me attempt to make a couple of suggestions on this. Last para in Section 2.2.5 (page 33) in rev09 mandates unsolicited data ordering with the following wording. "iSCSI initiators and targets MUST also enforce some ordering rules to achieve deadlock-free operation. Unsolicited data MUST be sent on every connection in the same order in which commands were sent. A target receiving data out of order SHOULD terminate the session." I have a couple of suggestions to reword - - The reference to a deadlock is true only for an implementation overcommitting the unsolicited data buffers. So, I would suggest dropping this reasoning. - The "MUST enforce" wording should also add that "targets MAY however see unexpected unsolicited data (UUD), following data digest errors on expected unsolicited data." - Finally, the "SHOULD terminate the session", IMHO, is too strong. This wording will imply that a session is vulnerable to a single digest error regardless of recovery level. [ I however disfavor attaching more assorted semantics to recovery levels (than the current easily understood "level = setof(recovery classes)" formulation). ] As you mentioned on this discussion earlier, a UUD is a protocol error (or may be not, if it is the effect of a prior digest failure). We can choose to Reject ("protocol error") the UUD and discard the data. The only problem with this approach is: following a digest error on unsolicited data for one task, all the following unsolicited data in flight would have to be discarded. One option is to imply that the target update the "expected unsolicited data" so the unsolicited data following in the pipe can be processed normally. So, I guess it would be saying: "the target MAY Reject the UUD". Comments? -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Mon Nov 26 17:24:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA07046 for ; Mon, 26 Nov 2001 17:24:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAQL0up23387 for ips-outgoing; Mon, 26 Nov 2001 16:00:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAQL0sl23379 for ; Mon, 26 Nov 2001 16:00:54 -0500 (EST) Received: by ariel.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Mon, 26 Nov 2001 13:00:48 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Subject: iFCP: iFCP Version 07 Date: Mon, 26 Nov 2001 13:00:44 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Colleagues: Version 7 of the iFCP protocol specification was submitted on 11/21/01 for inclusion in the IETF archive. Pending the IETF announcement of availability, versions of the document can be obtained from the T11 document archive at: ftp://209.26.30.221/t11/docs/01-495v3.pdf -- created with markups visible ftp://209.26.30.221/t11/docs/01-495v3.txt This version partially addresses the comments received from David Black as discussed below. A number of issues are still open and will be addressed in the next release. Major Issues: Section 11, Security -- Updated to reflect the latest security text specified in http://www.ietf.org/internet-drafts/draft-ietf-ips-security-05.txt Section 12, QoS -- Updated per review comments and subsequent reflector discussion. "Minor Issues" iSNS -- Added new section 5.3 describing the role of iSNS. 4.4a) -- Added FC Node definition 4.7.1 -- Added description of address assignment for fabric-attached loops. 4.8 -- Added description of class 4 and class 6 transport services. 4.9 -- Added description of FC logins. Figure 7 -- Changed "IP fabric" to "iFCP fabric." 5.2.1 -- Added description of how supported transport services are discovered by an N_PORT. PP 20 -- Corrected specification of requirements for Address-translation and Address-transparent modes. 5.3.1.1 -- Added material describing use of iSNS for domain I/D assignment in address-transparent mode. 5.3.1.2, 5.3.2.3 -- "SHALL" added to requirements for rejecting incoming frames having incorrect address mode. 6.2.2.1 -- Corrected per review comment 6.2.2.2, "Use of TCP Features" -- Revised per review comment 6.2.2.3, "Error Recovery and Cold Start" -- Deleted per review comment Sections 7 and 8 -- Order reversed per review comment 6.2.3 -- Identified entity being terminated or aborted (iFCP session) per review comment 6.3 -- Qualified reference to point to appendix B of RFC 1323 7.3 and 7.3.7 (PLOGI) -- Terminology: Augmented ELSs are now referred to as "Special" ELSs. 7.3.5 -- Fixed description of FARP-REQ. 7.3.8 -- Deleted description of REC ELS. 7.4, table 4 -- Modified table per review comment to delete references to "y" entry types. 8.1 -- Added Connection Handle reference. Appendix A -- Summary of Link Services. Deleted Link Services removed from FC-FS spec. Appendix B -- Deleted. Spec will be revised to include a reference to published material. Open Issues 5.3 -- Separate discussion of address mode tradeoffs from functional requirements. Delineate requirements for CRC recalculation in address translation mode. Work in progress. 5.3.2, and subsidiary sections -- Specify rules for translation table maintenance. Rewrite in progress. 6.2.1 -- N_PORT network address definition and NAPT support. Discussion with co-authors and proposal is in progress. Post for review on reflector. 9.2, Stale Frame Prevention -- Add guidelines on when to enable the detection of stale frames traversing the IP network. 10.4 -- Broadcast Server. Revise mechanism to provide reliability and functionality equivalent to an FC fabric. Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 Charles Monia Senior Technology Consultant Nishan Systems email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ece.cmu.edu Mon Nov 26 20:31:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA18269 for ; Mon, 26 Nov 2001 20:31:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAR0WVL08656 for ips-outgoing; Mon, 26 Nov 2001 19:32:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel8.hp.com (atlrel8.hp.com [156.153.255.206]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAR0WUl08650 for ; Mon, 26 Nov 2001 19:32:30 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel8.hp.com (Postfix) with ESMTP id ACFCF1F82B for ; Mon, 26 Nov 2001 19:32:20 -0500 (EST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id QAA10401 for ; Mon, 26 Nov 2001 16:33:49 -0800 (PST) Message-Id: <200111270033.QAA10401@core.rose.hp.com> Date: Mon, 26 Nov 2001 16:45:03 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: Comments on ips security draft Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Some quick comments on the rev05 ips security draft. Authors may please comment. - Section 2.3, page 10. "Conformant iSCSI security implementations MUST support ESP in transport mode. " I assume it should be tunnel mode.... - Section 3.3, page 17. The well-known target port for iSCSI may be updated to 3260. - Section 3.4, page 18. "[d] a specific connection be closed at the Target's request. LP Command [d] is distinct from [b] in that it indicates that the connection is being closed in response to a request from the Target for it to be closed. Due to asymmetries in the iSCSI protocol, Targets cannot close a connection on their own initiative." It needs to be reworded, since this is incorrect. [d] is the same as [b]. - I may be missing something, but the table listing IKE implementation sizes on page 28 seems completely out-of-context in the middle of a rekeying frequency discussion. Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Mon Nov 26 20:33:44 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA18339 for ; Mon, 26 Nov 2001 20:33:44 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAR0xkP10284 for ips-outgoing; Mon, 26 Nov 2001 19:59:46 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAR0xil10279 for ; Mon, 26 Nov 2001 19:59:44 -0500 (EST) Received: from MURALIRLAPTOP ([192.168.2.38]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id QAA04061 for ; Mon, 26 Nov 2001 16:59:38 -0800 (PST) From: "Murali Rajagopal" To: Subject: FCIP 11/21 Teleconference Minutes Date: Mon, 26 Nov 2001 16:58:44 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Minutes submitted by Jim Nelson, Vixel Corp. Agenda: 0. Roll Call/ Agenda bashing : 10 Min 1. Ralph: Quick status update on the released drafts: 5 Min 2. NAPTs ( 30 Min) 3. Resync (20 Min) 4. Security: FCIP, SLP, FCIP MIB (30 Min) 5. SNMP Response Required via FCIP Entity IP address? (15 Min) 6. Next Meeting Agenda/ Host: 5 Min Note: The actual meeting only lasted for an hour and items 1 and 3 in the above were not discusssed. Roll Call Jim Nelson - Vixel Bill Krieg - Lucent Dave Peterson - Cisco Raj Bhagwat - Lightsand Murali Rajagopal - Lightsand Andy Helland - Lightsand Bret Kethum - CNT Milan Merhar - Pirus Venkat - Rhapsody Anil Rijhsinghani - McData Bob Snively - Brocade 1. SNMP Response Required via FCIP Entity IP address? Dave Peterson - Not clear what to do with this issue in the SLP draft. Anil - Anything with an IP address implements a specific MIB. FCIP MIB should address the device, not just the entity. It is similar to a 10/100 interface. Dave - Will leave the subject out of the SLP draft for the moment. 2. Quick status update on the released drafts Ralph was not present. Venkat - When we added section 7 with the short frame, Some textual changes are probably required in Annex D which currently only described non-Short Frames. 3. Security: FCIP, SLP, FCIP MIB SLP - Dave - The current SLP draft is not consistent with the security draft because the security draft requires IPSec whereas SLP does not. At the moment we have fundamental difference in approach. No change for the moment. FCIP MIB - Anil - FCIP MIB at the moment doesn't discuss security relative to management traffic. It is clear that IPSec could be used for both authenticating and encrypting this information. This is open at present. SNMPv3 addresses security, but does not require it. Inband access could be disabled. It might be desirable to allow this, but not require this. Anil will coordinate with Mark Bakke relative to the iSCSI MIB. FCIP - Venkat - With the addition of the short frame it makes it easier for an attacker to open a connection. Thus there is more of a security problem in the absence of IPSec. The main issue is the possibility of unsecure joining of multiple connections into a link. There is no particular direct protection for connections against false new connections. Group pre-shared keys are also a problem, because any member of the group can initiate a TCP/IP connection and potentially foul up a link. One solution is to use IPSec, but prohibit group preshared keys. Bob - Without IPSec, not protected if you don't have a policy established. The security behavior is established by policies. May or may not choose to require security. If you don't have security it's because you choose not to have it including all parties that the entity is allowed to communicate with. Thus may refuse connections based on the policies. If the policy is security is not required, in the presence of NAPTs, then is vulnerable. 4. Neil Wanamaker will set up the meeting for next week. -- Jim Nelson Systems Architect Vixel Corporation Irvine, Ca 92618 jnelson@vixel.com 949-450-6159 From owner-ips@ece.cmu.edu Mon Nov 26 22:17:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA22229 for ; Mon, 26 Nov 2001 22:17:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAR1hSS13002 for ips-outgoing; Mon, 26 Nov 2001 20:43:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail1.aarohicommunications.com (cpe-66-87-94-158.ca.sprintbbd.net [66.87.94.158]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAR1gel12942 for ; Mon, 26 Nov 2001 20:42:41 -0500 (EST) Received: from LINUX15 (linux15.aarohicommunications.com [192.168.1.234]) by mail1.aarohicommunications.com (Mirapoint) with ESMTP id AAJ03921 (AUTH shaileshm); Mon, 26 Nov 2001 17:45:12 -0800 (PST) From: "Shailesh Manjrekar" To: "'John Hufferd'" , "'Rahul Bhagwat'" Cc: , "'KRUEGER,MARJORIE \(HP-Roseville,ex1\)'" Subject: RE: Selectively exposing Porgal Groups to Initiators Date: Mon, 26 Nov 2001 17:43:30 -0800 Message-ID: <000001c176e4$f1751dd0$ea01a8c0@aarohicommunications.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2627 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi John, In your configuration examples, in the slide about SCSI nexi, Model 3, entries 3 and 5 would form a parallel nexus, so as entries 4 and 6. The entries instead should have been : 3) iqn.1999-12.com.ajax.os1 + VID=2 + ISID=3 and eui.02004567A425678A+1 4) iqn.1999-12.com.ajax.os1 + VID=5 + ISID=1 and eui.02004567A425678A+2 ****** ************* 5) iqn.1992-12.com.ajax.os1 + VID=2 + ISID=3 and eui.02004567A425678A+2 6) iqn.1999-12.com.ajax.os1 + VID=5 + ISID=1 and eui.02004567A425678A+1 ****** ************* Right ? Regards, Shailesh Manjrekar Aarohi Communications. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] On Behalf Of John Hufferd Sent: Wednesday, November 21, 2001 12:43 PM To: 'Rahul Bhagwat' Cc: ips@ece.cmu.edu; KRUEGER,MARJORIE (HP-Roseville,ex1) Subject: RE: Selectively exposing Porgal Groups to Initiators Rahul, Marjorie's answers are correct, but you might want to be very careful with your terms. Your point three talks about limiting Portal Groups to specific Initiators. I am concerned about your use of the words iSCSI Target, which could apply to a couple of different things. In a Target Network Entity there can be more then one iSCSI Target Node (SCSI Device). Each Target Node can have more then one iSCSI (SCSI) Target Port connected to it. Part of the name of this iSCSI (SCSI) Target Port includes the Portal Group Tag. So if, in your point 3, the term iSCSI Target meant iSCSI Target Node, then you would be able to set up different iSCSI (SCSI) Target Ports (each with different Portal Group Tags) that can access the resources at the same iSCSI Target Node. The ACL would then probably be applied at the iSCSI (SCSI) Target Port. To be sure we are on the same page here, please reference the charts that have been placed at: http://www.haifa.il.ibm.com/satran/ips/iSCSIConfigurationExamples.pdf (Note: I use the term iSCSI (SCSI) Port to represent the concept of "SCSI Port" within the context of iSCSI.) . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "KRUEGER,MARJORIE (HP-Roseville,ex1)" @ece.cmu.edu on 11/21/2001 11:11:45 AM Sent by: owner-ips@ece.cmu.edu To: "'Rahul Bhagwat'" , ips@ece.cmu.edu cc: Subject: RE: Selectively exposing Porgal Groups to Initiators > 1. Is it required that TargetAddresses of an iSCSI target advertised to a > directory service include portal group tag ? Yes, information is necessary to communicate to the initiator which target addresses can be used to form a multi-connection session. > 2. Is it mandatory for an Initiator to use "SendTargets" to discover > TargetAddress for an iSCSI target (even if if has a set of addresses > either statically configured or found through a directory service)? To my knowledge no iSCSI document has declared it mandatory, but it's recommended to ensure the initiator has current addressing information for this target. > 3. Is it okay to restrict access to an Initiator (based on it's iSCSI name) > to only a subset of total Target Portal Groups supported by the iSCSI > target? Yes > > In this scenario, an Initiator may find out the TargetAddresses for an iSCSI > target using a directory service, and try to connect a normal operational > session to any of these addresses without using SendTargets. The > iSCSI target can return an error code for login as "Initiator not Authorized" > which in fact is not > completely true. Initiator is authorized to only use a subset > of portal groups. Correct, although "not completely true" is subjective. Another perspective is that this initiator is not authorized to use this *target port*, which is completely true. Marjorie From owner-ips@ece.cmu.edu Mon Nov 26 22:18:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA22270 for ; Mon, 26 Nov 2001 22:18:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAR2JXa15436 for ips-outgoing; Mon, 26 Nov 2001 21:19:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel9.hp.com (atlrel9.hp.com [156.153.255.214]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAR2JWl15432 for ; Mon, 26 Nov 2001 21:19:32 -0500 (EST) Received: from xatlrelay2.atl.hp.com (xatlrelay2.atl.hp.com [15.45.89.191]) by atlrel9.hp.com (Postfix) with ESMTP id 36F581F5EF; Mon, 26 Nov 2001 21:19:25 -0500 (EST) Received: from xatlbh3.atl.hp.com (xatlbh3.atl.hp.com [15.45.89.188]) by xatlrelay2.atl.hp.com (Postfix) with ESMTP id 065CC1F514; Mon, 26 Nov 2001 21:19:20 -0500 (EST) Received: by xatlbh3.atl.hp.com with Internet Mail Service (5.5.2653.19) id ; Mon, 26 Nov 2001 21:19:19 -0500 Message-ID: <6BD67FFB937FD411A04F00D0B74FE87805CCF249@xrose06.rose.hp.com> From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" To: "'Shailesh Manjrekar'" , "'John Hufferd'" , "'Rahul Bhagwat'" Cc: ips@ece.cmu.edu Subject: RE: Selectively exposing Porgal Groups to Initiators Date: Mon, 26 Nov 2001 21:19:17 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Those are not parallel nexus, they are both examples of the same iSCSI initiator port talking to two different iSCSI target ports. Parallel nexus would be two iSCSI sessions between the same port pairs. A "SCSI nexus" is defined as a relationship between an initiator port and a target port. Marjorie > -----Original Message----- > From: Shailesh Manjrekar [mailto:shaileshm@aarohicommunications.com] > Sent: Monday, November 26, 2001 5:44 PM > To: 'John Hufferd'; 'Rahul Bhagwat' > Cc: ips@ece.cmu.edu; 'KRUEGER,MARJORIE (HP-Roseville,ex1)' > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > Hi John, > > In your configuration examples, in the slide about SCSI nexi, Model 3, > entries 3 and 5 would form a parallel nexus, so as entries 4 > and 6. The > entries instead should have been : > > 3) iqn.1999-12.com.ajax.os1 > + VID=2 + ISID=3 and > eui.02004567A425678A+1 > 4) iqn.1999-12.com.ajax.os1 > + VID=5 + ISID=1 and > eui.02004567A425678A+2 > ****** ************* > 5) iqn.1992-12.com.ajax.os1 > + VID=2 + ISID=3 and > eui.02004567A425678A+2 > 6) iqn.1999-12.com.ajax.os1 > + VID=5 + ISID=1 and > eui.02004567A425678A+1 > ****** ************* > Right ? > > Regards, > Shailesh Manjrekar > Aarohi Communications. > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] On > Behalf Of > John Hufferd > Sent: Wednesday, November 21, 2001 12:43 PM > To: 'Rahul Bhagwat' > Cc: ips@ece.cmu.edu; KRUEGER,MARJORIE (HP-Roseville,ex1) > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > Rahul, > Marjorie's answers are correct, but you might want to be very careful > with > your terms. Your point three talks about limiting Portal Groups to > specific Initiators. I am concerned about your use of the words iSCSI > Target, which could apply to a couple of different things. > > In a Target Network Entity there can be more then one iSCSI > Target Node > (SCSI Device). Each Target Node can have more then one iSCSI (SCSI) > Target > Port connected to it. Part of the name of this iSCSI (SCSI) > Target Port > includes the Portal Group Tag. So if, in your point 3, the term iSCSI > Target meant iSCSI Target Node, then you would be able to set up > different > iSCSI (SCSI) Target Ports (each with different Portal Group Tags) that > can > access the resources at the same iSCSI Target Node. The ACL > would then > probably be applied at the iSCSI (SCSI) Target Port. > > To be sure we are on the same page here, please reference the charts > that > have been placed at: > > http://www.haifa.il.ibm.com/satran/ips/iSCSIConfigurationExamples.pdf > > (Note: I use the term iSCSI (SCSI) Port to represent the concept of > "SCSI > Port" within the context of iSCSI.) > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "KRUEGER,MARJORIE (HP-Roseville,ex1)" > @ece.cmu.edu > on 11/21/2001 11:11:45 AM > > Sent by: owner-ips@ece.cmu.edu > > > To: "'Rahul Bhagwat'" , ips@ece.cmu.edu > cc: > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > > > 1. Is it required that TargetAddresses of an iSCSI target advertised > to a > > directory service include portal group tag ? > > Yes, information is necessary to communicate to the initiator which > target > addresses can be used to form a multi-connection session. > > > 2. Is it mandatory for an Initiator to use "SendTargets" to discover > > TargetAddress for an iSCSI target (even if if has a set of > addresses > > either statically configured or found through a > directory service)? > > To my knowledge no iSCSI document has declared it mandatory, but it's > recommended to ensure the initiator has current addressing information > for > this target. > > > 3. Is it okay to restrict access to an Initiator (based on > it's iSCSI > name) > > to only a subset of total Target Portal Groups supported by the > iSCSI > > target? > > Yes > > > > > In this scenario, an Initiator may find out the > TargetAddresses for an > iSCSI > > target using a directory service, and try to connect a normal > operational > > session to any of these addresses without using SendTargets. The > > iSCSI target can return an error code for login as "Initiator not > Authorized" > > which in fact is not > > completely true. Initiator is authorized to only use a subset > > of portal groups. > > Correct, although "not completely true" is subjective. Another > perspective > is that this initiator is not authorized to use this *target port*, > which > is > completely true. > > Marjorie > > > From owner-ips@ece.cmu.edu Tue Nov 27 02:55:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA15545 for ; Tue, 27 Nov 2001 02:55:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAR6NYt00699 for ips-outgoing; Tue, 27 Nov 2001 01:23:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAR6NWl00694 for ; Tue, 27 Nov 2001 01:23:33 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA75252 for ; Tue, 27 Nov 2001 07:23:26 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAR6NdC75486 for ; Tue, 27 Nov 2001 07:23:39 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: unsolicited data X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Tue, 27 Nov 2001 08:23:41 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 27/11/2001 08:23:44, Serialize complete at 27/11/2001 08:23:44 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, The deadlock free requirement here goes a bit deeper than your comment may suggest. Unsolicited data needs are very hard to predict and ordering them may remove the need to do so to a large extent. As for the action - yes we may want to weaken the requirement to accommodate for digest errors. Regards, Julo "Mallikarjun C." Sent by: owner-ips@ece.cmu.edu 26-11-01 21:45 Please respond to cbm To: ips cc: Subject: iSCSI: unsolicited data Julian, I don't recall the earlier discussion being conclusive on this topic, so let me attempt to make a couple of suggestions on this. Last para in Section 2.2.5 (page 33) in rev09 mandates unsolicited data ordering with the following wording. "iSCSI initiators and targets MUST also enforce some ordering rules to achieve deadlock-free operation. Unsolicited data MUST be sent on every connection in the same order in which commands were sent. A target receiving data out of order SHOULD terminate the session." I have a couple of suggestions to reword - - The reference to a deadlock is true only for an implementation overcommitting the unsolicited data buffers. So, I would suggest dropping this reasoning. - The "MUST enforce" wording should also add that "targets MAY however see unexpected unsolicited data (UUD), following data digest errors on expected unsolicited data." - Finally, the "SHOULD terminate the session", IMHO, is too strong. This wording will imply that a session is vulnerable to a single digest error regardless of recovery level. [ I however disfavor attaching more assorted semantics to recovery levels (than the current easily understood "level = setof(recovery classes)" formulation). ] As you mentioned on this discussion earlier, a UUD is a protocol error (or may be not, if it is the effect of a prior digest failure). We can choose to Reject ("protocol error") the UUD and discard the data. The only problem with this approach is: following a digest error on unsolicited data for one task, all the following unsolicited data in flight would have to be discarded. One option is to imply that the target update the "expected unsolicited data" so the unsolicited data following in the pipe can be processed normally. So, I guess it would be saying: "the target MAY Reject the UUD". Comments? -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com From owner-ips@ece.cmu.edu Tue Nov 27 04:45:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA17403 for ; Tue, 27 Nov 2001 04:45:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAR8YVN08117 for ips-outgoing; Tue, 27 Nov 2001 03:34:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAR8YQl08108 for ; Tue, 27 Nov 2001 03:34:26 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id DAA62134; Tue, 27 Nov 2001 03:31:41 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAR8YNR33578; Tue, 27 Nov 2001 01:34:23 -0700 X-Priority: 1 (High) Importance: Normal Subject: iSCSI: Selectively exposing Porgal Groups to Initiators To: "KRUEGER,MARJORIE (HP-Roseville,ex1)" Cc: "'Shailesh Manjrekar'" , "'Rahul Bhagwat'" , ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Tue, 27 Nov 2001 00:33:32 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/27/2001 01:34:23 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Marjorie, Your logic is correct, however, it does look like I had a finger check as I typed in two characters. I intended the Initiator iSCSI(SCSI) Port with the ISID=3 to have a nexus with both the Target iSCSI(SCSI) Port ...A+1 and .... A+2. and the same Targets for the Initiator iSCSI(SCSI) Port with the ISID=1. However I think I typed the A+1 and A+2 backwards on entry 5 & 6. Shailesh, Thank you for doing such a careful review. I will have Julian up date the Web Site. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "KRUEGER,MARJORIE (HP-Roseville,ex1)" on 11/26/2001 06:19:17 PM To: "'Shailesh Manjrekar'" , John Hufferd/San Jose/IBM@IBMUS, "'Rahul Bhagwat'" cc: ips@ece.cmu.edu Subject: RE: Selectively exposing Porgal Groups to Initiators Those are not parallel nexus, they are both examples of the same iSCSI initiator port talking to two different iSCSI target ports. Parallel nexus would be two iSCSI sessions between the same port pairs. A "SCSI nexus" is defined as a relationship between an initiator port and a target port. Marjorie > -----Original Message----- > From: Shailesh Manjrekar [mailto:shaileshm@aarohicommunications.com] > Sent: Monday, November 26, 2001 5:44 PM > To: 'John Hufferd'; 'Rahul Bhagwat' > Cc: ips@ece.cmu.edu; 'KRUEGER,MARJORIE (HP-Roseville,ex1)' > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > Hi John, > > In your configuration examples, in the slide about SCSI nexi, Model 3, > entries 3 and 5 would form a parallel nexus, so as entries 4 > and 6. The > entries instead should have been : > > 3) iqn.1999-12.com.ajax.os1 > + VID=2 + ISID=3 and > eui.02004567A425678A+1 > 4) iqn.1999-12.com.ajax.os1 > + VID=5 + ISID=1 and > eui.02004567A425678A+2 > ****** ************* > 5) iqn.1992-12.com.ajax.os1 > + VID=2 + ISID=3 and > eui.02004567A425678A+2 > 6) iqn.1999-12.com.ajax.os1 > + VID=5 + ISID=1 and > eui.02004567A425678A+1 > ****** ************* > Right ? > > Regards, > Shailesh Manjrekar > Aarohi Communications. > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] On > Behalf Of > John Hufferd > Sent: Wednesday, November 21, 2001 12:43 PM > To: 'Rahul Bhagwat' > Cc: ips@ece.cmu.edu; KRUEGER,MARJORIE (HP-Roseville,ex1) > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > Rahul, > Marjorie's answers are correct, but you might want to be very careful > with > your terms. Your point three talks about limiting Portal Groups to > specific Initiators. I am concerned about your use of the words iSCSI > Target, which could apply to a couple of different things. > > In a Target Network Entity there can be more then one iSCSI > Target Node > (SCSI Device). Each Target Node can have more then one iSCSI (SCSI) > Target > Port connected to it. Part of the name of this iSCSI (SCSI) > Target Port > includes the Portal Group Tag. So if, in your point 3, the term iSCSI > Target meant iSCSI Target Node, then you would be able to set up > different > iSCSI (SCSI) Target Ports (each with different Portal Group Tags) that > can > access the resources at the same iSCSI Target Node. The ACL > would then > probably be applied at the iSCSI (SCSI) Target Port. > > To be sure we are on the same page here, please reference the charts > that > have been placed at: > > http://www.haifa.il.ibm.com/satran/ips/iSCSIConfigurationExamples.pdf > > (Note: I use the term iSCSI (SCSI) Port to represent the concept of > "SCSI > Port" within the context of iSCSI.) > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "KRUEGER,MARJORIE (HP-Roseville,ex1)" > @ece.cmu.edu > on 11/21/2001 11:11:45 AM > > Sent by: owner-ips@ece.cmu.edu > > > To: "'Rahul Bhagwat'" , ips@ece.cmu.edu > cc: > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > > > 1. Is it required that TargetAddresses of an iSCSI target advertised > to a > > directory service include portal group tag ? > > Yes, information is necessary to communicate to the initiator which > target > addresses can be used to form a multi-connection session. > > > 2. Is it mandatory for an Initiator to use "SendTargets" to discover > > TargetAddress for an iSCSI target (even if if has a set of > addresses > > either statically configured or found through a > directory service)? > > To my knowledge no iSCSI document has declared it mandatory, but it's > recommended to ensure the initiator has current addressing information > for > this target. > > > 3. Is it okay to restrict access to an Initiator (based on > it's iSCSI > name) > > to only a subset of total Target Portal Groups supported by the > iSCSI > > target? > > Yes > > > > > In this scenario, an Initiator may find out the > TargetAddresses for an > iSCSI > > target using a directory service, and try to connect a normal > operational > > session to any of these addresses without using SendTargets. The > > iSCSI target can return an error code for login as "Initiator not > Authorized" > > which in fact is not > > completely true. Initiator is authorized to only use a subset > > of portal groups. > > Correct, although "not completely true" is subjective. Another > perspective > is that this initiator is not authorized to use this *target port*, > which > is > completely true. > > Marjorie > > > From owner-ips@ece.cmu.edu Tue Nov 27 11:44:30 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08732 for ; Tue, 27 Nov 2001 11:44:29 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fARFaLB16605 for ips-outgoing; Tue, 27 Nov 2001 10:36:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fARApfl28307 for ; Tue, 27 Nov 2001 05:51:41 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA17998; Tue, 27 Nov 2001 05:51:37 -0500 (EST) Message-Id: <200111271051.FAA17998@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-boot-04.txt Date: Tue, 27 Nov 2001 05:51:37 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Bootstrapping Clients using the iSCSI Protocol Author(s) : P. Sarkar, D. Missimer, C. Sapuntzakis Filename : draft-ietf-ips-iscsi-boot-04.txt Pages : 8 Date : 26-Nov-01 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. iSCSI is a proposed transport protocol for SCSI that operates on top of TCP[12]. This memo describes a standard mechanism to enable clients to bootstrap themselves using the iSCSI protocol. The goal of this standard is to enable iSCSI boot clients to obtain the information to open an iSCSI session with the iSCSI boot server, assuming this information is not available. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-boot-04.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-boot-04.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-boot-04.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011126101343.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-boot-04.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-boot-04.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011126101343.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Tue Nov 27 11:47:18 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08955 for ; Tue, 27 Nov 2001 11:47:17 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAREUhh11459 for ips-outgoing; Tue, 27 Nov 2001 09:30:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thalia.fm.intel.com (fmfdns02.fm.intel.com [132.233.247.11]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAREUel11449 for ; Tue, 27 Nov 2001 09:30:40 -0500 (EST) Received: from fmsmsxvs043.fm.intel.com (fmsmsxvs043.fm.intel.com [132.233.42.129]) by thalia.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.46 2001/10/25 21:02:55 root Exp $) with SMTP id OAA07474 for ; Tue, 27 Nov 2001 14:30:39 GMT Received: from fmsmsx28.fm.intel.com ([132.233.42.28]) by fmsmsxvs043.fm.intel.com (NAVGW 2.5.1.6) with SMTP id M2001112706295210396 for ; Tue, 27 Nov 2001 06:29:52 -0800 Received: by fmsmsx28.fm.intel.com with Internet Mail Service (5.5.2653.19) id ; Tue, 27 Nov 2001 06:30:39 -0800 Message-ID: <1D28B91BEE38D411AC6D00902785733F029D6C23@fmsmsx90.fm.intel.com> From: "Thompson, Daniel J" To: ips@ece.cmu.edu Subject: remove Date: Tue, 27 Nov 2001 06:30:38 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk From owner-ips@ece.cmu.edu Tue Nov 27 11:48:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09069 for ; Tue, 27 Nov 2001 11:48:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fARFcH716777 for ips-outgoing; Tue, 27 Nov 2001 10:38:17 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail1.aarohicommunications.com (cpe-66-87-94-158.ca.sprintbbd.net [66.87.94.158]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAR1gel12942 for ; Mon, 26 Nov 2001 20:42:41 -0500 (EST) Received: from LINUX15 (linux15.aarohicommunications.com [192.168.1.234]) by mail1.aarohicommunications.com (Mirapoint) with ESMTP id AAJ03921 (AUTH shaileshm); Mon, 26 Nov 2001 17:45:12 -0800 (PST) From: "Shailesh Manjrekar" To: "'John Hufferd'" , "'Rahul Bhagwat'" Cc: , "'KRUEGER,MARJORIE \(HP-Roseville,ex1\)'" Subject: RE: Selectively exposing Porgal Groups to Initiators Date: Mon, 26 Nov 2001 17:43:30 -0800 Message-ID: <000001c176e4$f1751dd0$ea01a8c0@aarohicommunications.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2627 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi John, In your configuration examples, in the slide about SCSI nexi, Model 3, entries 3 and 5 would form a parallel nexus, so as entries 4 and 6. The entries instead should have been : 3) iqn.1999-12.com.ajax.os1 + VID=2 + ISID=3 and eui.02004567A425678A+1 4) iqn.1999-12.com.ajax.os1 + VID=5 + ISID=1 and eui.02004567A425678A+2 ****** ************* 5) iqn.1992-12.com.ajax.os1 + VID=2 + ISID=3 and eui.02004567A425678A+2 6) iqn.1999-12.com.ajax.os1 + VID=5 + ISID=1 and eui.02004567A425678A+1 ****** ************* Right ? Regards, Shailesh Manjrekar Aarohi Communications. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] On Behalf Of John Hufferd Sent: Wednesday, November 21, 2001 12:43 PM To: 'Rahul Bhagwat' Cc: ips@ece.cmu.edu; KRUEGER,MARJORIE (HP-Roseville,ex1) Subject: RE: Selectively exposing Porgal Groups to Initiators Rahul, Marjorie's answers are correct, but you might want to be very careful with your terms. Your point three talks about limiting Portal Groups to specific Initiators. I am concerned about your use of the words iSCSI Target, which could apply to a couple of different things. In a Target Network Entity there can be more then one iSCSI Target Node (SCSI Device). Each Target Node can have more then one iSCSI (SCSI) Target Port connected to it. Part of the name of this iSCSI (SCSI) Target Port includes the Portal Group Tag. So if, in your point 3, the term iSCSI Target meant iSCSI Target Node, then you would be able to set up different iSCSI (SCSI) Target Ports (each with different Portal Group Tags) that can access the resources at the same iSCSI Target Node. The ACL would then probably be applied at the iSCSI (SCSI) Target Port. To be sure we are on the same page here, please reference the charts that have been placed at: http://www.haifa.il.ibm.com/satran/ips/iSCSIConfigurationExamples.pdf (Note: I use the term iSCSI (SCSI) Port to represent the concept of "SCSI Port" within the context of iSCSI.) . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "KRUEGER,MARJORIE (HP-Roseville,ex1)" @ece.cmu.edu on 11/21/2001 11:11:45 AM Sent by: owner-ips@ece.cmu.edu To: "'Rahul Bhagwat'" , ips@ece.cmu.edu cc: Subject: RE: Selectively exposing Porgal Groups to Initiators > 1. Is it required that TargetAddresses of an iSCSI target advertised to a > directory service include portal group tag ? Yes, information is necessary to communicate to the initiator which target addresses can be used to form a multi-connection session. > 2. Is it mandatory for an Initiator to use "SendTargets" to discover > TargetAddress for an iSCSI target (even if if has a set of addresses > either statically configured or found through a directory service)? To my knowledge no iSCSI document has declared it mandatory, but it's recommended to ensure the initiator has current addressing information for this target. > 3. Is it okay to restrict access to an Initiator (based on it's iSCSI name) > to only a subset of total Target Portal Groups supported by the iSCSI > target? Yes > > In this scenario, an Initiator may find out the TargetAddresses for an iSCSI > target using a directory service, and try to connect a normal operational > session to any of these addresses without using SendTargets. The > iSCSI target can return an error code for login as "Initiator not Authorized" > which in fact is not > completely true. Initiator is authorized to only use a subset > of portal groups. Correct, although "not completely true" is subjective. Another perspective is that this initiator is not authorized to use this *target port*, which is completely true. Marjorie From owner-ips@ece.cmu.edu Tue Nov 27 13:57:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA17793 for ; Tue, 27 Nov 2001 13:57:10 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fARHWle27076 for ips-outgoing; Tue, 27 Nov 2001 12:32:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from patan.sun.com (patan.Sun.COM [192.18.98.43]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fARHWjl27072 for ; Tue, 27 Nov 2001 12:32:45 -0500 (EST) Received: from phys-ha10nwka.ebay.sun.com ([129.150.152.210]) by patan.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA14054 for ; Tue, 27 Nov 2001 10:32:27 -0700 (MST) Received: from aphelion (aphelion [129.150.144.99]) by phys-ha10nwka.ebay.sun.com (8.8.8+Sun/8.8.8/ENSMAIL,v2.0) with SMTP id JAA19468 for ; Tue, 27 Nov 2001 09:32:43 -0800 (PST) Message-Id: <200111271732.JAA19468@phys-ha10nwka.ebay.sun.com> Date: Tue, 27 Nov 2001 09:38:39 -0800 (PST) From: Carl Childers Reply-To: Carl Childers Subject: remove To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.3.2 SunOS 5.7 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk From owner-ips@ece.cmu.edu Tue Nov 27 14:16:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA19259 for ; Tue, 27 Nov 2001 14:16:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fARICLI00544 for ips-outgoing; Tue, 27 Nov 2001 13:12:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail1.aarohicommunications.com (cpe-66-87-94-158.ca.sprintbbd.net [66.87.94.158]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fARI82l00149 for ; Tue, 27 Nov 2001 13:08:07 -0500 (EST) Received: from LINUX15 (linux15.aarohicommunications.com [192.168.1.234]) by mail1.aarohicommunications.com (Mirapoint) with ESMTP id AAJ04250 (AUTH shaileshm); Tue, 27 Nov 2001 10:10:49 -0800 (PST) From: "Shailesh Manjrekar" To: "'John Hufferd'" , "'KRUEGER,MARJORIE \(HP-Roseville,ex1\)'" Cc: "'Rahul Bhagwat'" , Subject: RE: iSCSI: Selectively exposing Porgal Groups to Initiators Date: Tue, 27 Nov 2001 10:08:59 -0800 Message-ID: <000701c1776e$99dd0030$ea01a8c0@aarohicommunications.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2627 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit John, You spotted it right. The portal group tag in both the examples was the same.. just to make sure that people are not confused about the general portal group and nexus understanding. Thanks for correcting it ! - Shailesh. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] On Behalf Of John Hufferd Sent: Tuesday, November 27, 2001 12:34 AM To: KRUEGER,MARJORIE (HP-Roseville,ex1) Cc: 'Shailesh Manjrekar'; 'Rahul Bhagwat'; ips@ece.cmu.edu Subject: iSCSI: Selectively exposing Porgal Groups to Initiators Marjorie, Your logic is correct, however, it does look like I had a finger check as I typed in two characters. I intended the Initiator iSCSI(SCSI) Port with the ISID=3 to have a nexus with both the Target iSCSI(SCSI) Port ...A+1 and .... A+2. and the same Targets for the Initiator iSCSI(SCSI) Port with the ISID=1. However I think I typed the A+1 and A+2 backwards on entry 5 & 6. Shailesh, Thank you for doing such a careful review. I will have Julian up date the Web Site. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "KRUEGER,MARJORIE (HP-Roseville,ex1)" on 11/26/2001 06:19:17 PM To: "'Shailesh Manjrekar'" , John Hufferd/San Jose/IBM@IBMUS, "'Rahul Bhagwat'" cc: ips@ece.cmu.edu Subject: RE: Selectively exposing Porgal Groups to Initiators Those are not parallel nexus, they are both examples of the same iSCSI initiator port talking to two different iSCSI target ports. Parallel nexus would be two iSCSI sessions between the same port pairs. A "SCSI nexus" is defined as a relationship between an initiator port and a target port. Marjorie > -----Original Message----- > From: Shailesh Manjrekar [mailto:shaileshm@aarohicommunications.com] > Sent: Monday, November 26, 2001 5:44 PM > To: 'John Hufferd'; 'Rahul Bhagwat' > Cc: ips@ece.cmu.edu; 'KRUEGER,MARJORIE (HP-Roseville,ex1)' > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > Hi John, > > In your configuration examples, in the slide about SCSI nexi, Model 3, > entries 3 and 5 would form a parallel nexus, so as entries 4 > and 6. The > entries instead should have been : > > 3) iqn.1999-12.com.ajax.os1 > + VID=2 + ISID=3 and > eui.02004567A425678A+1 > 4) iqn.1999-12.com.ajax.os1 > + VID=5 + ISID=1 and > eui.02004567A425678A+2 > ****** ************* > 5) iqn.1992-12.com.ajax.os1 > + VID=2 + ISID=3 and > eui.02004567A425678A+2 > 6) iqn.1999-12.com.ajax.os1 > + VID=5 + ISID=1 and > eui.02004567A425678A+1 > ****** ************* > Right ? > > Regards, > Shailesh Manjrekar > Aarohi Communications. > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu] On > Behalf Of > John Hufferd > Sent: Wednesday, November 21, 2001 12:43 PM > To: 'Rahul Bhagwat' > Cc: ips@ece.cmu.edu; KRUEGER,MARJORIE (HP-Roseville,ex1) > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > Rahul, > Marjorie's answers are correct, but you might want to be very careful > with > your terms. Your point three talks about limiting Portal Groups to > specific Initiators. I am concerned about your use of the words iSCSI > Target, which could apply to a couple of different things. > > In a Target Network Entity there can be more then one iSCSI > Target Node > (SCSI Device). Each Target Node can have more then one iSCSI (SCSI) > Target > Port connected to it. Part of the name of this iSCSI (SCSI) > Target Port > includes the Portal Group Tag. So if, in your point 3, the term iSCSI > Target meant iSCSI Target Node, then you would be able to set up > different > iSCSI (SCSI) Target Ports (each with different Portal Group Tags) that > can > access the resources at the same iSCSI Target Node. The ACL > would then > probably be applied at the iSCSI (SCSI) Target Port. > > To be sure we are on the same page here, please reference the charts > that > have been placed at: > > http://www.haifa.il.ibm.com/satran/ips/iSCSIConfigurationExamples.pdf > > (Note: I use the term iSCSI (SCSI) Port to represent the concept of > "SCSI > Port" within the context of iSCSI.) > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > > "KRUEGER,MARJORIE (HP-Roseville,ex1)" > @ece.cmu.edu > on 11/21/2001 11:11:45 AM > > Sent by: owner-ips@ece.cmu.edu > > > To: "'Rahul Bhagwat'" , ips@ece.cmu.edu > cc: > Subject: RE: Selectively exposing Porgal Groups to Initiators > > > > > 1. Is it required that TargetAddresses of an iSCSI target advertised > to a > > directory service include portal group tag ? > > Yes, information is necessary to communicate to the initiator which > target > addresses can be used to form a multi-connection session. > > > 2. Is it mandatory for an Initiator to use "SendTargets" to discover > > TargetAddress for an iSCSI target (even if if has a set of > addresses > > either statically configured or found through a > directory service)? > > To my knowledge no iSCSI document has declared it mandatory, but it's > recommended to ensure the initiator has current addressing information > for > this target. > > > 3. Is it okay to restrict access to an Initiator (based on > it's iSCSI > name) > > to only a subset of total Target Portal Groups supported by the > iSCSI > > target? > > Yes > > > > > In this scenario, an Initiator may find out the > TargetAddresses for an > iSCSI > > target using a directory service, and try to connect a normal > operational > > session to any of these addresses without using SendTargets. The > > iSCSI target can return an error code for login as "Initiator not > Authorized" > > which in fact is not > > completely true. Initiator is authorized to only use a subset > > of portal groups. > > Correct, although "not completely true" is subjective. Another > perspective > is that this initiator is not authorized to use this *target port*, > which > is > completely true. > > Marjorie > > > From owner-ips@ece.cmu.edu Tue Nov 27 19:50:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01426 for ; Tue, 27 Nov 2001 19:50:09 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fARNtI028913 for ips-outgoing; Tue, 27 Nov 2001 18:55:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fARNtEl28906 for ; Tue, 27 Nov 2001 18:55:14 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id PAA09377; Tue, 27 Nov 2001 15:55:04 -0800 (PST) Received: from aimexc02.corp.adaptec.com (aimexc02.corp.adaptec.com [162.62.62.40]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id PAA16631; Tue, 27 Nov 2001 15:40:03 -0800 (PST) Received: from adaptec.com (ws10-20-142-76.corp.adaptec.com [10.20.142.76]) by aimexc02.corp.adaptec.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id V4VC0HXZ; Tue, 27 Nov 2001 15:55:03 -0800 Message-ID: <3C0427D5.9EA467DE@adaptec.com> Date: Tue, 27 Nov 2001 15:55:02 -0800 From: Tow Wang X-Mailer: Mozilla 4.78 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: "Robert D. Russell" , ips@ece.cmu.edu Subject: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hi all. I fully agree with the suggestions for issue 4. If we really need to reserve a value to mean "unlimited" or "infinite", could we use a value of all bits set to one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be great for most 32-bit processors and above.) This would make arithmetic comparisons more intuitive, IMHO. "Robert D. Russell" wrote: > Attached are the new issues that arose during the iSCSI plugfest > at UNH on Wednesday 31-Oct-2001. > > (Note: these issues do not take into account any modifications or > clarifications that occured in the standard due to the issues raised > on Monday or Tuesday.) > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > now allow: "A value of 0 indicates no limit." > > Is this useful? Does it buy anything? > > The difficulties implementers are having with this are: > > 1) It is a special case. > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > 3) It violates the min/max function normally used for the key. > 4) There is always a limit anyway. > > Consider FirstBurstSize, which can have a value that is described > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > is selected. > > I one side offers 0 to mean unlimited, and the other side > has a limit, it will reply with that limit, say 128 Kbytes. > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > The statement in the standard that "the minimum of the 2 numbers is > selected" is therefore wrong when one of the numbers is 0. > > Furthermore, when an initiator or target receives an offer for one > of these keys, it cannot simply check that the offered value is > legal by testing it against some minimum and maximum. It must first > check for 0 and then only if that check shows the value is non-zero > can it do the min/max range check for legality (i.e., 64-2**24). > > Finally, there is always a limit. If nothing else it is the > limit imposed by the 24-bit DataSegmentLength field of the PDU > requesting the transfer. It is useless to specify a FirstBurstSize > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > the largest possible DataSegmentLength in any PDU that can use > this value is 2**24-1. > > The suggestion is to just eliminate this special case of 0 and require > that the range 64-to-(2**24-1) be used instead -- it has exactly the > same effect in all cases, it is easier to describe in the standard > because it avoids all the extra words, and it is easier to code > because it avoids all the special cases. > > NOTE: the standard should specify the limit in the ranges for > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > of 2**24. The number 2**24 cannot be represented in the 24-bit > DataSegmentLength field and therefore can never be used. -- Tow Wang Adaptec Software Engineer Telephone: 408 957 4838 Mail stop 46 800 869 8883 x4838 691 South Milpitas Boulevard Fax: 530 686 8023 Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com From owner-ips@ece.cmu.edu Tue Nov 27 21:01:07 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA02896 for ; Tue, 27 Nov 2001 21:01:07 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAS0jb002200 for ips-outgoing; Tue, 27 Nov 2001 19:45:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (smtp.nishansystems.com [216.217.36.162] (may be forged)) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAS0jal02196 for ; Tue, 27 Nov 2001 19:45:36 -0500 (EST) Received: by ariel.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Tue, 27 Nov 2001 16:45:30 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Subject: iFCP: Broadcast support Date: Tue, 27 Nov 2001 16:45:19 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, November 20, 2001 3:56 PM > To: cmonia@NishanSystems.com > Cc: ips@ece.cmu.edu > Subject: RE: iFCP: iFCP -06 comments > > > > > > > 10.4 b) What happens when the broadcast frame exceeds > the MTU size? > > [... snip ...] > > > We had intended to use IP datagram fragmentation > > Oh ... I'm not sure that's a real good idea. Using IP > fragmentation is > one of the few ways to make UDP datagram delivery even less reliable > than it already is. This puts FC broadcast at the mercy of code that > doesn't > fragment correctly at both iFCP and network nodes > (infrequently used code > paths are subject to bit rot) and systems in the network that can't or > don't want to deal with IP fragments (e.g., firewalls and > NAT/NAPT boxes) > and hence drop rather than fragment. For source > fragmentation, discovering > the MTU to fragment to may also be a problem area. > > While I'm not going to put my foot down yet, I would advise > thinking about > using TCP to deliver FC broadcast frames over iFCP. If FC > broadcasts are > infrequent and not performance critical, perhaps iSNS could > provide the > FC broadcast forwarding service? This would also clean up > any issues around > whether an iFCP gateway knows about all the other gateways in > the fabric, > as the iSNS server always knows this. > We are considering a client-server variant of the above in which the iSNS server supplies the address of a an iFCP "broadcast server", rather than performing the actual broadcast itself. An iFCP gateway that supports FC broadcast would implement the iFCP Broadcast Client and may implement the iFCP Broadcast Server function. Broadcast traffic would be sent and received using iFCP. The server replicates any FC frames it receives and retransmits them to broadcast clients on other gateways for local redistribution. A client does not receive a copy of any broadcast frame it originates. The broadcast server and its clients would each have an N_PORT network address whose N_PORT I/D component was 0xff-ff-ff ( corresponding to the well-known address of the FC broadcast server.) Charles From owner-ips@ece.cmu.edu Tue Nov 27 21:48:21 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA04441 for ; Tue, 27 Nov 2001 21:48:20 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAS20pJ07337 for ips-outgoing; Tue, 27 Nov 2001 21:00:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web12802.mail.yahoo.com (web12802.mail.yahoo.com [216.136.174.37]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAS20nl07333 for ; Tue, 27 Nov 2001 21:00:49 -0500 (EST) Message-ID: <20011128020049.66610.qmail@web12802.mail.yahoo.com> Received: from [24.42.134.59] by web12802.mail.yahoo.com via HTTP; Tue, 27 Nov 2001 18:00:49 PST Date: Tue, 27 Nov 2001 18:00:49 -0800 (PST) From: Luben Tuikov Subject: Re: iSCSI v8 CRC32c To: vince_cavanna@agilent.com Cc: ips@ece.cmu.edu, pat_thaler@agilent.com, vince_cavanna@agilent.com In-Reply-To: <01A7DAF31F93D511AEE300D0B706ED920CF705@axcs13.cos.agilent.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Vince, Thanks. I'll look at it right away! -l --- vince_cavanna@agilent.com wrote: > Luben, > > Your questions may have already been answered by Paul > Koning who has > apparently reviewed your code in detail but attached, as > I promised, is a > Mathematica worksheet (pdf format) that produces results > consistent with the > iSCSI spec. You do not need to know the syntax of > Mathematica in order to > follow the process, as I have inserted lots of comments. > > The example I describe uses, as data, 32 decrementing > bytes. This is one of > hte test cases in the iSCSI document. I have added an > undetectable error > pattern which does not change the results obtained at the > receiver. I think > it would be useful to include, in the iSCSI document, > such an undetectable > error pattern. > > I am considering writing an informative Internet Draft > describing the > process. I would incorporate the contents of the attached > worksheet and also > explain the theory behind the process. This should answer > such questions as > why is the expected remainder at the receiver the > constant 1c 2d 19 ed in > hex. If you or others have suggestions let me know. > > For those not interested in the math I summarize the > process below. > > Vicente Cavanna > Agilent Technologies > > > The symbols I use: > ------------------ > > G is the iSCSI CRC32c polynomial. > > L32 is the order-32 poly represting all 1s. > > F is the poly representing 32 decrementing bytes which I > will use as my test > data. > > ReflectedF is the poly representing F, after reversing > the order of bits > within each byte. > > R is the remainder of the division performed at the > transmitter. > > ReflectedR is the poly obtained by reversing the order of > bits within each > byte of R. > > FCS is the complement of ReflectedR. This is the CRC that > you would compare > with the iSCSI document. In the case of this example of > 32 decrementing > bytes the FCS is a7 e4 e4 ae in hex. > > M is the transmitted message before injecting errors. > > Error is the error pattern that I add to M. I use G+x^5*G > after applying > reflection. Any linear combination of Gs will similarly > be undetected. > > M* is the received message. > > ReflectedM* is M* with the order of bits within each byte > reversed. > > Rec is the computed CRC at the receiver. You would expect > 1c2d19ed in the > absence of errors or if error pattern is undetectable as > is the case in my > example. > > The process, for the case of no errors, is as follows: > ------------------------------------------------------ > > At the transmitter: > ---------------------- > > F is the data. > > Reverse bits within each byte of F to obtain ReflectedF. > > Shift ReflectedF left by 32 positions. > > Add 1's to most significant 32 bits of the result > (implemented by > initializing CRC register to 1's). > > Divide by G to obtain the 32 bit remainderR. Note that R > is not the CRC > shown in iSCSI document! > > Reverse bits within each byte of R to obtain ReflectedR. > > Add 32 1's to the result to obtain the FCS (implemented > by complementing). > This is what is shown in the iSCSI document and, for this > example, is a7 e4 > e4 ae in hex. > > Form the transmitted message, M, by shifting F left by 32 > positions and > adding FCS. Note that M is derived from F rather than > from ReflectedF even > though the FCS is computed from ReflectedF. > > M* is hte same as M since we are not introducing errors > here. See the > Mathematica worksheet for the case with (undetectable) > errors. > > At the receiver: > ---------------- > > Receive M* > > Reverse bits within each byte of M* to obtain > ReflectedM*. > > Shift the result left by 32 positions. > > Add 1's to most significant 32 bits of result > (implemented by initializing > CRc to 1's). > > Divide by G to obtain the remainder, Rec. The result (for > the case of no > errors) is expected to be the constant 1c 2d 19 ed in > hex. > > In the Mathcad worksheet I actually introduce an > undetectable error pattern, > G+G*x^5, which I reflect and add to M to obtain M*. The > rest is the same. > Note that the error must also be "reflected". See the > worksheet for details. > > > > > > <> > > ATTACHMENT part 2 application/octet-stream name=TestingCRC32cDecrBytePattern.pdf ===== -- __________________________________________________ Do You Yahoo!? Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. http://geocities.yahoo.com/ps/info1 From owner-ips@ece.cmu.edu Tue Nov 27 22:09:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA04654 for ; Tue, 27 Nov 2001 22:09:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAS1dv205873 for ips-outgoing; Tue, 27 Nov 2001 20:39:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAS1drl05859 for ; Tue, 27 Nov 2001 20:39:53 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 4070333F7; Tue, 27 Nov 2001 18:39:52 -0700 (MST) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 03058523; Tue, 27 Nov 2001 18:39:52 -0700 (MST) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Tue, 27 Nov 2001 18:39:51 -0700 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Tue, 27 Nov 2001 18:39:51 -0700 Message-ID: <01A7DAF31F93D511AEE300D0B706ED920CF705@axcs13.cos.agilent.com> From: vince_cavanna@agilent.com To: ltuikov@yahoo.com Cc: ips@ece.cmu.edu, pat_thaler@agilent.com, vince_cavanna@agilent.com Subject: Re: iSCSI v8 CRC32c Date: Tue, 27 Nov 2001 18:39:49 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C177AD.917FBD80" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C177AD.917FBD80 Content-Type: text/plain; charset="ISO-8859-1" Luben, Your questions may have already been answered by Paul Koning who has apparently reviewed your code in detail but attached, as I promised, is a Mathematica worksheet (pdf format) that produces results consistent with the iSCSI spec. You do not need to know the syntax of Mathematica in order to follow the process, as I have inserted lots of comments. The example I describe uses, as data, 32 decrementing bytes. This is one of hte test cases in the iSCSI document. I have added an undetectable error pattern which does not change the results obtained at the receiver. I think it would be useful to include, in the iSCSI document, such an undetectable error pattern. I am considering writing an informative Internet Draft describing the process. I would incorporate the contents of the attached worksheet and also explain the theory behind the process. This should answer such questions as why is the expected remainder at the receiver the constant 1c 2d 19 ed in hex. If you or others have suggestions let me know. For those not interested in the math I summarize the process below. Vicente Cavanna Agilent Technologies The symbols I use: ------------------ G is the iSCSI CRC32c polynomial. L32 is the order-32 poly represting all 1s. F is the poly representing 32 decrementing bytes which I will use as my test data. ReflectedF is the poly representing F, after reversing the order of bits within each byte. R is the remainder of the division performed at the transmitter. ReflectedR is the poly obtained by reversing the order of bits within each byte of R. FCS is the complement of ReflectedR. This is the CRC that you would compare with the iSCSI document. In the case of this example of 32 decrementing bytes the FCS is a7 e4 e4 ae in hex. M is the transmitted message before injecting errors. Error is the error pattern that I add to M. I use G+x^5*G after applying reflection. Any linear combination of Gs will similarly be undetected. M* is the received message. ReflectedM* is M* with the order of bits within each byte reversed. Rec is the computed CRC at the receiver. You would expect 1c2d19ed in the absence of errors or if error pattern is undetectable as is the case in my example. The process, for the case of no errors, is as follows: ------------------------------------------------------ At the transmitter: ---------------------- F is the data. Reverse bits within each byte of F to obtain ReflectedF. Shift ReflectedF left by 32 positions. Add 1's to most significant 32 bits of the result (implemented by initializing CRC register to 1's). Divide by G to obtain the 32 bit remainderR. Note that R is not the CRC shown in iSCSI document! Reverse bits within each byte of R to obtain ReflectedR. Add 32 1's to the result to obtain the FCS (implemented by complementing). This is what is shown in the iSCSI document and, for this example, is a7 e4 e4 ae in hex. Form the transmitted message, M, by shifting F left by 32 positions and adding FCS. Note that M is derived from F rather than from ReflectedF even though the FCS is computed from ReflectedF. M* is hte same as M since we are not introducing errors here. See the Mathematica worksheet for the case with (undetectable) errors. At the receiver: ---------------- Receive M* Reverse bits within each byte of M* to obtain ReflectedM*. Shift the result left by 32 positions. Add 1's to most significant 32 bits of result (implemented by initializing CRc to 1's). Divide by G to obtain the remainder, Rec. The result (for the case of no errors) is expected to be the constant 1c 2d 19 ed in hex. In the Mathcad worksheet I actually introduce an undetectable error pattern, G+G*x^5, which I reflect and add to M to obtain M*. The rest is the same. Note that the error must also be "reflected". See the worksheet for details. <> ------_=_NextPart_000_01C177AD.917FBD80 Content-Type: application/octet-stream; name="TestingCRC32cDecrBytePattern.pdf" Content-Disposition: attachment; filename="TestingCRC32cDecrBytePattern.pdf" Content-Transfer-Encoding: quoted-printable %PDF-1.3=0D%=E2=E3=CF=D3 1 0 obj=0D<< =0D/Creator = =0D/CreationDate = (D:20011127133834)=0D/Title = =0D/Author= =0D/Producer (Acrobat PDFWriter 4.05 = for Windows NT)=0D/ModDate (D:20011127135912-08'00')=0D>> =0Dendobj=0D3 = 0 obj=0D<< =0D/Pages 84 0 R =0D/Type /Catalog =0D>> =0Dendobj=0D4 0 = obj=0D<< =0D/Type /Page =0D/Parent 5 0 R =0D/Resources << /Font << /F1 = 10 0 R /F0 6 0 R /F4 16 0 R /F3 14 0 R /F2 12 0 R >> =0D/ProcSet [ /PDF = /Text ] >> =0D/Contents 85 0 R =0D>> =0Dendobj=0D5 0 obj=0D<< =0D/Kids = [ 4 0 R 21 0 R 29 0 R 33 0 R 37 0 R 41 0 R ] =0D/Count 6 =0D/Type = /Pages =0D/Parent 84 0 R =0D>> =0Dendobj=0D6 0 obj=0D<< =0D/Type /Font = =0D/Subtype /TrueType =0D/Name /F0 =0D/BaseFont /TimesNewRoman = =0D/FirstChar 32 =0D/LastChar 255 =0D/Widths [ 250 333 408 500 500 833 = 778 180 333 333 500 564 250 333 250 278 500 =0D500 500 500 500 500 500 = 500 500 500 278 278 564 564 564 444 921 =0D722 667 667 722 611 556 722 = 722 333 389 722 611 889 722 722 556 =0D722 667 556 611 722 722 944 722 = 722 611 333 278 333 469 500 333 =0D444 500 444 500 444 333 500 500 278 = 278 500 278 778 500 500 500 =0D500 333 389 278 500 500 722 500 500 444 = 480 200 480 541 778 500 =0D778 333 500 444 1000 500 500 333 1000 556 = 333 889 778 611 778 778 =0D333 333 444 444 350 500 1000 333 980 389 333 = 722 778 444 722 250 =0D333 500 500 500 500 200 500 333 760 276 500 564 = 333 760 500 400 =0D549 300 300 333 576 453 250 333 300 310 500 750 750 = 750 444 722 =0D722 722 722 722 722 889 667 611 611 611 611 333 333 333 = 333 722 =0D722 722 722 722 722 722 564 722 722 722 722 722 722 556 500 = 444 =0D444 444 444 444 444 667 444 444 444 444 444 278 278 278 278 500 = =0D500 500 500 500 500 500 549 500 500 500 500 500 500 500 500 ] = =0D/Encoding /WinAnsiEncoding =0D/FontDescriptor 7 0 R =0D>> = =0Dendobj=0D7 0 obj=0D<< =0D/Type /FontDescriptor =0D/FontName = /TimesNewRoman =0D/Flags 34 =0D/FontBBox [ -250 -216 1158 1000 ] = =0D/MissingWidth 321 =0D/StemV 73 =0D/StemH 73 =0D/ItalicAngle 0 = =0D/CapHeight 891 =0D/XHeight 446 =0D/Ascent 891 =0D/Descent -216 = =0D/Leading 149 =0D/MaxWidth 965 =0D/AvgWidth 401 =0D>> =0Dendobj=0D10 = 0 obj=0D<< =0D/Type /Font =0D/Subtype /TrueType =0D/Name /F1 = =0D/BaseFont /TimesNewRoman,Italic =0D/FirstChar 32 =0D/LastChar 255 = =0D/Widths [ 250 333 420 500 500 833 778 214 333 333 500 675 250 333 = 250 278 500 =0D500 500 500 500 500 500 500 500 500 333 333 675 675 675 = 500 920 =0D611 611 667 722 611 611 722 722 333 444 667 556 833 667 722 = 611 =0D722 611 500 556 722 611 833 611 556 556 389 278 389 422 500 333 = =0D500 500 444 500 444 278 500 500 278 278 444 278 722 500 500 500 = =0D500 389 389 278 500 444 667 444 444 389 400 275 400 541 778 500 = =0D778 333 500 556 889 500 500 333 1000 500 333 944 778 556 778 778 = =0D333 333 556 556 350 500 889 333 980 389 333 667 778 389 556 250 = =0D389 500 500 500 500 275 500 333 760 276 500 675 333 760 500 400 = =0D549 300 300 333 576 523 250 333 300 310 500 750 750 750 500 611 = =0D611 611 611 611 611 889 667 611 611 611 611 333 333 333 333 722 = =0D667 722 722 722 722 722 675 722 722 722 722 722 556 611 500 500 = =0D500 500 500 500 500 667 444 444 444 444 444 278 278 278 278 500 = =0D500 500 500 500 500 500 549 500 500 500 500 500 444 500 444 ] = =0D/Encoding /WinAnsiEncoding =0D/FontDescriptor 11 0 R =0D>> = =0Dendobj=0D11 0 obj=0D<< =0D/Type /FontDescriptor =0D/FontName = /TimesNewRoman,Italic =0D/Flags 98 =0D/FontBBox [ -250 -216 1158 1000 ] = =0D/MissingWidth 376 =0D/StemV 73 =0D/StemH 73 =0D/ItalicAngle -11 = =0D/CapHeight 891 =0D/XHeight 446 =0D/Ascent 891 =0D/Descent -216 = =0D/Leading 149 =0D/MaxWidth 965 =0D/AvgWidth 402 =0D>> =0Dendobj=0D12 = 0 obj=0D<< =0D/Type /Font =0D/Subtype /TrueType =0D/Name /F2 = =0D/BaseFont /CourierNew,Bold =0D/FirstChar 32 =0D/LastChar 255 = =0D/Widths [ 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 ] = =0D/Encoding /WinAnsiEncoding =0D/FontDescriptor 13 0 R =0D>> = =0Dendobj=0D13 0 obj=0D<< =0D/Type /FontDescriptor =0D/FontName = /CourierNew,Bold =0D/Flags 16418 =0D/FontBBox [ -250 -300 713 1000 ] = =0D/MissingWidth 594 =0D/StemV 191 =0D/StemH 191 =0D/ItalicAngle 0 = =0D/CapHeight 833 =0D/XHeight 417 =0D/Ascent 833 =0D/Descent -300 = =0D/Leading 133 =0D/MaxWidth 594 =0D/AvgWidth 600 =0D>> =0Dendobj=0D14 = 0 obj=0D<< =0D/Type /Font =0D/Subtype /TrueType =0D/Name /F3 = =0D/BaseFont /ONHAEA+Math2Mono,Bold =0D/FirstChar 30 =0D/LastChar 255 = =0D/Widths [ 500 500 263 260 1041 260 1183 260 1214 260 1143 1143 1143 = 260 918 =0D1041 1182 1214 918 1041 1183 1214 1143 1143 1143 255 600 600 = 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 1000 600 600 600 600 600 600 600 600 600 600 600 = =0D600 300 300 300 918 600 600 600 600 600 600 600 600 447 723 447 = =0D447 447 764 764 764 600 600 600 600 600 600 600 600 655 1087 655 = =0D811 655 1205 1206 1206 600 600 600 600 600 600 600 600 874 1496 = =0D874 1123 870 1737 1736 1736 600 600 600 600 600 600 600 600 600 = =0D600 263 600 600 447 655 870 600 600 600 600 600 600 200 600 600 = =0D447 784 821 764 764 692 694 643 870 1265 1819 1223 1645 1205 1737 = =0D1205 1737 861 1333 861 1333 600 600 600 600 600 600 734 1102 1469 = =0D600 600 600 600 600 600 600 600 600 ] =0D/FontDescriptor 15 0 R = =0D>> =0Dendobj=0D15 0 obj=0D<< =0D/Type /FontDescriptor =0D/FontName = /ONHAEA+Math2Mono,Bold =0D/Flags 16388 =0D/FontBBox [ -250 -4039 2183 = 2212 ] =0D/MissingWidth 600 =0D/StemV 159 =0D/StemH 159 =0D/ItalicAngle = 0 =0D/CapHeight 2212 =0D/XHeight 1106 =0D/Ascent 2212 =0D/Descent -4039 = =0D/Leading 5251 =0D/MaxWidth 1819 =0D/AvgWidth 500 =0D/FontFile2 64 0 = R =0D>> =0Dendobj=0D16 0 obj=0D<< =0D/Type /Font =0D/Subtype /TrueType = =0D/Name /F4 =0D/BaseFont /PNHAEA+Math1Mono,Bold =0D/FirstChar 30 = =0D/LastChar 255 =0D/Widths [ 500 500 263 600 600 600 600 600 600 600 = 600 600 600 600 600 600 600 =0D600 600 600 600 600 600 600 600 600 600 = 600 600 600 600 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 = 600 600 600 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 = 600 600 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 600 =0D1000 240 291 600 600 600 600 1200 1200 600 600 798 600 600 = 600 600 =0D600 600 600 1200 200 1200 790 1200 600 1200 1200 1200 733 = 600 600 =0D600 600 600 600 411 600 600 735 740 600 600 600 600 790 720 = 600 =0D720 600 600 600 649 600 600 600 600 600 600 600 600 600 995 600 = =0D200 628 600 337 654 600 600 600 600 628 628 600 1000 600 600 600 = =0D600 600 600 600 600 600 797 679 600 600 600 694 692 790 720 600 = =0D720 600 600 600 598 600 600 650 600 600 600 600 600 600 600 600 = =0D600 600 600 600 682 870 870 870 600 600 600 600 600 600 600 600 = =0D600 918 ] =0D/FontDescriptor 17 0 R =0D>> =0Dendobj=0D17 0 obj=0D<< = =0D/Type /FontDescriptor =0D/FontName /PNHAEA+Math1Mono,Bold =0D/Flags = 16390 =0D/FontBBox [ -250 -234 1417 1000 ] =0D/MissingWidth 590 = =0D/StemV 159 =0D/StemH 159 =0D/ItalicAngle 0 =0D/CapHeight 853 = =0D/XHeight 427 =0D/Ascent 853 =0D/Descent -234 =0D/Leading 87 = =0D/MaxWidth 1181 =0D/AvgWidth 500 =0D/FontFile2 69 0 R =0D>> = =0Dendobj=0D21 0 obj=0D<< =0D/Type /Page =0D/Parent 5 0 R =0D/Resources = << /Font << /F1 10 0 R /F0 6 0 R /F6 24 0 R /F4 16 0 R /F3 14 0 R /F7 = 26 0 R =0D/F2 12 0 R >> =0D/ProcSet [ /PDF /Text ] >> =0D/Contents 87 0 = R =0D>> =0Dendobj=0D24 0 obj=0D<< =0D/Type /Font =0D/Subtype /TrueType = =0D/Name /F6 =0D/BaseFont /CourierNew =0D/FirstChar 32 =0D/LastChar 255 = =0D/Widths [ 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 ] = =0D/Encoding /WinAnsiEncoding =0D/FontDescriptor 25 0 R =0D>> = =0Dendobj=0D25 0 obj=0D<< =0D/Type /FontDescriptor =0D/FontName = /CourierNew =0D/Flags 34 =0D/FontBBox [ -250 -300 768 1000 ] = =0D/MissingWidth 640 =0D/StemV 109 =0D/StemH 109 =0D/ItalicAngle 0 = =0D/CapHeight 833 =0D/XHeight 417 =0D/Ascent 833 =0D/Descent -300 = =0D/Leading 133 =0D/MaxWidth 640 =0D/AvgWidth 600 =0D>> =0Dendobj=0D26 = 0 obj=0D<< =0D/Type /Font =0D/Subtype /TrueType =0D/Name /F7 = =0D/BaseFont /AOHAEA+Math1Mono =0D/FirstChar 30 =0D/LastChar 255 = =0D/Widths [ 500 500 263 600 600 600 600 600 600 600 600 600 600 600 = 600 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = 600 =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 = =0D1000 240 291 600 600 600 600 1200 1200 600 600 798 600 600 600 600 = =0D600 600 600 1200 200 1200 790 1200 600 1200 1200 1200 696 600 600 = =0D600 600 600 600 371 600 600 695 750 600 600 600 600 790 720 600 = =0D720 600 600 600 569 600 600 600 600 600 600 600 600 600 988 600 = =0D200 628 600 270 587 600 600 600 600 580 580 600 1000 600 600 600 = =0D600 600 600 600 600 600 600 631 600 600 600 600 600 790 720 600 = =0D720 600 600 600 598 600 600 602 600 600 600 600 600 600 600 600 = =0D600 600 600 600 590 796 796 796 600 600 600 600 600 600 600 600 = =0D600 835 ] =0D/FontDescriptor 27 0 R =0D>> =0Dendobj=0D27 0 obj=0D<< = =0D/Type /FontDescriptor =0D/FontName /AOHAEA+Math1Mono =0D/Flags 6 = =0D/FontBBox [ -250 -234 1481 1000 ] =0D/MissingWidth 617 =0D/StemV 91 = =0D/StemH 91 =0D/ItalicAngle 0 =0D/CapHeight 853 =0D/XHeight 427 = =0D/Ascent 853 =0D/Descent -234 =0D/Leading 87 =0D/MaxWidth 1234 = =0D/AvgWidth 500 =0D/FontFile2 74 0 R =0D>> =0Dendobj=0D29 0 obj=0D<< = =0D/Type /Page =0D/Parent 5 0 R =0D/Resources << /Font << /F1 10 0 R = /F0 6 0 R /F6 24 0 R /F4 16 0 R /F3 14 0 R /F7 26 0 R =0D/F2 12 0 R >> = =0D/ProcSet [ /PDF /Text ] >> =0D/Contents 101 0 R =0D>> =0Dendobj=0D33 = 0 obj=0D<< =0D/Type /Page =0D/Parent 5 0 R =0D/Resources << /Font << = /F1 10 0 R /F0 6 0 R /F6 24 0 R /F4 16 0 R /F3 14 0 R /F7 26 0 R =0D/F2 = 12 0 R >> =0D/ProcSet [ /PDF /Text ] >> =0D/Contents 103 0 R =0D>> = =0Dendobj=0D37 0 obj=0D<< =0D/Type /Page =0D/Parent 5 0 R =0D/Resources = << /Font << /F1 10 0 R /F0 6 0 R /F6 24 0 R /F4 16 0 R /F3 14 0 R /F7 = 26 0 R =0D/F2 12 0 R >> =0D/ProcSet [ /PDF /Text ] >> =0D/Contents 105 = 0 R =0D>> =0Dendobj=0D41 0 obj=0D<< =0D/Type /Page =0D/Parent 5 0 R = =0D/Resources << /Font << /F1 10 0 R /F0 6 0 R /F6 24 0 R /F8 44 0 R = /F4 16 0 R /F3 14 0 R =0D/F7 26 0 R /F2 12 0 R >> =0D/ProcSet [ /PDF = /Text ] >> =0D/Contents 107 0 R =0D>> =0Dendobj=0D44 0 obj=0D<< = =0D/Type /Font =0D/Subtype /TrueType =0D/Name /F8 =0D/BaseFont = /BOHAEA+Math2Mono =0D/FirstChar 30 =0D/LastChar 255 =0D/Widths [ 500 = 500 263 255 919 255 1064 255 1072 255 994 994 994 255 819 919 =0D1063 = 1072 819 919 1064 1072 994 994 994 255 600 600 600 600 600 =0D600 600 = 600 600 600 600 600 600 600 600 600 600 600 600 600 600 =0D600 600 600 = 600 600 600 600 600 600 600 600 600 600 600 600 600 =0D600 600 600 600 = 600 600 600 600 600 600 600 600 600 600 600 600 =0D600 600 600 600 600 = 600 600 600 600 600 600 600 600 600 600 600 =0D600 600 1000 600 600 600 = 600 600 600 600 600 600 600 600 600 300 =0D300 300 819 600 600 600 600 = 600 600 600 600 386 598 386 478 375 =0D704 704 704 600 600 600 600 600 = 600 600 600 602 972 602 761 595 =0D1145 1146 1146 600 600 600 600 600 = 600 600 600 854 1415 854 1098 =0D843 1677 1676 1676 600 600 600 600 600 = 600 600 600 600 600 263 600 =0D600 374 571 796 600 600 600 600 600 600 = 200 600 600 374 724 761 =0D704 704 600 600 571 796 1205 1759 1163 1585 = 1145 1677 1145 1677 =0D741 1215 741 1215 600 600 600 600 600 600 734 = 1102 1469 600 600 =0D600 600 600 600 600 600 600 ] =0D/FontDescriptor = 45 0 R =0D>> =0Dendobj=0D45 0 obj=0D<< =0D/Type /FontDescriptor = =0D/FontName /BOHAEA+Math2Mono =0D/Flags 4 =0D/FontBBox [ -250 -4091 = 2110 2469 ] =0D/MissingWidth 600 =0D/StemV 91 =0D/StemH 91 = =0D/ItalicAngle 0 =0D/CapHeight 2469 =0D/XHeight 1235 =0D/Ascent 2469 = =0D/Descent -4091 =0D/Leading 5560 =0D/MaxWidth 1758 =0D/AvgWidth 500 = =0D/FontFile2 79 0 R =0D>> =0Dendobj=0D47 0 obj=0D<< =0D/Type /Page = =0D/Parent 48 0 R =0D/Resources << /Font << /F1 10 0 R /F0 6 0 R /F6 24 = 0 R /F8 44 0 R /F4 16 0 R /F3 14 0 R =0D/F7 26 0 R /F2 12 0 R >> = =0D/ProcSet [ /PDF /Text ] >> =0D/Contents 109 0 R =0D>> =0Dendobj=0D48 = 0 obj=0D<< =0D/Kids [ 47 0 R 52 0 R 56 0 R 60 0 R ] =0D/Count 4 = =0D/Type /Pages =0D/Parent 84 0 R =0D>> =0Dendobj=0D52 0 obj=0D<< = =0D/Type /Page =0D/Parent 48 0 R =0D/Resources << /Font << /F1 10 0 R = /F0 6 0 R /F6 24 0 R /F4 16 0 R /F3 14 0 R /F7 26 0 R =0D/F2 12 0 R >> = =0D/ProcSet [ /PDF /Text ] >> =0D/Contents 111 0 R =0D>> =0Dendobj=0D56 = 0 obj=0D<< =0D/Type /Page =0D/Parent 48 0 R =0D/Resources << /Font << = /F1 10 0 R /F0 6 0 R /F6 24 0 R /F4 16 0 R /F3 14 0 R /F7 26 0 R =0D/F2 = 12 0 R /F8 44 0 R >> =0D/ProcSet [ /PDF /Text ] >> =0D/Contents 113 0 R = =0D>> =0Dendobj=0D60 0 obj=0D<< =0D/Type /Page =0D/Parent 48 0 R = =0D/Resources << /Font << /F1 10 0 R /F0 6 0 R /F6 24 0 R /F4 16 0 R = /F3 14 0 R /F7 26 0 R =0D/F2 12 0 R /F8 44 0 R >> =0D/ProcSet [ /PDF = /Text ] >> =0D/Contents 115 0 R =0D>> =0Dendobj=0D64 0 obj=0D<< /Filter = /FlateDecode /Length 65 0 R /Length1 67 0 R >> =0Dstream H=89=84V TTG=16=BD=F5=B7n=E8=06=BA=D9E=D4n=D9=DC=10=90M=14=03Q = =82=06=011"=8B=0A= =08b"-=0A= =0A= =06=11P=D1=A81&1=EE&=12%=C6=10B=B0=C1=15=DC=82;=A21j=D4(A=E3df2g2=87=E39= =89=CB=81=DF=F3=9A%38sf~=9F[=EF=BFW=EF=D7=BFu=EB=D7=AB=06=03`=852=F0=88=9B= 6=DDgL~=83=E11E=1E=10b3s=D3=F3=9A=E2=9F=06=03l=04=C0uf.+=D0=A5=9D=A8=DF = =08=9E=14=1B=9C=9D7?wt=FC=15=1E=10#(=7F=D0=FC=85=CB=B3=9DV4=AD!?=1D=E0=DD= s=B2=D2=E7]u=9D=F0=1B=A0=AC=A0=FE=A0=1C=0A= =E8=1C=FE*=93=DFH=BE{NnA=D1=E3=AD-=F9=E4=B7S=FE=8A=85=8B2=D3=B7=AF=DA^ = =A8Sh=FCK=B9=E9Ey\=14=B7=17=B0v=A3|=9D!=3D7=EB|]8=8Dg=3D=89=9E=89=CA[=94= _=10=E8Y=14=058=AE=A7=FC=BF=E7-=C9=CA{=F6=E4=80=0B=E0=BC=94=FC=BF=11OpM=10= =A1=14w=89=FE=14=F1=E8=B1|%=B29[=1A=11=0A= =FC=F7kfB=B4=0E=BA=0E]=87I=D2=CBz@k=D4=C5Q=98=03=EB=EE=B6'=B5=E8b=F4&&=A1= 7H=96=EF=CE=E9=7FQ'/=88=92Bia=A9R[Y=DBh=B4=B6v=F6=0E=8EN=CE=03\=06=BA=0E= =1Aaz=E2=8C7f&=CDJ= NIM=9B=3Dgn:2=E7ee=CF=CFY=F0=E6[=0Bs=0D=8B=F2=16/=C9/X=BA=AC=B0h=F9=DB=C5= +JV=96=96=95=AFZ=BD=A6b=ED=BAw=D6o=D8=F8=EE=A6=F76=BF=FF=C1=87[>=DA=BAm=FB= =8E=9D=BBv=EF=F9=F8=93=BD=95=9F=EE=DB_=F5=D9=81=CF=0F~Q=FDe=0D_=FBu=DD!c= }=C3=E1#G=8F=1D?=D1=D8t=F2=D4=E93g=BFi>w=FE=C2=C5K=97=AF=B4\m=BDv=FD=DB=1B= =DF=DD=BCu=FB=FB;w=EF=FDp=FFA=DB=8F=ED=0F=1F=FD=04=81=FD=04=B3=D8=82y=B6= =1D&=93=89Z=9D=B9%_ = =CBS=8F=08=89=E4V=C2=02=96PAM=DF=9C5l=A0=81=16=B6=B0#E=1D=E0=08'8c=00\0=10= =AE=18=84=C1=18=02=1D=F4=18=0A= 7=B8=C3=03=9E=F0=C20=0C=C7=08=8C=C4(xc4|=E0=0B?=8C=81?=02=10=88 = =04c,B0=0E=E3=11=8A = x=05a=08=C7=AB=98=88I=88@$=A2=F0=1A&#=1A1=98=82=A9x=1D=B1=98=868=C4#=01=D3= =91=88=19x=033=91=84YHF=0A= R=91=86=D9=98=83=B9HG=0621=0FY=C8=C6|=E4`=01=DE=C4[X=88\=18=B0=08yX=8C%=C8= G=01=96b=19=0A= Q=84=E5x=1B=C5X=81=12=ACD)=ED=ABr=AC=C2j=ACA=05=D6b=1D=DE=C1zl=C0F=BC=8B= Mx=0F=9B=F1>>=C0=87=D8=82=8F=B0=15=DB=B0=1D;=B0=13=BB=B0=1B{=F01>=C1^T=E2= S=EC=C3~T=E13=1C=C0=E78=88/P=8D/Q=83=AFP=8B=AFQ=87C0=A2=1E=0D8=8C#8=8Ac8= =8E=13hD=13N=E2=14N=E3=0C=CE=E2=1B4=E3=1C=CE=E3=02.=E2=12.=E3=0A= Zp=15=AD=B8=86=EB=F8=167=F0=1Dn=E2=16n=E3{=DC=C1]=DC=C3=0F=B8O=FB=BF=0D?= =A2=1D=0F=F1=08=E6=B5=85Pj=BA/T=F1=ABM=8F=F8}r=9B|[T=0A= =FF=10=AEs=AB=C5=1Cu=92=B8Hn=D0zs=D90=8A=90+=A4{=9D5=CA=F5/:-=F5=CF=E7=D1= =A2=FF=CB=CB=E9=EE=BD=DE=D9=A0=AC =DF=E9y = =B7I=9AjZ+=AAY=A0=E4=A3=A8=EC*V=DB=BD=C8=D7=1A=9F&=89=8B=E5l=D3xeL'T=CFL= =A6=E7=BB=B8=95LfG=10F=B5=E0>Ss=83L&=D6=C0=FCY;=A7fa=F4u=D5=B2=0C)=8A=E3= X*=A7=E4=ACi5KQ=C7=E2HSc=F7=EF4}=1F5=D4=D6=D3=EC[h=F6=E6=D8=1AZ=07=994=3D= O:]=A0=DEX=14=B0=F1=8Cc=C9t=7F=98Vs7i=F4=B3=A9=FB=92K)r=1C=C7=A4y=D2=15=E1= =84=DC"=AE=C4an=00=FB=85Us=19,=88=B3d = =B8{h=88=CD(7=DD=86=0D=91qIz=FD=C0=C8=88Y=DE=A3=A6$$EF=0C=D4=EBgySA(=A3=12= P&=E9i=1B(`=7FT=C1h3=F0=02|=DA|=DA=BA=1B?_=7F=AD^=EB=A1=D7=EA=CBxt=95qDM= =D2?o/=13=F5`,N=AE=E0=ABD=AAp=88=0E=F7=B6djK=17=E6jy=94=89Z=D1 = =AD=8Bq4=C0AmP=D9o=D1=EE=D76ky=AD=9D=A8=82=8D]=8A=A8=E4=E3=9D4=BF=87=9E=EB= =0A= }=10=AA=B5=0DICXW=D8=AF=CB=C2=9C=BBB=FC|Y=1A=D3i=B5=01n=81c=02=B5=1A=BD=C3= P=07=AD=BD=C2=91=1A=7F=BEJ=9Em4=B2}=1D=19=19=1Dr=05=F3c=B1=0BjrX,=F3=AB.= =97k=0B=A3=E3=16=CB=B5=E5=C4h81*!F=AE=88 = =F7=16D=CE=D1^=B4u<.=8Aj=18\=D7=C5(=0C=92=83=9D=C1=DE~=8F=BAF}E=CD=AB-`/= i,R=E0$=C4=0F=D6=C8=FF=C9=A8=8F=D2P=C9=81=D8=989=05=04{zR=EBI=04=CD=B4=9C= =1C=FD=F9=92=E4$yvG=86=9F=DF=92=95=B7=AE=8D=1C6=87=E5=06=06=C8F=F9zu9KX=1C= =E7=9E=19=C2=12"C=AB=B7=EF=92=8DA.=CETt8N.=E5=9B=84=17=A4=B7W=B8=B3=C2=80= =12=C1=C0=BB=0A= 6TP|=A8(=08=90RY=BCR#=FF=DA=96=16j&=D2E=14H~=07=02=DF$G=B1=13=F23=A6=94K= =A5=E2=9A=17m4c=8D\=CA=B5=F4=8D=C6=19X = i=EF*=81i=98=8E=F92=81=F1=A9=88=B7xi=B4@=1A+P=CF=B5=C8=91=CCB~=CA=1A=E5=D2= =1A=D1=AD=C6=CCM"=FD=9E=88=E6C71=FC=95 = >H=8C=E2=A3=C4D.Q=9Aa1=C3*=8B=CF=12=B3-UpW=BAE=A9=0A= =85e*=9E=05+"=14s=15y=8A2=85=A8P=86=87QI=3D=E3=A5b=AA4!=DE=9A=DE=99=E6=D2= =96=E6r=EBA=DA9R4=AC[I7O.0=C06=D8_=E2=1C=ECm=F9'=F5Eg=EB=D6=D7'=DF<#W=1C= a=07=1F=B6=B3=8D7+e=FD=A9?=C9=B94=D2=1D=B9=82k=EE=E6=12=17>n,=1Bk5=99M=B6= J=E1R=C4=14=8B=14u=0E=CB=B1=B2=14=DC-=DD=A2=15E=0A= =8E=0FQMQ-T=95=AA6=ABD=95=A5=AF=C0=C2=853=C3=14L=91=86x=9B=1E=1E.i=FDx=D8= s=0A= =B7 = =DB=C0=00=CE=CB=DF=D1=96kn.4=CEm=BD=98m,md=CD=7F=96=93=8DU=AC=B1=FD1=DBw= =ACY^=DAs=82=12l=E7=1FK=99c=13=FA=1B=06(=BB=CF=CE=AF~?=99g=B6u=DBb<=E4=C4= =F6:=8D=95=D6Hy=16=BD=87,3=1F=CC=EDu=F4w`=93=9C=F8=FCg=8D=D5=1F'q=DFu_=80= y=F7=D1=A9<=B0=17=F6=F4=C7!=11=C5fkQ=D8k=8F=F4Z=0F=14[xHQf=98}E=12=8A=CD= =B9=16K = =94=F3=87_=D8=E3=F7=CB=B7=A4=12=D9=CA=E2=08^B+~!=EC=A0=FB=E1=04=8A=A3=96= =90*=B4r\=7F0=0D=C5; = =7F!=DC=A4=98=D4=1F=B8C=B8Ah"=1C=A4=FC=D4=FE@b=7F0=0FB=18a"=C1=9Fb=B7=C9= =FA=10=02{=B94=FC=7Fp=CA=7F=835=8D=91=D0c=BB=EFO=FDop=F4^=94=F7&=D2=C4D=BA=C9=89/=FB=FC=16=A4=8A=97=91j=B6}=90=06=E1=9F=EDW= kt\U=15=FE=EEd=12=9260tYK=86=B6=CC=E1=D1=D0=B4c=1Em=A5=B4Z=93=B4i=1BH=DB= I=A6B=1Fd=DA=DC=CC=DCd=C6N=EE=1D=EF=BDc=92j!* = L=13=08=E0=8B=82=82=80=A8=15=05=E2+=E0=83=FAB=EA=0B=F0=FD=16W=F5=9Fk=F1C= =97=CB=1F,=EBw=CE=3D=D3=A6!=AEJ=7F=E0=1F=E7=AC=3D{=9F=B3=F7=D9=8F=F3=BA{= '=CB@=BD=F7=9C =A1=DA=99P=B9=01=BD=D5S=D4A\u(=C05=8F = Y=F3h=00=B4w=FCL06=CC=84=AA=7Fa=DF=FC=7F"E=9C=AA~=85=F8=1F=D8w=FE_=90<=FF= D=00zof=80=B4=AB=E8#z=FF=95=1F=F2=9C=96i=15=93=F4=87=BA(=B78XStj=90g=AB=D3= 8=12`=D9W1oa=BCw=9D=8E=BB=F2v=C2=04a=99=D2=D3;/=86=E4=05=EC=D7=1C=A7_=0D= =C4=8CQ=FAY=C6U=F4u=DE=D1=D3=98z=A75=C8=3DxL=C3=D1J&=ED=D5/=02=F3=8F=A9=B3= =EF=E1~#=83 = =F9Q=96=00Y=12\rNmP=B7=A7t{)h=C6y=BAm=D7=ED=EE=FF=B7=D7=A3=A9W=F5=04=D3=E1= 0Su=83oo5=D3j=C1=E7w=8D=F1W=F6%=B7=CE=A8?=F5=F6=DE=8CrEd0=BD=BFY=D3!=CE=FE= =A0=A6+=98=EC=7FT=D3a=D2=9F=D3t%=CB=82oi=BA=8A=E3?=D6t5=F5=9C=D0t=0Dm=FF= M=D3=F3=0C=C3X=AD=E9=F9X=14Z=A3=E9Z=D2=D7=CA=12-\C'=AAC}=9A6 = *=B2=9Af=14=15=B7i=BA=02+*&5=1D&=FD=B4=A6+QW=F1gMWq=FC=15MWC=84=EB5]=83=B1= =F0=16M=CF=E3=BB=F6=AC=A6=E7#^=F5=9C=A6kI=FF}=BB=E9gWmwlGd,/7h[=19=D1?*=DA= =ED=8Ck = =D1U=81=AD=1C=CF=13=0A= =84A=EA=F58=D61C=B3=94p9^=A4=84=A0=8C=AD,=ED=E4l=8B#Y=FEK=99]=94=C8c=80=92= &=86=D0=C8=91M=1C)=D0=1FW=E9=C9R^`9=D2h = n=C1:=B6=AB=94'=01=BDV=F9z=A6=16=C1rO=FAa)=CF=D3=D4=D1=88=04=CB=B9NF=B8=99= =B0r=8EU=D9=A84d=CE*=17?=07=C9=EB=E9=89=AB=D6=C5Q1=B7=D0=9FUh~=0D=1A=CE=16= =DF=CC=9D=FB_=EC=DB\=A7,=A74=9B=844uy=E4;=CA=F7=01%=E1=CF:=87=E53x=B6H=83= =F9.qQ=8D=CF=8C=AC=EC=85=C5=99=92=CAq=86=A9=A2/s=92JSF=EDG|Ft=8E=9E=E7=FE= W=1E=14=D4=FAd=E8A=9A=F3=C6=F1&z=D1=C4{=DA=C2s=BD=1Ak=F0f=BE=98F=88;=96V=E7= r@=BDy9=BC=83=F76=CF=F8m=F5=1A=BES=9D=15=9F;=F8.=AE=D3=08O=C0A=BC=1B=EF=C1= !=DC=88=9BX=CB=BE=17=EF=C3=FB=99U=DD=82[=F1=01=DC=86=DBQ=C2a=8Cc=02w=E0N= L=E2.=DC=8D{=98i}=08=1F=C6G=98e=DD=8B#=B8=0F=F7=E3c=F88=1E=C0=83=F8=04=1E= =C2=C3x=04=9F=C4=A3=F8=14>=8D=CF=E0(>=8B=C7=98=83}=1E=8F=E3 = <=89)|=01_=C4=97=F0e|=05=D3=CC=BE=9F=C6W=F15|=1D=DF=C038=86o2C=FB6=BE=83= =EF=E2Y|=0F=CF=E18=BE=8F=1F=E0=87=F8=11=B3=B5=E7=F1=02^=C4O=F0S=FC=0C?=C7= /=F0K=FC=0A= =BF=C6o=F0[=FC=0E=BF=C7=1F=F0Gf=F1=7F=AA=E8=D8=BC=ADm=E3=E2=CB=A2=B1=8B/= =8D=C6=A2"=1A=AB=8BEc=17]=12=8D-Z=1A=8D=BD=F1P4=B6pI4=B6=82=FC=06=F2=97=93= =7F%=F9=F5=E4/#=FF=0A= =F2/'?;8m=88=A9H>@=83=F6=B4=D17=15=B9=91=E8=C1=A9=C8=80Dmu=91=88=FD=B8=FD= =8C=FD=BC=1Dn=B5=13=F6=CB=F6I;|=A7m=DC=DAK=D1=B6=AB#(]X=12=A5=8A=EER=A14= Vz=A2t=ACT=89Rs=A9=AD=D4]=EA=1E=EF=9E=E8+=F5=8DK=C6=D8=E1=B1=F1=C9=D2=E4= =E1=C9=F1=17J/=95=16=0C=8F(s=A3=01=1AI)4&Q=DB=C2H=A4=C9=A8=89$=8C=EA=A6D= k"=91=D8=9F=08=DF=D4=AF=F8=07R=CA=ABL=D0K=07=A8/@f=80=FA=03M;=02=C9;=02=B4= ?@=BD=01=BA!@=C5=BD=D2=D8=06=1A=9DL=85b)=A3)=D5=9AJ=A4=F6=A7=9CT%z.=EC=11= =3D=CD=3D=DD=3D}=3DU"=D9=9C=0C=89m=CD=DBB=E82"Fk=ED=03=C6=CBFxu=FD=92=05= =D7=19=D3=C6=C9[&=BAv=EE=9E6=B0tOW=CF=EE'=F7=8Eu=04=F8=98=C2Oa=AF=81ST[=C7= =1EO=FD=FC=E2=CA=99?o=1F=9B"=C8=C0ln=F9=F7oz=F2=A0{=0Dendstream=0Dendobj= =0D65 0 obj=0D3901 =0Dendobj=0D67 0 obj=0D6751 =0Dendobj=0D69 0 = obj=0D<< /Filter /FlateDecode /Length 70 0 R /Length1 72 0 R >> = =0Dstream H=89\VyTTe=1B=FF=BDw=1D=86m=86=94=CDPn=88=B8=D1=80=84@=0E1)=A8=A8!=E2=02= .=A8=A8,=A1(=A8)=8A=0A= dZ=A0fV.=A1=A6=E1^|j=E3=86=1Bi=92+=A3=99=BB=88Ju=F2|=E7h=C7=FE=F84=FB=E4= =CE=F7=DC=99k=C7=BE{=EEo=9E=F7=BE=CF=FE<=EF2`=00=BCQ=01=1E=E9C=87[z=8D=1F= \"=D0=CC=1DB=DA=E4=A2=9C=E2c=C3=9E=C6=01=AC;=C0=3D=9FtzNQn=FCq=7F=0E=F0=A1OCu=F1= =8CY=B3S=CA=E3=1B=01=FFj@^P<3=B7xFY=FF=15@G=89=F4=FF=A08=C1=1D=83=08=83X= #=C6=D0L=B8=9B=F2=9B=91=C7=F9yA=94=0C=E0)=7F=A6=D5=E0=1FOfFj(B=1F=87>vJ=8A= =AA=00=D2=15v=8F=A69=B8%=DBQ=B5=E8a=C1=04=CD=93=AE=C5x=97=CC?=1Fb=F2=82(= =C9=06=0F=A3=A7=97=B7=8F=AF=C9=EC=F7J=BB=F6=FE=01=81A=C1=1D^=0D=E9=D8)Ty= -=ACsx=97=88=AE=DD=BA=F7=E8=19=F9=BA%*=BAW=CC=1B=B1=BD=E3=E2=13=DE=ECcM|= +=C9=F6v=DF~=C9)=FD=07=0CL=1D4x=C8;iC=D3=87e=0C=1F1rTf=D6=E81c=C7e=8F=9F= 01=07=93=A7=E4=E6=E5=17=BC[8uZ=D1=F4=19=C5%3g=CD~o=CE=DC=D2y=F3=CB=16,\T= ^Q=F9=FE=E2=0F=96,=FD=F0=A3=AA=EAe=CBW|=BC=F2=93U=9F~=F6=F9=EA5k=D7}Q=B3= ~=C3=C6/7m=FE=AAv=CB=D6m=DBw=EC=DC=F5=F57u=FC=EE=3D{=BF=B5=EF=DB=7F=E0=E0= =A1=FA=C3G=8E=1E;=DE=F0=DD=89=93=DF=9Fj=FC=E1=F4=99=B3=E7=CE_hr\=BC=F4=E3= =E5=9F=AE\=BDv=FD=C6=CD[=B7=9B=EF=B4=DC=BDw=BF=F5g=08=ECgh=C5=16=B4l=1F;= =9D=D4=F1=C7=A1=DA/}=0BDy=E2=88=90 =C3=00=0F=18=E1 /Zs>=F0=85 = f=F8=E1=15=AAh{=F8#=00=81=08B0:=E0U=84=A0#:!=14=0A= ^C=18:#=1C]=10=81=AE=E8=86=EE=E8=81=9E=88=C4=EB=B0 =0A= =D1=E8=85=18=BC=81X=F4F=1C=E2=91=807=D1=07V$=E2-$=C1=86=B7=D1=17=FD=90=8C= =14=F4=C7=00=0CD*=06a0=86=E0=1D=A4a(=D21=0C=19=18=8E=11=18=89Q=C8D=16Fc=0C= =C6b=1C=B21=1E=130=119=98=84=C9=98=82\=E4!=1F=05x=17=85=98=8Ai(=C2t=CC@1= J0=13=B30=1B=EFa=0E=E6=A2=14=F30=1FeX=80=85X=84r=DAW=95x=1F=8B=F1=01=96`= )>=C4G=A8B5=96a9V=E0c=AC=C4'X=85O=F1=19>=C7j=AC=C1Z=AC=C3=17=A8=C1zl=C0F= |=89M=D8=8C=AFP=8B-=D8=8Am=D8=8E=1D=D8=89]=F8=1A=DF=A0=0E=FF=C2n=EC=C1^|= =0B;=F6a?=0E=E0 = =0E=A1=1E=87q=04Gq=0C=C7=D1=80=EFp=02'=F1=3DN=A1=11?=E04=CE=E0,=CE=E1<.=A0= =0E\=C4%=FC=88=CB=F8 = Wp=15=D7p=1D7p=13=B7p=1B=CD=B4=FF[p=17=F7p=1F=AD=D0z=0B=A1=DC=D9,l=E5=17= ;[=F9Z=B5E=BD&=1A=84=DF=85KB=16=FB7=D7=EC=0C=E3O9=F5=C7=90=F6=1C=9E=7F>=AB= =11K=D4=81j=1E=BF=92=0B=E1=13_=F0=D4=1A)Vz=80_=C5E=C2=11=F5=82=E8=C5=AD=E0= |=9CK9=8E3HS=A4=F3,=96:WO=99=D5SF=FB)=87=06=8A=FC,=FD6=E2 = =1B=02=95=F5g=15=C2T=D9f<(\=A4=0A= =EDd=D5=D8J=9B-=0D=7F=90=CEr=D2M=A4=9A=D5QU=06=91~=3D=AD=BA=06=EA=CA=13f= q=E9=E6R=15Oc=1D=1B=C1=FA!M=84=BAD=F8K=BD,=EEt>=14k=D9=18=D4=B1tV=C3=CE=B0= =1D=D4=8F=83=A8=E3=82=B8$=CE=87=8D=E0,=D4?;=EC=AC=12'=EC=A6N=BE=3D=C3B=AB= =ABS=D2=B3=14=A5CJ=F2=E8=C8=9E=833=B2R=92;(=0A= =0D_=E2=EC=B5=E5=10=93=CE=81=0A= =DA=F9=15=92B=AB_F=90=CD(3=DA=04=BC=87=00=83=E9n=0B=BD=B0=B4XZ=A2=A3b=CC= =8A9\1+=15<=DA*8=A8=90=94g=F7*DE=DB;u=08=10=14~"=ED=94T[=98=D1=9B = =B2(=FAxxx=F2=A2''1=0E=B2d=10=04o#=BC=BC=0D=9EF=A3=C9=D7=EC=97`illil=A4A= =90%=D0/!=81^=8B=05=16=8B=958V=BF=80=84=E8(&=06=C8=11rD\x=9C=C8=C7=F0=E1= =82=A2=B6=D6=EF=DE=BEm=CFa=B55=99=F9{=DD=F2b=FE=CC=91=F6=A0=F2=E9=D3=CA=07= i=97=F2Y=0F=F5Z=BE=16=8B=03=9D=04=F0y=B4w=BB=DB=DA=F3L=C84=18=A5=10=18=B9= =85r=B1=C8D=9E=F7=F6=D2=BC7e[=DB=ACw=90dm"B=CE=14s=98Y=89U=CC=94=A5=00=B5= =C1=AE=9EdIv=D6=97=15=AA=07=D8=E0},U=3D=E4=B2=CD=0A= =E9=A4=0E=A1J=F9=1C`=994`=14tR=9B=A6=1FKz=CF=1D|=0C+=DC=A7IV9=D7P=14=BF=D3= i2=F60=1D-=8Fm=91B&=13x1S;i2=19=9F=C9q=14=0CG=07=0E=E3i=04=96D=9B=9DC=14= =97L=CBL`=08=B2X=1B=B3=AFfS=A8=F4=062=8B=A3%=C0*=0B=A6G=A2=E9=91l=12=1E=19= L=8FX|t=94=C8X{F=EDz^=C5=CFm;=C7=C5=15r=91\=0F=87Z=ABn=D2=CEw=07=EA(=DAf= W_=03m=9Eb=A6D1=FB=90G=D1b=B5X=91=F4=E8=AA=1E=B9=F9=EF=E85p=DD=ECm=97=ED= =B4.=1D=94C2=E50=80*=EA=00mQ=9B=96Y=06Lb = ?=97N=C8q=B6v6=8E=97DE`2=C7Og=8B$q=BA=1C"=1C%U=06=D9=E9=B4=F9=C8=F3=05=18= m|)=15^=E1$Z=EB=FB=A52A=F00]7=B5=C5[=83of[=83=DD=C3x=0A= &=A1=ADM=CB5=D8t=C7t'X=8B+@f,@V=C4=92=E7=B1=CD=E5j:=DB[=DE=CC_`=85=CD=E5= =EC=00=B3=977=83=DC=E8=B1=B0=01X)=9E=D2#=94=ED=06=81eGG=B9=AF;=C23=FE=F2= =98 =BE=D6=FF = =D0=E0=BA=E8v=CB%M=1A=DD{=EC=CF8=E7=06=B5=87=A7Q=BABr=1E=FA=8D=C8=B4[T=BB= K=3Dj=9C=1B=FE=FB=D0=D3=88=FF=BFu=7F=A1?=1E=15=DA=80=EB=E0=86=E0`=A7=A8B= =89=845=84<}<=9E0Ipp=12=D1=1B=82=D6=0D=07=1C$=DBI=A7=DB=89^#=9C!=1C!=AC=D2= =F5+ [=DD2XBh = =1Ct=CB=BBt=ABt=9E=86=16=DD=AEI=F7=9B@h=A7=8F5=14=11=92u=D9=81=841=BA=AC= &=A3=D9J=D7=F9=1A=ADu=C7=8BQ=84X=C22B=1A!=8BP=E8=8E=89=D6=A6=03=C3(=0E=D3= K>=AE=10=86=10"=05=87s=03=D1=B9=84=12]7=CCM=99=EE=C3=A9=EA=FE-:=7F=0A= =A1=1B=A1/=81=D3}j=F6V=12=FA=E8=D8=AC=FB=D1=EC=90=0D=CE=8B=A8=A6S=A7=F5=93= =3D=A4=1E=8C@=99^=AB*=B7=0C=8B=F1=A8=C1=03B=93=DB>=93=A4I=AE=9A=8F#h5=98= G=A0=9E=B1=08w^,=9Dd3X=8D=D6udH=19=C8=D0(=F1=F6k=F4=05=C4=16=D7=DC=11=BD= =9E=11=EE=BE=B2=07Z=BD=F8A=AE=18j=08=19b+=12=C5=87.=D9H=BD=16=9A=EC*=CD.= =ADS=97=DD=17=94=F4=3D4{b=B4K_=B3[=A6=F7=A4=80xy/=F5ZC=9D=FC=04=BB=A8=06= >Z=AC=A2=F4?=EE=AB-=B6=8E=A3=0C=CF=7F=EE7=DB-m=D3$M=DB%=A5m.=C6=C9q!=B7=D2= =D6=CE=85$=CD=CD8=CE=85=90=B6=99=B3;>g=F0=DE=B2;=1B=1F=DB=94=84=14(=05B = =BD=A7=85=84R=04R=DF=90=90=8A=C4=03=AFH}EB<=81=F2=8ADy=C9k=F8fv=8F=ED:)=A1= HH=C0Y=CD=EE=F7=CF=FC=F3=CF=F7=FF3s=E6=1F=13=B7=FD=B9_=B1=B5=C5=C7M=1C=D1= =EF=C6=8D=B4=B01=B4=8F=EB=A2=FB=EAXes=BC=B4xY=ECI=97=D2)=D8=0A= L=9F=1A=C6=FF=83=E1v=01v^J=E3=92=F1~=B7=F7E=B9=06=BD=87=0B=BFg=8F=F5=D6<= rW=B4=E5*=8BJnQ=E9=D71J=BF=06?=AA=E7=A7=98=C0=FEuv=B2t=BD8=AEK=BA=0Ft_=B3= =1F=B0=07r+=B2=BD0=DE=B3Q>=8E=CC=8B!=F3=FA=B8=A7=8Dl=E5U=E4!=FF=EC=F9+5=E8= =DE=FF=E3=E7=19<=DE=7F=FFc=FEe=AF!=FB-"=CF=CE=E1=A9=E0=CF=DC=C2=FE=1E=C5= =FD=88L=EBrzd=FE=BF=F8<=EB]g=08=B9=F9=F9=0C=E7=90=BF_=CAp=1E=99=FAk=19.=00= =BF=97=E1"r=FA=DFe=B8=84=FA=0F2\=81=9D=BFd=B8=8A=1C=FB=EF=19=AEQ=1E=E7K=8A= =EBlY=EE=E9=0C7=80O=E9=FBU=A1=0A= =12=85=DC=99=0C=13=B3=F2=B3=19=86=17=F9=CB=19=CE=B3u=F9w2\=00=FE = =C3E=B6<=7F=3D=C3%=B6=AE=D0=9F=E1=0A= []=D8=92=E1*=FE=E3Od=B8=96+aO=A7=B8=CE=06=CB=8D=0C7=80=B7=1C=E0=AA=D3<=10= =F8=81=E5=88X=B6}=E1X=AD=19k=D4w"a=EDKf}=C9#15hMK=D5=B1vs=97=87=BC=1D=C4= =D6N=A3k=ED=8E=82$=B4=B8=EFX=87=95=08;=C2=B7=8E=05=EEd=C4=BD!kG=10=CED=B2= =DDQ=D6=1A{=AD=D5=DC=BAu=D3=A0~o=EEiX=E3"=16<=B2;Cc=07=F7=8C=EE=1A]?Od{=E0= :K=EB=06?=B6=F2=A8=88b=19=F8Vshx=E3-=15=96=8E=F7=9Fqx=C1=AA=8C-n=D9I=AC=02= =CF=9A=0C|=95=0D=A3=07YJ=05=ED=91=95=C4"=1DL=9B=10=1EW=D2=E6=83F=18=17=DC= =11=D1=A0=19.@[t=B3=810=0A= =9C=C4V=F1=90=B5W=E9=91]i=0B?=86K*=B0=C2=04=1A=DCFg!K=F4=D75Sl=10=B5=D3=90= =B55=0B=99'=C7=0D=96=E3=F6=CAa'=80=3D=8B=ED\dWkD=A8O=A0aA=C77=E3=1CFo=81= =9A=0E=DEZ=E7=184\6 M=CE<6=84=9A=1D=A8 = =C1&2v:=D0=B7=D8=1Af=B3=B5=F86=D9V<=9B=0C=93=14o=BE=C9=86=85=CCS=B3=10=86= =B7=0D=0BCl=0C7=BC=3D=F0n=17=CA=FA[Dd=BB=B1=E0=DCVo=F0=DF=D0<=0A= &=91=89J`= Ew=D1=DDt=0F-C~=B7=9CV=D0J=BA=8F]=A1Ut?=3D=C0=9E=A4=07=D9=08Y=D8=D9=7F=A4= O=D3jz=88>C=0F=D3#=F4(=AD=A1=B5=B4=8E=D6=D3 = {=9D>KC=EC-=DA@=1B=A9I=C3=F4=18}=8E>O=9Bh3ma=CF=B2=E7=D8)=DAJ=DB=E8q=FA=02= =3DAO=D2S4=82,l;=ED=A0=9D=B4=0B=FF=9D=17=E8=8B=C8=88=FED{h/=3DM=FBh?=1D=A0= =83t=88=C6=E8K4N=87i=82=8E=B0=87=E8(=1D=A3=E3=F4e=F6=0A= =FB-=BBD'=E8+t=12=99=E8=B3=F4=1C=9D"N-=B2=D9=0B=ECer=90=9F=7FH=82=BD=C1^= d=EF=B3=8B4=C9~=C1=DE=A36uH=D2Wi=8A\=E4=86>=05=14=D2i=8A(&E = =9D=A1i=EA=D2=0C=CD=D2=1C}=8D=9E=A7=AF=B3=CB=EC=E7t=96=BDK=E7=E8=1Bt=9E^= =A0o=D2=B7=E8=DB=EC7=F4"}=87^=A2=EF=B27=E9{=F4}=BA@?=A0=8B=F4C=FA=11]=A2= =97=E9=15z=95^co=D3=EB=F4=06=BDI=97=E9-z=9B~L?=A1+t=95~J=EF=D0=CF=AA=89/= =CF=E0=BC=E5n]tq=9C=0A= _I=EEV=E2=C4=EE=A8=0EW=0D=8E=AAH=C6S8=E4;U;=F0=DBQ=02=95=E2=A8=1Bvxa=BBP= <=BF=A3#=CB=BB=C2X=BA=81=9F=1F=EB=C8=E2n=EEy<=BFK=F1=C2=DE@=F1=12=CEc=C5= =9B=C5}<=0Cyi?=F7Z=0E=CF=1DHr=07=93=F2!O=DAQ=E0=E7=C6dqB+=E5=C7;A=F1=B0l= =A3=F7=04O=CAGR=9B=A5X=D74s=C7e~,=96=85=13P=AC=EA=C4B = =11=11}8=E1C=E1;=D2N\=1E=15=B9a=D5=D2=B6l0q=84=ABxYd=DCB=D4=B4=0D7=B4=17= $=B8=15P=D5,N=19fn=CA=CCO=CAAJ=ABh=88=E7#p2=0C=F2=0A= =9C=92=8CS=E0=896=9C2=9F\W=E6Q]=98=85z9=96=9E=D4L=E2N=10=A9:R=07[!=FBh=19= n=92=C7=F58p=A5=13=BA=DC=16Q)=FD4=10T%=FD=84k=C5J<%\=A1=02=BF=E1=8AI=D5=13= =FALR=DA=93Jq=A8=FB=15A2=8EK=E2t=C2=DDf=C5i=C1=18=F2=A0j=80=C9T=D2uDU=FA= J=B4#4=CE=A3=E1B=9CxM=FD=1A=AEdYU=B3=07=86KX=08=81=DFL?=C35=DD'=8A=054=16= =E0p>=88=9A(=C3=05dbM=FD=1AN=1D=F2=84=13=9F=8E=CAzl'P=95l=E6=9A=C50=92=9E= =A8G=89+0=15|F8=05=DBMZeGr/=F0=9Db=07i=98*=C2=1D=B0=E5Q=14L=B7=902=A6H=BB= _6( k=E6kb=906:=C1=B4_w=82=A4=E5=0A= 3B=03>=84=887=88=EBe|:=91g=B8+|[=D4=8D=BA=8E=88=E86=0C=C6=B4=C8Y=D1=ED=B7= !I=DE=C62RI=E4cr=90=19W=F6b=BE=A6 = W=C63P=9F=16=12=AE=AB=88=C7q=BF-#=DB=15^=E2*=19=BA3=B5T=0C=DD$=AE=08/T3=B1= P=8D=F9H=81I=D1=04=B2_SC]=A2_B=F5c=D9=BA=A2=DB=13=AB=C8d=E3=A4=A5=BB=F6=F4= =8C=D0=D3=D2B=19=B3=EEa=CB=D5=A0=9B=C1"=F7=DB=AE=A8`J=1D = =B1/=12m=BDsa1=92=93=FDv=EF=16=93=8A =EF = E=8F=A6=8CX=C6=FC=E8=AD\s=836=92v=17sX=CD`=10=A5!=C2R=D2=F30/=E8=A9=A8=F5= =84$=EC=EBA3=C6=BC=96=9E=93=AA=E1=A5=F5=8B=8E=9C=9Ct=0A= =C8=C3EQz=BC-=EB!=AE(=BEnR=E1=02=16=DD=05=DCR}-l=97)=A1R=AD=C5=92=E8.=96= Z=AA=AE%=91=EA5=E6=B1'=9D=85=86=96*=1B,=BA=85=D8=0D=E0=80=A6f(=D7z=DBA=85= =F3Pt=E7a=0Bs=A19=19]=98_=10=B0=86=16=84=96=EA=CF(e=8A=1F=11=B1=C2=16=8B= =B0i=C8d=AA}=0B=028/jj=A9=82=BEp=FC=FA=DC=C4=AA=07=CF=3D=FE>]=1Dyb=A0=CA= =BA4=82bu7vG=BA=F9=B0{=AE=9Bcsw=CC=85s=17=E7=AE=CC]9[b=B3#scscg=C3=D9=0F= =E7J=FB&=FE<`=A3=9C=99X5p=FC=98=B6=B0r=80O\=1D8=8Cr=04=F5=87=F0}=E6=E0=3D= =03=A7=9E=FF=C7=18k=81=D2=0A= =9B=F8=C3Cv0&l=E275=03R=0E=02=FC=1E!"=F2=F3=03=D6=07=EC=0F8=1F=C0=E2i=0E= T=B2=8D=BF=CE=FC=01=BF9H=99=83(=BF@=88B=08=93B=88A=88CH@HBHA=08[=08X=A7=04= ?C=89=00=B0=F7eP=E2P=12P=C2=E6=E4'!=EFg=F6=80=DF=DE=0A= =E8=86M=FC6 = j=1B=BF-P=C4=DA=0El=9F=85=19=98=B24=03=CB=97=9A=82)=133=B02+=A02;=10=D3!= =87?#KB>=3DSB=9E=83]A^=86MB^=8EE@=9E=87QR=9E=95IO=9E=89AM=9E=9BKE=9E=8B=91= O=9E=99=D1@=9E=8DEU=9E=13=D8=1Cq=D0g=ACgd=E2=07V=FC=F6=C0=0A= ;=1F=C8=D9=CF=B8=81=85=83=9F=91=9FE=9E=C5=9E=D1=9E=C5=9F1=81=A5=9Fe=3D=CB= ~=C6=FB=8C=FF=19y=AC=8D=D5=A4=B8=BD=19w0=FEo=ED=F5=0E=8E=D8=C1=C8 = =1B=E9=1D=18=B1=D1=AC=C1=05B=1F=00=D3;=19=CC=18=19=E0,=078=0B=A8=AA=B8=A4= 4=AE8N=1B=07(f=D0..=01=D1=DA0=02"\=02=D3=06=0041J=07=0Dendstream=0Dendob= j=0D70 0 obj=0D4686 =0Dendobj=0D72 0 obj=0D8183 =0Dendobj=0D74 0 = obj=0D<< /Filter /FlateDecode /Length 75 0 R /Length1 77 0 R >> = =0Dstream H=89=ECWkPTG=16=FE=FA=DE=BE3=C3=88:(=08=BE=E2=8C=A8=88=A2=80d|D=08=88b=A2= <=C4=F11@=14Ey=AA=03=C8CEM=00=9FQTD=13=A3D=D1=A0=F1=BD=84=E0#=9A=A8ID=8D= F'j=D0=F8Db=ACM=D6-=DD=B2=B6*=B5=1BKg=F6=DC=99=D6=18=ABvk=FF=ED=9F=BD=3D= =1F=FD=F5=E9s=FA=9C=EEs=BBo=03=06=A05=CA = #q=CC=B8=E0=01=93=BB=D9z=90=E46!a=BA--=FF=D8=D8=7F=0C=02X=1F@z2}N=91qlDr= =03=C0{Q=FF_3=F3=B3l=DB=1F=0F8=01(=06j=87d=CD*=C9=F4]f=19N=ED(=B29=99=9D= =91=96~=B6=DF=C1=EE=80V=A2=FE=81=D9$=F0j=CF=B7R{0=B5{d=DB=8A=E6=E5=94=EC= =AE=A3=F6$@^5+ozZ=F2=99=94=DE=80=FE/d=7F=D6=966/_=0A= =92&=03=9E=F5=A4o=CCM=B3e=D4XW=DD=A4=F6%@=97=95=9FWX=14S:=F8=14=E0C1j=17= =E6=17d=E4=EF=AB=AA=AA=05=BA=1C = =FBF=8A=13l-=14=E8=94j%=8C$=3D=DD=B5=BC=0D=99R;O(=1A=0E=99=E6=CF=D45=F8=C3= c=B5=8C2"=EA=91=F1=91Scr=98=00M=13k!=B1=04=B7=A67=AD=16=3D=AC=13A=03<3g=B2= K=E7=8F=0Fu=CA\=D1hu=1E=FAV=9E=AD=DB=B45x=B5k=EF=ED=D3=C1=D7=AFc=A7=CE]=BA= =BE=D2=CDh=EA=EE=DF=A3g=AF=80=DE=81}=FA=06=F5=EB=1F=1C=12: = =ECU=F3=C0A=83=87=BC64<=E2=F5=C8=A8a=D1=C3G=C4=8C|=E3=CDQ=A3c=E3=E2=13=C6= $=8E=B5=8C=1B?a=A25)9=E5=ADI=93S=A7LM=C3=F4=F4=8C=CC=AC=EC=9C=193g=D9r=F3= =F2g=17=14=16=15=CF=99;=AFd=FE=82=85o=BFSZV=BEh=F1=92=A5=CB=96=BF=BBbe=C5= =AA=D5k*=D7V=AD[=FF=DE=FB=1B>=D8=B8=A9=FA=C3=CD[j=B6n=FB=A8v=FB=8E=8Fw=EE= =DA=BDg=EF=BE=FDr=DD'=F5=9F6=1C8x=E8=F0gG=8E~=FE=C5=B1=E3'=BE=FC=EA=EB=93= =8D=A7N=9F=F9=E6=EC=B9o=CF_=B0=7Fw=F1=D2=E5=EF=9B=AE\=FD=E1=DA=F5=1B7o=DD= n=BE=D3=F2=E3=DD=9F=C0=D9OP=17=9B=AB=B3}=E4t:=E9=AFQ=FDKmN=B5L=3D=0A= 4=D0B=07=0F=E8=D1=0A= =9E=F4=CE=B5A[=18=E0=85vhO+=EA=83=0E=F0=85=1F:=A2=13:=A3=0B=BA=E2=15t=83= =11&t=87?z=A0'z!=00=BD=11=88>=E8=8B =F4C=7F=04#=04=A1=18=800=BC=0A= 3=06b=10=06c=08^=C3P=84#=02=AF#=12Q=18=86h=0C=C7=08=C4`$=DE=C0=9B=18=85=D1= =88E=1C=E2=91=801H=C4XX0=0E=E31=01=13aE=12=92=91=82=B70 = =93=91=8A)=98=8A4L=C3t=A4#=03=99=C8B6r0=0331=0B6=E4"=0F=F9=98=8D=02=14=A2= =08=C5=98=83=B9=98=87=12=CC=C7=02,=C4=DBx=07=A5=B4=AF=CA=B1=08=8B=B1=04K= =B1=0C=CB=F1.V`%*=B0=0A= =AB=B1=06=95X=8B*=AC=C3z=BC=87=F7=B1=01=1F`#6=A1=1A=1Fb3=B6=A0=06[=B1=0D= =1F=A1=16=DB=B1=03=1Fc'va7=F6`/=F6a?=FE=84:|=82z|=8A=06=1C=C0A=1C=C2a|=86= #8=8A=CF=F1=05=8E=E18N=E0K|=85=AFq=12=8D8=85=D38=83op=16=E7=F0-=CE=E3=02= =EC=F8=0E=17q = =97=F1=3D=9Ap=05W=F1=03=AE=E1:n=E0&n=D1=FEo=C6=1D=B4=E0G=DC=85=9A[=F0R=E7= -=BEC^=EC=BC+=D7:=9A=1DM=8A=8E?=E6=B7x=12=BB/=EDv=FA=CB=8DN=F1=E8=12=9E@= =D9=F9[=B5R=EB=BC=EF=C8=94=97K=99=0A= =1CKY=B9R=A7ItVK%=9A=C3=CC=ACx=F2=9D=8E=16M=13=ED=AD:V-=0D=95=CCR4=F9=DC= B=99=1BI=19=1AM=D9I=A0=CCX(+=13)#)=B4*?=C3=C1Z1=7F=1E=A4=E5=FA=F9<=9B=A4= q=88aZ=E7=F3=87l#(=D7=CF=EDI=92HY=D9H3 = =DB=E7Z=EB=D9=1A=C4=B2=C54=F3p=CAH%=F9=8Dg=7Fv:=D9=02=98=E5=15H`=E5=0D=86= nm=83=FC=8D+W=C6$&=99L=9DcF$=F7=0B=8A=B5$=C5=8C=E8l2=11}=A1=A7>*=8D:i=BF= =97=D1=0E/=D3=98=E8-=D7=A2c=94=9EC=CB=14=D9=83Cg=B8=D3L?=047=077=87=86=84= y=99=BCz=9A=BCLe2=9E=96Ip@c=FA=AD=A5L1=A9{=C4=8E@:=89&=D1=BE=E8=15=D5Nf=DC= =AA=D3kZC/=0D=D2*=B2=DC=DA=D3=AB=DD=90=E0=0B=E1O=C3o#2=FC=02U=A1!=CC=E4=E5= =EFe2=9B=BChT=0EGE=9Cc=0D+=88c=C5=AC=C6Q=C1=8AcY=A1c=B5k\VC[=F2>E=D6=E6=10= =B3J=AD=C1=82=83=11=F9T=B57=93=DD=13=BB=1C=C6jbU=CD=0A= =E7=03=8A`#=ED=D2=94(=FDD=9E=C97=F0=9D=9C=F3#=CEGQ=FE=DC=CA=B8=ACX=D5=BD= le=B2U=92hz=12mi&=13#=A9=C4=19:=06=87=9F=BABq=D2=CF=8F=05=DB=9B}=C3=B5=DC= =F0P1<=D4=1A=F8C=9D=E1!=1B<94Da=CC=871=F2[!=17?=0D=90nx=B3k=ECj=B2c=89=A3= \=8D=95"=88vE01=AA=7F=B2=9C#o=92=F7=C8\6=C8VpY=B2=D2=19=AD2=AB=A2=90?E=A6= =A3=96=0EcN=EE=15:t=FF+=EF=8C=99=E9=C7=A3=9FTH7=9E=06=B0FG`=12+e=0B=93]=C7= 7=E1|C=F7=C2)m=C3=7F=85=9F=CEup=D7i=F3"=D4=BA=FE=D8=E3]=CE=16=87M=9F=AAi= =A2=A6=878=E1=99=FAUP=BF=0D=1E=D5=CE=96'=D0=A7=E2=E5=AF=C8=3D=0E=F5=DD=A0= OBg7=B8=9D=DD=E5vL%=9C#d=12=CA = =95=84=D9=DC.=0D=A1=FE=00=E2=7F#=D8=89=17=8A=FA*=81l=F1=0B=E12=A1=86=B0=9B= p=90p=9E=F0w=C2aB=0B=A1I=E8=AB=B6=15n{=D7=18=BET=EF%=18=84_=F2=850=C1U=D8= =08 B?=8E=90"t=BD = =DDD;A=D4=B5B?=95`&=9Cx=A1=AF=98POsQ=E5=16=F2;=E2w=1F=CC=DB-=03=C9=9Cj=AC= =8B=DD=F3F>=A1/a=03=E9L=A2:=8F=E0=EB=D6s=8Do=13z=FE=C2O=B4[=D75=DF=8D"=DE= 81=BF=F5=EE=F5e4=B6=14)=C6=DB=AB=83=F3:{@9=18=8F=05=D4^JXA}3=08a=1E=D5=AC= =D4=A3=DA5=EF@j=F7=D5Ls=AD=D7=DBn=E0=06=E1=81=C8=CBAu\=D2=B5=B0j5=EB=B0h= ,=B0=A8=B5=DA=A7=D6=CF=A0T=B9d=E7=C5z=AA=B6=D7=C8V=CD=9FY=FA=A7+=065w=16= =A5=12=11t}=B0=889n&$=12=AA=D4q=95F=F7=B8T=8F=15=BE=C3\=E3=ADv=D9=DBE=1E= =E6=BA=D7=86e=FE=9Ek=17=F6j=7FA3=ADA=B4=1A=AB=A2q=AD=D9j)=90r=D9=80=9E=C4= i=EE=18 = =90=C2w=90=AF=1D.=7Fv=F5=DD=17c=BF=8C#"=7F=10=F0V<]=ED=AE=EEw=CE=E5Wm[^=8A= =DF"=E6=AB=AE=E3T~=0E=D9"=7F=17=F9e=B5O=1A=FA=02=CC/ = =DA=9D+=B5v=F1=00=97=FD=11=1A?=15)=9ATe=88=0A= =927=BBm=D5Z=DD=0BR=9C=D8=13K=9F=8D=A1=A9=A3=9B=04=E8&=F1=EFK<=95i=FF=B1= l=A7r=F7=FF=E5=7F^=D4S=F6=1E=DD=E6=14=BA7JT=0Ct=C73=D2ql=A0=FB>s=F5=FA=B1= ^=CF=CF=E2Exv=3Dg=A4=B9Hp=89=BE=D4=EB=04=97I=BEIpN|=8F=E0=0A= =DDQ=8F=0A= =AE!=F9i=C1utc=BD*=B8=07=DDO~=16\=CFd=16.x+t=90"=05=F7$>A=FD=7F=81{P=10\= =9A)8=83Q=CE=13\B=1B=B9Rp=99=E4[=04=E7=C4=8F=0B=AE=C0On=16\C=F2_=05=D7=A1= ;=F7=11=DC=03=FB=F9 = =C1=F5=92=86=DF=13=BC=15=824=F7=05=F7D=90=D6'>=AD(;4>/7=CF=98=9EQ=98=93=95= =9B=91n=9CVb=1C=96=9B^=90a=8C-=9E=9F=9B=93V=9013=E8_=D5W=E9s=14=C7=15=EF= =B7;{KZ=1F`=E3=03kbc=07=82"=B1=1B,=0C6X=E2=08=C82=96,q=19_=F4=CE=B4v=DB=9A= =99=1E=BA{=D0=8A=1C=C8q=1C=E7"8>b|$=90=AB=92=AA|=80=E4C=0A= W=A5=CA=FC = =F9=98=CA=97=B8*=7F@=F8=0F=C8=EB=9E=DD=95,=A8R=F2!=1F=B2[3=EF=BD=EE=D7=EF= =FD^=1F=D3=EF=B9=0B\=B7=DC=834=A01m=0A= =E5=EE=B7=BA=EEA)=92=D8=A5=91=EF=CEj=16=B7X=E4=1E=17=C1=9C=A4=E1=B0=BBO=C4= =8B=927[=DA=DD=ECmqk;w=8E=0E=99=F7=8E=AE=86;=C3=14=A3=D2k=0D=8FO=1D=1A?0= =BE=B5=07d=865=93=80=CA=D5=CD=AB=E5cL*."=B76\=DF=B6=BA=EF=16'=FF=9B(=97=AD= r=E5R=D7K=94=16=A1;'"=DDqc=9C=AC=86=82=FD=D2M=14K=9D=19=13,=A4=9A{t=C8=0A= 3=8C=FAL=0EYw=02=FB=E4=AD=06b)=FC=C4=D3j=D8=9D=D0=C6s=C0=3D=16)=0CI=0B7N= P=83*=9C=1BW=CC=B98=1E=1Du=F5S=A3!]t#=A1=DD=06s%=F3=B9=D2=927=12=8D=A3=0D= =1E=91h3=C8e=EDX2=A5=DC=98=C9=90+;=CFh=EE=96=A5=EB/=B7=B4=8Ew=8D=8C,,,=0C= /t=96=DE=13=E1=ED[=F1=C3J1=11oa=A1v=18=0B&S4=B9=B6=9CRXJ5Qf(=B9=F8=81]=C4= =F78=CA>=16T=0C=F9I,=A9=CE=A2=CCq=BCi=99=C7=A2=CF=C5=12=8B[k.=16s=14=CB0= =8A%=18E;=02=ED=B9X=E6-=DB5=1A=12=DB=13=D4pQ'=B2~fq4=C3=96=16=BE=8D=CEq=D4= =08=B0=B4=93=A8=11bI=E9b=B1h=CA=BAEl=E1=B6=E0=D3=D8=B6=19K=C0-HkX^=EE=C4= Bs=A8=C7=EF=B8=C5=86=8B=85=8BA=C1,n=0F-=0Cc\SX=98=8C=E3gk=1C=8B=D7[g=C4=8C= h"=D2=C0=8EYK{=AD=FEchM=DAy=106=C6=1A"=A8=93mk=8E[;=92=FF=A7=B5=BC=1DVn-= S|<=B4=A5=B0_=D88=E7=AC=86^=15M7=92=B5f%=1D/=91&=B6}ed]=14=0CG=1A=8E=E3=08= j=A3=EF=F6=CCXK=BE]=B3=A1=15=D1=89=CE8=F9=1F!=88=ED=FC=F8=88=C0=C3q=CA=EE= =E4 = =1BQ=1As`=3D=9ByR=9DU=D2vFb=1C=91=DA=A0=B6GZm=81=BE=DC=8E=FF4=A2=D5=F6W"= 5=91=99y=8A,f3kf=84=B4~=B8=9Des=96=1A8Vw|w=E7G=D8=B6=AE'=17=9F=B6=F5d=BC= *=EB=D5 =0A= =AD=95=E5=FD=9C=A2[=FB=D4=F5=93=B2=3D=BF=1A=ED=EC"#=F8_=B0=FFa|=BEx=EA=3D= =BB=0F=FE=1B=DD=B4=08$7_=C2=13u=BB=DF?m=CA=91=C1K=D7=81=0C=C9c=BDZ$%p=F0= *=EE=83=1C=19 U=F2/L$=EE"w=93ud=3D=B9=87=DCK6=90=FB=C8=FD=E4=01=F2 = =D9H=1E=C2=84=D5=C5=B4=F4a=C8C=01=8AP"=BF=872T=A0=0F=FAa=00=AA=98=EC=DC = w=C1=DD=B0=0E=D6=C3=3Dp/l=80=FB=E0~x=80\=82=07a#=FC=00~H>=84=1F=C1=8F=E1= <=FC=04.=C0=DB=F0Sx=07=DE=85=F7=E0}=F8=19=F9=04>=80=8B=F0!|=04=1F=C3'=F0= s=F8=05\=82=CB=F0K=F8=15=FC=BA=94D=FC=0C=DE=AB4=A8=B06^=9B,=D2=9C=06E=95= x-=DD=A2=BA=8Fb=93=E4j=1E/=F3V=C9=13QS&=A8=92=1B=0F=E2=16u=F62M=B3=FBZ=BC= p V<=10Qv=BA=C5s=07i=18=D2=EC=01M=9D = =A1i=1E=EF]Mk=B9I=1A=C74=FF,=0D=1B>=CD=1CN2=CF%=85=A9=90{RD=99i=9E;b=94=B2= 3-=91=9B=E5M=1C}=84&=85=A3=A9=CD=BC2-=B5=CC = =9E=9DV=DC9=89=8A%=93@0L8X?=DE=E41=8B|=EE=99=DC*G-=AA=86=B1=E5!=12=9F=05= =9A=16X=07[=8C-M=8B=0D=FB=1D=8E=D8=1Cl=AA=E5=E6-=B2 = E=16%=05=91=C2=CAY=E0Y=89=98,=82=ACFLI=07=93=08Y=13=83=B2$=D3=E6Ylv=CE=A2= zA=F1=90=1B$=AA%=A4=AE`=8A=E0i=CC2=1A=16=1B=A7=AA=A2D=C0=FD8=A0=1E=93=F9= =94=F4=E1=A4j=1E%=D4(=16=D5<=0B=98=16Q_=C0=E6tW=E8=B7=19gW=CA=AB=D8=8C=CB= !H=A5=F2=ECtB=83Z=D1o=A01=CCwJ=02=17S=F3=C0g%=1Ei=D6=94=D8=D9=E3=EA=8EJ=C2= =9Ay=D5=8B=9D=EC=A9=D6e=EAy=DC=08"=AA=A5=A4^6c=A4b=A8=B1=CC=D6=B3B=D6=F0= =A9;=98q=D5=CC=AB=9E=06=142_=9D=96=05=E3=DB=17=BA=D8Y=B9Z.=96=9D=F034`=91=C7*V=DD=CC=08k=F7Y=1E=97=85=9Fe=ED=01=0F%N=9B=B8=8Dt"#\=1C= =CC=80=8B=13=B8^=F3(=17g:Le=81q=0C]K=AA=D4=80=C7=A5=17=B00 = 4=8F=83=C5r*=C6A=A2=8A,=8C=F5=A2b=BA=AF7S=88$g'r=C0@=C3=B6=C4=BC=98=1E=C0= m=1B=B0vW,a=C6=AA=92=86=19=DA=D5=B3BW=CB=08=05\=F5=10=8F\=19u;l=8EF=CD=80= =15qI}=8Eb=BFdMsr=D1=A2=E4s=03^=B7DIE=04=EFc*.=E7=ADX=C0=F51G=B9=1C=88&&= =E7=01=AEa=A9=C3=0A= =99N=11n%=B3=0E=3D=C1,E=B9+$q=7F=97=B5>zZfMJ=16=97=D1=CF=F9|n=CEw0=DFf9=1E= =D2&=AF=C4X=8AD=A6K=C7=CBg7=98C=A2;=A2i=FC=AAEK=D1%}=3D=FAk=94=9B=D6=97=A3=AB= =D1=8D=C8=99=3Dr=0D.=FF=A9=FAbJN"9=F5=E7=EA=CB{=EF=1C<=96=B6<3a=C9Q#=8D=AD= =AF=8E=B1i=F66=BB=D1t^=1B=FD=BCJ=EAw=D43=8Fo=B7=0A= =DBG-=99H=C9=A1=94=1C=DFa=C9=8E=D4=D4=11l<5V=AD=CE=8D^=1E=9CD=0F{=FDu=83= ~=AA=F8TJv=A6=E4=89=94<=B9=CB=92=3D=BB-=D9=95=92=DD=A6o=ECd5C6=0D=E6=F09= =97=87=BC=B3i=B0R=1E=18,=C3=C0`=16=B6=0D=8E=C09=C8T=F1*=9F=C2k=F2=94s=0E= >=83=AB=CEM(U=A1=EA=0C:#x{?=EDLa=C7=05=E7=8A=F3=19=FC=03nB=7F=F51=D8p1=A2= =07=1E=BD=02=D7=E0=E6=9B=E7'gO\]=DA=F8=C2=E4=F3'=FE8=BA=B4?=A5=D7-=FD=94= =8C=02=E9qc=3D=AE=A3u=1A[=94=D2=C9V=FC=D9=D7=EA=9F=DA=AAH=97=D5=89N=F0=F5= o=CA1s=EB=0Dendstream=0Dendobj=0D75 0 obj=0D4335 =0Dendobj=0D77 0 = obj=0D7798 =0Dendobj=0D79 0 obj=0D<< /Filter /FlateDecode /Length 80 0 = R /Length1 82 0 R >> =0Dstream H=89=ECViTTG=1A=BD=F5^=BF=D7=DDB=D3=D0,-=B6 = =0D=8A=1B=8A=80=E0=02=06$=02=82=88=D0n(=DD=0A= Q=C1=A8=EC=B8"=02"=1A5=EEK=94=98=C4=B8k=8CQ=83=BB1=1A=8D=8A=84=184&. = =C1=99=8C=B38=1EO=0E=A3b=B0{=BE=D7=0D&x=E6=CC=9C3=BF=E6=C7=BC=EA=DB=B7=BE= =AA=AF=EA=D5=AD=E5}=05=06@=85=12=F0H=1C5=DA?p=C1/Y=1A*=A9#$L=C9L=CB9=9B=F4= l=00=C0z=01\=CB=949=05^iK=0Ew=05d=BET=A6I=CF=C9=C8=EC=9Bt=8D=07=84=81=E4= =EF=911k~=FA=DD=C8=F9Md=8F%=FF=90=E9=D3=D2=A6V/=7Fc4 = _I=F5!=D3=A9@=BF=FE=CF=E4+=BF@v=D7=E9=99=05=F3Nf=FC=A9=86=EC=9F=01~=E1=AC= =EC)i=D9=89=D9y=80]=1A=F5=7F53m^=0E{=CA=AAh=80=3D=C9=DF++-s=DA=A8=9B=B5c= =C8=8E=05=14=DE9=D9=F9=05=C1=BE=F3=A2=01=D7=99=E4=FF('oZN=97=D8=13=D4=B7= v<=D9I4N=B0=B5=10=A0=10*=84 = *=E9fc~;=D29R)P=3D=0D=FE_<=E3=0D=B1^=88x=E2=F5=C4"=EA=CDz=C0=F9=82=D7i*=E6= =C0=AC=D5.=B6f=AC=13ADk!1o=F5i=FFP%/=13D=B9B=D9=C1=CE^=E5=A0vt=D28=BB=B8= =BAi;=BAw=D2u=F6=F0=EC=E2=A5=F7=F6=E9=DA=CD=B7{=8F=9E=BDz=FB=F5=E9=EB=DF= / = 0=A8=7Fp=C8=80=81=83=06=87=86=0Dy#=B1X,=F4=EF%=FD= =93-#=E6=A9F=80=089=14P=A2=03=EC`O{=CE=01j8=C2 = =1A8=D3=8C=BA=C2=0DZt=84;:A=87=CE=F0=80'=BA=C0=0Bzx=C3=07]=D1=0D=BE=E8=8E= =1E=E8=89^=E8=0D?=F4A_=F8=A3=1F=02=10=88 = =F4G0B0=00=031=08=83=11=8A0=0C=C1=1B=08G=04=86"=12ob=18=A2=10=8D=18=0CG,= =E20=02=F1=18=89=04=8CB"=92`=C0h=8C=C1X=8C=C3x$c=02&"=05F=980 = =93=91=8A4=BC=85)=98=8AiHG=06=A6=E3m=CC=C0L=CCB&=B2=90=8D=1C=E4"=0F=F9(=C0= l=CC=C1\=CC=C3|,@!=16=A2=08=8BPL=E7=AA=14=8BQ=86%(=C7R,=C3;X=8E=15X=89w=B1= =0A= =AB=B1=06k=B1=0E=EB=B1=01=1B=B1 = =9B=F1=1E=B6`+*=F0>=B6=E1=03|=88=8F=B0=1D=1Fc=07vb=17vc=0F=F6b=1F=F6=E3=00= >=C1A|=8AC=F8=0C=87q=04G=F19*q=0C=C7q=02'q=0A= =A7q=06g=F1=05=CE=E1K=9C=C7=05|=85=8B=B8=84=AFq=19Wp=15U=B8=86j|=83=1A|=8B= =EB=F8=0E=B5=B8=81=9B=F8=1E=B7=F0=03~=C4m=DC=C1]=DC=A3=F3_=8F=FBh=C0Oh=84= =B4=B6=90=15[=EE=C9v=F1e=96F~=87=B9=DE|KP=C8=1E=CB=AEs=CFE=9D=EA=A9=F3ps= =A5s=11=97=8E=04=01=E6r=F1N=CBA=C5=F2=17-=1D=F4=CDS=05=C8=BB=B5~#=0C=C9Q=C3= tz=FD=84>=F4=11(=A1c_"=EAi=EB=CB=A1=8B=B0=97A=CE=04~=9CR=86q=0A= Gs=3D=FD=E0_=EF_=1F=D0/=C8I=EF=D4M=EF=A4/=E1=F1=B2=84=A3Q=89=FA=E6=86=12= =81=BE9=8C=95=99=CB=F9\=01t=10=B6E=CC=EF!=F4Pq=0E=9C=83=CA=93=F3T=F9q~=AA= 0.T=15=CB=C5=AA=AA=C5=CB=9A=CB=9Do=8B=B5N=B5=1E=7F=15=1B=9D=1A=3DZ=B8fe=8B= =CA=ED=B9=C8=06=AAbU=19=1Es=3D=CA=3D=F6x=88Zy=8AbI=9Cg=16\=1D=B3=D4.[=B5= =07=B4=A7=B5U=DA;=DA=BFi=9B=B5r=ADN=EEx=8C=8E=AB=CB^=DD = =1D=A7K=91=DB)=F9=9E<=C7'uq4=E7=86]zi=0A= =AB=CB=0Ds=D2=0C2!=FCe=F8=DF=E7=84w|9(=A0=1F=CB5=99X=1E=F3=F6=0Dv=EA=EF=DB= =3D8$=848d=80=AB=B7=E8=EA=E4"=CA=DD=DC=88=DC=82=F8=DC=9D[=0D=A9=E3=8C=1B= =F7e=97&ED=8EZ:=C3\=BEx!+=0DOR=8CT=C6=0Cg=A5E=8B=F8=E81=93=CC=85=91=A9=9E= =DE=93=86=9A=0BMcI=BB=91=B4=C7Y=B5o=8A(=B8=C8n=B0=07=EC=91=F2=91=EB3&=0A= =8CWw=EC=E8=CB|=D5=03=84=10=F7h!=C6=FD=94=F2=94=AB=1D'p=EE=1AA=E3=EE#=F8= =B8=07=0A= =81=EEQB=94=FB=1Ev=9C=D5)=EB\=9BX=93=D0=E4=FE=CCC=B3_}J]=A5=BE=A3~=A1=16= =D4H=F1\=12=A7=C8=92=BB=BAei]=D4v=D0=CA5v)=9E:=91=F7=95d{=D9d=87=D5I=A2=DB= 4=93h=13=A96=91n&ZE=FA=B4=89=F6m=9D=83@=ABh-=1F=17=3Dk=D7=96=D2=EC=A8=F8= =B8=98=82=85K=8F=A7%=B1m=DC=C8Xs=F1=B2=B9=D1=BCq4+=1F=92=D2=B3=EB[=83Y=F9= =D8 = =B2=E8=FC=95=E6=E2=98x=C1=1AG=08g=A6=F4=10'=AB=C3=FE=01w=855=82|=FA=F4=EC= n=89=0Fo=8E=DDo.j=88V=1B=9C=A5=F0=A9l=0D5L=0A= O=0D=14=04=1D4=E6=A2=172=B5=E1U}=957=93=A6= u=D4=E7$=18m=FB=C4:=8E=F4V=0C=97=98=02=04=DAl=C9_=A8=86=81=C6=FF;=DD=ADh= =EDG=B9=0B=06=07=0DiJ=80=C1=FE=101i=94=C6=D9=C6=C2c=04(O"=A0=8D=A9=ED=D3= V=DC=A11=84=DA=80GB=03=9D=9DZ=BA=B2=9E=B7=EE=FDJ<$=FCL=A1)_=02=85=16=D0=05= =E6=BFI=DBm=89=E9=FE=9F=FE=17=92=F5=AB=FA=07=BA=18R(=A7=FF,=BEJ=DB=9B=A8=A2=F0=9B=85>m=A9=04=95=A5a=CBeoil).HUJ=05=04=04,=B6= =EC4=C04=996#I&=CELl=0B=A8(`"=8A=E2=BEKE=DCp=AB=B8=D5}=DFp=C3}W=D4=0F~=D4= =7F=80=EF=DC=99@)}=9E=AA_L=9E=E4=9E{=CFz=CF=BDg=E6=BC=9C=DB=DCR=CF=A4#=CF= =DEm=C8=E3=02=0F=9B=DCm.=ED=A5=EE=CD.=ED=E3=FA].=ED'=FD=A8K=0F`s=FC=92K=17= p=FD=3D=97.$=FD=8DK=17=D1=F7=1F.]=EC=F1x&=BB=F4@=0C=F3=96=B9t = =E9=996P=F1=171=88B=EFR=97=F6@=F8=9A\=DA=8BA=BE=CD.=ED=E3=FA=0E=97=F6=93= =DE=E7=D2=03P=EA;=E0=D2=05\=FF=CD=A5=0B!=FC^=97.=C2=16=FF8=97.=F6z=FD{\z= = =C2=05{]=BA=84=F4=C1=C5=8A=15=9F=B6XO=E9"=A6=9AZkJ=8D=89=E6=0EQ=97=8A=19= =AA=10=0B3=1BS=9Ab=A8=1B=C2=A2M=B3=E2b=9E=92P=D2J=ABn=8A9RX=CC3=F4LZ(=A9= =98h=B4=D4t\M=89=15z=A2=C5P=92=95b=B6=9E=EE0=B4=D6=B8%=CA=A2=E5=A2z=C6=8C= 3=C2=F6=FFt=91=17=11=0D=AA=A9*F4^yn=FD=FC=BA=B9u=15GBiP[3 = =C5=E8=BD=DC{=BE\5LMO=89=EA=CAiS{=F3z=FBp6=F4=DF=F7s4M=9A)=14=11=CD=98=96= =9E=14-z=CAr=F3fg=AD=B7S=F2=0D=911U=C7=99mBM*=96=16U=C2r=D2=A0*1=D5=08Kw= :y=C6=F1=06=D2=86=1E=CBD-=B3R,=B0l=CF = -=AA=A6L=9E=91=A5=8Bt=86=12=8A=C9$=08=BDEP=9F=8E=F2=F2=8E=D1=A4=D2!R=BA%= =9AUa=A81=CD=B4=0C=AD9cQ=DB=8EG=CFX=B6=92P=DB=D3=86j=9A"=AD=1AI=CD=94 = =A5=B9=E3=CE(nY=E9=9A=AA=AA=B6=B6=B6=CA6=F7=88=A3z=B2=EFUB9=85=00,=CEF=7F= 1!=99=0D=CB=84=04l&=C1Z+=E7*g=820=AE=83=FFu=9C=C7=08=D9T=D2=82=CD|=86p-E= 9E=AEm = =B0=14=84q=9A=B4'=08=18=15=82=05=850O=A1%=9D=16=05=A1=E4Q=CB=B6=84=C1=F5= =0C%=04eR=D2S#=B5U=AE=C4=F9o=CB=AC=A0D=82=F0=D1=A0D=92=B0U=10=90=DA=D0=B1= =83+=9A=04=95=16=D7=CA=083=CB9V=13=C2=CE = =98=0D=1F=A1=A7=CBX=8F=B5"=08_=EC8T=19y=946* t=EB = p=EB=08u=EBP=D1GVl=8DV=C6=9A=90:=FDI=F7=C7_Nk=86=CC=84.wY=CD=08=A6aj=BFz= =FD=ED=A3=E7 = =FD=1F=E7=D3=D7m=D2=A4e=85=BF(m=99=E4=EB2=F6=16)a=F5=BAo=F9=BB=D6=DFN=1D= }=83cF=AE=F7=DCY>=0A= =95=9A6=A5QC=91=BB=CFs=1A=A4=A5=98<=85p=8F=DD=E9=AE=9E=F1=8F"H=CB=FC=C4=18= A=94z=A6=BC=9D=0B=E4=8E=9C=3D'=A4g;O=A6[G=96=CCH=9A=1A=8E=0DEr=0C)=AD=D3= =97p=FD;;=EAm=BFg=A4=F6=CE=EC<=A5d=CCv=D6l=0DC=FA=D1d=96=ED=FAh=A6=AE=E5= =FA=CE=E7G=97kyO=82=BFv=E9=C9=F6jJ=AFvDIi=E5=E8=0Du=A2=EB=BF=8E=ECz=B4h=A3= =06U=FC=B6=C9o%=7F=C7VqT=DE=81=7F#=EB=E05=1Cnb}=F4=F5=F9]v=0B=F6;=D5=CF=B7= n=01{=85B=BEY=8B=F9=16-=C1 = =18=84=00;=80=13q=12N=C6=10=0C=C50=0CG)=82=18=81=91=18=85=D1=18=83=10w3=96= =9D=E3xL=C0DL=C2d>S=CA1=85U=18=C6)=8C=A2=8A=D5Y=CD{}*N=C3=E9|=C2L=C7=99|= =BA=D4=E0,=9C=8Ds0=13=B5=EC*=EAX=BF=B3YCsq=1Ekg>=EF=C2=F9=AC=C2E=BCu=17=B0= =AE=97=E0B=E6=AB=11K=B1=8C=F5=BF=02+=B1=0A= =AB=B1=86]Q=04k=B1=0E=EB=A1x=BC<=B1=A8=BC=97-=F2=D9=A6=E1"=D6m=82=FBO=C9= =A7=DE=C5=F2=AEX<=C1K=98=A7v=DE=80=8D=D8=84=CD=B8=14=97=E1r=E2=D4+p%=B6=B2= c=DA=8E=AB=90E=0EWc=07=AE=C1=B5=D8=89=EBp=3Dv=E1=06=DC=88=9B=D8E=DD=82[q= =1Bn=C7=1D=B8=93}=D4=DD=B8=07=F7b7;=E6=FB=B0=07=F7c/=1E=C0=83x=08=0F=E3=11= =ECco=F5=18=1E=C7=13x=12]x=0A= =FB=F14=9E=C1=B3x=0E=CF=A3=1B/=E0E=F6[/=E3=15=BC=8A=D7=F0:=DE=C0=9Bx=0Bo= =E3=1D=BC=CB=DE=EB}|=80=03=F8=10=1F=E1c|=82Oq=10=9F=E1s|=81/=F1=15=BEf?=F6= -=BE=C3=F7=F8=01?=E2'=FC=8C_p=08=BF=FA=E6=CC]4+=17=D8=E6=199.=18=1A16=18= =0A= =8A`=A84=14=0C=0D=1F=13=0C=0D=1B=1D=0C=0Dm=0F=86=86=8C=0A= =86=A6=90_N~=19=F9=93=C9=9FD=FED=F2'=90?=9E=FC=ED-=87=02-=91=CE@=C0=B3=DB= =F3=A7=C7=17=A8=AA-=E92|EvV=BC=B3=90=1D=9C=15Y=91[=92=DD=92=DB=95=DD=95=EB= =CCv=E6=8Ak=B3=B5=B9=FAl}n=1D=A7]=D9=BF=B2=85=A9H=B7=A7s=7F`k=93=1C69=B3= =8D=CE=D0=E1=0C=AA3D=9D=A1=D9=19=14gX=EF=0C=EB=9C=A1=C9=19=D68Cf=B5=1CV=DB= =B3Y=E3=02l=02=07GDdj=C4=DF=80=88M=F9j:#]=11/o=0F=EF=8Bo=99=A7=DBsx=FB=CE= =85=8D+=BB=3D=18=BD=CA=B4=D6=9A=FCV=1C=FB=91=AB0=CD=8A=E3>=E6=DF=90+=D0=AD= =0Dendstream=0Dendobj=0D80 0 obj=0D3601 =0Dendobj=0D82 0 obj=0D6312 = =0Dendobj=0D84 0 obj=0D<< =0D/Kids [ 5 0 R 48 0 R ] =0D/Count 10 = =0D/Type /Pages =0D/MediaBox [ 0 0 792 612 ] =0D>> =0Dendobj=0D85 0 = obj=0D<< /Filter /FlateDecode /Length 86 0 R >> =0Dstream H=89=ECWko=DB=C8=15=FD=05=FA=0F=FAVy=BBR8/r=D8=07P=AC=D3=B4^$=80=9B=08(=0A= =EFvAS#i6=12i=90=94=15=EF=AF=EF%=E7M=8F=E9=C4I=BF-=12=1B49s=E7>=CE=3D=F7= =CC=0F=EBY2=EF=FF5=BB=19=9A=CB=F9=EC=D5{q(:y/.=EBC=DD=C8=A3=E8=1AY=CE=1B= 9{=F5&=99=A3=F9z;_&=AB=04#=8A=E7=EB=12=F6=AD=CF=FD=AFf>=CBW=19=1B,=0D=0F= =8C=0E=BF1=9B=AF=8F=B3=C5z/=DB=8B=F5=AF3=D8=89=08E=FD=CE=DE=0A= =C1)=ED=0D=CC=D0=8A=D3=14=F5=866=B3=C5=FC\7=1F=DB=BD=10=9D=DE=82I=92=99-= =14=93a=07]1=C2=99=D9=D1=EE=EB=B39=00s=96=DA=D5Y=96=0F=CB=F1*=C7=C4=1E=D0= =ED=85^=9C=B0=9C=9A=C5=98=A1T{=E3=DB=BE=AB=0F=0FU}=94=C5a=D8cV=A3=DCz=92= =A596=AB=8FE=B77=8E$in#=CDa=89r=04=82=C9=CC=EA[=B1=97=D5f=CAq=B2J=C0=FC=C8= =F1=C1"=02=17=FB=0A= =80+8=E6=B7=FCp=F9=E1=CA=AC=A6=9Ca=BD=1A=E7=98kW=FC=E5=97=EF/ = .=CD=FA=84=10=9B=17=C48=D7=BE@=91"yY=99]=98=11}H=92&=CA'=BB=FCJGI=11=E1= =C62,B=C3=AAd=05=86=A9YZ=D5gc=11Bcfu=922e=D3&=BB+>=9AB"=CA=89M6MLB=FC=AA= =CB=AA=ABm=AA1u=A9=A6Y=04=84EY=D6=A7=AA=B3=8E=F0=D4=16=07P=C3tB|=B7o=A5=01= ,xcA=95=A4L[=F7=0B=DF=88=EDA=94=9D=AC+=B3%=CFmZ=A0BHC=CB/=FEO=8BF=DC=8B=A6= =95=D5=CEl=A2=08=DBM=84=A4z=13=818<=A7Z=1B=03=CBl=8E = I<=D6z=B2=03D=9A=9Cf=CC=E54=CDt=F2=03=B8=17z)=19zG-e=94pS=D5=BEI=8C'=0F=9D= =F8=E9=C2=ACG=D4=86=CB=92=CC=B4=86=E7xa=FB=02=0A= =9F=B9=AE=CB=B5=D7=BE=17e}=BC;=88=A3=A8:/7=06=8C=C3=FAe=AA=FD^"=05=E2~_#= =8E=05=B4=9Fh=0C4Yb=E3M)Vu=B3=E5=AD=B7S}=8Ab}=FA"=82=E9=B1=E6a=99=EB=EC=84= 4=B3=91=F7=B2=B5=E0=89e=93=04x=D3=D9T(H,=01=03=9D=A2H>e%;pG=FE=E6=D2=19=0D= :=84=9AGN=10(=D2L@0=E1=91=A0=81n=E2t=8A=83D=9A=00=87=8E=D6=01=12=A2=D1=05= s=81=DB=C3Mk=93=84=D9\=D0\=1FmV=A1ve=C9=19=F9=F0=C0La=D5=87=C7=DA=16=11{= &=07=8C=0C=E1=F8=E0=DE=14=9Dm=85=81=EA=B5=AF9=A2=91=A8Z7=DA8wD=C4=13=93)= =9Fg=BB}=D1M=10l=C8q=86eS=EE=10=C1=B3=0CGX=F6=D4=8A=89=EC=86=80=B0=D9e)=B3= 3-c=1C=C7=B0/=DAn=BAS|=C2q=9DB=88=85%=C3=C4=E8=02=DF=E3=BB=A6.Ek=E7|=86=AC= +4=A5X=A3=DE=07YYW=ADl=BBv=AA=BFIPG=D5=E4=AA=F1=A8=9D=AF=06=9Bh=85=C0=BC= YK=B0=E9=F1,=B7C=0A= =BC2E=F4=E1=B9=11e3"(L=1C3P=A2w=B1 = 9=3De=1A=E7=A1J6\=8C3=03+?=DC=F3^=96=FB = =A8=84=D4}e=D9=C6=E1=95p=CDP!Tv=A2=12M=D1=D9v=E0n=EC=D0L=8F=1D=12=18?=99= )=05k=10=90 = =D5=BEpd=BA=DC=B7=7F|=B0Ht=E2=90=91=94D=88=A3=12g=E3x=EAd!a8=8F=B4=C3=F6= T=F9C=16q=EAB=CD=B4=F6=0C=1D=BF=AA:=B1=83!=BB=AE=AF=81=96=A7=94M=1Am<=0C= =84=E7D=9F=D6 = a6=8BCk=C9=8A'=16=03,=C3Y$=84b=B3=F9~=A2OCb=F1=04N=B4=F3|=F8=BA=CEC = =F5=B4G=86"Y=EF=9A=A2j=8F=B2=EB=C4=C6f=9F=E4=FE|=D5=F3=06=08t=98 = =FD=AE#4k=B1=13=C6=FD=A8=9C = Q9=91=E4=89=03;=D7d=90=04K=EF=8A=A6=93=E5=E9P=98=E9=8D=9C=E8"I=8A#=02J4M= mVG=E72=0E=12=F4h.G=C9=9A=8E=E8=EC=19=B2=F6=17[=CC = =07Kj=F4b=12=E4=FF=A3=A7=89)=B6=CA=1F=13-=A0q02=CE=F2`|&=8C9=84=D1=CC=B8= =E1s^U[=BE=CE=9C=08=A5=99%=BD@=E4N=CF=C4=90=F3:=90=B8=160=86P=15=D6Ss=A5= =08=F9n=1A=BE=BE=A2=F1=D4=06%=EErC=F1=13jc<=F8U=83=12=16=B9=95=B9=C1=8F=88= =D7=CD=98=B1H=11=A1?=A5G1_4c@=8AY=98+-=1FQ=92=B2=9D=00x8=A6=BF=16=E0(K=BC= =0E5=0288=A2=DD=D7=A7=C3f=0A= /$8c_=DC=8B)=00=E0=002Um=BD=C7=8E=92(C=11=00=88=ED=16=E05m=DB=C7=80-=D1=E7= =83kHz=EA=92=CEc=AA=A4=11=ED=E9=F0=8C=C6=F8=8A;=C4=97=C9=E9=BE=86=D4=A7=E5= L=D9=F4um=B9=17=E5G{=E9A=8C=DB=CC=11=96=F0H=0D=EF=EA=E6y=84=FBe|a=98=88p= =D7=0EN=88=FAq=16=87]=DD=C0=1D=F5=B8=8A=DF=1EhPFc8=C9=B9?=AB=90=9BU6)=FF= =A9O=06z=9C=B8=D9=93=D9>=0B=D4~]=FDa=923C>=A9=84#=C1=E8=08=7F=E2"Cm=9Ei=9E= =B2H=EEN=FD=E5=B5=ED=F4=E5.=96=0D_I=F4k^=BDAs=80=D2v=90=B5=04yP = =AES=EFFm=05=CF=1E=E7=99)=8D=F3=94=EB=AF^=8D=C6- = =CF^2=F6#1=10=EE=B5P=F0L{=83=ED=F8=D8=B4W=96I=B7n=C0=AFd!/~^=FF=F8=F4=F1= =E5c=FB=DE=E9=85J`b=13HrdK=CF8E=91=B9=DD>T]=F1i=B2=F8~=89^^=FC'=C7I=88=01= =7F=9CdN=CD=90=CC=0A= =14=9F=EC=CFu=F3=B1=DD=0B=D1=99=E1=99d=C4Jn=9C=A6=FA=00=BF5=DE=8Bb=E3=DD= qb=FDO=02=FA=F4=FA=1F=94=8D=F5=08=D1<=8B=F4=7FY=1F=FB[=94=8B!:=B0=06=05=1A= =1BX=FD"w=89=E2=E6=02=E0=A7=B5=3Dm=B7=B2=14F=F9=BB=CB=8B=E2=0E=CAT>=97LE= 1=C0=D2=C8=96=F8=8D$=0D=C8`t%I=F2=D4S=BA=B9f=1B=1E=94a#=B6=B2=1ADF;=D6=96= =CA)=C2,=CB=1B=F1}=B3X^<=F5?=83=AB=CB=C4=E7=DF=F7=FC=BE=E7[=EC=E9=A9=F6=D5= =1B=AC=E8r=E8=E94=1D=D8`=D67[2=CF=E79=99=93=14=AEu=D0=B7=C7=D9=E2=AA=EA=C4= =0E(m]_=830UdK=D4=EE=19_=C1f=D3C=7FS=DF=B4=E5=9Eq9!=86=E6=0F=B2=ED=D0/=C0= =F5=DF_,)=F4^=B6=B8-Z=11=BE=90]{-=9A=7F=D7=CD=E6=17=ED=A59=07=B1=15!=C6=D8= =E2=F5=F8=A0=1CY'=FE=A4=BE=D1=88=13=8B=BF=86=FB=90=EF=FC=BBzs:=880:=B2b~= hK=9C=AF=92=DE=1A=F4s=DA=EF=EC=BF=F0=E7=82=C6*D=F0~E=86=17d=FC=82=AA=17=B0= =11^@=F0=A0=01G=D1Su=9E=F2=E5/O=9F=08f=82=00=82=08=FF9=91=99=EF=9E=CE=0C= DQ=97p=15=D1=FE=DD=17=8D,n=0F=A2=D5>=1As=B9_=A0=EFB7=82=A3=DE=86G-1W=C1=F9= IUy=0B=DC=05%=CB=E2e=04=0D=E4N=D6`}-wRM#=E7E=E6{=F1=BF=E6^=CB=BF9=14]'=AA=F0|=AAZ=F5=D9j=FBX3{=B1o=FD=8Bj=E8=9D=F9=ADjX=89= =B6=83{=8A=FE=AB=F7VU=D4=15q=0B=F1=07=DF=C3"=A1=DC=87=D6=97=94=89=A4=8F=F9= N=11=D6KZ=F3=BA=D8=BC=15=DB=EEee=BAyL=9D=04=8F=02e=13q=A2=B0*=D1j=A2q=C5= <=E7=FF8=11=D8=BFNu'A=FD=8E#=9B=A2=1B=8F=CF=DF=8Aj=D7=ED''=CF=04t=C9=CB=A0= {=13=C9=A3=B3=91x=EB^C=CE=DF=C2=CF=EBP8=F4W=12=F6=AD=DB=00=DB[=E1]=B11,t= =96=DD=DE=C0=FB7=D1=D40z=CC=AC=B4=EC=D5=DFVL=07=F4=18=D3=CFm=ADze_=DC=0B= c=EEx:t=F2=EE`=D7=D7[=F3=F4=08P0=EF=F9=CB=86Z=9A=3D&85=D9_Bp_=87=10=FA5=E4= =B6DC=9D!=8E=E1K=FF=FE=C3=E98=AE=B7=97=A5)o=AE=8B=A6=1B;C>k=EB=8D/=94=14= l=E5=08=B5tE=1F=C5=E5P=97=B18=E6=DC=97=9B=C5=A7=8B%=1A = =F2=DF=F1P=F7MO1H=B4=C4=1E)-=9F:=1C=A2=F9lpM = =BE=006=93=D2T=FAz=06]=0C=0D`=FF=8E=89P=E6=99=BE=01=11j=18a=895=D3=F9X=7F= a=D7=EB=DE|=18=E9=17o=F2=E9=1E=16=C7[pP=7F=F1 = =A2=B8=BBk=EA;=90=A8=9D0=BA=E5=AE>=BB=B5=AE=DB?9=0D=B4Q,Q=94=E5 = <=18=B6j=FE=00=A8/=87=A7=959=BA=1Fa=C7=BA=ED=9C^WYH=1E=A9=D8=9B=C5V6=C6=E7= T=E9*=B5G=B6=01o=E9=98j3=D9[=B9=AB=E4V=96Eew=94=B5=D8=C2=9Ba=CA=04nk+=A7= vDor`=C9^&8f=DC=CB=DD^8=87=9E=CC=CBh=AA=FA=E3=E0y=12\2=DD=88>=1Ct;"=B5=AB= =BFr=E6 = UWN=D5=02p=EB=EC=1F=E0,=F8=95=11s=F5\=83=BB=B2=DA]=BE=BF$=B8|-=CAF=1C!=03= =F0=E6=87=87N\=F7=AA=AF=A9V=D5=AD=B2=9EX=EB=84S6XO=C1j=9Ac=7F=BE=FF}=3DK= =86=03=DF=FFC=F5=DFy=8E=92=F9;=D8=FB+=FC=FC8=BF=F99=99o=E0=B2@=F8=E0=10=04= p=9C)=B7=E0=F10=FB`=FEH=07=D8=0E=DE=F7=BF=1A1=DB=CE=FE'=C0=00=F7=89T=CB=0D= endstream=0Dendobj=0D86 0 obj=0D2621 =0Dendobj=0D87 0 obj=0D<< /Filter = /FlateDecode /Length 88 0 R >> =0Dstream H=89=ACWmo=A3F=10=FE=05=FC=07t=9F=F0=B5p=BB,/&m=A5=EA=92=F6=AE=A7DJS=A4~= =B0N'b=D6=0E'=1B,=C0=97K=7F}=87]=96}=C1=10':Y=B6=D6=B0/=CF=CC<3=F3=EC=FB= =D4Bv=F7=A9=B7=16=B6=0B=DBzwGwY[|=A3=97=D5=AE=AA=8B=3Dm=EBbm=D7=85=F5=EE= O=DF=C6v=BA=B1]=E4!=1CE=91=9D=AEa]=FA=D8=FD=D4=B6=95=B0m=12;!v=E8=87^=1C= =DA=E9=DEZ9=7F=95-=DD=D2=BAI=AB=DBj=F7=B4@=CE=C5=05=FC=1C=9BlK=17=9F=D3O= =B0m=C0=B7=B5=B0=EF=85=DD^=B9=E5=FC=B6H=BF=CA=03-=E4%8=8A=F9=BB=95=F3=06= =D6=A7=0FE=B3p=C3%=F1=88=B39=96=EB=B6=A8J=F8=8F=90=B3=AE=CAo=B4n=87=B7=99= =18=EC=8A=A6=15=E3j#FE=0F=CF=C0=E2=0B$=AE=8E=04{`=F9=80d_=C9-=9Bb[=16=9B= b=9D=95-=DB=CB%=B1=B7$=84=D8.=F6=A2n=0D[=C1P=B2=F9=ED=03=1D=90=D1Mk=9C=1F= =9Fs<`=AF=143=E1=14/v=0E=E0=E5=B2=DA=17=D9N=BCz,=DA=07=F1=F6=BE(=B3=FAI=BC= YWt=03=90=0BZ=82=BF=90=E31=C7=0E=B0=F4=B95=3D=D4=B4=81=99Y=EFk=C3=91=9D=3D= =CC=EE%=C7=A8=99=AD=FA=BE=0B\=0F5=CBs=9A=F3=7FQ=0F=93=CFA2w=14= =9Anc=94=84=BE=F6Op^=8D=A0=CC8=19=C2=E4U=0C=C7=EA=99/ = =B7=9E=83=8F=83=1A=8A=9F=EB=F0P=88=B5f?=A5Bq=A2=92=F2%q"=C1=E9=1A=E5=BF=AA= F=DDf=F9=F5(a=CE=8D=D4=AA=A7=88R=85=89o=18=1A=AA=9D=C2=B0=13=9F=91=CE=D8= =0C=9A=02=FE=A7=19=C3=FE>V-KY=C3=B2p=C62=19s=E7=9A=96[=C8=BC=B9=9E=F6=E3= =D9=BB:=E1G=B9=07R=E6]=81=CF=AF=E1{eh=0E,=BB=8AY=EBa[qi=1A=A4=E6=8B=CA=BD= ,I = =D5D:=A85=F3?ZW=B4=19=D3=7F=B8%=B0<=10=0F=07MV=D2=0C=84=E4p=DD=98=93X=06= =B7=08>=01=F8=CC$=1A=E5=0Fye=FE=D4-=13=9B=FA=E1=A1\=FDl=FE=F8=13B=CE=D42= H=AD=18c=16M=15@%=FD^=DD=DD=B7=B4=A4u=D6=CA=E06=C7=FB=0Ezc6=C47=0BW=94=C1= =C1=0C=D4=3DU=AE=83z=0C=FD=E0=15=E1=0Bz=AD=AC=C6=F0&;=98=A6=9F=A7=B0@=A4= =A4=D5=15]=8F=D5=A5=1E=80=A5=17=CA=F4=03A=8B=C0=9FFQWu=D8=ACC=BB=9Bf=0D=F7= '=E1AU=FFt=B7=9CS=F2)+s)I=F4[=ADL=A6=9C=AE=8B=BD=B8=E8=E5=C5v=ECp=B0o=BA= "=8F=BD=EE=061=AF^=AA=AF{=F6=E1^=EB=80|IP=D0=C9=17+=E1NG6=1B=84>=F8,=8CI= 7N=F7=96=93B=9E=17=E5=F6=F2=EE=92=F8kpyM=F7P=A2=E1=C9=FB=A7=96=DEf-8=A5=F4= =CA{=BE;=1Av'=CB = d=BBG=B0k=94=F8=3D>=D6=F1=FEH-=C4=0E=BC=FB=00=E8=E1=9CG=1B#=FB=06=D6~=85= =EF'{=F5=19=D9=B9m=C5=84 = =EE=10=0C=D8[=1C=16=0Cw=D6?=E2O=C42=96=A1=EF~jjm=AC=FF=05=18=00.=05=B2=F4= =0Dendstream=0Dendobj=0D88 0 obj=0D1312 =0Dendobj=0D101 0 obj=0D<< = /Filter /FlateDecode /Length 102 0 R >> =0Dstream H=89=CCWmo=DB6=10=FE=05=FA=0FB?=D9]=A5=EA]v=D6=0D]=92u]=91=00Yf`=1F=8C=AE= =A0%=DAa'K=86$7=F1~=FD=8E=A4DQ=94=AC=C4J=BB=ADE=12=8B2=8F=C7=E7=B9{=EE=EE= |=A1Y:=FD=9Fo4['=BA=F6=FA=16'=A8$_=F0E=96d9=D9=E22'=91=9E=13=ED=F5;G=B7=F5= =C5Z7,=D3=B2=83 = =D0=17=11=EC[=DC=D3_=B9=AE=CD=99=99=B9>wu=DF=F1=CD=D0=D7=17[m9=B9=C9=92=C3= "=FB5-=F1=06=E7=C5=D4=9A=9C=9D=C1=AF}=816x=FAq=F1=01=CCz=DC=ACf;t=13=18=8B= =B5=C9=0F=D3=C5=E7=E6D=CD6=E1=C0=90=BF[N^=80=81M=9E=EDw=C5=D4=F0-kR=DE=E1= =A9=01=0E=99=E1dER=94=1F=F8S0=D9=C1=D9=F0=95=99k=BA=93(=C3=EB5=89=08N=CB= =A2^#)7 = =8C=B1=D5l=AD8=E6=CC=A5=E3'o=DA=AEYf=D0=BC[=91=B2=B8=C1=F9=1FY=1E=F3o=D5= &=02=D3=B3=C5=B7~=1C=BA=1CJ=E3=DA=91=98=14=BB=04=1D=AA[2g=F9:=8E=C8=16%=F5= =E3=8B=CA=DD=A0=B2g=F8=B6=E9=C2?=DDp=F8=A9=CCn=FB=8E=0C2=FE=B1=86=8C?=0D@= =D6=F2=E1=08da=0D=D9=CC=F4=DBx=05=02/=D8=E2=1E=C3+=1C=C0+=18=C0=0B|=93=D0= j=C8m=E1=C5]=ACa7=DC=80S=070=CD(\=F4=A8=1C=7F=81(=C5=E7=87=12=9F=83o=FC`= =B7=DA1k=E2=F3=ED@=10P=08=EDO=ED=AD=AE=80=E3R=DD=C9=CE=E6=EF=CE=DAA=D3=82= J=C9=08=C3v+=EFm=FE-=FA=9D=EB,=DE'=F8=D8=C9=CCg=C3=A5;=D8N=BA6;=EE=CDr=92= =90=A2=B4!=D7^M=0DX=05 = =E9=82=C3=178=B2t=C1U=17+i=1E=AB=0EH=0C<=EE@E=06d=80=D7= =14=02=D7Q|=9A=F5$=B3=D7=87=C9w=03x=D9=8F=A7PcS=08=D1K=D5=A2x=E3:=AArHb=FE= =AF(N=1D&=A0=88H=B4ueV=7FJ=F1=83=10=A1;=B2=B9=A3Z=C3Eh=BBOJ=B2K=B0=D4`U=9F= =04=F4B=86=E62=86=A7=C8=90=EF=8B=16=A2=1D=ED=FD=89+=E5Q=A7=3D=17=F9=F0=BC= =DC=F5=D4=DC=95n=F6=A4=DC=ED=B4D=BC=FF=18=97=BDyIJ=92=A5m=9F=FCf=F7=A3=C9= =E3=C9=CD=D3=EC=7FU=FF68=C59*=B1h=97=8B=FD=8A=BA\t#n=D6L=0A= =A5=DA=B4=D8=AD=B08=A9=06=F6=0A= m0=AA=02^=A3=9D=8A=D1=D3=FA=94=BAt*=3D=AE=AFP=15=9ETC=BE6U=D5LRK=03c=A1=1A= wi=AF=D2=CC9=18Ew|=16=AA=C8=EC*=C5=C8=16=D3=0Dyf=A8t=85=A3=E8z=97=A0=B2=C4= J^=9D=D4=B0=04=CF=A9=F1=F2=9D=BB=95=F4Ta_=A1=E8=AF=AE=B2=17$=DD4=EA-7=9B= ,=AF=80=A2=A8=E9=B3{r=AE=BC=C3=CD=F4Z=90=1C=C75=FD=B4=B5Vxu=1D=19=BBSx=F5= =C2=FE=0A= 0=FB=0F=1A=D1=05=1D=8A=06x=1DT=DB=87=A9=C1=C0=FA=B3=FAKz=FA=A6=96=B5=CE=E0= +[#=CA=E8b=A9=02=D1=9E2=9B N=BA=8E1=AE=D1Y=C2tjA=0C=D3=1Fn=BD = :=A1i=DFDf=A2=1CC=3D=E8=0BY)=18=0B=F2wOO=C2k=1D=0B=EBf=91=A4`q=0B=E1=0D=A9= =D0=04/=CCS=3D=E1=FE=A0F=B4;=B6=A7 = =FC~=A5=9A=8F=EA=01z5=AE=9Bk=DD=DEsY=A5=91=05\)e=FFyrS5=86=87=1En=CA=FBL= &=A4=E8=AA=92=A0=18=88B}LGYZ"=922=C6=AA=CD8=DF>Q=9F=F8c=8F>y=DE=E8=0Eu=CE= CTf=F3=A7=1D=BD=FEH=9D=B8I=F6=85=9A=CCs=85=A1=A0=95=16=97S=C3=F1-=A0K=A9= =A5r=F76=98W(=8E=FB=C8=AAp=05&=1A~6=B8=9Ds=BC=C8=F7 = j=87=A6=FFT8=0D=B7=12i=83=05}O=894=AAM=86#P=16]=C7=F9=A1=C4=E7=B4=E3=B0L= {rv=06=D8=ED=0B=B4=C1=AA?=AD=AC=18=A8=11=CB=C9=0B0Q=E6(-=D6=99@=00b=D1=A6= = =D3=9B=A6=D9=96=A0=A4=1Bq(=87=C6'G=F9=A1=89=BBM=8E=AB`&i=03=A2=08=EC=14=DF= O=8DP1=CBu=88=F6M=EC=0A= p=AE=D2=86.e=96=B2<=A63=99=EA=8BR=BB=99=03J=13=C6=F9=CC=B3=FDn=A0=A1n%W=05= x=0C=F8=98=DC=E8=E2=8E=88=B7wH4}+=8C=C5i=11J=92=A67=00=84=1Ak=EB=04Gl|=81= g=0F`=E7=8C=055=E7=AC=9C=00=E3,=8C=F9=BDeVj(m[=C4`=0F3=94=BB=A3=BCTU=E0=18= 3=8D\=08=C3`=8E1#=E5=00=F5=DA=16"P=8Fu=CB'2S=B9=C0M=7F-f=D8Z=C5=0D=98e=CC= T=03R=0F3bt=EAa=A6]=E0=8D=AA=8E=03)!=CF=D3=15I=17=D9{=FC=A0vT=CD=083=D8Q= =91=94=CA=DB=A7=81~=AC=D3+K:}6P=A0=94=0C7l=AB=AB=D3=D7Y=BCW=1B:=D7=14=85= =EC-=A09=EB=B48=8Dp2i=B6=07t=FE=CD=C0=CD_1=E1;=D2=DC=DA=E3=86=96<=DB^=92= =0D=95=C2=96O=BE=8C=D9`=ED=E1t=C8=D5=C7Q*=8F=7F=8C=96o=3DbB=96=E2|=97=E3= =BE=96O=EE=0FP=15=F31=07=A2+=D2B=A3H*IB=BA=DF=AE = [=DB5=C3=99=8D=1BX|Y=08&=E7=A8=C0=EF@=B3=D4 =17=BD=F2 = %<=1C$F=EC=A0=9F=92=AF=D0=07=C4=A4=D8%=E8=D0=D4=8B=0A= )=B8@=8D=938]X=B7O=9A=EC=C0C=A5=9A=D5=A1d=F3=3D=96i=D9s=CB=D3=17=11=9DV=18= F=96=CE>=F8=0E=00=EB=87.=93=9E-Lc=B8=A0=8D=FB=C5=ED=85=EBD=97=B8i=E5iCp=83= J=88=97=D4LW=DC=BA%=AC=BB3=CFg=D6=03=B0=1A=CC=9D=CA;=97~=EF=E7=85f=B1=03= o=7F=E1=9D=F2=BDn[=FA5=EC=FD=0C?=1F=F4=E5GK=8Fu-tg=CC!=B8=C0V=E3n=C1=C7D= =FB=BD~=08X=C22=EF=E9=AF=1Ckk=ED=1F=01=06=00x=FB=A2i=0Dendstream=0Dendob= j=0D102 0 obj=0D1852 =0Dendobj=0D103 0 obj=0D<< /Filter /FlateDecode = /Length 104 0 R >> =0Dstream H=89=C4W=DBn=DB8=10=FD=02=FD=03=D1'9=88TJ=94d;=AD=81"=C9f=8B"=01=8A=AC=DE= T=A3Pd=DAf!K=06%=E7=82=C5=FE=FB=0E=87=BA=C7vR4h=11D7=923=E7=9C=19=CE=D0=E7= =A1A=89=FA=93+=C3!=82=18=EFoy=1A=97=E2=9E_=E4i.=C5=86=97R$D=0A= =E3=FD=95K=1C=12.=89Em=EA=04A@=C2=04=D6=85=0F=EA"=891E3S2e=F6=D8'=BE=EB=AB= [=B81"=F3Nda=FE=99?=8E=A8yv=06=97]=11=AF=F8h=1E~=01=9B=9E=B6iLm=D7W=86=16= =869=1B=85?Zo=86c=83=B3=B1=1E=8B=CCw#k=C2l=D6=DA=B4|J=CD=85(=B6i=FCT=C0=1B= =8E=8A=AC~Z=E3=14=FD=B1l=C6c=B9=DAmxV=0E@=B8=AE=C2=ACQX=C7P=C4=B5=A1T=14= =A5=86=90/G=16=88b=8F=01=1A8=A2=A6]=CFyWy = *S=16=F3l=CF=01c=96=AB=EFh=B2C=08W=FD=04%=F0> 4n =D1 = c}J=C1=AB(=E1=B3"=E5WzW=A4=B4=83Z=16=CBe=DA=0A= p=01G=0C=FD =95K=9Ehw=AC=9A=EA=D9A=E3=CE=FC=D4W=97=DA~=0BSd=0A= =C2=F7=E1=EA=0E=91=CB=E1jt=AD=C7=CE=F4=98=B7=CF=F2 = =B3,=87V=E0=1D=3DK=CD=B9=C9=17=BB=94=F7=9D3=DB=AFe=FA=04"L=06"t=85D=F9=9C= =FEr=D7=9E:=CD=F8=C7#=CCO=D5=985=D6B=01(=BC=0F=8Cz=0D=A6C=1B=86v=DD]=C9|= s)V*z=3DL~W=B3#=D1=88=AAp=00=ED=D3=91=05K = =17=DC=8A=7Fc=EBPX:=D4>=F4=DD;=DDp~>=12=B2=93c=DBPd%=97[=C9=9B|-=D7|_=1A= =C7E]&V=9D"=D0&w=DCn=DD,=96O=F5=E7l=B7=B9=E3rX#&}dl=1F=EA=EBA=A2yc=C8=A0= n@=CF=E3=82_=E5r3L=F2=A6=FC=1C=0D=89N=87ND=1C=BA?$=11=C4=C4r}=0A= =12=F7Y8=BA=F2=BCB=E3=AA=0Ei=01U=11=AA=94=02=02=B5N=8D=F7=C6z/=BC/=0A= =05=08U2w=F5=19=A4=12=B4=1C_=95=98z]=E4|=B9,=B0=05G=EA=DD=F9>=1FA{0=C9=D9=0C1=B5P=D0=E6C=B3=87= =C2k=12=9ED=E0=9C=0E=FE=91=17=C1=C5=CF=B1=B0=AEpD7=9DH=E1=7F=EE=C7=AA=E6= ZN'=A4=BF=E0=0E=16=FF=8B=15=E4T]]=BC2=BCz=FF=9Dj=CE=DF`=D3=9ADUS=B0=9C=E6= d=04<=DC=C7R=C4w)W=81"'=DFF=07t=E9=E0=F5=DF=02,b=9Da|D" = =AF=E1=83=8E=D1=E9=E3=FCC=171=7F,e=9C`&=91=A4]=80=EF=90k=0D%=B5=F8=D5$=DE= JtT{v=CB=EF=B9,x=84=AC=FA=E8=A5=1EB=B4E=0E=1F=D6b=B5=86=06=A4=C0=CB=85~@= V8#=96H%=CF`b=CA=97%=AE=12=0B=FEGx=B1=D9=D7xq=0D("=9DS=CC=EDS=DB=C6=8B=C3= A=D9C=13=FA=FA=06'=085a#=8ABd=AB?=C2=CC=03f=B2=14=A5=C8=B3H=EF=14oH=AD=1A= =D6xU8V2=DFm[~=DE=EF=DD/7=F16=AA=7FM=E8m=3D=EF=E2M=F2=0C=D2L=CB=CF=E3d=AD= =EE=08=B8=8F=B7=C4=04=C4=1Fm=C4=FE=BD=CA=9Bd=AE+=B9Su=01U=FF=A9=A7&=0F=EB= =BF=0B=92=F9cV=FD=D44C^=94=90(=17=B7=17=CCM=A0=E1I=AE=9A=16|9=7F*=F9=D7=B8= =84=B4=CA=EC=ECN[=A7=8Du6=F1|=B4=1E=80=D5`=EAV0<5=EF=AF=D0=A0=E8=F0=F6o8= =BA=80=9F=07=E2Pr=03k=7F=C0=FF=17=12=CD)Y=10c=CC&=08=088l=0C=0D=0B=1ES=E3= =9F=FA%=C0=C3=08=A2W=17=C9=8D=A5=F1=BF=00=03=00j=E1=A7=9A=0Dendstream=0De= ndobj=0D104 0 obj=0D1117 =0Dendobj=0D105 0 obj=0D<< /Filter = /FlateDecode /Length 106 0 R >> =0Dstream H=89=ECW=DBn=9CF=18~=02=DE=01=F5=8AM=02=993=90=A4U=1B=BB9)Q"g=A5^Xi=85wg= m=12=0C=16=E0=D8=EE=D3w=0E=0C=0C=B0=C6=D8=A1wQS/=BB=FC=E7=FF=FBO/=D7=0Ep= =E5=7F=E5=A9=03=DD=D4u=9E=1E=F1,=A9=D3=EF=FC=A0=C8=8A2=3D=E7u=99n=DC2u=9E= =BEB.t=D7;=D7=07=01=80=8C1w=BD=11|=EB+=F9=A7t=9DX=89=89=DD=18=07!u)=A2=F2= c}=EE=1C{=17EvsP=F0=DD=AEZ=17o=F8=F5=0A= =04=D0{=F6l=05=BC=CB*9=E5=AB/=EBwB8=D1=C2=1D=88=03=10a,=85n=1D=EF=D7=D5=FA= k=A7=D9=01A=0CY=A8=DF=1D{=BF=AC=FC=08=07x$=DF'q=1C=C4=DE=A9p=A2Z=F9T=D1=A4= =B9y:=93=14=FA=B1>=E3=E6q#=F9=D3M=CA=F3Z=F2=00=E0=15;=F3.iy=8Br=CBK=F3=05= =A3=95/=C2=10=84=CA=80=BC8O=93L8=15=98_?$=DFx#=EB=B2j=15I=B1=FA=FDI=9A7=F1= =10=9E=E8 = =B0=C6O=1F=12=1D=05=1F=05Dz=AC=FC=1D=F9=A9%N=FA)=94=DF=EEek=D0=9D~=D26=D0= =8D=9F=E2;=91=AE=EA=17=C6U=F5e=E0=AC~j=9D=D5=8E=9A=84=92=C6=BD=F7=EE=FA=91= =97d=D9=C7=9CW=9F=84=16=9Dv=DCP1M=A5!=F1=FB=10=12=B4=83K=95=FE=CB=FF=E9=F3= "=0B2=DE=E1=90W=B0=1A=DEg=FA=1D=D9'w=00C=1F=82@=E40t}=A8?%=CD=87b{=99=F1= =BEr=1CP=DBj=1F=F78=A2=DB=AD9=F6=B2=B4=AA?=EE=DE=E65?=E5e%0=F2=A4=C3M]}=E2= =E5_"I=BD=9F=93=CA=D4=92Q=0Fc=DB=89=17=13=81{=D2=B7=1B=06=C8=18=FEf",=8F= =FA=12=A1U=B9=C2=81b#p=D2=A4=FF{R=A6=C9I&@=D2=AF=F68=C0x(=0E=EFS=F5~=90=01= L{=A1=1CD=ABgrd=A3g=A2=A1xki=E1=04z&=90w=ECA=95=8BA=FC=DB=F4=8FR=3D=00=ED= P+=B6=90=F0BH>=1C=D4=0D=0C = k=EDz>L=9E=15=D3{=A5=AF=93y=EC=9D=F2=9C=97I=DD=95ub=1Ed=AC=AD=02=1F=B4o8= =17r=9D=DBd=8F=DB=DEo=B7=1B'bm=F3=019=88=D4P=C2r=1A=F5=B5=B8=3D1=A3=D1%=88= [=A4=0F`=D3=D3x/h=C6=E3=E6`=D7lO=0B=D343=A6=9D=04=D8=F3=11=C0,=EE=07=A7=9A= '=9B=B3n~=A8=12=D2M=85yWi=96=B5=0D=A6m=EB%=BF(y%=86=08=DF=9A=9F=AE=D2=FA= l=84=14h=B5,k=EA%=E5M+_=06=A4=8F = L=BA=94=DC'=EC=84=EA=FE=D9=0B=BB=EC=89=03=8CM=C5=DBn=C0ho=BC=C3=1F=EF=8B= =E9=B76=90o=CDC=95=A4M3=EF=C6x/^=A3%=E9!!=12=F3`=14=A2WC?=E2=89fi=8DK=D3= k=D7=C5x^Gv=E9Lv=CDA=DFV[=C5=13=BD=B6=A8=DC=A9`=98=1F=AC=1A=1A=A4%=B2-;=D4= =A3=16H7=95=AE=3D=93=DFo=CC=F0=89=EE9=CA=18{=FF=00=13=1B*=DC=1F=9E~Q=FD=D2= =AEgV#=15^$=F6=D2x{=1BE=B6G=FFO=1B=3D=F6=0A= =E1=ED=BE=ED=13E=02^=BD=D5=F3=F6i=B0=D7=8D=B0=9B=06V=B95n=B0i7=C2 = 7=D8=9E=8A=D2^=A8=D7=A0=19=08=01=C0=90D=B2=C9=8B=05CU=AC8L=D4UB=E4_D=C5^= $O=13=8F=E7=AA=1D=0B=06B=016s=84=11=D1=80=E4p=88=02=16=A3v=92=08=EF$=B0=C4= =F5=02=00%j=84=083=9A=A1=13=05=18=84=C6Q=F7=FC=A6=11=0B=19=08=8DXLQl=C4B= !=A0=A1=DD]=E6=9B:-=F2=86=03=C4,6=1C(=8A=91=E2=10=FB=A60=C4=E0=CE=DD=F2]= =9A=A7=92=A9jL=02=88=E2n=A8=F9=98=05=14GjE"B=97=CE=A1=BF=BA=ED_=C8=C4=B9= t=FB=EB=9F(d=8A=1Ek=98= 7=F4=D6=99=DA=147=14Uo=8A=1B=C4=A1=E2=11=AD@T=B7=E1=11G=EB=B6=B5=08E=D4X= =84=99=AE@=14 = =0C"C=9D=E6=86=16=C3=96=16"mL[=A6=E9=E7=83=CFo=EF=D8?c=DD=9C=DCX=CC=B2=08= =A8=A6=F4z=FED=EEo=16=D7+=1F=AA=D1=F2=F7=CAW=FD=18=A3=C1X=E9M=95=C7=C3=A9= bv=9B=3D=92Pt/I=F6=1Eu=DD=88hE=85#Q=EDR5W=906=0E=B1=C5$=D1=C5=02=85=97=0B= =D48{=0Fu=0F,% =C6K=05=0A= =DE=0FQS=92=C8b!=87=E3=EC=3D4P=F0=E1=92=06=EE=8D=93=F7=C0@=8Ds=87=EE=EB=9C= =FA=1Cg=CEZ=F7=EF=17=A6q=05=CF=13%N=CB=DEB=E83=D8_Y=0DA=B8=8F=F9=F1p=11=B5= =DE]=B7S=80=10=3D=B8=98Y!=D5=03=14=B7=97=08=1B=94sA4k=D6Sc=F7=F7=B6=B5Cq= vI=0E=1A=C9=0F=C9=F4C=FA1j=95G3=94=E3h1=CDT=A1=CB(=8Fg(=A7=0D=E32=FA=99\= =DB=8DzQ=16w=EB=0F=17=8D|=DCE=1E=C2=BB=B5=8B=CDgA=EF=11$=B6=F7x=86~D=16=F4= =1E=E1=B0=D3Nfh'K=E6^=9Fl=AD=FE=19=C0Gl=C9=DC=A3=C8=CA=FD=0C=E4=A3x=C9=DC= c`=E7=1E=CD@>=86K=E6=1E=A3.=F7b+=B9[;^2=F7=98=D8=B9G3=90=8F=19=98T=DEN=E2= ;U=87]=DA=C5=8Ex=B7=E2=08=05=D3~=CFW=1D=F7r>c=CE=10=B0=90jq.u=8A=C3=19=8A= Q=B8=94=D7=04=F7r=3D=A3=CE = ](=D7=84u=B9=C6=0D=C8A=A7=18=C5,=D2=8A=0D=97z=A0D/=05Q=A0=F5=FF=D1=9C=9C= 0=C4=CC=9C=9C=98=D1X=1D{=A0wrn=F9i=C9y{HF=A4=3D$c=84=F5YkH1l=C4b=8Ab#=96= =12=AAoS=18@=8A=DB=DBtS=E4U=9D=E4u=C3=01hL=0C=07=A2P=9F=A88=88=08k=EF=D4= =8B"=BB=C9=8B=F34=C9Z=E3=A9e|=A8=8D'=3D=E3K~Q=F2=8A=E7u=9A=9FN=B9@=03=F1= +3\=A4Q=80pg=14=C10j=A2=C3$WCzrS=F3=AA!=179=C6=86=9C=11=99=E9+=B9*=0A= =A7Mr=DDbg=AC@=94=E1=C6=0A= A=0C=9B=08=01a=BC=A1M=B2=AC%=86=04=19b=D4=D8=D1=0F'=AC4=14=A0=86=82=F2=0D(= =E3=81=A2=F6=C5=94e=90=12=81=1Fm=BF=E4Z=F3J=06=E6=E0=E8=00=A3=CD!=DF=94=FC= \=87=EA=A5=F0=EASR=D7=BC=CC=83=FC=A4=072+=80=8E=98]V=B0U=E3=F9s=ED=00=05= =BB=A3=D7=1A=D5W.=04=EE=07=C1=FBU=FC=FF=CE=3D=FE=02=DC=AD=EB=848R=B0=14[= =F0=B9Ce7=92=8F=99=F3=D9|aj=E9W=18=96=7FJ=EE=EC=9C=FF=04=18=00 = RW=A5=0Dendstream=0Dendobj=0D106 0 obj=0D1823 =0Dendobj=0D107 0 = obj=0D<< /Filter /FlateDecode /Length 108 0 R >> =0Dstream H=89=D4W=EB=8E=DB=C6=15~=02=BD=03=FFU=9BV=CA=CC=99=1B = 4@`=BBu=128=88=E1=0A= =E8=8Fu=02p=A5=D1.=03.=B9=A5=B4^=EF=DBw87=CEp)j6=11=0A= =14=86mR<=97=EF\=E6=9Co=DEl=16(=EB=FFt=B7=0B=9CU=D9=E2=DBO=B2.=8F=D5=17=F9= =B6=AD=DB=AE=BA=97=C7=AE=DAf]=B5=F8=F6=9F=90=E1l=B3=CFVh=8D0=E7<=DBl=95=DE= =E6=A9=FF=A7=CB=16=856Sd=05Y=0B=961`=FD=7F=9B=FB=C5=F2=03=81=AB=CD=EFJ=9F= =1A=FD=05=ACQNH=AF=B6[,=BF3=DF=AC=ED=05Z=ABO=EE[Y=D7=BF4=F2=F0=B1=AD=9F=8D= =14=B1R|M1=17V=EA=FB=B1=056Xw=AE=9D"^+=E4N=F1=9D=F9=C6=ED=B7=15=B6_W=D4=00= =ECe=B0=91=11S=F0=FE=1A=EBG=DF=BE=BEFO%*=D0=EA=F3K)%}~U=A4=FA#=CA=F4=03=06= =B1f=19-=C0=E5=16"7ae|=3D0=A1kPJy=A15=FE=84s=CA=B5=A1=C1;I=F0=CE=C8%\=F3= =C8/M=F0+p/}=D2=F5=B8Ts=CEs=1A'=9D%=B8/=D0=FA2=DE=01=91Q=D6=F9y=F7=80=8A= =D9=9A=BF=C6?@=E8\$8=87=FC2=9E=A9=EE=90=C0y=9E=E0=9C=F2K=D5=1DX=11=D7=BD= Hp=CF=D9=A5=EA.=F2Q=DD1J=F0=9F_&=F7=04=8Ds=8F=F1y=EF=04_=AA=EB = =8C=BB=1E'=8C=BA=DE=FEE=BCS6=8E>a=D4=11v=A9=DA=13=FE=A2=F6 = #=8F=88=93=B5_=11lV=DA=0A= =9B=ED=984=F7=D4=E4Q = =84=1E%=1AD=CA=E0=C3=A6p=82:=E8=7Fj=F4=02=8F=00$=8C>LL=ED.=04=80=E6=A3=1C= $=0C@=CC=8AY=FF=E9kO=E0=C8w=C2=FC=C39^=CFG=9F=EE=BD=18=D7?e=00"z=19=EF=80= =A3=D2C=CA=F0=03~=A9=D8=81=8C=EA=0E = =E3=0F=E8=85=EA=0E<=AA;$=8C>=10=17=AB;=E4=A3=BAC=C2=F0=83=E2Bu'(=AE{=CA=E0= =C3=17=AB;=81q=DD=13f=1E!=17=AA;aq=DD=13=C6=1D=E1=17=AB;=11=E3=BA'=CC:=92= =CF=D7=FD5[=AF=88+=9F0=ED(:W=F9W=F8=A7=D8=90=E6=00B=C2=C0=A30_=FC=D7=00=A0= =E6=CE2=00 = S=8F=B2s=1D=F0=1A=08=9C=8Er@=EC=E0C=03=04=C0=14=0C=04=A7=AB=1F=98F@=89=D1= =DB=DCU=07=E7=8C=D3=DE=D86=EB=FD2=A4=FC>e=0B=BC=CE=A9=1As=06Vfd{=DB9=D3=E1= i=8CB=14Z=16=AD=0B = ^=F6x'=AD0a*=F9V=98Q=06=D60#=B9=EB=F9l=DB6=87c=D9=1C=AD=06*r=E14=A0=C0Z=81= DH=1E=CA=E3Qv=CD=1C=1C=B2V=B7=DA=FC%=1C=8C=0A= /L=10=CE'=E0tr+=AB/=B2s=1A=02=11=AF=C1=FB$i=F3=0A= =8E=D78=DC=B5=8F=F5=CE=C1G>^@=98Xq$x=E1=C4=E5=D7=07=B9u=D1b=8E=BDA=8F=CDN=1EU=9B=947=B5\;=17=18=E7>=04$=88 = =97E=E1=BE)=0F=D5=B6=AC=EBg?E=04=10=0B=0BD=91[X=A1=AB=EA8=17B=9C=CD?6=04= =10=1Ez=0B=D0dou=F2=F0X=1F=E7=DB%=94o=F7=FE|=B2=E0|=0A= >=D1.=A5?=FC,(in!=AB=E8=DC=E8=CDv=D5=97=EAP=B5=CD=DC=99 = =11=8E=1B=9Fh(z=01=D3=B6=88=B2=89#=F1~.<=14=9D=F9v=B6=C3=E3=F0l=9EuS=00=CE= =FD=B9=1C=CEN=08=F8=A15=BDaR=A7m#-=B6=E2=C2=1CHu=01U=81s-=DE=C9=07U=19=D9= =1C=AB=E6v.=E1,jt=97p=02=E0=FB=95=A9{=DAD=C6=0F=F2?=8F=B2=D9=CA=B9=D4=D0= =C8=B8I=8D=CE8=E3=AE=B5=950=9E=AA|]=0F=B3=B0(=FC,=14nb=85=86=F1_\s=13=8D= =D5=E2FS=D3=ADl=CC=BC=1F=A61=B8d=87=C7=C5=A7L=04)+=B8=CD=03=10=E4=9D=1F=EE= =AA=FDQ=EE=86=D0li=FA=D0=08Ll=91Z=EE=FD=B9u}=1A=03 = =ABN=C0=F5S=81=FD=19=A09=9D=C8=D9C{=A8=8E=EA=0C=1C=D6=86U=C0Kbc=1A=C6=B1= =1Bu=07=B5=D4=B0'=15}=7F=BDm=E5~=7F=D8=B4?=C8=AF=C6=06=B1D'=1F=A8=EE=F7=91= u=05=8D=E5=9E=02}T6=9A=F6=BE*=EB=9F=DB]lA=ACO=1AP=E0\}>=10{=1F=A3=F6+^=17= =D8=7F=FDf=AC9=B0=AFkE=BFV=EArE=96=BF=D9=FF=95%=B4=FC=DB=D5J=89=A8=B7=F7= =E6=85=AA=EB=8CX*p=8F=F5=E3=E1=EA=D7=CDO=83=AB=BE=1C=96y=D1=C1=E5g=D5=AB= '=D3=B9p=C7=1E=E2X=C3=9C\/=DF)=D7=EF=AC=AF=DCJ=AC=88j=CB=1E=FC=0A= =8C=B7=DEJ>&=97Af=F1=19r=89Q=7F=C1P;=C91U=CC=FD=F2=19=B3Z=8C=B1=B9=93PKD= =AF=87=EA=F1=E5=DF=A3=BBV=B0= =A1'=AFZ=84=F4=FD=D4=EB=0Dl=08=98_.=C1U+=DC.=BB=F2=E8V(=04=AB=88"a=A4=FD= ~=B9=97=87Cy=EBi=16=1F=B8=82v=A2=97F=B8=8FJ=CF@=A7Y=D3=D4=12=07=8C=07=06= =82H1=B1=C4=EB=EA0K=DDptE=F1=DC=06=E5=C1]I=90=A9=05^5Gy+=ED=3D=E2=C5=B2=0D= =A9=D8=B5=DA=B6H=FD=ED=DB@=EF=B9=C2S=0FZ`=B7rC=1C=9F=97w=D5=ED=9D=F4=D0=81= =0E=17=04J=C8=14=DBk=BB=9D=BF=18=02=CA=83=BC=B8=0B=82fN1=F8=F9=0BB=14=AD= ' = S=C43f=D7=9E=A4=9E`=88a?=05=0Cq=8AX=C4T=C1=11=8B=93<(=14=B6<=E8Txx=FA=F2= =A0=F8=B1=B7=0B=85=C0S=97=87=AEl=0E=F7=D5=D1r"}!#=FE=D2db=D6=D40d=CD=FB=AA= ;=1C?_%=F3=96=0C=FA&Q=87S~}=E8p=CC=18H=7F=E0=8D=DD=EFN=F3=85=A5=DF=EBN=CF= E=BB=9AQ=AA=C6{>=F8=F6Cl.=DA=DE#=E2=A2=8EJ=C0=0F=FA"=AF=98=E6,=DDc=ED=9F= =F7=EAFk=1Foe#=BBRSy=FB=8B=1E4=11}Y=A8i=FA=92*M1=92=E5=87=98g=AC=C0j=86\= =C4=A7GL=A7=87=CF=A4=07=BCa=F3qe&=B7=12=E8=E7=CD/=FB=1F=83=D90=E0=CFC=DA= =F5=A2rAh=9B=F2=A6=96qx=10~=9F!=A8=D7=B6az.h=B2G|=D7=04q=E4s=16=AA=90V=E2= =F0=85=C0=C8=A8=BA=FFxj{=AD=16P=CF=03W=0C!=D5+q=ED`=E8=D9=17=AD=12=9A=B0= =8D =0F=C6L=E9=DAA=CFq=FB=AC=A6=B4}=0A= =1A=EBKY?j=AD=91=88I=C7J=1D=B35_>=DD=C9=C6=18=AE=86=86l=BC=D6=BEk=EF=8D=A8= Pq;=1F=AD{*F1=11a=18nr?zn=CCu=BF$2=E3k=D5=AA=EC*=A0=95=8CFo$z=83=AB=81y=00= =C3W=1Ay=F0=0B=1A=FFB=8BP=9F=E6=D1=9B=88=DEx=F4=16a=A2=11&=1Aa=A2=1AS=9F= ;l=C2=DA|=C86=DF=A8=B0(=8E=C4P=F8F"X$=0F=C3"b=1C=04=E1/~=89=E0=91=08=1E=89= =E0=11=88=DE"L$=C2=04=11&=88R=05Q=AA = J=15DX=80=8E=AEJT=04G=E0=95,=0E=0B=ECH=E3=FF-=8DCl = D=AAe=F9=04=8D{h=EB=E7=A6=BD=AF=CA=DA=A1=9E"Q4=82=92H=A2=14=ED=A2/H=94^=CD= 9=F7=C1*=8Cl=82F=DDT=CE=F8$o=18Q=9DT=DE=10=06=11=F0=06S+=16=D4=CA"=0A= s=BAm=BBN=1E=1E=DAf=D7=EFR=CB=95(=F2=810b=0A= =ACny=04y-5=E3f=19[=C8=04=836=9B=A2=DF#=C6=F6?=A1=DFI=8CJ=9F=16=CA=DDeI=95= =EE=F0Qv=FFV%=8F=B7=B4=DA=13> =0Dstream H=89=AC=97=DBn=1B=C9=11=86=9F@=EF=C0=DCQ=CEr=B6=CF=87=8B=00=8Bda=D8=8B=00= 1=02=02=B90=F6=C2=96)=99Y=99=14Hz=1D=BF}=FAP=DD=D3=3D=B4=C8=1A=A8=B0=C0J= =D6=F4=F4=FFw=9D=FA=9B=BF=AFo=D8"=FEwx=B8=E1=8B=ED=E2=E6=E7=7Fo=1E?=9C=B6= =7Fn=FE=B1=7F=DC=1F=B6_6=A7=C3=F6nq=D8=DE=FC=FCZ,=F8b}=BFX=B1=81qc=CCb}=17= =DE[=7F=8B=FF;,n|=DA=C6/=BC=1C=AC^h=A1=E3=8F=F5=97=9B=E5=C7=0F=C7=CD=ED=FA= =BFa=03=957=B8=11Cx=DD=C6=F7>=DD,=FF=96=9F=C1=E67lpRJx&=F23=F9=A3go=FA=3D= =D9=A0]}=F6=AA=DF=93=0F=BC=EA=BD_nw=B7=AB=A0?=D8=ECl=15=DE=1Bd=90=FA}=FD= [=B3=9F=1E<=AF=1E_M}4Z=FF=CC=CF=0C<[q8=DDJ=0C*=EE=D0=9Cc<#=8B=EF=87=15=E9= A=C8=8A=1Ew=AD=B3_'q=11>=C7=A3=8D=CB=FF=E2=9AXOJ)=19=EB=E9=C6=C4= =AA=89=85=94~=F1n=10z=A1=0C=8F?b-=E9=BC=A9=3D=AF=C4Z=7F=9C=857=B4)=D5=F7= =D7=DE=06=CB=FBc=C49=B7=AD=B4AH=8BT=FB4=EA25Pc=C0"=0C(1\=D6o=12t=D5=81=D6= =93=E8;=84=83=BC=9A=CA=82U=AD>=97=08=03=EEj=0E=E6=18=F0f=E8B=C0=D5u=0B=82= =19=C2=18=08=EE;=03=88=16=10=C2=11=C6@(>=89=01=A2=12=85=A6=AC=03a=BA:=10= =1Ca=C0R=D6=81p=93:=10=02a=C1S=D6=81d]=1D=08D/HNY=07RN=EA@ = zA*=CA:=90=BA=AF=03D/HCY=07=D2N=EB=C0#,8=D2:=F0]=1DHv=DD=80b=94u=A0=C4=A4= =0E$=A2=1D=95=A4=AC=83=F0=E7=CE=00=A2=10=95=A6=AC=03e&u = =113QY=CA:P=AE=AF=03=C4=E5=AC=AD=03D/hAY=07Zvu=A0=10=BD=A0=15e=1D= h=3D=A9=03=85=B8=17=B4=A1=AC=03m=BB:P=88=99=A8=1De=1D=186=A9=03=85@e=C3)= =EB=C0=88=BE=0E=10=CDh=E4=95:Xi=1F=8E=B5=8AgC=91"=931=0A= =CAf=07=88V=E0\=A6=10(Y|=BF=88T=85=A9=EA=1A=D1=07=E5=FB=82H]=B9=E6=F4=1A= =C3=E9=DA_=14=C7=7F(Y>=0A= #.=02=EER=B9=92H=FB6=E7=1AC=E6L=D1H=0B=DE=A4=1B=D1oB=18=AAS=0B=D9=E5=1A=F3= 1=A0=88r-L=93k=C4=85',Y=AE=85ksm0_ = =9E(=D7=92=8D=B96=88=FBEr=B2\K=D1=E6=DA`=BE9$Q=AE=A5=1Esm=10E&=0DY=AE=A5= =EDr=8DB=FC=CB=B9=9E=C5=F7c=B6-=0A= =EE=AFe{=0ETr=9F=EF$=D0=C7=90=BD=B8=9C=F0YX=CF=F3=0FPG=D4=BA=D2=D7=B2>=8B= =EAUwz=C4@W=96,=F1=CA=99=EE=F4=98/=0A= O=98{=CD=FA=DC#=E6=AB=E6d=B9=D7=B2=CF=3D=E6[B=11=E6^=EB.=F7=0EQ=F9=DA=90= =E5^=DB.=F7=0E1h=B5=A3=CC=BD=EFr=EF=10=95o=18Y=EE=8D=E8r=EF0=DF=0F=F2r=EE= #=BD=8B=8C=EF=C1=C8=1C=82=97=A2|=108D=07=16=88=0FC=90=84=E0=1BuD=FF=15=88= =A7P=CF=04=DF=E8c>a=B4=7F^|=1E=C1=8F=C2=1E=D1z=00=F1/=97=F6=93=9C{D=E7%=88= =7F=B1t"=F8F=18=F1=E9=02=10=FFri9=C9=B5=C7|=BA(=8A\'=82o=84=F1=10=FFri7=CD= 5=A2=C2=13=C4=BFX:=11=FC(=CC=19=A2=C4=A5`=17'=0B^<=08=0B=DD=EBc=BE"=14M=B1= I=3D)=B60g=11=EAV=90=8C=B5=00=E6iB=B6=F2=88N=93=DEQ=8Du=C5y/=8F=B8=DD=94= P=97b?G]=DA=B3=E4c=C0R3=AA=8BE=199=CD?=A2=EF=94=B54=EA=CEO=F3=CF=11=1FU=9A= =91]=EB=BA=9F=F3=9Cc=C8Rx=A2=FCku=D6=FC=1C=03=97=9A=0C,=B4=99=F6?G4=A0v4= =FD=AF=FDY=FFs=14=DC=92=F5=7F=80=DB^=1E=D1~F^=E8=FF=88=B6 = j#=DC=CE![=EE=0B,s=8E=81K=1EZ=90=EB=E1=E5|=CFEn=C2=D6=00=86.3=DD=13y=D0=A6= 3 = =10S=80=9B=D4=864=FA.7bk=01=03=BA>7"=89=07=C1=DC=A4=0E=04=02=04=84=10T=FA= RM=EB@`x[9=BA:H=00=DA=1A=C0=A0=AFUdu = =9C=3D=AB=03=04=11H=C6=E8=EA@=F2=E9<=10=88=A1$=05=D5<=90=F2l=1EH=0C=91j=C2= y M?=0F$=0A= J=E9=E6=81=F4g=F3@"=EEE=C5=08=E7=81=E2=D3y = =11=DD=10.=B3K=FA=E8=0F=83=F0=E7^=1B=03=C6=FA=DA(=C0=CB[6M=80B=DC = =CA]=99=05x=03=DEL=A2=AF0h=C8=AF=0D=02=B4=01-=C4=B4=0D=15=A2=0D=B5=BC8=07= =F0=F2=CA=F7=DA=18.5=82=AC=03=B5=D5g=05=80(=7F=ED=1C]=07=1A=C6=A75=80 = 3=C3=C9=F4=85=3D+=01=04=99=19=C5/=B6=E1=CA=B0AJ@=D4=88=AA3(=95=81=0F=8D=01= =B4=88=A8=8C=17=EF=14=8CZ=E51p=06=80J=E4 = =12j=95G4"=E0)=91:=F0i5=80=E8=C6=02=A74=0E=80N=AB=01=C4u=90=D0=94H=1D=D8= =B4=CA#=88=A8=80)=91=83H=A6U=1E1=06=00K=89=D4=81K=AB=01=C4=18(PJ=E3=00=A8= =B4=180=18"=14d=FD_=98=B4=CA#`=AC=00)=91=03=D3=F4=BFA=DCD=80=A3D=EA=BE=EF= =7F=83=A11F=D9=FF@=A3=D5=00=A2=01=13=8A^P=9F=C7=A2U=19=D1{=05DI=C4=81D=8B= =BE=C5c(=89|=E6=D0=AA=8EA@~=B5=F1gSh=D5=C7`=A0=BC=DC=F7=F3=18=B4*#Z=1E=00= =94=A6=E2=0B=81V=033=F0=93=C6=01=F0g5=80=E8=B9=04=9FD=EA@=9FU=1E=C3=BE=80= =9E=CF:x=01{J=CF=81g=B9C=D2=A7t=96=0A= =3D[u<|=92=18=08=E4=D9=AA=A3=D9=93D<=83g=AB=8FGO=0A= =03=99;[}=C4=1C=88=E4I"=9E=B1=B3UG=0C=01=00O=12=03=81:[u=0Cu'=EE$=11=CF=D0= =D9=EA#F=00`'=85=81=CC=9C=AD>=06{=05M=DF=03r6=EA=1E1u=00:I=0C=98=AE=EF=3D= b=EAd=E6$=11=F7=D3=BE=F7=88=CA=07=E4=A40=90y=B3=D5=C7 = =AF=BC=D0=F7=B3p=B3=15=C6=A0=AE=BE=DC=F2si=B3=95=C7=F0=AE=BB=D8=F33a=B3=15= G4=1C=E0=E6=CB=D5=815Gy=C1=10=1D=17i=93@;=A0f+=8C=C1=DC=04=9B=14=B5=0E=A4= =D9=EAc`7=B3&=85=81=0C=9A=AD>=A2=D9#j=92=88g=CEl=D5=11=AD=0E=A4=F9=8C=81= =0E3=E70=A6=B5=05\=05C=F4}=82L+=87=97=D3v=C1=CC=D6=00=A2=F9=0A= g=D2x=88=A4=D9=18=E0=18=D0=CD=A8I=A3=0F=B0=D9Z=C0=D0.=D0&=89=07=E0=CD=D6= =02b=16$=E0=A4=D1=07=E4l=0D`=88=17=98=93=C6C=A4=CE=D6=00=06z3v=D2=E8=03x= =B6=16=10#=A1=90'=89=07`=CF=D6=02b=1E$=F8=A4=D1=97g=F3=80c=E8W=13=CE=83D= =A0=8D=01=81=01`K7=0F=0A= =84=B6=16=10#=A9P(=89=07=E0=D0=D6=02b$%=10}^=7F=1E=8A=B6=DA=88YTX=94B=1E= h=B4u=80=C1pwe=16=CC=05=D2V=1F=D1=85=85H =0C=14&m=1D = =80wCx=BD=CC=D3_=BA=BDc=A1=D6g=EF=97=AF=C3<=F9=E9v=15/e=B9ti=AA= =8C=DB=84=BEu=15=0B~M=A1=0A= =08=E9yx=3D=14M=DA&>y=93=05T=15=D0=E3[=AFzq=DE=18{=BF|{=BB=0A= K=83=F0n=13=AE=1D=F8=3D=DC*=F0=DB=F1is=B7=BD=FF^=1F=84=A9W=D6=7F=FD=F2qs= =08=FFbl=19=06:=FC=B5=D8/F=B8o=C3=B0=BA=14=86=8F=DB=D3=ED*=FCa=B0=CBP=9E= =9F=8E=A3fh=B4V?=AFy=FA?=E1=E5=B2=DB6=0CD=D1/=F0?h=99=04=B0=C1=97^@7mZ=14= -=12=A0H=BD3=B2=90e=06u=A1=C8=82=A4=D4=C8=DFw(=F11c=D3=CE"FF"=A9=AB=AB!=E7= =8C=D9=1C=F6r=AF=BB^=0F=BA=1D=87[=88=D5=CD=CA=0Dz|=F7=C3=AB=DD=CEd=BC=9D= Qu=9D=AE=FA=01=BF=F0=EA=9Cp=CD=0F=B9=A8=94=08=98=92=A5= =11=AE=81V=E4=D0=FC=9B=D3=E6=9Ck=85UN=FA=B2=F6=AAA'=FD=D0d4=CF=B8]S=8A(=FD= =DE?=DD{=DC+=03=D02=9EF=CC=A8=AB=A6~k*C=E3=AB+=BBH=91=F7=FC=E1=B3Q=15!=1B= =95U=C3=88=1Ax@=F3=C1=E6=A7\=EB=0D=97=D2=EF=A4T=F8=EE=0Cg:`a=AD=077=9E=85= =BE@=B0=CCu~=B8K=E9=F5K=A3q'=14=A58E{=15Oq=12=DA=19?=B8L=9D=F5=F8=01=F5=A1= =1D=F6=C3=A8=DB=DA=93s4}S=B2=BF=CD=86=BD=96=06=E2$=0D=A6O=14N=19%]=DE=12= d>t=F0]=FB=B3=B4=B1=E4=9F=9E=93=BF=CD=9BI4=17i=10]f=91s=B7=D9=8F=BA=AF=C6= =B7^[r=E5=BE*=F0=92)=F4,=9Bp2=9Fs=C8=CC^=EBa=04=1C=82'JQ=7F=D5u=AF_=01=9F= =E0=CA=178]~U#,=DD=AE=DA=ED=C5r=93=958=0Fr3=EE=DBz=C1=A6=0A= =F4=F4=1D=0C=82=F2sL8K=1Ea=EE_=F8=FB=99l=9EY=B2K=1690=94=A9P=00B=AF=8B=14= =1A=C1=E9=DFf=F1=DB=05=99=A9Y=D3=02=D3O=AF=17/=8B=FF=02=0C=00=EC=9Am=0F=0D= endstream=0Dendobj=0D110 0 obj=0D3768 =0Dendobj=0D111 0 obj=0D<< = /Filter /FlateDecode /Length 112 0 R >> =0Dstream H=89=CC=97=DD=8E=1B=B9=11=85=9F`=DEAw=BBAv=B4$=8B=BF=17=01=12;p=B07=B90=FC= =02=DA=99=96=DDY=CD(=91d=EF:O=1Fv7=8B*=B64=3D=A5u=01=1B=18=B0=0D=A8=D9=E7= =90=A7=AA=F8=F5=9B=0Fwj5=FC9|=BC=D3=AB~u=F7=E3=FBn=B79=F5_=BA=B7=FB=DD=FE= =D0?u=A7C=FF=B0:=F4w?=BE3+=BD=FA=B0]=DD=AB=B5=D2=DE=FB=D5=87=87=BC=EE=C3= =AF=C3_=87=D5]=1A_=93V = =D6=C1=AD=9Cq=C3?=1F=9E=EE=BE=7F=DFmw=DD=C3=A9{|=F7=A7=0F=FF=CA=AF=B1=D3= k=EE=FCZE=80a=F5=E3=DD=F7=7F=99~+=12wj=9D=B4=0F=E5=B7C=F7=A5;=1C=BB7_O=DD= =9B=FEt=9C=9E=84=F2d=1Cd=A6=E7=FE:=7FG6=89=EFx=D7=AERk\=F4=F7=E9=07_~=B8= =D7auo=D6vP=1F~=D6=D3=CF=A1=AE=CB=96=D1=F3=9F=DB=A5=CDo=BF=DD=B2=AE=EE`\= 5=9C=AF=B5=16=86=F3=CD=874=FE=A8V=E3=7F=B4 = =D9=B8S=01=CF=D6424=99=9A=87=06=BB6=C3";=AE=F8=06q=EB=A7=17U=F5=C8Pw = !=ED=1B=DD=C4=D0=0Dz,=C3=97=A4=E7Q-=89G=DB=1E=BAV=0C=FD=B4|=EA7=E8=1B=15= =1Au=F7=BA=BA=D1^j=F7=C6=A4=D9=EE=3DC=1F=92=D8=EE=9Di=D4=03C=DD=8Beo=C2<= {F=CD=9B(=97}j=B27=8C~=07%=96=3D=E8Y=F6=C62=F4=8DX=F6`=9B=EC=0D=A3=F2=C1= =89e=0F~=96=BDaT>=04=B1=EC!6=D9=03c=EA@=12=CB=DE=AAY=F6=A0_=D7=B7Z,{=0BM= =F6=C0=A8|k=C5=B2=B7n=96=3D=00C=DF=8BeoC=9B=3D=A3=EFl=94=CB>=CD=B3gL]=A7= =C4=B2w=A6=C9=DE2*=DF=81X=F6=CE=CE=B2=B7=8C=CAwN,{=E7=9B=EC-=A3=F2]=10=CB= =DE=C5Y=F6=961u]=12=CB=DE=EB6{F=E5y=B3=94=FD=BD=8Bk=3D|=08=DC=EB=B5=1B>8= X=CC=A7`=3D=F4=C1=F0=F7h=83=83=BA=1A=86C=B0=F9=00=8B=F9o=A2N=E3=A9>=07ya= =FC=E4=92=D2=B7=B19=01=C7=81^=97=16=E5=F9=BC=1F4=95f=CC^=1D=F5zy=EF|=F1=D4= f=EF=18=C3=D7=E4=DA=13=11=CF=E4N=A59=ACm=BC=D4=CE=0D=CC2=E7=C0=B6=15=CA<= S;=95=E6pv=10=CB=DC=C46s=CF=18=F9& = e=9E=89=9DJs=18_=8Be=0E=A6=CD=DC3=9A=0D@(=F3L=EBT=9A=F3u=E1=C52=870=CB=9C= =F3y=11=973=BF=05=EFS=93:=A3=D3=ADz-=F5[=18oBu=E2=80=D1q=D6,=07=7F=8B=FE= =04=EBg=FD=C0!|=F7Z=FA=B78=98p=9D8=E00~=10+=80=02=ECD=9FC=F9I=B0=06=0A= =B2=13=07=8C*tZ=AC=06=0A= =B4=13}=06=E68+X=03=05=DB=CF=0E"=E7[=C3=8B=D5@=01w=A2=CF=F9=D6=88=925=90= f5=10=19}=E8=95X=0D=14x'=FA=8C.=F0=B0\=03=F7n=DC=D4@=FC=B7=E0~T=93=03F=17= = =EC=E7=FC=8C=10=EC=A3:=E7S=A3=A0=BE=90=FA=84=FA=A8=CF=F9=D4=18@=7FA=FC6=D0= /=C2=89=F3=851a=BE=88t=A2=99'F=D5=8F=90/!=3DB>=0A= s=BE.&=C4=17=91=06=9Aub =CF=08=F8"=D2=9Ed=CD=F9=B0 = bY=17=BCGq=CEgM=12=CAz=84{=14f=B4vA{=11iC=B3=D6=8A=D1_`=CDk=83=E5=06=B8=B7= =13=E6=A0>=E3j=03/Tl=105Q=E6|X$=F7=DA@g=8B[=15=C6wU}F=9BY=A3=04G=BA=05h=A2= =E7|]=D8 = =A6=EER=9B<=E7=D3"H^=A8=99=AD=89<=E3F=B3=19=83=C4=D4=9D6M=FE=9A=03uF=F2J= w=CD=98=D7=9Aq=BB9g=C4=D4}=DB=F9=9A=F1a=E5B=14=CC=DF%=D2=FE=9AC=94=CA=CA= =E5=EFu=DB=FF=9A=D1~=1E^=E9=FF=81i=F3=AA0Rm6s=0B=D8z=87=A0=AC5=E3=F2=CB=0F= =E55F=00k=A79@=E59tk=A7Q = =E1=C0=F9F=DEp=18=D3=8F=A3@B=3DNs=80=1A`=8C=02=9D=A6Q = =E0=C0=A88=CB=DFp@=D7=18=19u=B0=F3=FC=0D=07wm=94=CA=7FdN*=CF=01=DE`=85=F2= 71\=E4=CF=18=05=A0=94T=FE=A0=E7=FDo=18=FD=0FF=A6=FF=01.=FA=DFp=10=D8=89=F5= ?=F8Y=FF3P=00=82T=FFC=BA=E8=7F`=0C = =AB=C4=FA=DF=EAy=FF=03c=00YX=E8=7F>=05[=DB*s = =D4-=B7>_<=A8=8B=A3gT=BE=8D=8B=BD=CF=97O~~=EE=8C=C2wz=B9=F1=D9=F2=CE=98y= =E3=01=A3=F2=1D,=F4=3D_=DC=A6F=D9r=E8=CF=1B=A1=9Es=C1=CD=83=B7=8C;=C7=C5= (=D5s^=E9Y=F6=96q=EBx-=A4n=C2q=D8=D8I=F6=FFH= =A6U=9E=03=C6A=B0=FF=11K=AB=01F=FF#=93=CA8(LZ=0Dp=A0=18=96=FB=FF6 = =AD=CA=8C=C9=834*"^h=B4=EAsP<=BE=D6=FB=B7=A2hUgL=1E=E4P = y=E4=D0=A2o=14=87=83a=B9=EFo=83=D0=AA=CC=C1=DF=89@e*=1E = =B4=1A`=F4=1C=E2=A7=8C=83=82=9F=D5=00=A3=E7F=F6=14R/=ECY=E5=19=8D=87=E0=F9= =A2=83=06<=B3=99[=D8=D3=00=D2=ACQ=8C=FBo=C4O=A3=C4=D8=93=CAs=E8=B7=E0=A7= =84=83=81=3D=A9<=87~'=FC=94P/=ECI=0Ch=C6=14@=FC=14pP=D8=93=1A`=0C=83=11?= %=D4=0B{Ry=C6(@=FC=94p0=B0'=95=E7=C0=F7=84=9F=12=EA=85=3D=A9=01=C6-=84=F8= )=E0=A0=B0'5=C0=E1_#=D3=FF=C8=9ET=9E=D1=FF=88=9F=12=0E|=DB=FF=9A=03=DFA=AA= =FF=91=3D=89=01=C3=E8=7F=C4O=01=07=85=3D=A9=01=0E=FF=C2B=FF=DF=C6=9ET=99= =03=BEn=B9=F5ofO=AA=CF=C1=DF=B8=D8=FB=B7=B2'Ug=C0/=E2=E77=CB#{R}=0E=FE=C2= B=DF=DF=C6=9ET=991q=0A= ~=0A= T<=B2'5=C0=989=88=9F=02=0E=0A= {R=03=8C=A93=E2=A7=84zaO"=0F=8C=96G=FC=BC=EE=E0[=D8S+=04Z=03=8C = 0=B2=E74=00=A5=F0=93:`=CC=00=C4O!=13=03=81R=07=8C9P=08T=C8@=81P=EA=811=0B= =10BeL=14=0E=A5=1E=18Sa=E4P!=03=05E=A9=03FW"=8A=0A= =99=18h=948=B0=0C=18(4*d=A0=00)=F5=C0=F8=1E@ = =951Q=98=94z`=0C=A8=91I=85=0C=C0=C5\=B0=8C=C9=84X*d=C2=B7s=C12&S!S!=03=E9= b.X=C6lB8=951Q=F8=94z`=CC=A6=91O=17=0C=DC=86=A8T=9CA=0A= =88=A8"=FA=85R=A9=05=C6T*=94*=E2`=02Ub=C01=86=12=82=AA=84=03dUj=811=93FV= =15=D1=1Fp=95=8A3=86Q=C1U=99=1E@b=A5=1E=18=E3=08=89U=C6D=81V=EA=811=90Fh= =152P=B8=95:(=E3H=9D=1Dx=93=F4=E4=00=17=A7)=C3q=A2=C5=BA=F6=9F=DDo'=14=CC= =A79,=C9=FFS^e=FF=BF=AE=EE=F4:=DA|=17O=CEV?=0DO=8E?=C7=F1=E5=E3"=AF=DC=F8= =A8Z=E7MZ|=F4a=B3{=F8=BC=DB=9C=BA=B2=C4D=E7q=89=0D!=8DK=A0y=FB=E9=13>=8C= N=C6=F7=EB=80V=1CD=AC=D5=D5=A1{=DA=F4=CF=8F=DD=A1,=B1N=D5%=DE=9A=C9=92]=1B= P=11=97=EC=B7K^=F4Z=E7=F7_z=D1=EE=BCW=18E.=BC<=F6_=FAc=BF=7F=FE=A1,=81h=1D= .q=C1=C7=B2=D7=E0=93=C1%?=7F-=87=AE=95=CE=AA=D3=A9G=A3]y}~=85=C7g=FF=F1= =C3=D2=1E=F5Z=E5=17s=F7H=9F%=E7=ED=92=AD=E7=ED=F4=B5=F3=FE=F7~=F7=F5y=FF= =D4ov(=A0=AD=A9=02=CA=E3=81=D3@=F7?=9FrF=DD=E3=F2=B1=D0=15=D3=B1=0CoW=D9= '=9Ez=CA=E7V=8EE=85=80=CFn=1E=1F=FB=E7=8FX=BB=F8=F2!=B1d=A0=BC=9BF:}=DC=8D= [4)a=A1C~=E1e=FC=FA=BB#=9A=B6=AA=9E=A3=03=B8=E2=E3=B4=FF=3DG=0E&=85=FA^c= =AE=9C=F8=D3=FEx=C2|B=885=9F=A0=A6=A7M=E3=E2=D8=7F|=EE=B7=FD=C3=E6=19=17= =E5=C6<=07=E4=8C-=01=B5=A1=1E=FBS.=DC=E3r=13}c=81=DD=8F=1D=AD=EB=0Et=B8=DA= EC=85=D5=FD=C2=B9=1E=BD=8Be=BF4=A1=F7=DDv=D7=3D=9C=BA=C7w=D8=A7=B1=EE=16= =82E=EBt$m=B6=A7:.=F4=F0=C5=84=8F=EB=F2=B8i=DC=1F?=F5=DB=13=A9=B0L=AA=0A= =9C=C7=1BK{HW=8Aj{=D8u=0F=BF=D4T=DCy=97.=E0.= iE=1D=BB=FF|=EE=9E=1F=BA=F5=C2=0Dj=1B=F3?m=97=86a=9B=F9=D7=FD=E7y=DA=D7.= =E5=DC=F2=DF=D5A=A1=829?=9E=FC=95=C6=D9=EEw=BB=FD=AFK=C7m=9A=9B=8A=8C,=AD= =CF=F5=A4 = =16/=ED=81<=9F=86=03=C1.=0B=E7y=02=BE=CC=13h6=B9=DB=EF=EB=81=E7G=AA=99=EC= l2S=BB=F7=B4X"=97v3=97=D5=13=B6=D6=C7+=D5=91=E3=DB=0C]=B5^=F2=DB=F6=D6=DF= =F2=AD3,Y=9A=98=ED=0E=A7=89=F9=D2=1D=D56=D9=1FuG]G=1F=89;=8A=8E=9B=FF=97= ;=EA=05=D0=B3=13=D0\Nl;=D0=FB=F0=F4=00z=8F=DD=F3#=864=E9=8F!YDCh=1A=A2?=D6= =B7=9F=9B=07=E0=DA=88=CF=C5=D8=7F=D9=EC=BAz8=D7bm=CF=E6<=BF=B5S=95Rs=F3=E8= +=B5=D5?=E7s=DC=EC=FA=FF=96;=ED=A5=F3=B1/M=DB=0C=86=BA=EC=16=0C\=EB=A7=B7= =EF=DF=E2v=13=D4=E6=CB=0F=84+=F7=EB=A1=FB=D8=1F=CF7=F2=B5=DD=C2=F5"=9Ef=3D= "=B9=BF=D6H=B99=16=9B=BA=F5M=9B=FA=E5\iXS=AE=E7I5=DD=9C=E3=00=BB=08=B6=AF= =EFU=E5=A3=EC~=DA=C4=95g=9F=F6=8FK=D3=A2N=C3:'4=9C=C1*=DF9=F8N?=AC=C0=C9= y=E8O=9F=9E=BAS=FF=80=AD=0A= =B6=9E=9F1=E1=7F=BCW=DBn=DBF=10=FD=02=FD=03=1F%7b=B8=B37=B2@=81=C2=B7=A4= =86=8D=06=B6=1F=0A= 8=0E=E0=DAt=AD=C6=96=02I=8E=95=BF=EF=EC=95=CB=95=CCl=D0m=1FlP=E4=CC=CE=ED= =EC=CC=19w=EB=C2=AB=FA=F20=BB}=18J=07=F4L=FC_0=C73}GBZ=B1k=03=FA=E3=F7s=04= =B8.=FE=DBc(0=DF=F76=DFBt=D7[=89WES4=B4=A0=04=8B=F64=1A=7F6*=CC=A8(=AF=89= p=D8=FB=A5w\=FF=DB=D1=E6=CBbn=C3y{L=AD=08RO=E1E~=8D=D5=BBoW=E3=80=C9V=E3= 7=93)=A5=08=FB=F1fr}y=12=9C'K|=EF=F2}=18=BBZw=DF~=8AmI=97Gb=BE=08=FBeJ=04= =AE=9C=A87=85=92=A9p=94=0Cp=11=EB;=19iz=E0=F9=8F=A4)=F0=EB=83_ = =CF=0C=C6=C3=D8x=ED=E5=06r5=EE=93=FE=CE =D7=AD=F6^=8B=FD=0A= =F39=ADU^?M0nQJ=C5=BEM=86=DD!=D0ej+=87A=8C=A7fFwz$=FC:=EC=81=B1=FC=C9=BA= =F29=AC=F7;=F3=03=0B=81=12=98=A1=E7=C7=E7U=DFAuU=14=80Gu=98=D6=8Fq=C5B=AC= =AB=AB=D1e=16=FAY=AF=82=AC_!=A4=A6=04=E9=E3=F8=FDVV=88x=3D=BA=F0=84=F5=03= v=81)=D7=A1=E9'=DD=CC=E6w=D8=EF=ED=0B=9C=D2=F6)=90R=13v=A5= :q=E4=05=F6=EB=BE=0F=BB=A2=18=9FF=08=E75=AA=85=F0=DE=18=019pq=C4=AEo=1B=DF= =ED=193E=10=A6=B2U=A1=1F=08=A1%=F6H`=08e=DDLh=CFP=AF&=AE=F5=10=D2=18=1D=AA= =AF=D6=D3=BF=F4=007=D2=CE=BCL0=8F=13<=97m=AEq=DE=99'$=C1=BE=C0=D2e=F3@F=F9= '=90=E0AM=B3=D9ox=94=01=F6}=FBP=F1|=19=00RG=19=E0 = =1E@6=0C=00=8B1=D0$=D8=E7=191=00"=C2=00T = =1E=C8l=18@=1E=D8=CF=00$`=10=9A=8C=18@=FE=16e =A1=0D)=FE=98=CB>=8D0=00 = =18D=FA=971=03<=C6@B'=A4"=1B=06=A8=8C1P=9B=D3*?=C3=99j=BC=DA=BC=D3=D4=0F= =9C=05=B7Xi=CE=17/=CE=18pj=E972|a9/R^=BF=0D=FC=E6=F69=D9=F8e=80Ip=CB@=C8= =A5=E7m{7@=BC=A1=B7=05=04=C4=1B=A0=DB\d=B5k=BF\=1Ar6=B4ZB=CFk=BBZ=EA=D3=B9= _=D1=80H=B1cE=EB=E8=C3=D0=D6=F0=8A=F3=1C=C0;O=9Bf=87=F3=F7=8B=E5=D3=B0=E7= =A1=B4=F5\=BB[=D7=B5+N#vy~|p=E1O=AE=99?=B9FO=8Dp=B8=1D=AD=1Fn=063HveP/[=C4= =AF=A1=04v=B91=BB8=B8p8=A1=82=FBjrn=17@=A4I=82x=F1=D5=97=D6=AD=8B=98T=BF= t1=B7=88=96=04=0F=F7=C2=0F=8B=97U=F2=C6e=06&=B8=FB=E6I}=B4Y=88=90=BB=0E-= `=CB=F6k=BB\=B5=FB=DF=D6=ED=FEl=BD=EA=F3=C3=BAc=F5=83=9B=C5=16=AB=EC=AFZ= =DD=D2${=84=B2Ot=B2=13J"=99KS=02=A1=B0=8C=92H=92=85N=06=C6E=1A=9F=CCb=D9= =90=C9=C0x=9D`=9C[=1A=98=C1=BEh=CA=9E=F9=04=1EC$/3Y=AF=EB=A8=EEI\=BA=C9=93= { q=EESx4@=95)z=A01=EAI=02=F2pv=E7=B1=CEy=1C}=02{=00=91=AB=F6 = =B7j=9F=C2=A1=EBz=10=FA=BE=FD}=9F=3D=C6=C5O!=D0=94=90=C1=9E=93n=1E=E2=DA= 'q7J3E=CF=E2=E2CB=DF=A1=D8w=F2D/=E2=DA=D3=EA=D5=81=1A=CFR=C6-=DB=B4T=A3=9B= =A1=90:C=8F6_n=E6w=FD!H=CB=94=C9yJ!=B2=D9=1F=7F=B0=E3*\=85C=BF=1A=BF=99L= q=A0=96r|=B6=B8{~|^M=AE/O=BA=03=15_=D1Q=1B=A4=99=C3?=02=17=AF=E7=C7=D1)=D8= =9A=EAu=E7=C2=E1d=CA=ABj=FC=BEo=0DYS=10=DC^?=06R=12=FF=ED=0A= =19=FA=BA=C53jZ=D21=12=A0=C9=14=151=08E=CA=EC[U=0F=FB8[=B9'E5=91=83=BB_=CB= =C5=D3=0E=CD=8E=EE=FA=176a=91=BB=14:=90=ED=BD=1E=EE=F84=E21=BC1=D0=08=C9= =CC=E6?&3=E0=97=9B=84=A9=E2=C8=0CdY=CC4=9F=E9=EC=D3D>=93=C9=B8=A54=9D}=9E= Ni2=B9`XM=E7ABo=B5=AC&=93=03=96=D8t=1E=90=84=E9=A2=89M=1E=07=1C=B7 = =1CH@=81=E56=99\=A0=F1=3D =AC^=D3=9BL=0EX=86=138=90=80D=CBp2=B9 = =B7p=900g=1D=C9=19=F0=E1=87yN=D0=8D=12(=B6=E39Y<=80=AD~=98=D0=10=1D=D5=C9= =E2=01=8B=81=00 =97=C1=B1=9D,=1E=88=18=07=90p=17=A8=CC=88=83f=0B=07 = =97=81U=F9p=C0=B6=E7b=C2=BA=C3 = =1F=0E=18=DD=C2A=C2=CA=C1X>=1C0=1E=E3=80=DA=DBXy=FA=075=B7=0E8U=FD=C05Wn= =BC=A2=A2O=A8=89=0A= =15=80T=0A= =CA.=10=FC=FA=A2=B8=1B=AD=A4=E3u=C5=ED=A2=BD=BF=9F=DD=CE=DA=F9ze=95=187<= S) =06F=CB=85Q,=EE] = =D8=8Bj%=A7=EC4*3=FA=F0J=8A=C6=C9Z=1A=AE=A3F=DF=AD0P=01V=B8=01=EA(j1=9B{= Y=EE=0F=06!=98=91ub=0F=ED=E6g=93=17=E2=F3B=9A=8Ai=05-:=C5=01=C7)=9AcxGD=03= Z=EF=B2]=ADg=F3=BF=0E=CE=0F(=DC=1E=B6=B7=C8+1d|=B3=FFm=DD~=B8Y=AF=DB=E5=BC= =9C=FF=19g=9C=D6=8C=9Bb5=E6,=E3=85=EE=D4G=97=A3J=17=E1=FC=9D)=F4KA=AA=E2= =0Cu=FF=C6=BF=93=E2=EA=BA*=EE=8A=91=A4=FA=AAr=E4=96O#=0E=A5y|=1C]=B8=1F=1A= C=06)=EA=DF=B2=1D=DD=8F=FE=11`=00=EB0N=CA=0Dendstream=0Dendobj=0D112 0 = obj=0D4619 =0Dendobj=0D113 0 obj=0D<< /Filter /FlateDecode /Length 114 = 0 R >> =0Dstream H=89=AC=97[o=DBV=16=85=7F=81=FF=83=1E=A5t=C4=9E=FB=05=98=02E=DC6=99=A2}i= =FC=E6i=015f=1A=0Dl+=90=956=F9=F7C=9E=9B=0E)[\=846=0A= 4=B4xY=8Bg=EF=BD=F8=9D=D77Wl=D1=FF=B7=FF=EB=8A/=B6=8B=ABo=7Fk=EF7=87=ED=DF= =ED=F5=EE~=B7=DF>=B4=87=FD=F6=FDb=BF=BD=FA=F6'=B1=E0=8B=9B=0F=8B5k=187=C6= ,n=DEw=F7=DD=FC=D3=FFo=BF=B8=F2=E11~=E1ec=F5B=0B=DD=FFs=F3p=B5=FC=B4=BB=FF= z=BDk?|x=BA=D9=BDm=BF=ACn=FE=D7=3DK=C6g]=B9=FE=AA=EE=01wW=CB=EF=E3=99=A4= r=C5=9AN=C3=A6s?]=BF=1B=DE=C7=CB}=B7=CB=1FVk=A1=D9=F2=ED=EA=F7=9B=9F=BBK= T=B9D=F1=F2=80W=C3=87=F3=86=97=87=DF.=B7=EF=AE=DF=FDg=B5=D6=8C-=9F>=B5=EF= =BB#'=1B=B9|=DA|}=CA=C7=1B=9B=8FZuz=B4i=C7=DA=A6=11z(-=CB{u=F7=C8t=EE=97= x=CE=A5sk=99=DEz-=A2=F9=FE=1A=17=AF1=CF=DD=BF=E9=CF=F5=05QJ=C9=BE = W&.=0C[=84=03=CEd=D3=15=83=89&=D6=82=9B|C=AE`=A9=1B=E7aM5S=FD=95=B7=CB=7F= =AD=D6R=F6/g=C3=BB=9D=13=E9j-P=15)=9EWi'U=94=C1U=B4j=9E=13Q=93"=C6=81=0A= 6=B9=99=FF=1E=CE=C7=15@T=BC}^e=F2E=04g=B0=8A=E0=FE=19=89=CD=A4=84=14ho = =C5=06=0A= ]o7vz=A9DW=C6gJ^=E6=E5y-S=F7=F1=F2=DF=C3=C1_sW&=AC=1B#=19=A6=E8=D7x=8D*=13= =E6=8F=C1=F1=DD8=95=AAs?~=F9=B4y=BC=1B=8Ew?o/=E4=D9qno=97_V=EB=90=1C=7F=AC= =D6=DD=1Bt=8B!=C5(B=C41=19_=8D=9F=14=9C=A7d=1C{=AF=CE}=F3=F2}=B7!R=FBZ=98= = =FF=EB=EE=EE=F3=FD=E7=A7=A1=87nycYl=1D=A5=FF=15=DA=0C=1E<(=03OW=89=97S/$= =B6}6=B1=3D=98=D8=87=8Fm=CE=DE=C3~=F3=F8=F4=B0=3D=1C=DA=BB=F82f=F9=D0>=3D= m=FE:=C9d=85=07=B2)=81l=A3=A7:=90=D3'=CC=9EYr=F3=DC=B9/Sa=CDe=DF=EE=AAk`= =11=FB]=0C=94^=88m=1Fn=D2&=7Fo/=B2 = E=AD/=01}=E9=C8=C4u=C8=ACJ_=03=FAQ=9B=CA=82=F1=CD=C0=81=05=1CX=DD=D0=19p= n=D4=03=9C=01=16=98=91=CBZ=8C;A=02= =9D=A0=E5d'=CC=F1=A0|m@=01=80=A0=F5d#=CC1`=F9=90=92=14=90=08=DAMv=C2=1C=0B= ^=0D=0C=00=81`=18e#=98n=0B;\=03`=1A=8C=98=E8=83nO=93=B6=A7=FD=0A= C=C0=C8=FA=1D=A7R6=9A=00=9A1o3=94=CC=D6/=02Va=8A=BA=86=F6=0Bq=80=88=D4=95= =AB=DE^=03=9FF=AE=FDYq8=8D=B8=E5Gad=A3=E2=C2=D0=90H=FB=BA=E6=1A=E8}=C1=14= =8D=B4=E0U=B9=81=8E=17=C2P=BD=B5=90u=AD=0D=F0=F9=13=8A=A8=D6=C2=1Ckm=00=06= =15=96=AC=D6=C2=D5=B56=C8^=CC=13=D5Z=B2c=AD=0D=D0d=92=93=D5Z=8AA=AD=81T=93= =92=A8=D6R=1Fkm=81O=AB4d=B5=96=B6=AE=B5E=B6[=EE|=ADgD=A9=F4=C7j[=A0=C7=15= =9B=AA=F6=0Cq=C5}=FC&%} = =C8=958_=F09=EA=8A=C7=7F=92:=C2=B5z=AA=EAs=F4=8D=1A=BC=3D@=B5=CA=92=15^9= 3x{dg=E3 = k=AF=D9=A0=F6=0Eh{=CD=C9j=AF=E5=A0=F6=0E=E8|=AD=08k=AF=F5=A0=F6=0E!iCV{m= =07=B5w@=CAkGY{?=AC=3D=C2=D0=8C=AC=F6F=0Ck=0F=CC=9D=91=E7k=BF=D6=E1=85z|= =EF=8C=CC!x)=F2=9E=C0=CD=80=F8.=04I=08=FE=A8=EE=81o^=86x=0A= =F5H=F0=95>=B2=89=D0=FEe=F1y=04_ = #=BB=87=08=F1=97K=FBQ=CD=3D0=F7=01=E2/=96=0E=04_ =03=CD=96 = =FEri9=AA5g=08=C5kq=B6=D9fP=BC=0A= =0F=AA=F5!=98wT=A3&<=1F=CA=03=BD.=99:=B7=F8s=08=8F=DB=F0"=B5=01=A0=E7=A5= dT=C3.=95=1C7=00=90=F8R[=1Au=E3O=EA=8F=EC,=1CY=D4=06=C6=AE=E49=90=B5=AA=C3= #=1Au%=C4=B8=FE=1C=98?E=17=F6j=1C=F6=DDg=0C0`=04=8D=BA=3D=99=7F=0E=C4=AE= rd=F3=AF=D9p=FE9=D0=FD=9AS=CD=BF=16'=F3=CF=01=E4=D1=8Al=FE=B5=1E=CF?=07=06= P=1B=9A=F9=D7=F6d=FE=050=80=DA=93=CD=BFa=C3=F9=17=C0=F8=19~f=FE=D7=DA6=9E= w=EB=D3=03=E7=1C=DA=E4>=03,=17=08p=F1=AE=04\7=9737=17=B1=08=B5=01`=08y$n= "=0F=DA=0C=0D=00=DB=1EnB=19h=F4]=0C=E2=DA=02=90=04=DC=C7 &=F1 = =98=1B=F5=81=04fA=08A=A5/=D5=B8=0F$=02=83=CA=D1=F5=810|h=00=98=04a=15Y=1F= =08g=C7} =11 c=8C=AE=0F$=1F=E7=81=04=C6Q=0A= =AA<=90=F2$=0F$0=8ER=13=E6=814=C3<=90=C0WQZ=BA<=90=FE$=0F$=F0]T=8C0=0F=14= =1F=E7=81=82=D8=F0l=1E=C0[=B3=EE=E7=A16=B0/Sz*=0A= py=CB=C6=05P=08=98=BA=89,=C0=0Dx3^}`=0A= 5=9F=0A= =02=D8=80=16b<=86=0A= =18C-=CF=E6=00.=AF=FCP=1B=01c#=C8&P[}=D2=00=08=99v[=03=B2 = 4=8C=8Fz@=03_d=C3=CF=E8=F7x=A82=1EvV=E6=10"=CB&=80=18=08x=D8=D9=17t|X=E4= =81$=C8pH=E4=A0=A7=C3"=8F=C0qDC"=F5=C4=86=C5=00=00=04=19=0Ci=1C$2,=06=80= =0A= XH=A4=9E=B8=B0=C8=03I=90=A1=90=C8AO=85E=1E@=81=84=84D=EA=89 =8B=01 = =862=10=D28HD=98=0D=18=84H=05=D9=FCg=1E,=F2=08=8Dj=CA=F9=0F4X=E4=11=16=B5= =84=F3=9FY=B0=18=00=FA?=83 = =8D=83D=82=C5=000=01=01=03=CF=A8=CF=E3=C0=AClg@ =89x=A2=C0=A2=8F# = =89|d=C0=A2=0E=CC]=06@=0A= =F9L=80E=1F=01Py~=EE=E7=F1_Q=C6=E1=8F=A6=E33=FDe=03=0E=E0=AE=8C~4=0E=12=FB= =15=03=00s=05=F0{Q=FD=02=F2=93=9E7=D9=07=02_=1D=FBIg=A9=C0=AFVG=D8+=A2=1F= =89=81=8E=FBju=04=BC=02=F9=91=88G=EC=AB=F5=81=F9K=E0Ga = R_=AD=8F=90O=C7}$=E2=11=FAju=00{=12=F6=91=18=E8=98=AFR=F7@=00D=EA#=11=8F= =C8W=EB=03=F3=9F=A0=8F=C2@$=BEZ=1Fd>=12q9=9E{=8F0=97&=9B=FB=9E=F7ju=E0=EB= =13=89=8FD=DC=8F=E7=DE#=BC=C5=C8=E6>=D2^=AD=0FL^=CF{/=8A=CF=82=BD=A3=B0`= =C0=D0%=DC#=D0=8E=ACW=CB#=B4=E9=CE=CE=FCL=D4=AB=C5q=D8=BB\=3D=91^-=0F=B2= =1E=81v=07z=B500=EA=11=F5(z=3Dq^=AD=8F=A0f$=3D=0A= =03=11=F3j}`=D8z=D0#=11=17vTv=0EL=9CQ=FC=CC=C4=AD=0Dkdoa=1D=1E=8A3=A6=B5= =19\=05G!=D3=CA=E6r=D2=CE=98Y=1B=98=C1=994=1Ez=D2=AC=0D=E0=A8I=A3=9F`=B3= =B60=836I<$=DE=AC-=00Y=10=80=93F?!gm=00=08=83=CC=9C4=1Ez=EA=AC=0D = =C8=1D=B1=93F?=81geA=00=91=90=C9=93=C4Cb=CF=DA=02=02=BF=82*=0F2~=D6=06=80= @=CA=FCI=E3=C1=0C=F3@=00=81=94=10=94F=DF=9F=E4=81=00f!S(=89=87=C4=A1=B5=05= =04=84=E5=D9<=98=87=A2=B56=02=C1z*=0A= f=D3h=E5@"4=EC&=B2`.=90=D6=FA=C0=14f"%0=90=99=B4v=00=8Ca=80R=0A= =F9=1EKkm=04=88#=97=92t=7F&=D3=DA=02=82=C6 = MI<$8=AD-=00!=10=E8=94F?=F1im=00=88=80=0C=A8/y=18 = jgg=0E=A5=1A=99|=00q=10=10U=FB=EC=9D=82Q=B3=BC=02f1=03*=91=83=9EP=8B<=C2= =C7=11O=89=D4=13=9F=16=03=08=1F'8=A5q=90=E8=B4=18=00=E2 = =A0)=91zb=D3"=8F=90q=02S"=07=3D=99=16y=84=8B#=96=12=A9'.-=06=80=18=C8PJ=E3= Qi1=00=CC=7F@R"u9=9C=7F=8D=10=B1=A6=9C=FF@=A4E=1E=E1aK8=FF=99G=8B=01 = =802=8C=D28H4Z=0C=00=01=14P=F4=8C=FA<=16-=CA@=F2d=10%=11O$Z=F4=11=08wS=B3= ?=97C=8B:0w=19B)=E43=85f}=03@x@P=12=F1=9EA=8B2=82=DF=11@i:>=13h1=80=D0w=C2= O=1A=07=89?=8B=01`=E8=03|=12=A9'=FA,=F2=C0=C8g=F4|=D1=C1%=EC=A9|=E2Ya=80= =CF=7F=A0OE=87=9E=95:=F0=F5=CF=F0Ia=A0'=CFJ=1D=C8=9F=C4=9E=14=E2 = <+}=84=FC=13z=12=18H=DCy=D4=B7@=02=05=F2=A4=10O=D8Y=A9=03)=94=C1=93=C2@O= =9D=95:=90=00=89;)=C4=13tV=FA@=04d=EC$0=90=98=B3=D2=07=BE=FD=81:)=C4=E5x= =EE-=90:=19:)=0C=98=C1=DC[ = u=12sR=88=FB=F1=DC[=84{=18=D9=DC'=DE=AC=F4=81=DC =C4=F9=92=F8<=DC<=0A= ; = p2p^=AE=9Dh=B3=92=07=12'=F1=E6=E5=EA=116+q=84y=F8=F9=81=9F=CD=9A=95<=90w= =816/=D7=EEQ=B3=12=06=82.=C1&A=AFg=D2=AC=F4=81=A0=CB=ACI` = =81f=A5=0FD]@M=0A= =F1=C4=99=95z=8A:qF=DD=CBp=93`=194=7F=DC=EFw=FBx=9FJ6B=14G=13=DF=0D=9E84= =B8o=FFn=F7O=ED=EB=AF=87=F6=F5=F6=F0=14=AF=94=E9Jw=EC=A1=EF=C7=CF=08=00=1B= =CF=BD=19*W=9D=F7=CD=CB=CA=B7=DD=DA=AC=BB7k=EC=F2=8F=D5=DA=C9F.=F5=EA=F7= =9B=9F=AB=07=89=A6;o=D3=A3^=8D=1FUD=DE=0CM=0F=AC=FD=10=D6=9F=BB=1E=B9M=FF= =B0=FE=C7=B7c=BF=D5=0D#=19^Y=B8]nVk=1D=9C=DEo=1F=DB=CD>=FF=F5~=F7=F0=E7=F6= qs=D8=EE=1E=F3O=BB=0F=DD=11c=9D=B7=F4=C3=E6=F1.=1F>}=DC~8=B4=E5=CF~=FD=C7= w=86=A37+=B6l=F2=1F7=1F=B7O=F9=B8=0D=D5N=7F|=DA=1C=0E=ED=BE=DC~=BC=EA=F3= =E3]{h=DF=1F6=7F=DE=B7=ABu=F7=F2=8DY=FE=F3=B1}<>=BE=BE=A5=F3z=F8=D8=E6=9F= =AE=7F=BB=8E=B7=D8=E5=A7=DD=FD=D7=C7=DD=C3vs?=AA=8E=B1qE=07=EB=F6l=11~=19= m=83l:=BB=16=8D=E2=A9&_=0637=EC=D1=8B=18=9E=87A=10=CE=C5=F1B=F6N>=DE!=1B= =82-D=F7Q=CE=DA=C8=CEI:*a=CD"=C6Fmd=DF=A4=13=F8=02=F2=B7=DDB=AE=85f=9D=8D= =BE)=CE=F9=B0=B6=98@6ONR;=10l=D0=01=1C=D9Bqz=17r=E8=02hD=A1@=17=D3=9B=19= =AD=EBf=E0=C8N=CA=E8=86j=08:|=1E=BC;=C0=17=FD=B54=E2iKV=C4=91]T=07u=E4=83= =E5=D0=C7=FFi/=BB=DD6=92#=0A= ?=81=DE=C1=97=B4=B1$=A6=FF=BB=81=04X=C46b,=D6@=B0=F1=9D=B2=17=8CMy=15H=A2= = =CA=88=FD=F6=E9=E9=AE=1Ev7=FF=CED=B5=17=B6(=91=C3sf=AAN=F5W=C8>=A5a=1F=97= =1F=83i=FB=0FY=A8,W=FF)=D7=F6=1F0=08=94=87=FB=0F=AF=01=ADh=D3a=80,7=E2=CF= =F0=A1Z=1F=C8=96=A3=F9=F2H=0B=DB$=0F=8C=03m=B9=F2=A8}=93=03 = =0C=03=1D=F8r`=DA=D3@"=BB=06=FFi`=DA=D3@=02i4=E8i0=C3=85m]=00=90`=1C=D7L= 0=BE=CD=000=13L=F8=13=B2=187=B9=DA=87=02f=82=95|Y=B4=AA=C9=A2=02F=81=D5\= Y=B4=B6=C9=A2=02=B2h=1D_=16=ADo=FAO=01=FDg=03{=0A= \=BB!=A8=D0l=80G]=04=95=A9=CA=8F?=C6=8B>6=1B=E5=19=B10=92=95$=FBo=CA=F39= =C4`:=A8=F6=0A= =7F=ED=F6Rz=C4=EF=BF?=8E=9Be=B3{=8DO'=BF=FBs{=D1P/l=AD=E5=AB!=DB=AC=D6=AD= =FDUA=94=AB=AE=17=EF=D3=EE9,~*=FB=E1=C7=ED=97ow=DFv=EDr=18=EF)=DF=BD=9F=BC= =FCK=1A{=FA=C9=8A=D5=B0=DF=17=E5=E9]=F2z=F1./=AB=1F=BAmT=1CYF=A7=87=B5=12= v=7F=07=D5=9A=BB=FBv_^no=CA=AB=F4~=BE=B7=FB=CDn=B7=FE:}||=D2=F5=87=F2=CB= Mz"=E5=97=EF=8FO=F1=A2=CD=97=ECr=BD=9B=AE-=DF=F9=B8=BD=FB=F1=B0=BD=BF]=DF= uw=A0l=9D=9D7=A7=9F=C1=E2=D7=B6=ED=A3B=BE=F9=A5\=E9=B1TH=F6B=1E;z(=0D=06= @@=EC=CA=B8=C7=B8=E3=D1=9B=DA=E7=E2f*\=AD=0B=1C=C0B=E6=BC=BD\=9A=B8k=AF=0E= =8C=1C=A1=C7=BC=9E=14=9F1=EFD\=C3=DA=87=8E,=A3=C22=A9;=D7=DD<=B2=04=08=EF= =98=EE^=C6E=AC=BB{=E0=E9K1=F0=A8K=D9=DF=3D@=1CRq=D5^=EA=BE=F6=08=F5K=C3=A4= n=FB=DAK = =A8;=B6=DA=87=BE=F6=D2=02;=E0=C0S{%=FA=DA#=A4=A3$W=ED=95=EAk=AF=80=E4)=CD= =A4n=FA=DA+=A0=F3=95=E5=AA=BD=F2}=ED50=F5T=E0=A9=BD=1E=FA=DAk=E0=C4=D1=82= =AB=F6Z=F6=B5=D7@=E7k=C5=A4=AE=FB=DAk=A0=F3=B4=E1=AA=BDv=07=B5=07zO{=A6=DA= =87=BE=F6=06=A0=1C3p=D5=DE=88=BE=F6=06=98;F2=A9=AB=BE=F6=06=E8|=A3=B9jol= _{=03t=BEq<=B57=FE=A0=F6@=E7=9B=C0U{;=F4=B5=B7=C0yo=05=93=BA=ECko=81=F3=DE= *=AE=DA[=D3=D7=DE=02=C9=B3=96=A7=F6=D6=F5=B5=B7@=E7Y=CFV=FBpP{`=EA=B9=D3= =DB=D52=1E=C6i=B3[=8A=BC=07B=BC?=A8=F1~b'=C8l=C2!=CB=86P=C9z=A6=E5=17=AF= =1C=D26=06=80=04=94=15=8D=C9=80=F6=DD3=00b = L8=AB=8F=AF=9BN4=DA=C0=F0=17^=AC=CE=DF=3D=AE=1E=FA=FA=03=03P=0E=9AG]=8A=B6= =F4=C8=AA'-=D7=BDK=D5=D7=1DY=F64S=DD=A5m=EA=EE=91E=CF=B1=D5]=FA=AE=EE=1E= Y=F5=02S=DD=D5=D0=D4=DD=03=C8=A1=04[=DD=95=EC=EA=EE=915S1=D5]=99=B6=EE=C8= =92g=D9=EA=AE\_w = s=CA=9F=AF=FB=9C=3D/=B4=95G@=7F=B8T=F99=B4/B>=BA&=0B=018=F0=B4<_=FCY=CB=96= =C8G=D7=DE=00=00=3D=DA\=EA=809=16=AC=EE=9F=01p=E8h=C7=D6=04=DA=DB=FE=19=00= =13@=07=C6>0=C3A=1F=00=ADh=04[=1F=18=D5=F7=81=18=80#=C0h=C3=88^=C6=F8=D6= =01p=08=98=8B=87=D0=AC=05=ACo=C5=08=A4=C8=06=E6.=F1=1F<=11m=EC=84=BE=10@= =1C=ACd:=0B=AD=B2=AD8p = =94=CD=89E=DF=CA=83=12=00Q=B0=EE|=1A=97y=C8=8C=BB=C8=9CED=A8lA@k=88=8B(=1A= =8A=ED=97=ED =B9 &yd =D1y=0Bbr`l%=0Fd@=D8=B0=E2S=F7=B9 = &=03=C0<=16!/a<=0E=E4=E0=9B=FA#k=88=94l=EAJ=B7=F5G6=91=B86=F2=D5?=AD=03=93= <=B2=888=CDW=7F=E9]S=7F = =C4O=0D=03c=FD=95h=F2/=81=00*=C9=96=FFH=F7M=FD%=10@e8=F3=AFl=95=7F=89,D=8E= 1=FF*=B4=F9=97=C0F=A4=07=CE=FCk=D1=E4_=02=F9=D7=EA|=FE=E1#0=FEy=AF=AC=90= }=C0\=8C>.=EE=86=E6=D1+=A0=F3=B5=BF=94}\>S=F5=A4=8E=D0=97=B8=18|X=DEd=80= =D8=EB=03=E8e=D4=F9=DC=E3=E2:T=CA@=E6Ld%=B6=8E7=CE=B4=85=072g=BCg=CC=9C=1D= DS{=E0=D0=B3=82O]=BA=B6=F4=08uj=C18s=AD=A9r=AF=81=DCY=EB=CE=D6=7FiU=81=DE= hc=06=F7=8A=E0=0BG=0B=0D,a#=FAF=FE=E2=E2=DEF=1Ea=CF=8C=BE,=0E"=F76=F2=08= y&=F4eQ=CF=DC=DB=18=00bH=E8=CB=E1 = soc=00D_=16=F5=CC=BD=8D<=C2=9E=19}Y=1CD=EE=AD=E5=0Dp=FCf=F4eQ=CF=DC=DB=18= @=C83=A3/=87=83=CC=BD=8D=01 = =FF#=FA=B2=A8=AB=83=FC=1B=1C}Y=1C=D86=FF=06=00=80=8C=BE,=EA=E1 = =FF=06=18@=84=BE=1C=0E2=F76=06@=F4=3D=A9>=8B{=1Be=00=00=08}=19=C43=F76=FA= =C0=E8=C9=E8=CB = =9F=B8=B7V=B7=08z=8A=F3=C1=9F=CB=BD=8D>=02=A0=EAL=EEgqo=A3=8C=90gB_=8E=8E= '=EEm=0C=00-O=E8=CB=E1 = soc=00D_=16=F5=CC=BD=B5=BC=03F>=A1/=8B=03=D3=E6=DE=01#?=A3=EF = =F5=FF=9F{=BD)(-=1C=90=BF=C4=BD~(=CE9=D0=B7v=80=B0'=A1/=93=89=91~k=07@=10= =88~=99=0C=10=00W=1E<@`=05=80yL=10=03=D7=1E=00=08K=0C=CCd=800=B8v=00D=B2= `0=93=89=91=84k=07@*=89=84=99=0C=10=0C=D7=1E=10=1A#=18=E61A<\{=00R=99x=98= =C9=80:=98=0B=1E8=1D=0A= =123=99=B0=ED\=F0=00=19=11=153=19=08=07s!=00s=A1=801=8F = b=E3=DA=030=17=12=1B=9F10=0F=8Fkq =0C=05=8FY=F4=89=90k=0B=00=A9=11!=B38=C8= =90\=1B@HM\=1C=08=B39=B9=B6=00=A41q2=8B=FE=88=CA=B58=10DBe=9E=0C=14Z=DE{= =90=03=10=C4B=CB<&=08=98k=0F=08=B1=0A= >=03=C4=CC=B5=03 = =8D=85=99=99L=18=DD:=00=0E&=C2=E6=93=06*r=9E=83=CDN=90=05`=1E$f=B6=AE=D8= =E6`=E6I=1E=C1U=02f&=07#0O=F2@=18=89=96=99=D4=89=96=8B=011=03=95y=1C=10*= O=06=10J=1D9=99I=9D8y=92G=10=95 = =99=C9=C1=08=C9=93<0=02=88=90=99=D4=89=90'=03=08=9A=12=1E=F38 = <=9E=0C=00=F9Ol=CC=A4=AE=DA=FC=0B = =FF=05=8C=99=1C=D8*=FF=02=E1r=C7=98=FFB=C5=C5=80=9C=81=C4<=0E=08=89'=03(= =0F=9FQ=9F=C7=C3=9320y=0A= =0C=B3=88=13=0CO=FA=C0=E8!=12f=91=CF$<=A9=CF=C0`=0E=F9=82=C1=93>=CA=C0,=E2= #=03O=CA8=00=F3t|=01=E0b@=CD=A0_=1E=07D=BF=93=01 s }=99=D4 = }'y=84=BC=89{=99=1C=98*=F7=0A= =87=DE=93=EA=15=F4F=1Bs=B8=D7=8C_=9C}=A0=E4k=14=1B=F6V=EA3=C0=97=C3=C0H=BD= =95:=90~=E2^=0Eq=82=DEJ=1F=E1n=C2^=06=03=C4=BC{}=0D$0Q/=878!o=A5>=03z9=0C= =8C=C4[=A9=E3=CC=CB!N=C0[=E9=CF@^=06=03=C4=BB=95>J=BC=1C=E2=AA=CF=BD=9E=01= =BC=1C=06l=93{=8D=F3.=87x=E8s=AF=81=DC=17=DCe0@=AC=BB=D77(=ED=9E=12=9F=87= =BA=95=F0=0C=D8}=B96=91n%=8F=B3=EE=CB=D53=E8V=E2=C0=C0)=A8=FBb=F5=C2=B9=95= <0o=12=E9=BE\{=C4=DCJ=18A=EC=0C=BA=0C=BD^(=B7=D2G@=9B8=97=C1=00A=EE^=DF=02= =9C=9D0=97C=9C=18=B7RG = =9B(=97=C3=80i=F2n=11=C6=CE=90{\=BC"=DC9x=9B=891[=002=9F=F8V=85=D5=CB=11= =BF=10nm=00=C8}A\=1E=0F#=E4=D6=06=10=C2=CF=94=CB=A3O=9C[[@0=9F@=97=C5=03= =A1nm=01=18C=89uy=F4=89vk=03=C0=1C*=B8=CB=E3a=04=DE=CA=80=03=06=11=11/=8F= >1om=01=98F=05zY<=10=F6=D6=16=80y=90=B8=97G_=1D=CC=03=07=CC=83=82=BE<=1E= l;=0F=1C=B2y8=BEyP=F8=B7=B6=00=8C=A4=02=C0,=1E=08=81k=0B=C0HJ=0C|Z=7F=1E= =05=D7=DA=C0,*=18=CC!O = \;@6=10=7Fa=16=CCe=E1J=DF=03=B3=A8=C00=83=81=82=C3=B5=03`=14%=1E=E6=90=D7= =A1=D5=06=B0=88=90=98=A5=FB=0B=14=D7=16=801X=A8=98=C5=03qqm=01=18=84 = =8Cy=F4 =8Dk=03=C0 = ,l=CC=E3=C1=B4s=C0=03c=90=F0=98G=DF=1F=0C=02OcP=9E=B1=10TzrB=A7K=1E=B7w?= =DEn777=BBO=DB=0F=9B=EF=F9rEv=FC>=14?7_=1C=8D=C6=AFu=F4=DE=C7=FC=9E=DE=8B= =9EdI=D2=F6t=FBo=1A=BD=E3 = =E9=D3=13=CBv=AF=17=EF^/=CD0,>=BC=FE=FD=D3/{=CD+Q=FBy=D3z=15+1=BDw=BDx=FE= c=13=BF"=1AQ=8B=DD=B7=FB=D7=CBx=DD=CA-=B67=E5=8F=E9=FD=FC=C7=FB=CDn=B7=FE= =BA=C9=8A=EB=87/=CDG=F2=CB=CD=D3=D3=F6i=FA=E5=FB=E3S=BCd3}p=BD+=AF=FE=88= =8F=96^~=1E=9F=F6=ED=E7=DB=CD=C3=F3.=7F=F5^|]=A4=C7=B2=C6k=FF=13=FF=FD=F2=EA=FA=F7=E1=D5=97WWN=A5= =F9n=E28=B8=BF=CA=B6=E2=CB=BB=AB=7F=96_R=0Ff0=1A=FF{=DA\=DD\=FDO=80=01=00= =A5=E5=D9s=0Dendstream=0Dendobj=0D114 0 obj=0D5967 =0Dendobj=0D115 0 = obj=0D<< /Filter /FlateDecode /Length 116 0 R >> =0Dstream H=89=AC=97=DBn=1C=C7=11=86=9F=80=EF=B0w!=1Dp=DD=E7=03=02=03=86=14=E4`=C4= @ = =10=C8=85=EC=0B=9A=1CJ=93=90\aw#Yy=FAtOW=F7V=0F=97=CB=1A=B1`@=12=BC=3D=FD= =FF=D3=7FU=CD=D7o=AE=CE=C4*=FF=B7=FDp&W=E3=EA=EC=FBw=C3=FD=F5~=FC<=BC=DD= =DCo=B6=E3=C3=B0=DF=8E7=AB=EDx=F6=FD_=D4J=AE=AE=EEV=97b-=A4snuu=93=9E=BB= =FA=92=FF=D8=AE=CE=E2=B4M\E=BD=F6ve=95=CD=7F]=3D=9C=9D=BF=1B=EE=EE=87=9B= =FDp=FB=F3=C5=D5=BF=D36=E6=B0=8D1F=E7m=CE\^=9C=9F=9F=FE!=8D[=AB=BCG\=97-= =BE=CBOb=DD=A6&=AD=C6Z?=14 = pz&=D6Q:=9F=0D=DE=9E=9Do=87=CF=C3v7=BC=F9=BA=1F=DE=8C=FB]Y=A9aeX=A7=9D=EB= =CA=1F=E7=BB=A0=DF=C8/=A1=ACX?y=87&x=F4e=94=C5=07=F7=FE=FC=CF=17=97V=88=F3= =BF]=FCz=F5=D3A=F3Lb?=DF=F5^=E5Z=04=AD=CBo=EF=CF=AF=1F7=FB=8F=C36m=13=F4= Z=A7=13=98=92=187=8F=F3=1D=F1c3=A7=E9=F5=ED=E1=B7=7F=94=DF=1C=FCv=A9=D5=DA= =E4#=BE=84=BF=F3=9A=DFkZ=CF=9CL=0CS=BA=C2=D7=D4T=D9=D4=9F8=1A)=F2=13fZ=FE= =C7=DE=83(=9BS=94=A5=F4X=D7=10tU)=E7=D7Kk[6j=EAR=13=E4=A1=13^/o=E7=87.-A= =DE=05=A6=B7=0Fb=FE=F6=9E = =1F=E5=C9=B7Oe=A9=89=06=94=D0=F3=F7=0F/=1BP=F2t=FAK=0C(=B7=EE=8B^=10=F4=F5= =E9=F8=97=E8=9B=D8=A9K=82=BA=3D=9D=FE=12u/goO=E8y=15=F8=E2=8F=A6S=8F/=AB= k=C1=96=BD=96=B3=EC5=E1=ED=B5b=CB^=EB.{M=98{=DA=B0e=AF=DD,{Mh}=ED=D9=B2=D7= =A1=CB=DE=10=FANG=B6=EC=8D=98eo=08=9D=97>=A5\ooT=97=BD!T=9E=D1l=D9=1B;=CB= =DE=10=B27=8E-{=E3=FB=EC = 3=DF=04=BE=EC=E3<{=C2=DC=B1=82-{+=BB=EC-=A1=F2=ADb=CB=DE=9AY=F6=960w=ACe= =CB=DE=BA.{=EB=08=EA=9E-{=1Bf=D9[B=E5=DB=C8=96=BD=13}=F6=84=CAw=92-{=A7g= =D9;=C2=DCq=86-{g=BB=EC=1D=01=B5=9Dc=CB=DE=F9Y=F6=8EP=F9.=F0e=1F=BB=EC=1D= =81=F4=BD8=95=FD=A5=F3E=FFR=96=CB =89=F9=13@=E5=19=98=FF=9Cl=10=1AP=CA = =D3M=88=D5=FCk=CEA*=87=F5 =C3=17ni\=FA =A2=F0 = x=CAE=D7=C6=93=F2=F4+=97=97X=9Ar=D9=0C=B9iy=C4c=9F=BD'4=80J=B5=C7"=AE$=8E= =DD=13=E6n=B9=9C=F1=88=EBY=E6=84=AA=CBw3=1Eq=873=0F=94=1B=A6g=CB\=85>=F3= @=B9bF=A6=CC=B5=C0=99=07=CA=FDJ=B2e=AEU=9Fy = =0C=DC|'=E3=11=B7]=E6=84![.d<=E2~=969=013=F2}=8Ci=C0=EA=D8=A5Nh=B5r=1Dc=92= 72N_=AB=E6 =12:._=C8=B8=F4=130=F9N=9F0=E5=CB=95=8C=CB=813=B3=13 = =0C=FA|)=E3=D2=0Fnv=02=84=D6+=D72&=07V=CCk=80r=CF=90l5`=F5=BC=06(=F7=0C=C3= X=03=D6=CEk=800=04=F2=E5=8CK=DF=CFj@=0A= =CAE7=0A= F=D2tBw=06(=BC=FF=E2=F7g=89=BE=9AU=A1=14=84FpF1=D2=AE=B3v=DD[ = t=82s=A7:=E12=DD=C9=8Ct~=BAw,=B9t=04=01=16=08=A5(=A5O#i=BA|=BD=FE=C2=11=CB= =81=82=BC$ = =904=E5=CA=C3=E4=C0:$O=B9o=B8=B8=E6S=0F=AA4U5@=B9u=C4r=E5=E2q=A0D=C0=F9K= =CA=BDC)6um=FA=FC =1D=A0=D2%=91/=FF=E9=06=D0=E4 = =D5=AF=BC=E1=CB_=05=DF=E5=AF=08=83X=0B=C1=98=BF=96]=FF+B=03j=C5=D6=FF = =EA=BB=FC=15=E5=1Eb9=FB_;=D4=FF=8AP=FD=DA3=F6=BF=8E}=FF+B=FD=1B=C1=D9=FF= Fv=FD=AF=080d=F4=E9=FE'_=86=D2=FFF=CA=84=DE3=F6=C5=D6=A7=8B{=D1=1F=3D=E5= 2=14^=EA}=BA<05=A8k=0A= =81=C9=17=1B=9F,o=95=EA=1AO=13=1A=CF=EA=D3}O=177=11)=13>y=D6)=BE=9E=B3=DE= v=C1k=CA=FD#=04=C6=9EsBv=D9=13z=CEI>u=E5=FB=E8 = =8D=E7=D2=FD=87o=E6:=8B=FA=DE=10*=DF9=7F2=FF=8C=BD=E9=97=0C=BD=C9=C6=12=EE= u=B6=92=B44=14=F6=CC=E8=EB=14=1B=F7by=0A= =FA=01=FAr8=C8=DC=8B=E5 = =9F>@_=0Eu=E0^l=80p=FF=AA=E8=CB=E0=00=B8=17=1B=A0=B0gF_=0Eu=E0^,OaO@_=0E= =07=99{=B1<=E1=F3=07=E8=CB=A1=0E=DC=8B=0C=D8=05=E8=CB=E0=00=B8=17=1B=A0=B0= =A7=E2=E9=FF=CA=BDX=9E=D0=FF=15}9=1C=B8=BE=FF-=1D}9=D4=E3=93=FE=B7=0B=D0= =97=C1=01p/6@E=DF=E7=D4=97q/V^=80=BE=AF=17=07=EE=C5=FAt=F4}=BD|=E1^=A4=EE= =16=A0=EF=AB=E5+=F7b}=C2w=7FB=DF=D7=8Bg=EE=C5=CA=14=E8.=E8=CBP=F1=95{=B1= =01=0A= {=03=FA28=00=EE=C5=06(=EC)=99=D4=81{=B1<=A1=F0+=FAr8=B0}=DF{:=FA=1EW=FFv= =EE=B5=AA=A2=B4=F4T=EE5=BE:=E7@_=EC=80=82~=80=BEL&2=FDb=07=84=8F=0F=D0/=93= =01=00`=EC=81=F0=FD=A9=00=CCc=02=18=18{=A0@hf`&=03=80=C1=D8=01=85C=01=83= =99Ld=12F=0E=02a*=03 3=19=00=18=C6=1E(4=060=CCc=02x=18{ = t=E5=C4=C3L=06=F4=93=B9=10=08]Y=91=98=C9=84=EB=E7B = =F4$P1=93=81=F8d.D=C2W=A2=821=8F `c=EC=81=F0=8D=98=D8=F8=84=81ex=8C=C5 = =0DY=F1=98E=1F=08=19[ =80=12=102=8B=83=02=C9=D8=00a TH=E6pP9=19[ = =CC=83=89=93Y=F43*cqB#=02*=F3=F4@=A5=E5=83=07%(=17=15=A0e=1E=13=00=CC=D8= =03=A1=11'`f2=00=CC=8C=1D=10=BA=B123=93 = kz=07=84f=04l~=D6=00"=E7%=D8=AC=03X=A0=10kf=E6=048=8A=8F=99=9B<=05=15=01= =98=99=1Cd`n=F2=04J=04ZfR=07Zn=06=08=90XQ=99=C7=01=A0r5 = =A3`=E2d&u=E0=E4&O=98=02=15=92=99=1CdHn=F2=84=06=04BfR=07Bn=06=16=E01=8F= =03=C0=E3f=80B=A6=8A=AD=FF+=1B7y=0A= =96Z=CE=FE=9F=C0=B8=C9=13=FA=1F=A8=98I=3D=F6=FD/ =FD_=91=98=C7=01 = q5=A0=A8<|B}=19=0F7=E5=050=CC"=0E0=DC=F4=E9$=CC"_H=B8=A9/=C0`=0E=F9=8A=C1= M=9F=CA=C0,=E2=99=81=9B2a=E2=00=00=F3T|=05=E0f=80B=E0@=BF<=0E=80~=9B=01=C2= =D4=99=D0=97I=1D=D0=B7=CA=EB=05=DC=CB=E4=C0=A2=BE=D7t=E8}V=1DAo=B2=B1=84= {=95=AE=1C=9DP=84=88=BEJ=B0q/=96=A7=907=A0/=87=83=CC=BDX=9E=D0=86=80=BE=1C= =EA=C0=BD=D8=C0=02=F4ep=00=DC=8B=0C=18*=FAr=A8=03=F7b=F9=05=E8=CB=E1 = s/=96'L=01@_=0Eu=E0^l=800=07*=FA28=00=EE=C5=06(=EC=ADx=FA=BFr/=96=A7=90=B7= e=EB=FF=89{=B1<=1D}9=D4=E3=93=FE7=0B=D0=97=C1=01p/2` = =FD?=A1=EFs=EA=CB=B8=17+S=A0=DB=9En=FD=C5=DC=8B=F5)=E8=1DN=F6=FER=EE=C5=EA= =0B=D0=F7=D5=F2=95{=B1>=15}_/=9E=B9=17+=D3=D1=97=A1=E2+=F7b=03=0B=D0=97=C1= =01p/6@E_=0Eu=E0^,O=989=15}9=1C=D8=BE=EF=1Da=E2=00=FA=1EW=FFv=EE=95=A2=A2= t=02=01"=F7=96=E1=CB=85=BE=D8=01=85=BC=01}=99Ld=FA=C5=0E(=F0]=E8=97=C9=00= =000=F6@!p=00`=1E=13=C0=C0=D8=03a L=0C=CCd=000=18; L=84=8A=C1L&2 = c=07=84=A1=00$=CCd=00`=18y=F0=84=9E=AC0=CCc=02x=18{ = P=C1=C4=C3L=06=F4=93=B9=E0)W=02=CB9=17&*=C6=0E(w=02=CF8=17*=18c=0F=84=D9= T=C1=98=C7=04=B01=F6@=98M=13=1B=9F0=B0=0C=8F=B18a(U^=EF=AB=9E=14=D1=E7g=F2=BFEPfu=F5eu=A6=D6Zx=0F=D6Vw=9B=FB= =FB=CD=97]}=C2=99=EC=E0&=0B=1A+=F4=F4=80=EE=1E=18=A7=B5=E9=F7=94=94=AB=9B= =1B=EF=E3=B4V=AC=A3=D2=B2=AE=DD=7F=1C=EA=C6V&=0C*=1Bk=A5=C3=B48=15=9F=0E= =B5cWo=DF=BD=85=9D=A5k=1Bk=1B=AAk=99=16=D7=B57=9B=87O=FF=DD=0F=B7=D5=8Av= =AAYI=BE=A6'Lg=BB=1C=CAs=B6=E7=8E=CB=D9=1D\=08Y=1D=A7=E27u=F1v=B8=19=C6=CF= =C3=B6>aBhO=18 = ggr=BB=C0=03=C3=EF=9F=B6=C3n=D7|=0Bw=F0=92=EE=AA=C7l=D7=D3=96=DE=EA=B6=B9= =F3=0E=EC=E0C=B9=AEg-=B46u=AD=B4!@2=DEEU=D7~=DA=DC=7F}=DC<=8C=D7=F7=EBV,= 2=B4=A7=84=D7=E5`=9A=F57=D7=BB=F1=E6=FA=FE=FEk]=AE=8A=9F=E9-=84=83=97=C5= =E1=FF=BDm=9C=AE?m=E3=E9=8D';8=FB=FD=F5=7F=86S=F9=A8=A7=F9=88=A8e;=BB`=F5= =91=8A=82|Z=95=A4=80=DA=CE=E9=0D=C1s:=95=DA=A8=AB=87=94=CE=F5=87V=02=A1U= =95=F6=C6=C2=FA=90=12mG~=B7o=F1=A7=FC=9B=1F=E1,T=ADH=A7~=F0sw?=DC=EC=C7=CD= #<=A2=91#+<=E4=7Fx=E4}Rx=BC=BD=B8=F4>=FD=E3=F6=F6=E2=D7=AB=9F=A6=13=D5=C1= =D4.=95=D9=3D=BC=08*M=ADZN*=C6=1A=94=F6=FEH=DD=C8?=D4"=D3F=B4=B3=B7=BA=1E= )=AE=C7=FD=E6d=1Bu=FB=1E=B2=D2=EA0=87=D2=97=FFHT=0F=9B]m=D0=A4=D6=0EEy=A1= =E0 = =95=16m=E7=DD=F8=E1q=BCK=F5=F8=D8=1E2r=CAJL=AB/=9D)=BB_=CA=D2=7F=F9=A9=DF= =C6}=1Bs=D2=1F=C2Ji=C5#=8E~9=DF=7F=1CO=CE=C5>=DCo=98=8BY=DC=AA=C3=CB=1A+= =8F=18=19=EE=EER=D1=C0=FA"^=D6=17O=93=11=BC~s=D7=DE=D2=8A=D6"i.=D6=CD;=D7= =8F=E3>=0D=81=F1=7F=E3=E3=87S=FE=D3TJu=FF=CA=B9~=AC=BC=FACl=E5ul=96=F6=CE= S=D5=FEr=D1>=AC^=D5i=A4|<6=AB=FF=D4ZN=A8C=CB=A9:=8C=F0=94No=F7=D8F=A9=F4= =87Q=EA=8D?=E2y=F7q=BC=DB=1F=A6b9=BCi=D4iud=F9=FDpX}=AC=8F=FB=C3+}=9C7v=07= =DF=C6=C2=A7=BC=EF=B7O=9B=DD=98=87=CB=8E8]=CAp=A9=C3KG4=BD=EA=81w=9F=8Ca= {=B7=D9>=9C=AA=12}=ACJ=A6o=A9m=FD=AF=E2=B1"=B9=1D?=8F;4=18=E1`&=EB=DE=05= =D8=1D?=F1[=FD=12=C9=E4=3D=C0)=06%=ED=91=A9=F5=D7u=A1)=F5=94=E8=CA=C0=A8= X=17=F5DS=CAO(=F5n=B8)=CF=19=A0=BA=94dhT=F7C=B7gO|=FFl=9F=D7=9F7=B7e=9D=86= u=89=1B=0F{=FC8=DF=E3=F0=DB=FB=84=8D=97!=9D=E8=F9=FFY/=A3=DE=A6a = =8E=7F=82|=87>=B6=85T=F69Nl=89'=18=DA=846 m}=1B = A=9A=A1=C2=DAJm'=E0=DBs=B6=E3=C4N=D3=D8=D5=F2=B0.=DD=92=FC|=F7?=9F=FF=F7= m=96R=E5=EF=A6X=0C=EA=04p=D7=D3=0C_^=AC=FA=F7=EF=93H=9D=CC=CF=07= 2=7F=CB=C0=CF=04m3q;=94=ED=B7=B3=14=DF=82=E8=EB=19Q_8!S=14=EF=E5=F9=E5=E0= =AFE=95g=9D=18I=9BL~=01=9E=9F-=B2=C6_BW%w=05W=B3=148A=BD=FC=D0=F1=08k1=9D= =D0=B1=014b>N=D5=86K=B9=CE=9Fj=BA=F5e=E3=95=EB=EF=E8=84=EB+=E7=FE=C6=CA=EA= =B8=D1u=DA=9Bgi=AEk=AF5=8C]e=84[=D0'e=E8=FC=EF=D6=1F=91=D2\=98=D5=A7`"T=F7= Po=F4=F2=A5}=CD=88E)S=B3=15SG=FD=A6=D1ah=C2=A3T=9A'=B8=1D=CA^=C5=C7=B3=C0= =C2Y=04=1C;=DAXd=AEw=AA=85=F3=088=CF=CD=13=E3=F0s=DD=04j|=1E=81/=F8b<=BA= =10=AE=EEE=04^=0E=C3=9Bn=12B=03=1E=B9=96+=C2\=A0,=94=F5x4pWsJ#=F0,=0F=95= {<>=F3=92N#v=1Bp1^=F4=85W=F14=A2=E6@=D0=F1=A2=97^=AB=A1=11=DA32=9E=F6=8C= =FA=DA=CB=08<=8C=A7=3Dc=9E=F6=10Qz,=0Bj=7F=C1=8Eg=B9=A7>D=A8=CF=8A=A0=FA= =97,@d-=3D=A2=E10=19=14=FF=02:=8EXN=BB=05q=DE=93=F8=9E=17=C7&k=E8=D4Q=FF= a=87c=E7a=B9=BB=A9=FE=FA'=BAhK=E1=C4=BCz=C6=B2]=0D>>=0Dstartxref=0D61992=0D%%EOF ------_=_NextPart_000_01C177AD.917FBD80-- From owner-ips@ece.cmu.edu Wed Nov 28 08:30:10 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA06264 for ; Wed, 28 Nov 2001 08:30:05 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASC3DX25111 for ips-outgoing; Wed, 28 Nov 2001 07:03:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailnpd.hcltech.com ([203.197.145.76]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fASC3Bl25104 for ; Wed, 28 Nov 2001 07:03:11 -0500 (EST) Received: from npd.hcltech.com (skotra-pc.hcltech.com [192.168.100.57]) by mailnpd.hcltech.com (8.11.0/8.11.0) with ESMTP id fASBwOe07793 for ; Wed, 28 Nov 2001 17:28:24 +0530 Message-ID: <3C04D1B1.F73A6EF0@npd.hcltech.com> Date: Wed, 28 Nov 2001 17:29:45 +0530 From: "Subrahmanya Sastry K.V" Organization: HCL X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.2-2 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI:connection recovery algorithms in draft-09 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Hello, In the connection recovery algorithms ( Appendix G of iscsi-09 draft), the connection states have been mentioned as ASYNC-MSG_RCVD, RECOVERY_START, XPT_CLEANUP( which were defined in draft-08). But these states are not referred to in draft-09 anywhere else though they exactly map to certain states defined in draft-09. I feel its better to modify their names to the appropriate states defined in draft-09 to make them more readable. Comments ? -Sastry From owner-ips@ece.cmu.edu Wed Nov 28 10:42:52 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA15044 for ; Wed, 28 Nov 2001 10:42:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASEhbB05330 for ips-outgoing; Wed, 28 Nov 2001 09:43:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mtv01ex01.mindtree.com ([202.56.255.246]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fASEhXl05324 for ; Wed, 28 Nov 2001 09:43:34 -0500 (EST) content-class: urn:content-classes:message Subject: implementation on windows MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 Date: Wed, 28 Nov 2001 20:13:32 +0530 Message-ID: Thread-Topic: implementation on windows Thread-Index: AcF4Gw1kpfHy3OqfRiW9LVfxy1Jv+w== From: "Murali S" To: Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id fASEhZl05327 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 8bit Hi, I want to implement ISCSI on windows NT/2K. Is it is required to write a SCSI miniport driver which gets the input from the SCSI port driver . This SCSI miniport driver is the driver which is the implementation is iscsi protocol and communicates through the TCP/IP layer. Can any one tell me as how can get the access to the TCP/IP layer in the drivers, like creating a socket, etc... From owner-ips@ece.cmu.edu Wed Nov 28 12:39:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA24379 for ; Wed, 28 Nov 2001 12:39:53 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASGVl614350 for ips-outgoing; Wed, 28 Nov 2001 11:31:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from astutenetworks.com (2.astutenetworks.com [216.142.74.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fASGVil14341 for ; Wed, 28 Nov 2001 11:31:44 -0500 (EST) Received: from amirdesktop (dhcp62 [172.21.2.62]) by astutenetworks.com (8.11.2/8.11.2) with SMTP id fASGVbZ10864 for ; Wed, 28 Nov 2001 08:31:37 -0800 From: "Amir Shalit" To: Subject: unsolicited data out PDU delivery Date: Wed, 28 Nov 2001 08:31:35 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700 Importance: Normal In-Reply-To: <01A7DAF31F93D511AEE300D0B706ED920CF705@axcs13.cos.agilent.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Any update on adding context information (i.e. cmdSN, static target tag, etc.) to unsolicited data PDUs in order to facilitate simple task lookup by the iSCSI target? Thanks, Amir From owner-ips@ece.cmu.edu Wed Nov 28 13:18:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26969 for ; Wed, 28 Nov 2001 13:18:58 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASHWGu19074 for ips-outgoing; Wed, 28 Nov 2001 12:32:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel13.hp.com (palrel13.hp.com [156.153.255.238]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fASHWEl19068 for ; Wed, 28 Nov 2001 12:32:15 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel13.hp.com (Postfix) with ESMTP id B94751F645; Wed, 28 Nov 2001 09:32:00 -0800 (PST) Received: from swathi (pal1nai160238.nsr.hp.com [15.244.160.238]) by core.rose.hp.com with SMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id JAA29012; Wed, 28 Nov 2001 09:33:30 -0800 (PST) Message-ID: <000701c17832$95b130c0$eea0f40f@rose.hp.com> From: "Mallikarjun C." To: "Subrahmanya Sastry K.V" , References: <3C04D1B1.F73A6EF0@npd.hcltech.com> Subject: Re: iSCSI:connection recovery algorithms in draft-09 Date: Wed, 28 Nov 2001 09:31:59 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sastry, Thank you for pointing out, it was my editing oversight. I will fix them. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization Hewlett-Packard MS 5668 Roseville CA 95747 ----- Original Message ----- From: "Subrahmanya Sastry K.V" To: Sent: Wednesday, November 28, 2001 3:59 AM Subject: iSCSI:connection recovery algorithms in draft-09 > Hello, > > In the connection recovery algorithms ( Appendix G of iscsi-09 draft), > the connection states have been mentioned as ASYNC-MSG_RCVD, > RECOVERY_START, XPT_CLEANUP( which were defined in draft-08). > But these states are not referred to in draft-09 anywhere else though > they exactly map to certain states defined in draft-09. I feel its > better to > modify their names to the appropriate states defined in draft-09 to make > > them more readable. > > Comments ? > > -Sastry > > From owner-ips@ece.cmu.edu Wed Nov 28 13:24:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA27434 for ; Wed, 28 Nov 2001 13:24:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASHpIn20491 for ips-outgoing; Wed, 28 Nov 2001 12:51:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fASHpFl20481 for ; Wed, 28 Nov 2001 12:51:15 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by dirty; Wed Nov 28 12:51:39 EST 2001 Received: from aura.research.bell-labs.com (aura.research.bell-labs.com [135.104.46.10]) by grubby.research.bell-labs.com (8.11.6/8.11.6) with ESMTP id fASHooo27243; Wed, 28 Nov 2001 12:50:51 -0500 (EST) Received: (from sandeepj@localhost) by aura.research.bell-labs.com (8.9.1/8.9.1) id MAA12945; Wed, 28 Nov 2001 12:50:49 -0500 (EST) Date: Wed, 28 Nov 2001 12:50:49 -0500 (EST) Message-Id: <200111281750.MAA12945@aura.research.bell-labs.com> From: sandeepj@research.bell-labs.com (Sandeep Joshi) To: amir@astutenetworks.com, ips@ece.cmu.edu Subject: Re: unsolicited data out PDU delivery Sender: owner-ips@ece.cmu.edu Precedence: bulk > Any update on adding context information (i.e. cmdSN, static target tag, > etc.) > to unsolicited data PDUs in order to facilitate simple task lookup by the > iSCSI target? > > Thanks, > > Amir You could rely on the fact that unsolicited data PDUs come in the same order as the commands, and directly match the data PDU to the first outstanding command in a connection. Sec 2.2.5 Unsolicited data MUST be sent on every connection in the same order in which commands were sent See http://www.pdl.cmu.edu/mailinglists/ips/mail/msg07197.html -Sandeep From owner-ips@ece.cmu.edu Wed Nov 28 15:07:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA05806 for ; Wed, 28 Nov 2001 15:07:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASJHrJ27700 for ips-outgoing; Wed, 28 Nov 2001 14:17:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fASJHkl27692 for ; Wed, 28 Nov 2001 14:17:47 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA67632; Wed, 28 Nov 2001 14:14:59 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fASJHMY83418; Wed, 28 Nov 2001 12:17:32 -0700 X-Priority: 1 (High) Importance: Normal Subject: Re: unsolicited data out PDU delivery To: "Amir Shalit" Cc: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Wed, 28 Nov 2001 11:16:30 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/28/2001 12:17:32 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I believe, that most of us do not understand there is a problem, since what I think you are asking for is what the Initiator Task Tag is all about. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Amir Shalit" @ece.cmu.edu on 11/28/2001 08:31:35 AM Sent by: owner-ips@ece.cmu.edu To: cc: Subject: unsolicited data out PDU delivery Any update on adding context information (i.e. cmdSN, static target tag, etc.) to unsolicited data PDUs in order to facilitate simple task lookup by the iSCSI target? Thanks, Amir From owner-ips@ece.cmu.edu Wed Nov 28 15:11:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA06083 for ; Wed, 28 Nov 2001 15:11:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASJZ6K29233 for ips-outgoing; Wed, 28 Nov 2001 14:35:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from astutenetworks.com (2.astutenetworks.com [216.142.74.2]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fASJYml29209 for ; Wed, 28 Nov 2001 14:35:03 -0500 (EST) Received: from amirdesktop (dhcp62 [172.21.2.62]) by astutenetworks.com (8.11.2/8.11.2) with SMTP id fASJYbZ21439; Wed, 28 Nov 2001 11:34:37 -0800 From: "Amir Shalit" To: "Sandeep Joshi" , Subject: RE: unsolicited data out PDU delivery Date: Wed, 28 Nov 2001 11:34:36 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4807.1700 Importance: Normal In-Reply-To: <200111281750.MAA12945@aura.research.bell-labs.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit A simple solution will be to let iSCSI initiator assign target tags in the form of 0x800000xx that can be sent along with unsolicited data out PDUs. A storage switch that has to handle N*M sessions, will struggle buffering and matching unsolicited data PDUs with command PDUs. Amir -----Original Message----- From: Sandeep Joshi [mailto:sandeepj@research.bell-labs.com] Sent: Wednesday, November 28, 2001 9:51 AM To: amir@astutenetworks.com; ips@ece.cmu.edu Subject: Re: unsolicited data out PDU delivery > Any update on adding context information (i.e. cmdSN, static target tag, > etc.) > to unsolicited data PDUs in order to facilitate simple task lookup by the > iSCSI target? > > Thanks, > > Amir You could rely on the fact that unsolicited data PDUs come in the same order as the commands, and directly match the data PDU to the first outstanding command in a connection. Sec 2.2.5 Unsolicited data MUST be sent on every connection in the same order in which commands were sent See http://www.pdl.cmu.edu/mailinglists/ips/mail/msg07197.html -Sandeep From owner-ips@ece.cmu.edu Wed Nov 28 15:12:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA06233 for ; Wed, 28 Nov 2001 15:12:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASJWmc29007 for ips-outgoing; Wed, 28 Nov 2001 14:32:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel13.hp.com (palrel13.hp.com [156.153.255.238]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fASJWjl28998 for ; Wed, 28 Nov 2001 14:32:46 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel13.hp.com (Postfix) with ESMTP id 3447F1F944 for ; Wed, 28 Nov 2001 11:32:40 -0800 (PST) Received: from swathi (pal1nai160238.nsr.hp.com [15.244.160.238]) by core.rose.hp.com with SMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id LAA18061 for ; Wed, 28 Nov 2001 11:34:09 -0800 (PST) Message-ID: <000f01c17843$70943e70$eea0f40f@rose.hp.com> From: "Mallikarjun C." To: References: Subject: Re: iSCSI: unsolicited data Date: Wed, 28 Nov 2001 11:32:38 -0800 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, > Unsolicited data needs are very hard to predict and ordering them may > remove the need to do so to a large extent. Completely agreed. There are obvious performance advantages in mandating order, as earlier discussions clearly outlined. I am not questioning it. I however could not really see a deadlock per se - my comment was to understand what it is. More comments would certainly help. > As for the action - yes we may want to weaken the requirement to > accommodate for digest errors. Thanks. I meant to weaken the requirement on the response to a UUD, not the ordering requirement itself. Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization Hewlett-Packard MS 5668 Roseville CA 95747 ----- Original Message ----- From: "Julian Satran" To: Sent: Monday, November 26, 2001 10:23 PM Subject: Re: iSCSI: unsolicited data > Mallikarjun, > > The deadlock free requirement here goes a bit deeper than your comment may > suggest. > Unsolicited data needs are very hard to predict and ordering them may > remove the need to do so to a large extent. > As for the action - yes we may want to weaken the requirement to > accommodate for digest errors. > > Regards, > Julo > > > > > > > > "Mallikarjun C." > Sent by: owner-ips@ece.cmu.edu > 26-11-01 21:45 > Please respond to cbm > > > To: ips > cc: > Subject: iSCSI: unsolicited data > > > > Julian, > > I don't recall the earlier discussion being conclusive on this topic, > so let me attempt to make a couple of suggestions on this. > > Last para in Section 2.2.5 (page 33) in rev09 mandates unsolicited > data ordering with the following wording. > > "iSCSI initiators and targets MUST also enforce some ordering rules to > achieve deadlock-free operation. Unsolicited data MUST be sent on > every connection in the same order in which commands were sent. A > target receiving data out of order SHOULD terminate the session." > > I have a couple of suggestions to reword - > > - The reference to a deadlock is true only for an implementation > overcommitting the unsolicited data buffers. So, I would suggest > dropping this reasoning. > > - The "MUST enforce" wording should also add that "targets MAY > however see unexpected unsolicited data (UUD), following data > digest errors on expected unsolicited data." > > - Finally, the "SHOULD terminate the session", IMHO, is too strong. > This wording will imply that a session is vulnerable to a single > digest error regardless of recovery level. [ I however disfavor > attaching more assorted semantics to recovery levels (than the > current easily understood "level = setof(recovery classes)" > formulation). ] As you mentioned on this discussion earlier, a > UUD is a protocol error (or may be not, if it is the effect of > a prior digest failure). > > We can choose to Reject ("protocol error") the UUD and discard the > data. The only problem with this approach is: following a digest > error on unsolicited data for one task, all the following unsolicited > data in flight would have to be discarded. One option is to imply > that the target update the "expected unsolicited data" so the > unsolicited data following in the pipe can be processed normally. > So, I guess it would be saying: "the target MAY Reject the UUD". > > Comments? > -- > Mallikarjun > > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions Organization > MS 5668 Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > > From owner-ips@ece.cmu.edu Wed Nov 28 16:38:38 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA14861 for ; Wed, 28 Nov 2001 16:38:38 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASKk3e04934 for ips-outgoing; Wed, 28 Nov 2001 15:46:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ariel.nishansystems.com (goretex.nishansystems.com [216.217.36.167]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fASKk1l04928 for ; Wed, 28 Nov 2001 15:46:02 -0500 (EST) Received: by ariel.nishansystems.com with Internet Mail Service (5.5.2653.19) id ; Wed, 28 Nov 2001 12:45:55 -0800 Message-ID: From: Charles Monia To: "Ips (E-mail)" Cc: "David Black (E-mail)" , Charles Monia Subject: RE: iFCP: iFCP -06 comments -- N_PORT Network Address Date: Wed, 28 Nov 2001 12:45:54 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, November 13, 2001 5:22 PM > To: ips@ece.cmu.edu > Subject: iFCP -06 comments > > > > 6.2.1 > An N_PORT is identified by its network address consisting of: > > a) The N_PORT I/D assigned by the gateway to which > the N_PORT is > locally attached and > > b) The IP address of the gateway's iFCP Portal. > > Use of an IP address to identify a remote gateway needs to be > reconsidered. > Please add something to allow multiple iFCP implementations > to exist at > different TCP ports on the same IP address (or explain why this has to > be prohibited). This is strongly related to the NAPT issues that the > FCIP folks are in the midst of working through. > We propose to modify iFCP to add the TCP port number as a component of the N_PORT network address. When performing N_PORT name resolution, iSNS now returns this parameter as part of the query response. The iSNS spec also provides guidance on coordinating address space views between the "external" and "internal" sides of the NAPT box. Charles From owner-ips@ece.cmu.edu Wed Nov 28 19:16:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA28048 for ; Wed, 28 Nov 2001 19:16:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fASNEbD16429 for ips-outgoing; Wed, 28 Nov 2001 18:14:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fASNEYl16420 for ; Wed, 28 Nov 2001 18:14:34 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 90BF33760; Wed, 28 Nov 2001 16:14:33 -0700 (MST) Received: from axcsbh5.cos.agilent.com (axcsbh5.cos.agilent.com [130.29.152.150]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 6388B5FD; Wed, 28 Nov 2001 16:14:33 -0700 (MST) Received: from 130.29.152.150 by axcsbh5.cos.agilent.com (InterScan E-Mail VirusWall NT); Wed, 28 Nov 2001 16:14:33 -0700 Received: by axcsbh5.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Wed, 28 Nov 2001 16:14:33 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A009062EEB@axcs04.cos.agilent.com> From: "THALER,PAT (A-Roseville,ex1)" To: Paul Koning , ltuikov@yahoo.com Cc: ips@ece.cmu.edu Subject: RE: CRC32C code -- please check Date: Wed, 28 Nov 2001 16:14:32 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Luben, I think I see the problem: Luben> Ok, so the receiver got M''(x) assuming no noise. Luben> M''(x) = Q(x)*G(x) + I(x). (this is (3)) Luben> Now we work the algorithm again: ..., find the remainder: Luben> Clearly, from (3) the remainder of M''(x) modulo G(x) is I(x). ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this is not true. Luben> Then we complement the value as per the draft: Luben> ~I(x) = 0. Luben> And we end up with remainder 0. Remember I(x) is the polynomial of order 31 with all the coefficients equal to 1. G(x) is a polynomial of order 31 with some of its coefficients equal to 0. Therefore, G(x) goes into I(x) once. The remainder of M''(x) modulo G(x) is I(x) - G(x). So that should explain the "magic number." Regards, Pat -----Original Message----- From: Paul Koning [mailto:ni1d@arrl.net] Sent: Monday, November 26, 2001 7:12 AM To: ltuikov@yahoo.com Cc: ips@ece.cmu.edu Subject: Re: CRC32C code -- please check >>>>> "Luben" == Luben Tuikov writes: Luben> Here is some C code at the end, the draft quoted and my notes Luben> alongside it. ... Luben> So, here we get M'(x) = x^32 * M(x). Getting R(x) is trivial. Luben> So we have: Luben> M'(x) = Q(x)*G(x) + R(x) = Q(x)*G(x) - R(x), (1) Luben> since plus is same as minus in modulo 2 arithmetic. >> - the bit sequence is complemented and the result is the Luben> CRC Luben> Now, from boolean logic we know that b XOR 1 = ~b, then, Luben> ~R(x) = R(x) XOR I(x) = R(x) + I(x) (2) Luben> where I(x) is a polynomial of the same degree as R(x) and Luben> whose coefficients are all equal to 1. So, to get M''(x) which Luben> we send we have: Luben> M''(x) = M'(x) + ~R(x) (draft) = Q(x)*G(x) - R(x) + R(x) + Luben> I(x) (using (1) and (2)) = Q(x)*G(x) + I(x). Luben> or succinctly, Luben> M''(x) = Q(x)*G(x) + I(x), (3) Luben> and this is the message which we send. Not quite. Your M' is taken from M by complementing the first 32 bits and appending 32 zeroes. That's what you feed into the CRC calculation -- but what you transmit has the original (uncomplemented) first 32 bits. Luben> On the other end we Luben> get M''(x), assuming there was no noise, and all Luben> link/TCP/Ethernet transformations are transparent for the Luben> sender and the receiver. ... Luben> Ok, so the receiver got M''(x) assuming no noise. Luben> M''(x) = Q(x)*G(x) + I(x). (this is (3)) Luben> Now we work the algorithm again: ..., find the remainder: Luben> Clearly, from (3) the remainder of M''(x) modulo G(x) is I(x). Luben> Then we complement the value as per the draft: Luben> ~I(x) = 0. Luben> And we end up with remainder 0. I think I see the problem. In CRC generation, you process the message (M). In CRC checking, there are two approaches: 1. You process M and compare the 4 bytes that follow (the CRC) with what you computed. This is fine, if you know ahead of time where M ends. 2. If you don't know where M ends ahead of time (as in the Ethernet MAC layer) or you simply prefer to process the whole received message, you run everything including the received CRC through the CRC algorithm. Note that the string processed in this case is 4 bytes longer than in case (1), because it includes the CRC. That's what you described, but I believe you missed a step. The CRC algorithm says: take the message (M'' in this case) and multiply by x^32, then divide by the CRC polynomial. You missed the multiply. In other words, you should have: (M''(x)*x^32) mod G(x) rather than simply M''(x) mod G(x) and that is *not* zero. I haven't done the math but I expect that you will find the answer to be the non-zero constant that appears in the spec. Ethernet works the same way, except for a change in the polynomial. You suggested earlier that Ethernet is using a non-standard algorithm because it comes up with a non-zero remainder at the receive end. That is not the case. You should not need any fudge factors to make the answer come out according to the spec if you get the algorithm right... paul From owner-ips@ece.cmu.edu Thu Nov 29 00:17:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA22477 for ; Thu, 29 Nov 2001 00:17:28 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAT48nS05133 for ips-outgoing; Wed, 28 Nov 2001 23:08:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from svldns02.veritas.com (bay-bridge.veritas.com [143.127.3.10]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAT48ll05127 for ; Wed, 28 Nov 2001 23:08:47 -0500 (EST) Received: from megami.veritas.com (megami.veritas.com [10.182.128.180]) by svldns02.veritas.com (8.11.6/8.11.6) with SMTP id fAT46W312715; Wed, 28 Nov 2001 20:06:32 -0800 (PST) Received: from vagarwal([10.212.84.139]) (1501 bytes) by megami.veritas.com via sendmail with P:smtp/R:smart_host/T:smtp (sender: ) id for ; Wed, 28 Nov 2001 20:08:30 -0800 (PST) (Smail-3.2.0.101 1997-Dec-17 #15 built 2001-Aug-30) Message-ID: <000701c1788b$cd55bce0$8b54d40a@vagarwal> From: "Vivek Agarwal" To: "Murali S" , References: Subject: Re: implementation on windows Date: Thu, 29 Nov 2001 09:40:36 +0530 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit you can use some thing called a TDI interface that can be used in the scsi miniport driver that will communicate with the remote machine and deliver inputs to the local machine. Regards, Vivek ----- Original Message ----- From: "Murali S" To: Sent: Wednesday, November 28, 2001 8:13 PM Subject: implementation on windows > Hi, > > I want to implement ISCSI on windows NT/2K. > Is it is required to write a SCSI miniport driver which gets the > input from the > SCSI port driver . This SCSI miniport driver is the driver which is > the implementation is iscsi protocol and communicates through the TCP/IP > layer. > Can any one tell me as how can get the access to the TCP/IP layer in > the drivers, like creating a socket, etc... > > From owner-ips@ece.cmu.edu Thu Nov 29 01:43:34 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA03412 for ; Thu, 29 Nov 2001 01:43:34 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAT5TDn10207 for ips-outgoing; Thu, 29 Nov 2001 00:29:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lucy.trebia.com ([65.192.191.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAT5TBl10202 for ; Thu, 29 Nov 2001 00:29:11 -0500 (EST) Received: by lucy with Internet Mail Service (5.5.2653.19) id ; Thu, 29 Nov 2001 00:27:25 -0500 Message-ID: <63616B261CEFD411834D00D0B7A86CF7243690@lucy> From: Sudhir Srinivasan To: ips@ece.cmu.edu Cc: sudhir.srinivasan@trebia.com Subject: FCIP: Echoing a Changed Short Frame? Date: Thu, 29 Nov 2001 00:27:17 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Section 8.1.2 of v0.7 of the FCIP draft states in the last three paras on page 25 that the echoed Short Frame must match exactly the one that was originated; otherwise the connection SHALL be closed. Section 8.1.5 on pages 28-29 describes how the receiver of the Short Frame can change the contents of the frame and then SHALL echo it back to the originator (with the CH bit set). What's the purpose of the Change bit and echoing a changed Short Frame if the originator will simply terminate the connection? Is the expectation that the originator can capture the changed info and try to set up the connection again with the new information? If so, would it not be easier to just have the originator respond with another Short Frame with the new information and repeat this back-and-forth until the echo'ed SF has no changes or either side decides to give up? Thanks, - Sudhir ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trebia Networks, Inc. Sudhir.Srinivasan@trebia.com 978-929-0830 x139 www.trebia.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From owner-ips@ece.cmu.edu Thu Nov 29 03:59:01 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA16697 for ; Thu, 29 Nov 2001 03:59:01 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAT7tHC18701 for ips-outgoing; Thu, 29 Nov 2001 02:55:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAT7tEl18693 for ; Thu, 29 Nov 2001 02:55:15 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA273346 for ; Thu, 29 Nov 2001 08:55:07 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAT7tNd155168 for ; Thu, 29 Nov 2001 08:55:23 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.8 June 18, 2001 From: "Julian Satran" Message-ID: Date: Thu, 29 Nov 2001 09:55:22 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 29/11/2001 09:55:29, Serialize complete at 29/11/2001 09:55:29 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk 0 as a stand in for "whatever" is out of version 09. We don't see any need for it. It was originally provide for compatibility with the T10 docs when we had values "residing" in mode pages accessible by SCSI commands. Julo Tow Wang Sent by: owner-ips@ece.cmu.edu 28-11-01 01:55 To: "Robert D. Russell" , ips@ece.cmu.edu cc: Subject: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Hi all. I fully agree with the suggestions for issue 4. If we really need to reserve a value to mean "unlimited" or "infinite", could we use a value of all bits set to one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be great for most 32-bit processors and above.) This would make arithmetic comparisons more intuitive, IMHO. "Robert D. Russell" wrote: > Attached are the new issues that arose during the iSCSI plugfest > at UNH on Wednesday 31-Oct-2001. > > (Note: these issues do not take into account any modifications or > clarifications that occured in the standard due to the issues raised > on Monday or Tuesday.) > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > now allow: "A value of 0 indicates no limit." > > Is this useful? Does it buy anything? > > The difficulties implementers are having with this are: > > 1) It is a special case. > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > 3) It violates the min/max function normally used for the key. > 4) There is always a limit anyway. > > Consider FirstBurstSize, which can have a value that is described > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > is selected. > > I one side offers 0 to mean unlimited, and the other side > has a limit, it will reply with that limit, say 128 Kbytes. > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > The statement in the standard that "the minimum of the 2 numbers is > selected" is therefore wrong when one of the numbers is 0. > > Furthermore, when an initiator or target receives an offer for one > of these keys, it cannot simply check that the offered value is > legal by testing it against some minimum and maximum. It must first > check for 0 and then only if that check shows the value is non-zero > can it do the min/max range check for legality (i.e., 64-2**24). > > Finally, there is always a limit. If nothing else it is the > limit imposed by the 24-bit DataSegmentLength field of the PDU > requesting the transfer. It is useless to specify a FirstBurstSize > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > the largest possible DataSegmentLength in any PDU that can use > this value is 2**24-1. > > The suggestion is to just eliminate this special case of 0 and require > that the range 64-to-(2**24-1) be used instead -- it has exactly the > same effect in all cases, it is easier to describe in the standard > because it avoids all the extra words, and it is easier to code > because it avoids all the special cases. > > NOTE: the standard should specify the limit in the ranges for > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > of 2**24. The number 2**24 cannot be represented in the 24-bit > DataSegmentLength field and therefore can never be used. -- Tow Wang Adaptec Software Engineer Telephone: 408 957 4838 Mail stop 46 800 869 8883 x4838 691 South Milpitas Boulevard Fax: 530 686 8023 Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com From owner-ips@ece.cmu.edu Thu Nov 29 09:06:41 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA00721 for ; Thu, 29 Nov 2001 09:06:41 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fATDVO918552 for ips-outgoing; Thu, 29 Nov 2001 08:31:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (io.iol.unh.edu [132.177.123.82]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fATAUfl08542 for ; Thu, 29 Nov 2001 05:30:42 -0500 (EST) Received: from io.iol.unh.edu (IDENT:rdr@localhost.localdomain [127.0.0.1]) by iol.unh.edu (8.12.1/8.12.1) with ESMTP id fATAUa6D012781; Thu, 29 Nov 2001 05:30:36 -0500 Received: from localhost (rdr@localhost) by io.iol.unh.edu (8.12.1/8.12.1/Submit) with ESMTP id fATAUaJ3012778; Thu, 29 Nov 2001 05:30:36 -0500 Date: Thu, 29 Nov 2001 05:30:36 -0500 (EST) From: "Robert D. Russell" To: Julian Satran cc: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: If I understand correctly what Tow was asking for, and your response to him, this means that the statement: "A value of 0 MAY be used as a "don't care" offer in negotiations." should be removed from the description in Appendix D of draft 9 for the 6 keys: MaxRecvPDULength MaxBurstSize FirstBurstSize LogoutLoginMaxTime LogoutLoginMinTime MaxOutstandingR2T The statement in section 2.2.4 on page 31 should also be removed: "For numerical negotiations, the value 0 MAY be specified by the offering party as a "don't care"/"unlimited" value for parameters that explicitly allow it; in this case, the responder may choose any legal value for the parameter." Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Thu, 29 Nov 2001, Julian Satran wrote: > 0 as a stand in for "whatever" is out of version 09. > We don't see any need for it. It was originally provide for compatibility > with the T10 docs when we had values "residing" in mode pages accessible > by SCSI commands. > > Julo > > > > > Tow Wang > Sent by: owner-ips@ece.cmu.edu > 28-11-01 01:55 > > > To: "Robert D. Russell" , ips@ece.cmu.edu > cc: > Subject: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest > > > > Hi all. > > I fully agree with the suggestions for issue 4. If we really need to > reserve a > value to mean "unlimited" or "infinite", could we use a value of all bits > set to > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be great > for > most 32-bit processors and above.) This would make arithmetic comparisons > more > intuitive, IMHO. > > "Robert D. Russell" wrote: > > > Attached are the new issues that arose during the iSCSI plugfest > > at UNH on Wednesday 31-Oct-2001. > > > > (Note: these issues do not take into account any modifications or > > clarifications that occured in the standard due to the issues raised > > on Monday or Tuesday.) > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > now allow: "A value of 0 indicates no limit." > > > > Is this useful? Does it buy anything? > > > > The difficulties implementers are having with this are: > > > > 1) It is a special case. > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > 3) It violates the min/max function normally used for the key. > > 4) There is always a limit anyway. > > > > Consider FirstBurstSize, which can have a value that is described > > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > > is selected. > > > > I one side offers 0 to mean unlimited, and the other side > > has a limit, it will reply with that limit, say 128 Kbytes. > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > The statement in the standard that "the minimum of the 2 numbers is > > selected" is therefore wrong when one of the numbers is 0. > > > > Furthermore, when an initiator or target receives an offer for one > > of these keys, it cannot simply check that the offered value is > > legal by testing it against some minimum and maximum. It must first > > check for 0 and then only if that check shows the value is non-zero > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > Finally, there is always a limit. If nothing else it is the > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > requesting the transfer. It is useless to specify a FirstBurstSize > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > the largest possible DataSegmentLength in any PDU that can use > > this value is 2**24-1. > > > > The suggestion is to just eliminate this special case of 0 and > require > > that the range 64-to-(2**24-1) be used instead -- it has exactly the > > same effect in all cases, it is easier to describe in the standard > > because it avoids all the extra words, and it is easier to code > > because it avoids all the special cases. > > > > NOTE: the standard should specify the limit in the ranges for > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > DataSegmentLength field and therefore can never be used. > > -- > Tow Wang > Adaptec Software Engineer Telephone: 408 957 4838 > Mail stop 46 800 869 8883 x4838 > 691 South Milpitas Boulevard Fax: 530 686 8023 > Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com > > > > > From owner-ips@ece.cmu.edu Thu Nov 29 09:10:37 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA01312 for ; Thu, 29 Nov 2001 09:10:37 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fATDVau18575 for ips-outgoing; Thu, 29 Nov 2001 08:31:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fATB66l10426 for ; Thu, 29 Nov 2001 06:06:06 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA20348; Thu, 29 Nov 2001 06:06:02 -0500 (EST) Message-Id: <200111291106.GAA20348@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-iscsi-slp-02.txt Date: Thu, 29 Nov 2001 06:06:02 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Finding iSCSI Targets and Name Servers Using SLP Author(s) : M. Bakke et al. Filename : draft-ietf-ips-iscsi-slp-02.txt Pages : 20 Date : 28-Nov-01 The iSCSI protocol provides a way for hosts to access SCSI devices over an IP network. This document defines the use of the Service Location Protocol (SLP) by iSCSI hosts, devices, and name services, along with the SLP service type templates that describe the services they provide. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-slp-02.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-slp-02.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-slp-02.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011128143605.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-slp-02.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-slp-02.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011128143605.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Thu Nov 29 09:11:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA01421 for ; Thu, 29 Nov 2001 09:11:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fATDJun17876 for ips-outgoing; Thu, 29 Nov 2001 08:19:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lucy.trebia.com ([65.192.191.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fATDJql17870 for ; Thu, 29 Nov 2001 08:19:53 -0500 (EST) Received: by lucy with Internet Mail Service (5.5.2653.19) id ; Thu, 29 Nov 2001 08:18:04 -0500 Message-ID: <63616B261CEFD411834D00D0B7A86CF75760C0@lucy> From: Sudhir Srinivasan To: "'roweber@acm.org'" , Sudhir Srinivasan , "'ips@ece.cmu.edu'" Cc: Murali Rajagopal Subject: RE: FCIP: v0.7 editorial suggestion Date: Thu, 29 Nov 2001 08:17:59 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, If the bit positions are going to be changed, I still think it would be better to simply swap the positions of the Ch and SF bits in the pFlags bit. This has both benefits -- it allows for the "growth" of the SF bit as you outline below AND it makes the table consistent with the bit field definition and hence makes it easier to read/verify Figure 9. Regards, Sudhir ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trebia Networks, Inc. Sudhir.Srinivasan@trebia.com 978-929-0830 x139 www.trebia.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > -----Original Message----- > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > Sent: Wednesday, November 28, 2001 6:24 PM > To: Sudhir Srinivasan > Cc: Murali Rajagopal > Subject: Re: FCIP: v0.7 editorial suggestion > > > Sudhir, > > The FCIP authors have agreed to move the Ch bit from > bit 25 to bit 31. This change has the additional > advantage that the SF bit can grow to the 'frame type' > field if a compelling need arises. > > The new figure will look like this. > > |----------------Bit--------------------| > | | > | 31 30 29 28 27 26 25 24 | > +----+-----------------------------+----+ > | Ch | Reserved | SF | > +----+-----------------------------+----+ > > Fig. 8 pFlags Field Bits > > I don't know if this will fully lay to rest the > confusion you experienced, but it should help. > > Thanks. > > Ralph... > > Sudhir Srinivasan wrote: > > > > > Ralph, > > > > Deleting the table is not advisable, IMO. The table > > is a handy quick reference and is very useful, much > > preferable to searching through a bunch of text. It's > > precisely for this reason that it is somewhat important > > that the table matches the actual bit pattern. Most (all?) > > standards that I've read do it that way. > > > > It's a suggestion only to improve the readability of > > the standard. No big deal if you want to leave it the > > way it is. > > > > -S > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Trebia Networks, Inc. Sudhir.Srinivasan@trebia.com > > 978-929-0830 x139 www.trebia.com > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > > -----Original Message----- > > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > > Sent: Sunday, November 25, 2001 4:21 PM > > > To: Sudhir Srinivasan > > > Cc: Murali Rajagopal > > > Subject: Re: FCIP: v0.7 editorial suggestion > > > > > > > > > Sudhir, > > > > > > Since the entire contents of table 1 duplicates > > > specifications written out somewhere in the text, > > > I would be happy to delete the table. > > > > > > Ralph... > > > > > > Sudhir Srinivasan wrote: > > > > > > > > > > > OK, let me spell it out: > > > > > > > > In figure 9 (that's nine, not eight), the pFlags > > > > value is correctly shown as 0x01. The normal tendency > > > > is to look up the two LSB combination (0 and 1) in > > > > Table 1, which is the 2nd row, which is the > > > > "Always Illegal" combination! So the intelligent > > > > reader (one capable of understanding NAPTs) figures > > > > something's wrong and has to go through the exercise > > > > of matching bit-for-bit and soon determines that > > > > 0 and 1 in the example means 1 and 0 in the table. > > > > > > > > So, for the sake of readability and cleanliness, > > > > I recommend either swapping the table columns or > > > > swapping the bit positions in the pFlags field. > > > > Since by your own admission the bit positions in > > > > the pFlags field is fairly arbitrary, the latter > > > > option should be palatable? > > > > > > > > Regards, > > > > -Sudhir > > > > > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > Trebia Networks, Inc. Sudhir.Srinivasan@trebia.com > > > > 978-929-0830 x139 www.trebia.com > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > > > > > > -----Original Message----- > > > > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > > > > Sent: Sunday, November 25, 2001 7:42 AM > > > > > To: IPS Reflector > > > > > Cc: Sudhir Srinivasan > > > > > Subject: Re: FCIP: v0.7 editorial suggestion > > > > > > > > > > > > > > > I believe that the logic and conceptual structure of > > > > > Table 1 will be less clear if the columns are swapped. > > > > > For better or for worse, the positions of the bits in > > > > > Figure 8 represents the order in which they were > > > > > defined not the logic and purpose of the bits. > > > > > So the figure and the table are disjoint, as would > > > > > be the case for any design that evolved over time. > > > > > > > > > > Finally, I find it impossible to believe that anyone > > > > > capable of understanding NAPTs could become confused > > > > > by Figure 8 and Table 1. > > > > > > > > > > Thanks. > > > > > > > > > > Ralph... > > > > > > > > > > Sudhir Srinivasan wrote: > > > > > > > > > > > Ralph, > > > > > > > > > > > > An editorial suggestion: > > > > > > > > > > > > The pFlags definition is: > > > > > > |----------------Bit--------------------| > > > > > > | | > > > > > > | 31 30 29 28 27 26 25 24 | > > > > > > +-----------------------------+----+----+ > > > > > > | Reserved | Ch | SF | > > > > > > +-----------------------------+----+----+ > > > > > > Fig. 8 pFlags Field Bits > > > > > > > > > > > > with the "Ch" bit to the left of the "SF" bit. > > > > > > > > > > > > I suggest the first two columns in Table 1 be swapped > > > > > > to reflect this. When looking at Figure 9, it was > > > > > > slightly confusing at first to map the pFlags value > > > > > > shown in the Short Frame header back to Table 1. > > > > > > > > > > > > Regards, > > > > > > Sudhir > > > > > > > > > > From owner-ips@ece.cmu.edu Thu Nov 29 13:29:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA29791 for ; Thu, 29 Nov 2001 13:29:40 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fATHCue05855 for ips-outgoing; Thu, 29 Nov 2001 12:12:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([64.214.104.163]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fATHCtl05849 for ; Thu, 29 Nov 2001 12:12:55 -0500 (EST) Received: from MURALIRLAPTOP (dummy172.lightsand.com [192.168.1.172]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id JAA20104; Thu, 29 Nov 2001 09:12:41 -0800 (PST) From: "Murali Rajagopal" To: "Sudhir Srinivasan" , Cc: Subject: RE: FCIP: Echoing a Changed Short Frame? Date: Thu, 29 Nov 2001 09:11:50 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <63616B261CEFD411834D00D0B7A86CF7243690@lucy> Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sudhir: I think your guess about the purpose and subsequent behavior of the changed CH bit is correct. This behavior keeps the protocol simple without getting into an ellaborate negotiation cycle. -Murali -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Sudhir Srinivasan Sent: Wednesday, November 28, 2001 9:27 PM To: ips@ece.cmu.edu Cc: sudhir.srinivasan@trebia.com Subject: FCIP: Echoing a Changed Short Frame? Section 8.1.2 of v0.7 of the FCIP draft states in the last three paras on page 25 that the echoed Short Frame must match exactly the one that was originated; otherwise the connection SHALL be closed. Section 8.1.5 on pages 28-29 describes how the receiver of the Short Frame can change the contents of the frame and then SHALL echo it back to the originator (with the CH bit set). What's the purpose of the Change bit and echoing a changed Short Frame if the originator will simply terminate the connection? Is the expectation that the originator can capture the changed info and try to set up the connection again with the new information? If so, would it not be easier to just have the originator respond with another Short Frame with the new information and repeat this back-and-forth until the echo'ed SF has no changes or either side decides to give up? Thanks, - Sudhir ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trebia Networks, Inc. Sudhir.Srinivasan@trebia.com 978-929-0830 x139 www.trebia.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From owner-ips@ece.cmu.edu Thu Nov 29 15:38:28 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA09930 for ; Thu, 29 Nov 2001 15:38:27 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fATJXiM17595 for ips-outgoing; Thu, 29 Nov 2001 14:33:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.bld.us.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fATJXgl17588 for ; Thu, 29 Nov 2001 14:33:42 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e34.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA41964 for ; Thu, 29 Nov 2001 14:30:53 -0500 Received: from d03nm041.boulder.ibm.com (d03nm041.boulder.ibm.com [9.99.140.41]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fATJXaH28130 for ; Thu, 29 Nov 2001 12:33:36 -0700 Subject: iSCSI: Digests and Multiple Connections To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Prasenjit Sarkar" Date: Thu, 29 Nov 2001 11:33:20 -0800 X-MIMETrack: Serialize by Router on D03NM041/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/29/2001 11:33:35 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk The standard says: When Initiator and Target agree on a digest, this digest MUST be used for every PDU in full feature phase. Also, it appears that digests are a session-level parameter. Scenario: The first connection of a multiple-connection session agrees on digests and proceeds to full-feature phase with digests. What about the subsequent connections that try to login? Do these connections suspend digests during non-full feature phase and turn it on during full feature. The standard is silent on this issue. Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose From owner-ips@ece.cmu.edu Thu Nov 29 17:53:16 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA19108 for ; Thu, 29 Nov 2001 17:53:16 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fATLdBG28072 for ips-outgoing; Thu, 29 Nov 2001 16:39:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fATLdAl28067 for ; Thu, 29 Nov 2001 16:39:10 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id WAA17614 for ; Thu, 29 Nov 2001 22:39:03 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fATLdIR73468 for ; Thu, 29 Nov 2001 22:39:18 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: Digests and Multiple Connections X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 From: "Julian Satran" Message-ID: Date: Thu, 29 Nov 2001 23:39:07 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 29/11/2001 23:39:10, Serialize complete at 29/11/2001 23:39:10 Content-Type: multipart/alternative; boundary="=_alternative 0076D7E642256B13_=" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multipart message in MIME format. --=_alternative 0076D7E642256B13_= Content-Type: text/plain; charset="us-ascii" Prasenjit, Digests and security are connection level characteristics. Regards, Julo Prasenjit Sarkar/Almaden/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 11/29/2001 09:33 PM To: ips@ece.cmu.edu cc: Subject: iSCSI: Digests and Multiple Connections The standard says: When Initiator and Target agree on a digest, this digest MUST be used for every PDU in full feature phase. Also, it appears that digests are a session-level parameter. Scenario: The first connection of a multiple-connection session agrees on digests and proceeds to full-feature phase with digests. What about the subsequent connections that try to login? Do these connections suspend digests during non-full feature phase and turn it on during full feature. The standard is silent on this issue. Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose --=_alternative 0076D7E642256B13_= Content-Type: text/html; charset="us-ascii"
Prasenjit,

Digests and security are connection level characteristics.

Regards,
Julo


Prasenjit Sarkar/Almaden/IBM@IBMUS
Sent by: owner-ips@ece.cmu.edu

11/29/2001 09:33 PM

       
        To:        ips@ece.cmu.edu
        cc:        
        Subject:        iSCSI: Digests and Multiple Connections

       


The standard says:

When Initiator and Target agree on a digest, this digest MUST be used
for every PDU in full feature phase.

Also, it appears that digests are a session-level parameter.

Scenario:

The first connection of a multiple-connection session agrees on digests
and proceeds to full-feature phase with digests.

What about the subsequent connections that try to login? Do these
connections
suspend digests during non-full feature phase and turn it on during
full feature.

The standard is silent on this issue.


  Prasenjit Sarkar
  Research Staff Member
  IBM Almaden Research
  San Jose

--=_alternative 0076D7E642256B13_=-- From owner-ips@ece.cmu.edu Thu Nov 29 18:45:40 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA22257 for ; Thu, 29 Nov 2001 18:45:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fATMltr03675 for ips-outgoing; Thu, 29 Nov 2001 17:47:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fATMlol03646 for ; Thu, 29 Nov 2001 17:47:50 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA12174 for ; Thu, 29 Nov 2001 17:45:04 -0500 Received: from d03nm041.boulder.ibm.com (d03nm041.boulder.ibm.com [9.99.140.41]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fATMlmH230488 for ; Thu, 29 Nov 2001 15:47:48 -0700 Subject: Re: iSCSI: Digests and Multiple Connections To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Prasenjit Sarkar" Date: Thu, 29 Nov 2001 14:47:31 -0800 X-MIMETrack: Serialize by Router on D03NM041/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/29/2001 02:47:48 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, My question was about digests only and you ought to put that assertion in Appendix D to avoid confusion. Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose Julian Satran/Haifa/I To: ips@ece.cmu.edu BM@IBMIL cc: Sent by: Subject: Re: iSCSI: Digests and Multiple Connections owner-ips@ece. cmu.edu 11/29/2001 01:39 PM Prasenjit, Digests and security are connection level characteristics. Regards, Julo Prasenjit Sarkar/Almaden/IBM@IBMUS To: ips@ece.cmu.edu Sent by: owner-ips@ece.cmu.edu cc: Subject: iSCSI: Digests and Multiple Connections 11/29/2001 09:33 PM The standard says: When Initiator and Target agree on a digest, this digest MUST be used for every PDU in full feature phase. Also, it appears that digests are a session-level parameter. Scenario: The first connection of a multiple-connection session agrees on digests and proceeds to full-feature phase with digests. What about the subsequent connections that try to login? Do these connections suspend digests during non-full feature phase and turn it on during full feature. The standard is silent on this issue. Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose From owner-ips@ece.cmu.edu Thu Nov 29 20:10:53 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA27370 for ; Thu, 29 Nov 2001 20:10:52 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAU04P009090 for ips-outgoing; Thu, 29 Nov 2001 19:04:25 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lucy.trebia.com ([65.192.191.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAU04Nl09082 for ; Thu, 29 Nov 2001 19:04:23 -0500 (EST) Received: from trebiaachadda (140.186.40.85 [140.186.40.85]) by lucy.trebia.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id X6ZMS1ZR; Thu, 29 Nov 2001 19:02:36 -0500 Message-ID: <007e01c17932$7d40d0d0$9a7fa8c0@trebiaachadda> From: "Anshul Chadda" To: "Prasenjit Sarkar" Cc: References: Subject: Re: iSCSI: Digests and Multiple Connections Date: Thu, 29 Nov 2001 19:03:44 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit The field "Use:IO" implicitly makes a key a connection level characteristic. This holds for Digests as well as Markers. But I agree it is not explicitly stated. Anshul ----- Original Message ----- From: "Prasenjit Sarkar" To: "Julian Satran" Cc: Sent: Thursday, November 29, 2001 5:47 PM Subject: Re: iSCSI: Digests and Multiple Connections > > Julian, > > My question was about digests only and you ought to put that > assertion in Appendix D to avoid confusion. > > > Prasenjit Sarkar > Research Staff Member > IBM Almaden Research > San Jose > > > > > Julian > Satran/Haifa/I To: ips@ece.cmu.edu > BM@IBMIL cc: > Sent by: Subject: Re: iSCSI: Digests and Multiple Connections > owner-ips@ece. > cmu.edu > > > 11/29/2001 > 01:39 PM > > > > > > > Prasenjit, > > Digests and security are connection level characteristics. > > Regards, > Julo > > > Prasenjit > Sarkar/Almaden/IBM@IBMUS To: ips@ece.cmu.edu > Sent by: owner-ips@ece.cmu.edu cc: > Subject: iSCSI: Digests > and Multiple Connections > 11/29/2001 09:33 PM > > > > > > > The standard says: > > When Initiator and Target agree on a digest, this digest MUST be used > for every PDU in full feature phase. > > Also, it appears that digests are a session-level parameter. > > Scenario: > > The first connection of a multiple-connection session agrees on digests > and proceeds to full-feature phase with digests. > > What about the subsequent connections that try to login? Do these > connections > suspend digests during non-full feature phase and turn it on during > full feature. > > The standard is silent on this issue. > > > Prasenjit Sarkar > Research Staff Member > IBM Almaden Research > San Jose > > > > > > From owner-ips@ece.cmu.edu Thu Nov 29 22:07:25 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA04004 for ; Thu, 29 Nov 2001 22:07:25 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAU1dA015299 for ips-outgoing; Thu, 29 Nov 2001 20:39:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siaar1aa.compuserve.com (siaar1aa.compuserve.com [149.174.40.9]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAU1d7l15295 for ; Thu, 29 Nov 2001 20:39:08 -0500 (EST) Received: (from mailgate@localhost) by siaar1aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id UAA05704 for ips@ece.cmu.edu; Thu, 29 Nov 2001 20:39:01 -0500 (EST) Received: from compuserve.com (dal-tgn-tks-vty16.as.wcom.net [216.192.230.16]) by siaar1aa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id UAA05666; Thu, 29 Nov 2001 20:38:49 -0500 (EST) Message-ID: <3C06E46E.4E5B0161@compuserve.com> Date: Thu, 29 Nov 2001 19:44:14 -0600 From: Ralph Weber Reply-To: roweber@acm.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector CC: Sudhir Srinivasan Subject: Re: FCIP: v0.7 editorial suggestion References: <63616B261CEFD411834D00D0B7A86CF75760C0@lucy> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Sudhir, The people building FCIP devices place importance on having the SF bit be the low order bit in a byte. They believe that if SF should become a frame type field in a yet to be considered replacement for FCIP, then there is value in having that coded value in the low order bits of a byte. In short, I proposed what you suggested and got shot down in flames. Ralph... Sudhir Srinivasan wrote: > Ralph, > > If the bit positions are going to be changed, I still > think it would be better to simply swap the positions > of the Ch and SF bits in the pFlags bit. This has both > benefits -- it allows for the "growth" of the SF bit > as you outline below AND it makes the table consistent > with the bit field definition and hence makes it easier > to read/verify Figure 9. > > Regards, > Sudhir > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Trebia Networks, Inc. Sudhir.Srinivasan@trebia.com > 978-929-0830 x139 www.trebia.com > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > -----Original Message----- > > From: Ralph Weber [mailto:ralphoweber@compuserve.com] > > Sent: Wednesday, November 28, 2001 6:24 PM > > To: Sudhir Srinivasan > > Cc: Murali Rajagopal > > Subject: Re: FCIP: v0.7 editorial suggestion > > > > > > Sudhir, > > > > The FCIP authors have agreed to move the Ch bit from > > bit 25 to bit 31. This change has the additional > > advantage that the SF bit can grow to the 'frame type' > > field if a compelling need arises. > > > > The new figure will look like this. > > > > |----------------Bit--------------------| > > | | > > | 31 30 29 28 27 26 25 24 | > > +----+-----------------------------+----+ > > | Ch | Reserved | SF | > > +----+-----------------------------+----+ > > > > Fig. 8 pFlags Field Bits > > > > I don't know if this will fully lay to rest the > > confusion you experienced, but it should help. > > > > Thanks. > > > > Ralph... From owner-ips@ece.cmu.edu Thu Nov 29 22:43:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA06442 for ; Thu, 29 Nov 2001 22:43:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAU2jWd19573 for ips-outgoing; Thu, 29 Nov 2001 21:45:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAU2jUl19569 for ; Thu, 29 Nov 2001 21:45:30 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id DAA242162 for ; Fri, 30 Nov 2001 03:45:22 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAU2jcR18506 for ; Fri, 30 Nov 2001 03:45:39 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: iSCSI: Digests and Multiple Connections X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 From: "Julian Satran" Message-ID: Date: Fri, 30 Nov 2001 04:45:27 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 30/11/2001 04:45:30, Serialize complete at 30/11/2001 04:45:30 Content-Type: multipart/alternative; boundary="=_alternative 000F2DB642256B14_=" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multipart message in MIME format. --=_alternative 000F2DB642256B14_= Content-Type: text/plain; charset="us-ascii" Prasenjit, The use is clearly indicated as IO in appendix D. However I will remove the statement about keys being session only - as all the keys state clearly what their use is and the statement creates some confusion. Regards, Julo Prasenjit Sarkar@IBMUS 11/30/2001 12:47 AM To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu From: Prasenjit Sarkar/Almaden/IBM@IBMUS Subject: Re: iSCSI: Digests and Multiple Connections Julian, My question was about digests only and you ought to put that assertion in Appendix D to avoid confusion. Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose Julian Satran/Haifa/IBM@IBMIL Sent by: owner-ips@ece.cmu.edu 11/29/2001 01:39 PM To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: Digests and Multiple Connections Prasenjit, Digests and security are connection level characteristics. Regards, Julo Prasenjit Sarkar/Almaden/IBM@IBMUS Sent by: owner-ips@ece.cmu.edu 11/29/2001 09:33 PM To: ips@ece.cmu.edu cc: Subject: iSCSI: Digests and Multiple Connections The standard says: When Initiator and Target agree on a digest, this digest MUST be used for every PDU in full feature phase. Also, it appears that digests are a session-level parameter. Scenario: The first connection of a multiple-connection session agrees on digests and proceeds to full-feature phase with digests. What about the subsequent connections that try to login? Do these connections suspend digests during non-full feature phase and turn it on during full feature. The standard is silent on this issue. Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose --=_alternative 000F2DB642256B14_= Content-Type: text/html; charset="us-ascii"
Prasenjit,

The use is clearly indicated as IO in appendix D.
However I will remove the statement about keys being session only - as all the keys state clearly what their use is and the statement creates some confusion.

Regards,
Julo


Prasenjit Sarkar@IBMUS

11/30/2001 12:47 AM


        To:        Julian Satran/Haifa/IBM@IBMIL
        cc:        ips@ece.cmu.edu
        From:        Prasenjit Sarkar/Almaden/IBM@IBMUS
        Subject:        Re: iSCSI: Digests and Multiple ConnectionsLink
 





Julian,

My question was about digests only and you ought to put that
assertion in Appendix D to avoid confusion.


  Prasenjit Sarkar
  Research Staff Member
  IBM Almaden Research
  San Jose


Julian Satran/Haifa/IBM@IBMIL
Sent by: owner-ips@ece.cmu.edu

11/29/2001 01:39 PM

       
        To:        ips@ece.cmu.edu
        cc:        
        Subject:        Re: iSCSI: Digests and Multiple Connections

       



Prasenjit,

Digests and security are connection level characteristics.

Regards,
Julo

Prasenjit Sarkar/Almaden/IBM@IBMUS
Sent by: owner-ips@ece.cmu.edu

11/29/2001 09:33 PM

       
       To:        ips@ece.cmu.edu
       cc:        
       Subject:        iSCSI: Digests and Multiple Connections

     



The standard says:

When Initiator and Target agree on a digest, this digest MUST be used
for every PDU in full feature phase.

Also, it appears that digests are a session-level parameter.

Scenario:

The first connection of a multiple-connection session agrees on digests
and proceeds to full-feature phase with digests.

What about the subsequent connections that try to login? Do these
connections
suspend digests during non-full feature phase and turn it on during
full feature.

The standard is silent on this issue.


 Prasenjit Sarkar
 Research Staff Member
 IBM Almaden Research
 San Jose



--=_alternative 000F2DB642256B14_=-- From owner-ips@ece.cmu.edu Thu Nov 29 22:54:06 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA06744 for ; Thu, 29 Nov 2001 22:54:06 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAU2r6l20071 for ips-outgoing; Thu, 29 Nov 2001 21:53:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAU2r4l20062 for ; Thu, 29 Nov 2001 21:53:04 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id DAA18546 for ; Fri, 30 Nov 2001 03:52:53 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAU2r8R43626 for ; Fri, 30 Nov 2001 03:53:09 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 From: "Julian Satran" Message-ID: Date: Fri, 30 Nov 2001 04:52:56 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 30/11/2001 04:53:01, Serialize complete at 30/11/2001 04:53:01 Content-Type: multipart/alternative; boundary="=_alternative 000FDD7342256B14_=" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multipart message in MIME format. --=_alternative 000FDD7342256B14_= Content-Type: text/plain; charset="us-ascii" Robert, Sorry for the confusion. The value 0 was on purpose changed from its "unlimited" meaning in mode pages to a syntactic "don't care" in some negotiations to simplify exchanges where one of the parties does not care. In that case the "burden" of the decision is upon the responder. Julo "Robert D. Russell" 11/29/2001 12:30 PM To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Julian: If I understand correctly what Tow was asking for, and your response to him, this means that the statement: "A value of 0 MAY be used as a "don't care" offer in negotiations." should be removed from the description in Appendix D of draft 9 for the 6 keys: MaxRecvPDULength MaxBurstSize FirstBurstSize LogoutLoginMaxTime LogoutLoginMinTime MaxOutstandingR2T The statement in section 2.2.4 on page 31 should also be removed: "For numerical negotiations, the value 0 MAY be specified by the offering party as a "don't care"/"unlimited" value for parameters that explicitly allow it; in this case, the responder may choose any legal value for the parameter." Thanks, Bob Russell InterOperability Lab University of New Hampshire rdr@iol.unh.edu 603-862-3774 On Thu, 29 Nov 2001, Julian Satran wrote: > 0 as a stand in for "whatever" is out of version 09. > We don't see any need for it. It was originally provide for compatibility > with the T10 docs when we had values "residing" in mode pages accessible > by SCSI commands. > > Julo > > > > > Tow Wang > Sent by: owner-ips@ece.cmu.edu > 28-11-01 01:55 > > > To: "Robert D. Russell" , ips@ece.cmu.edu > cc: > Subject: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest > > > > Hi all. > > I fully agree with the suggestions for issue 4. If we really need to > reserve a > value to mean "unlimited" or "infinite", could we use a value of all bits > set to > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be great > for > most 32-bit processors and above.) This would make arithmetic comparisons > more > intuitive, IMHO. > > "Robert D. Russell" wrote: > > > Attached are the new issues that arose during the iSCSI plugfest > > at UNH on Wednesday 31-Oct-2001. > > > > (Note: these issues do not take into account any modifications or > > clarifications that occured in the standard due to the issues raised > > on Monday or Tuesday.) > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > now allow: "A value of 0 indicates no limit." > > > > Is this useful? Does it buy anything? > > > > The difficulties implementers are having with this are: > > > > 1) It is a special case. > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > 3) It violates the min/max function normally used for the key. > > 4) There is always a limit anyway. > > > > Consider FirstBurstSize, which can have a value that is described > > as "<0|number-64-2**24>", and for which the minimum of the 2 numbers > > is selected. > > > > I one side offers 0 to mean unlimited, and the other side > > has a limit, it will reply with that limit, say 128 Kbytes. > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > The statement in the standard that "the minimum of the 2 numbers is > > selected" is therefore wrong when one of the numbers is 0. > > > > Furthermore, when an initiator or target receives an offer for one > > of these keys, it cannot simply check that the offered value is > > legal by testing it against some minimum and maximum. It must first > > check for 0 and then only if that check shows the value is non-zero > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > Finally, there is always a limit. If nothing else it is the > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > requesting the transfer. It is useless to specify a FirstBurstSize > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > the largest possible DataSegmentLength in any PDU that can use > > this value is 2**24-1. > > > > The suggestion is to just eliminate this special case of 0 and > require > > that the range 64-to-(2**24-1) be used instead -- it has exactly the > > same effect in all cases, it is easier to describe in the standard > > because it avoids all the extra words, and it is easier to code > > because it avoids all the special cases. > > > > NOTE: the standard should specify the limit in the ranges for > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > DataSegmentLength field and therefore can never be used. > > -- > Tow Wang > Adaptec Software Engineer Telephone: 408 957 4838 > Mail stop 46 800 869 8883 x4838 > 691 South Milpitas Boulevard Fax: 530 686 8023 > Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com > > > > > --=_alternative 000FDD7342256B14_= Content-Type: text/html; charset="us-ascii"
Robert,

Sorry for the confusion.
The value 0 was on purpose changed from its "unlimited" meaning in mode pages to a syntactic "don't care"
in some negotiations to simplify exchanges where one of the parties does not care.
In that case the "burden" of the decision is upon the responder.


Julo


"Robert D. Russell" <rdr@io.iol.unh.edu>

11/29/2001 12:30 PM

       
        To:        Julian Satran/Haifa/IBM@IBMIL
        cc:        ips@ece.cmu.edu
        Subject:        Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest

       


Julian:

If I understand correctly what Tow was asking for, and your response to
him, this means that the statement:

"A value of 0 MAY be used as a "don't care" offer in negotiations."

should be removed from the description in Appendix D of draft 9 for
the 6 keys:

MaxRecvPDULength
MaxBurstSize
FirstBurstSize
LogoutLoginMaxTime
LogoutLoginMinTime
MaxOutstandingR2T


The statement in section 2.2.4 on page 31 should also be removed:

"For numerical negotiations, the value 0 MAY be specified by the
offering party as a "don't care"/"unlimited" value for parameters
that explicitly allow it; in this case, the responder may choose any
legal value for the parameter."


Thanks,

Bob Russell
InterOperability Lab
University of New Hampshire
rdr@iol.unh.edu
603-862-3774



On Thu, 29 Nov 2001, Julian Satran wrote:

> 0 as a stand in for "whatever" is out of version 09.
> We don't see any need for it. It was originally provide for compatibility
> with the T10 docs when we had values "residing" in mode pages accessible
> by SCSI commands.
>
> Julo
>
>
>
>
> Tow Wang <Tow_Wang@adaptec.com>
> Sent by: owner-ips@ece.cmu.edu
> 28-11-01 01:55
>
>
>         To:     "Robert D. Russell" <rdr@io.iol.unh.edu>, ips@ece.cmu.edu
>         cc:
>         Subject:        Value of "unlimited". Was Re: iSCSI: current UNH Plugfest
>
>
>
> Hi all.
>
> I fully agree with the suggestions for issue 4. If we really need to
> reserve a
> value to mean "unlimited" or "infinite", could we use a value of all bits
> set to
> one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be great
> for
> most 32-bit processors and above.) This would make arithmetic comparisons
> more
> intuitive, IMHO.
>
> "Robert D. Russell" wrote:
>
> > Attached are the new issues that arose during the iSCSI plugfest
> > at UNH on Wednesday 31-Oct-2001.
> >
> > (Note: these issues do not take into account any modifications or
> > clarifications that occured in the standard due to the issues raised
> > on Monday or Tuesday.)
> >
> > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize)
> >    now allow: "A value of 0 indicates no limit."
> >
> >    Is this useful?  Does it buy anything?
> >
> >    The difficulties implementers are having with this are:
> >
> >    1) It is a special case.
> >    2) It causes discontinuous ranges (for example, [0,64..2**24])
> >    3) It violates the min/max function normally used for the key.
> >    4) There is always a limit anyway.
> >
> >    Consider FirstBurstSize, which can have a value that is described
> >    as "<0|number-64-2**24>", and for which the minimum of the 2 numbers
> >    is selected.
> >
> >    I one side offers 0 to mean unlimited, and the other side
> >    has a limit, it will reply with that limit, say 128 Kbytes.
> >    Therefore, the result is not min(0, 128K) but rather max(0, 128K).
> >    The statement in the standard that "the minimum of the 2 numbers is
> >    selected" is therefore wrong when one of the numbers is 0.
> >
> >    Furthermore, when an initiator or target receives an offer for one
> >    of these keys, it cannot simply check that the offered value is
> >    legal by testing it against some minimum and maximum.  It must first
> >    check for 0 and then only if that check shows the value is non-zero
> >    can it do the min/max range check for legality (i.e., 64-2**24).
> >
> >    Finally, there is always a limit. If nothing else it is the
> >    limit imposed by the 24-bit DataSegmentLength field of the PDU
> >    requesting the transfer.  It is useless to specify a FirstBurstSize
> >    (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because
> >    the largest possible DataSegmentLength in any PDU that can use
> >    this value is 2**24-1.
> >
> >    The suggestion is to just eliminate this special case of 0 and
> require
> >    that the range 64-to-(2**24-1) be used instead -- it has exactly the
> >    same effect in all cases, it is easier to describe in the standard
> >    because it avoids all the extra words, and it is easier to code
> >    because it avoids all the special cases.
> >
> >    NOTE: the standard should specify the limit in the ranges for
> >    MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead
> >    of 2**24.  The number 2**24 cannot be represented in the 24-bit
> >    DataSegmentLength field and therefore can never be used.
>
> --
> Tow Wang
> Adaptec Software Engineer       Telephone: 408 957 4838
> Mail stop 46                               800 869 8883 x4838
> 691 South Milpitas Boulevard    Fax:       530 686 8023
> Milpitas, California 95035      E-mail:    Tow_Wang@adaptec.com
>
>
>
>
>

--=_alternative 000FDD7342256B14_=-- From owner-ips@ece.cmu.edu Thu Nov 29 23:45:22 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08266 for ; Thu, 29 Nov 2001 23:45:22 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAU3iZw23361 for ips-outgoing; Thu, 29 Nov 2001 22:44:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f167.pav2.hotmail.com [64.4.37.167]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAU3iWl23356 for ; Thu, 29 Nov 2001 22:44:32 -0500 (EST) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Thu, 29 Nov 2001 19:44:27 -0800 Received: from 208.54.59.26 by pv2fd.pav2.hotmail.msn.com with HTTP; Fri, 30 Nov 2001 03:44:25 GMT X-Originating-IP: [208.54.59.26] From: "Bernard Aboba" To: cbm@rose.hp.com, ips@ece.cmu.edu Subject: Re: Comments on ips security draft Date: Thu, 29 Nov 2001 19:44:25 -0800 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 30 Nov 2001 03:44:27.0047 (UTC) FILETIME=[4F28CB70:01C17951] Sender: owner-ips@ece.cmu.edu Precedence: bulk >- Section 2.3, page 10. "Conformant iSCSI security implementations >MUST support ESP in transport mode. " > I assume it should be tunnel mode.... There is now a discussion going on on the SAAG mailing list about whether it should be tunnel mode or transport mode. We'll talk more about this at IETF 52. One significant issue with tunnel mode for iSCSI doesn't arise in iFCP or FCIP, since with iSCSI initiators, addresses may be dynamically assigned, whereas with iFCP and FCIP, the expectation seems to be that the IP addresses will be static. Static addressing for tunnel mode is a lot simpler. For one thing, since the addresses and configuration are static, you don't need to configure tunnel addresses and parameters on the fly. Another implication is that you can use main mode with shared secrets, and have a unique shared secret for each endpoint. For iSCSI with dynamically assigned addresses, life becomes more complicated. The implication is that if an iSCSI initiator with a dynamically assigned address were to use tunnel mode, the tunnel mode gateway at the other end would need to assign an IP address and otherwise configure the tunnel. The IPSRA WG has recently put out a Proposed Standard that addresses this issue, draft-ietf-ipsec-dhcp-13.txt. So if you want to do tunnel mode within iSCSI and have it interoperate with the IETF Proposed Standard for Tunnel mode configuration, you'll need to sign up for this too. The problem is that the goal was to be able to reuse existing IPsec gateways -- if you have to significantly update the gateway, then this cancels one of the advantages of doing tunnel mode. In the absence of referencing the IPSRA WG standard for tunnel mode configuration, there would not be enough detail to ensure interoperability between iSCSI security implementations. It would be highly likely that some vendors would ship; their existing gateways which typically do mode config (a proprietary protocol that is not on standards track and may never even be published as an RFC) while others would do DHCP config, and we would not have interoperability. Given that configuration is an essential part of IPsec tunnel mode setup for dynamically addressed hosts, it is probably required that we reference the proposed configuration method normatively. This would imply that if we go with anything other than the IETF Proposed Standard, any documents, such as iSCSI, which referenced these proprietary methods could not become IETF standards. So if I believe this means that if we go with tunnel mode, we probably need a normative reference to draft-ietf-ipsec-dhcp-13.txt as well. As the saying goes "think carefully about what you ask for -- you might get it." > >- Section 3.3, page 17. The well-known target port for iSCSI may > be updated to 3260. This will be reflected in draft -07. A version of this in progress is available at http://www.drizzle.com/~aboba/RDMA/draft-ietf-ips-security-07.txt. > >- Section 3.4, page 18. > "[d] a specific connection be closed at the Target's request. LP > Command [d] is distinct from [b] in that it indicates that the > connection is being closed in response to a request from the Target > for it to be closed. Due to asymmetries in the iSCSI protocol, > Targets cannot close a connection on their own initiative." > > It needs to be reworded, since this is incorrect. [d] is the same as > [b]. Do you have specific language you'd like to suggest? > >- I may be missing something, but the table listing IKE implementation > sizes on page 28 seems completely out-of-context in the middle of > a rekeying frequency discussion. Moved the table to be closer to the relevant text. > >Thanks. >-- >Mallikarjun > > >Mallikarjun Chadalapaka >Networked Storage Architecture >Network Storage Solutions Organization >MS 5668 Hewlett-Packard, Roseville. >cbm@rose.hp.com _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp From owner-ips@ece.cmu.edu Fri Nov 30 02:36:35 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA26514 for ; Fri, 30 Nov 2001 02:36:35 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAU6laV04735 for ips-outgoing; Fri, 30 Nov 2001 01:47:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAU6lXl04730 for ; Fri, 30 Nov 2001 01:47:33 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id BAA14262 for ; Fri, 30 Nov 2001 01:44:47 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAU6lVH188996 for ; Thu, 29 Nov 2001 23:47:31 -0700 Importance: Normal Subject: iSCSI DataSN Can equal 2**32 To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Thu, 29 Nov 2001 22:46:45 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/29/2001 11:47:31 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, At 3.7.5 there is a statement that data sequence MUST contain less then 2**32-1 numbered PDUs. I did not understand that since the count starts at 0, which should mean that 2**32 PDUs could be possible. So I do not understand the -1 in the count of PDUs, and I then do not understand "less then" (which I guess would mean 2**32-2). I think the number should be able to go from 0 to 2**32-1, inclusive, which is 2*32 different PDUs and not "less then 2**32-1". Perhaps the statement should read that the sequence number of the Data PDU can not be greater then 2**32-1. Or the total number of Data PDUs in a Sequence can not be greater then 2**32. But perhaps I missed something. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Fri Nov 30 05:16:39 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA01201 for ; Fri, 30 Nov 2001 05:16:39 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAU9TFN13716 for ips-outgoing; Fri, 30 Nov 2001 04:29:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAU9TEl13711 for ; Fri, 30 Nov 2001 04:29:14 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.140.23]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id EAA15444 for ; Fri, 30 Nov 2001 04:26:28 -0500 Received: from d03nm065.boulder.ibm.com (d03nm065.boulder.ibm.com [9.99.140.49]) by westrelay02.boulder.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAU9TBH76828 for ; Fri, 30 Nov 2001 02:29:12 -0700 X-Priority: 1 (High) Importance: Normal Subject: iSCSI: Flags in the Data-in PDU To: "Julian Satran" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd" Date: Fri, 30 Nov 2001 01:28:25 -0800 X-MIMETrack: Serialize by Router on D03NM065/03/M/IBM(Release 5.0.8 |June 18, 2001) at 11/30/2001 02:29:11 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk The current draft in 3.3.8 states that bit 3-6 is not used and should be set to zero. I think this is left over from before we added the A bit which is bit 6. The Draft should be changed to say bit 3-5 is not used and should be set to zero. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Fri Nov 30 09:00:33 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA11166 for ; Fri, 30 Nov 2001 09:00:33 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUDQ0T07742 for ips-outgoing; Fri, 30 Nov 2001 08:26:00 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUDPvl07737 for ; Fri, 30 Nov 2001 08:25:57 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA101466 for ; Fri, 30 Nov 2001 14:25:45 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay01.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAUDPiR32134 for ; Fri, 30 Nov 2001 14:25:46 +0100 To: ips@ece.cmu.edu MIME-Version: 1.0 Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 From: "Julian Satran" Message-ID: Date: Fri, 30 Nov 2001 15:25:32 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 30/11/2001 15:25:42, Serialize complete at 30/11/2001 15:25:42 Content-Type: multipart/alternative; boundary="=_alternative 0049A5E142256B14_=" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multipart message in MIME format. --=_alternative 0049A5E142256B14_= Content-Type: text/plain; charset="us-ascii" Bob, It was claimed that a value of 0 - used uniformly independent of the selection rule (min or max) would be easier to handle. I personally don't care - but would like to avoid another raging war on defaults. If you find some other supporters I am ready to remove the don't care altogether. Regards, Julo "Robert D. Russell" 11/30/2001 01:17 PM To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Julian: I would like to request that using the value of 0 as a "don't care" in numeric negotiations be removed completely. You said it is there to "simplify exchanges", but I do not see how it simplifies them because it makes a special case where none is needed. If the numerical choice function is "min", then to indicate "don't care" simply offer the maximum value in the allowed range (which clearly the offering side must be able to deal with if it really doesn't care). The responder they replies with any number it wants to choose from the allowed range, because by definition that will be the "min" value. When the numerical choice function is "max", the offering side simply offers the minimum value in the allowed range, etc. By making 0 a special case, extra code is necessary on the receiving side, and extra wording is needed in the standard, yet there is no gain in functionality. This is why I would like to have it removed -- it is an unnecessary special case. Thanks, Bob Russell On Fri, 30 Nov 2001, Julian Satran wrote: > Robert, > > Sorry for the confusion. > The value 0 was on purpose changed from its "unlimited" meaning in mode > pages to a syntactic "don't care" > in some negotiations to simplify exchanges where one of the parties does > not care. > In that case the "burden" of the decision is upon the responder. > > > Julo > > > > > "Robert D. Russell" > 11/29/2001 12:30 PM > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest > > > > Julian: > > If I understand correctly what Tow was asking for, and your response to > him, this means that the statement: > > "A value of 0 MAY be used as a "don't care" offer in negotiations." > > should be removed from the description in Appendix D of draft 9 for > the 6 keys: > > MaxRecvPDULength > MaxBurstSize > FirstBurstSize > LogoutLoginMaxTime > LogoutLoginMinTime > MaxOutstandingR2T > > > The statement in section 2.2.4 on page 31 should also be removed: > > "For numerical negotiations, the value 0 MAY be specified by the > offering party as a "don't care"/"unlimited" value for parameters > that explicitly allow it; in this case, the responder may choose any > legal value for the parameter." > > > Thanks, > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > > On Thu, 29 Nov 2001, Julian Satran wrote: > > > 0 as a stand in for "whatever" is out of version 09. > > We don't see any need for it. It was originally provide for > compatibility > > with the T10 docs when we had values "residing" in mode pages accessible > > by SCSI commands. > > > > Julo > > > > > > > > > > Tow Wang > > Sent by: owner-ips@ece.cmu.edu > > 28-11-01 01:55 > > > > > > To: "Robert D. Russell" , > ips@ece.cmu.edu > > cc: > > Subject: Value of "unlimited". Was Re: iSCSI: current UNH > Plugfest > > > > > > > > Hi all. > > > > I fully agree with the suggestions for issue 4. If we really need to > > reserve a > > value to mean "unlimited" or "infinite", could we use a value of all > bits > > set to > > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be > great > > for > > most 32-bit processors and above.) This would make arithmetic > comparisons > > more > > intuitive, IMHO. > > > > "Robert D. Russell" wrote: > > > > > Attached are the new issues that arose during the iSCSI plugfest > > > at UNH on Wednesday 31-Oct-2001. > > > > > > (Note: these issues do not take into account any modifications or > > > clarifications that occured in the standard due to the issues raised > > > on Monday or Tuesday.) > > > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > > now allow: "A value of 0 indicates no limit." > > > > > > Is this useful? Does it buy anything? > > > > > > The difficulties implementers are having with this are: > > > > > > 1) It is a special case. > > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > > 3) It violates the min/max function normally used for the key. > > > 4) There is always a limit anyway. > > > > > > Consider FirstBurstSize, which can have a value that is described > > > as "<0|number-64-2**24>", and for which the minimum of the 2 > numbers > > > is selected. > > > > > > I one side offers 0 to mean unlimited, and the other side > > > has a limit, it will reply with that limit, say 128 Kbytes. > > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > > The statement in the standard that "the minimum of the 2 numbers is > > > selected" is therefore wrong when one of the numbers is 0. > > > > > > Furthermore, when an initiator or target receives an offer for one > > > of these keys, it cannot simply check that the offered value is > > > legal by testing it against some minimum and maximum. It must > first > > > check for 0 and then only if that check shows the value is non-zero > > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > > > Finally, there is always a limit. If nothing else it is the > > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > > requesting the transfer. It is useless to specify a FirstBurstSize > > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > > the largest possible DataSegmentLength in any PDU that can use > > > this value is 2**24-1. > > > > > > The suggestion is to just eliminate this special case of 0 and > > require > > > that the range 64-to-(2**24-1) be used instead -- it has exactly > the > > > same effect in all cases, it is easier to describe in the standard > > > because it avoids all the extra words, and it is easier to code > > > because it avoids all the special cases. > > > > > > NOTE: the standard should specify the limit in the ranges for > > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 > instead > > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > > DataSegmentLength field and therefore can never be used. > > > > -- > > Tow Wang > > Adaptec Software Engineer Telephone: 408 957 4838 > > Mail stop 46 800 869 8883 x4838 > > 691 South Milpitas Boulevard Fax: 530 686 8023 > > Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com > > > > > > > > > > > > > > --=_alternative 0049A5E142256B14_= Content-Type: text/html; charset="us-ascii"
Bob,

It was claimed that a value of 0 - used uniformly independent of the selection rule (min or max) would be easier to handle.
I personally don't care - but would like to avoid another raging war on defaults.

If you find some other supporters I am ready to remove the don't care altogether.

Regards,
Julo


"Robert D. Russell" <rdr@io.iol.unh.edu>

11/30/2001 01:17 PM

       
        To:        Julian Satran/Haifa/IBM@IBMIL
        cc:        ips@ece.cmu.edu
        Subject:        Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest

       


Julian:

I would like to request that using the value of 0 as a "don't care"
in numeric negotiations be removed completely.  You said it is there
to "simplify exchanges", but I do not see how it simplifies them
because it makes a special case where none is needed.

If the numerical choice function is "min", then to indicate "don't care"
simply offer the maximum value in the allowed range (which clearly the
offering side must be able to deal with if it really doesn't care).
The responder they replies with any number it wants to choose from the
allowed range, because by definition that will be the "min" value.
When the numerical choice function is "max", the offering side
simply offers the minimum value in the allowed range, etc.

By making 0 a special case, extra code is necessary on the receiving side,
and extra wording is needed in the standard, yet there is no gain in
functionality.  This is why I would like to have it removed -- it is
an unnecessary special case.

Thanks,
Bob Russell


On Fri, 30 Nov 2001, Julian Satran wrote:

> Robert,
>
> Sorry for the confusion.
> The value 0 was on purpose changed from its "unlimited" meaning in mode
> pages to a syntactic "don't care"
> in some negotiations to simplify exchanges where one of the parties does
> not care.
> In that case the "burden" of the decision is upon the responder.
>
>
> Julo
>
>
>
>
> "Robert D. Russell" <rdr@io.iol.unh.edu>
> 11/29/2001 12:30 PM
>
>
>         To:     Julian Satran/Haifa/IBM@IBMIL
>         cc:     ips@ece.cmu.edu
>         Subject:        Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest
>
>
>
> Julian:
>
> If I understand correctly what Tow was asking for, and your response to
> him, this means that the statement:
>
> "A value of 0 MAY be used as a "don't care" offer in negotiations."
>
> should be removed from the description in Appendix D of draft 9 for
> the 6 keys:
>
> MaxRecvPDULength
> MaxBurstSize
> FirstBurstSize
> LogoutLoginMaxTime
> LogoutLoginMinTime
> MaxOutstandingR2T
>
>
> The statement in section 2.2.4 on page 31 should also be removed:
>
> "For numerical negotiations, the value 0 MAY be specified by the
> offering party as a "don't care"/"unlimited" value for parameters
> that explicitly allow it; in this case, the responder may choose any
> legal value for the parameter."
>
>
> Thanks,
>
> Bob Russell
> InterOperability Lab
> University of New Hampshire
> rdr@iol.unh.edu
> 603-862-3774
>
>
>
> On Thu, 29 Nov 2001, Julian Satran wrote:
>
> > 0 as a stand in for "whatever" is out of version 09.
> > We don't see any need for it. It was originally provide for
> compatibility
> > with the T10 docs when we had values "residing" in mode pages accessible
> > by SCSI commands.
> >
> > Julo
> >
> >
> >
> >
> > Tow Wang <Tow_Wang@adaptec.com>
> > Sent by: owner-ips@ece.cmu.edu
> > 28-11-01 01:55
> >
> >
> >         To:     "Robert D. Russell" <rdr@io.iol.unh.edu>,
> ips@ece.cmu.edu
> >         cc:
> >         Subject:        Value of "unlimited". Was Re: iSCSI: current UNH
> Plugfest
> >
> >
> >
> > Hi all.
> >
> > I fully agree with the suggestions for issue 4. If we really need to
> > reserve a
> > value to mean "unlimited" or "infinite", could we use a value of all
> bits
> > set to
> > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be
> great
> > for
> > most 32-bit processors and above.) This would make arithmetic
> comparisons
> > more
> > intuitive, IMHO.
> >
> > "Robert D. Russell" wrote:
> >
> > > Attached are the new issues that arose during the iSCSI plugfest
> > > at UNH on Wednesday 31-Oct-2001.
> > >
> > > (Note: these issues do not take into account any modifications or
> > > clarifications that occured in the standard due to the issues raised
> > > on Monday or Tuesday.)
> > >
> > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize)
> > >    now allow: "A value of 0 indicates no limit."
> > >
> > >    Is this useful?  Does it buy anything?
> > >
> > >    The difficulties implementers are having with this are:
> > >
> > >    1) It is a special case.
> > >    2) It causes discontinuous ranges (for example, [0,64..2**24])
> > >    3) It violates the min/max function normally used for the key.
> > >    4) There is always a limit anyway.
> > >
> > >    Consider FirstBurstSize, which can have a value that is described
> > >    as "<0|number-64-2**24>", and for which the minimum of the 2
> numbers
> > >    is selected.
> > >
> > >    I one side offers 0 to mean unlimited, and the other side
> > >    has a limit, it will reply with that limit, say 128 Kbytes.
> > >    Therefore, the result is not min(0, 128K) but rather max(0, 128K).
> > >    The statement in the standard that "the minimum of the 2 numbers is
> > >    selected" is therefore wrong when one of the numbers is 0.
> > >
> > >    Furthermore, when an initiator or target receives an offer for one
> > >    of these keys, it cannot simply check that the offered value is
> > >    legal by testing it against some minimum and maximum.  It must
> first
> > >    check for 0 and then only if that check shows the value is non-zero
> > >    can it do the min/max range check for legality (i.e., 64-2**24).
> > >
> > >    Finally, there is always a limit. If nothing else it is the
> > >    limit imposed by the 24-bit DataSegmentLength field of the PDU
> > >    requesting the transfer.  It is useless to specify a FirstBurstSize
> > >    (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because
> > >    the largest possible DataSegmentLength in any PDU that can use
> > >    this value is 2**24-1.
> > >
> > >    The suggestion is to just eliminate this special case of 0 and
> > require
> > >    that the range 64-to-(2**24-1) be used instead -- it has exactly
> the
> > >    same effect in all cases, it is easier to describe in the standard
> > >    because it avoids all the extra words, and it is easier to code
> > >    because it avoids all the special cases.
> > >
> > >    NOTE: the standard should specify the limit in the ranges for
> > >    MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1
> instead
> > >    of 2**24.  The number 2**24 cannot be represented in the 24-bit
> > >    DataSegmentLength field and therefore can never be used.
> >
> > --
> > Tow Wang
> > Adaptec Software Engineer       Telephone: 408 957 4838
> > Mail stop 46                               800 869 8883 x4838
> > 691 South Milpitas Boulevard    Fax:       530 686 8023
> > Milpitas, California 95035      E-mail:    Tow_Wang@adaptec.com
> >
> >
> >
> >
> >
>
>
>
>

--=_alternative 0049A5E142256B14_=-- From owner-ips@ece.cmu.edu Fri Nov 30 09:01:09 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA11201 for ; Fri, 30 Nov 2001 09:01:08 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUDDeD06893 for ips-outgoing; Fri, 30 Nov 2001 08:13:40 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUAvsl29386 for ; Fri, 30 Nov 2001 05:57:54 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA02117; Fri, 30 Nov 2001 05:57:48 -0500 (EST) Message-Id: <200111301057.FAA02117@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-ifcp-07.txt Date: Fri, 30 Nov 2001 05:57:48 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iFCP - A Protocol for Internet Fibre Channel Storage Networking Author(s) : C. Monia et al. Filename : draft-ietf-ips-ifcp-07.txt Pages : 90 Date : 29-Nov-01 This document specifies an architecture and gateway-to-gateway protocol for the implementation of Fibre Channel fabric functionality on a network in which TCP/IP switching and routing elements replace Fibre Channel components. The protocol enables the attachment of existing Fibre Channel storage products to an IP network by supporting the fabric services required by such devices. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-ifcp-07.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-ifcp-07.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-ifcp-07.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011129143756.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-ifcp-07.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-ifcp-07.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011129143756.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Fri Nov 30 09:05:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA11407 for ; Fri, 30 Nov 2001 09:05:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUDDlk06904 for ips-outgoing; Fri, 30 Nov 2001 08:13:47 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUAvwl29396 for ; Fri, 30 Nov 2001 05:57:58 -0500 (EST) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA02129; Fri, 30 Nov 2001 05:57:53 -0500 (EST) Message-Id: <200111301057.FAA02129@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ips@ece.cmu.edu From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ips-isns-06.txt Date: Fri, 30 Nov 2001 05:57:53 -0500 Sender: owner-ips@ece.cmu.edu Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : Internet Storage Name Service (iSNS) Author(s) : K. Gibbons et al. Filename : draft-ietf-ips-isns-06.txt Pages : 79 Date : 29-Nov-01 This document provides a generic framework centering around use of the iSNS for discovery and management of iSCSI and Fibre Channel (FCP) storage devices in an enterprise-scale IP storage network. iSNS is an application that stores iSCSI and FC device attributes and monitors their availability and reachability in an integrated IP storage network. Due to its role as a consolidated information repository, iSNS provides for more efficient and scalable management of storage devices in an IP network. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-06.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-isns-06.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-isns-06.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20011129143809.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-isns-06.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-isns-06.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20011129143809.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ips@ece.cmu.edu Fri Nov 30 09:13:26 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA11877 for ; Fri, 30 Nov 2001 09:13:26 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUDDuO06921 for ips-outgoing; Fri, 30 Nov 2001 08:13:56 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from iol.unh.edu (io.iol.unh.edu [132.177.123.82]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUBHcl00473 for ; Fri, 30 Nov 2001 06:17:38 -0500 (EST) Received: from io.iol.unh.edu (IDENT:rdr@localhost.localdomain [127.0.0.1]) by iol.unh.edu (8.12.1/8.12.1) with ESMTP id fAUBHX6D011226; Fri, 30 Nov 2001 06:17:33 -0500 Received: from localhost (rdr@localhost) by io.iol.unh.edu (8.12.1/8.12.1/Submit) with ESMTP id fAUBHXqH011223; Fri, 30 Nov 2001 06:17:33 -0500 Date: Fri, 30 Nov 2001 06:17:33 -0500 (EST) From: "Robert D. Russell" To: Julian Satran cc: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: I would like to request that using the value of 0 as a "don't care" in numeric negotiations be removed completely. You said it is there to "simplify exchanges", but I do not see how it simplifies them because it makes a special case where none is needed. If the numerical choice function is "min", then to indicate "don't care" simply offer the maximum value in the allowed range (which clearly the offering side must be able to deal with if it really doesn't care). The responder they replies with any number it wants to choose from the allowed range, because by definition that will be the "min" value. When the numerical choice function is "max", the offering side simply offers the minimum value in the allowed range, etc. By making 0 a special case, extra code is necessary on the receiving side, and extra wording is needed in the standard, yet there is no gain in functionality. This is why I would like to have it removed -- it is an unnecessary special case. Thanks, Bob Russell On Fri, 30 Nov 2001, Julian Satran wrote: > Robert, > > Sorry for the confusion. > The value 0 was on purpose changed from its "unlimited" meaning in mode > pages to a syntactic "don't care" > in some negotiations to simplify exchanges where one of the parties does > not care. > In that case the "burden" of the decision is upon the responder. > > > Julo > > > > > "Robert D. Russell" > 11/29/2001 12:30 PM > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest > > > > Julian: > > If I understand correctly what Tow was asking for, and your response to > him, this means that the statement: > > "A value of 0 MAY be used as a "don't care" offer in negotiations." > > should be removed from the description in Appendix D of draft 9 for > the 6 keys: > > MaxRecvPDULength > MaxBurstSize > FirstBurstSize > LogoutLoginMaxTime > LogoutLoginMinTime > MaxOutstandingR2T > > > The statement in section 2.2.4 on page 31 should also be removed: > > "For numerical negotiations, the value 0 MAY be specified by the > offering party as a "don't care"/"unlimited" value for parameters > that explicitly allow it; in this case, the responder may choose any > legal value for the parameter." > > > Thanks, > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > > On Thu, 29 Nov 2001, Julian Satran wrote: > > > 0 as a stand in for "whatever" is out of version 09. > > We don't see any need for it. It was originally provide for > compatibility > > with the T10 docs when we had values "residing" in mode pages accessible > > by SCSI commands. > > > > Julo > > > > > > > > > > Tow Wang > > Sent by: owner-ips@ece.cmu.edu > > 28-11-01 01:55 > > > > > > To: "Robert D. Russell" , > ips@ece.cmu.edu > > cc: > > Subject: Value of "unlimited". Was Re: iSCSI: current UNH > Plugfest > > > > > > > > Hi all. > > > > I fully agree with the suggestions for issue 4. If we really need to > > reserve a > > value to mean "unlimited" or "infinite", could we use a value of all > bits > > set to > > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be > great > > for > > most 32-bit processors and above.) This would make arithmetic > comparisons > > more > > intuitive, IMHO. > > > > "Robert D. Russell" wrote: > > > > > Attached are the new issues that arose during the iSCSI plugfest > > > at UNH on Wednesday 31-Oct-2001. > > > > > > (Note: these issues do not take into account any modifications or > > > clarifications that occured in the standard due to the issues raised > > > on Monday or Tuesday.) > > > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > > now allow: "A value of 0 indicates no limit." > > > > > > Is this useful? Does it buy anything? > > > > > > The difficulties implementers are having with this are: > > > > > > 1) It is a special case. > > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > > 3) It violates the min/max function normally used for the key. > > > 4) There is always a limit anyway. > > > > > > Consider FirstBurstSize, which can have a value that is described > > > as "<0|number-64-2**24>", and for which the minimum of the 2 > numbers > > > is selected. > > > > > > I one side offers 0 to mean unlimited, and the other side > > > has a limit, it will reply with that limit, say 128 Kbytes. > > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > > The statement in the standard that "the minimum of the 2 numbers is > > > selected" is therefore wrong when one of the numbers is 0. > > > > > > Furthermore, when an initiator or target receives an offer for one > > > of these keys, it cannot simply check that the offered value is > > > legal by testing it against some minimum and maximum. It must > first > > > check for 0 and then only if that check shows the value is non-zero > > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > > > Finally, there is always a limit. If nothing else it is the > > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > > requesting the transfer. It is useless to specify a FirstBurstSize > > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > > the largest possible DataSegmentLength in any PDU that can use > > > this value is 2**24-1. > > > > > > The suggestion is to just eliminate this special case of 0 and > > require > > > that the range 64-to-(2**24-1) be used instead -- it has exactly > the > > > same effect in all cases, it is easier to describe in the standard > > > because it avoids all the extra words, and it is easier to code > > > because it avoids all the special cases. > > > > > > NOTE: the standard should specify the limit in the ranges for > > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 > instead > > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > > DataSegmentLength field and therefore can never be used. > > > > -- > > Tow Wang > > Adaptec Software Engineer Telephone: 408 957 4838 > > Mail stop 46 800 869 8883 x4838 > > 691 South Milpitas Boulevard Fax: 530 686 8023 > > Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Fri Nov 30 11:40:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA20183 for ; Fri, 30 Nov 2001 11:40:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUFecY17846 for ips-outgoing; Fri, 30 Nov 2001 10:40:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUFeZl17841 for ; Fri, 30 Nov 2001 10:40:36 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id E552D355D; Fri, 30 Nov 2001 08:40:34 -0700 (MST) Received: from axcsbh1.cos.agilent.com (axcsbh1.cos.agilent.com [130.29.152.143]) by msgrel1.cos.agilent.com (Postfix) with ESMTP id C4ADD5E1; Fri, 30 Nov 2001 08:40:34 -0700 (MST) Received: by axcsbh1.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 30 Nov 2001 08:40:34 -0700 Message-ID: <9F8400020EC5D311AF62009027C3961605F26730@axcs09.cos.agilent.com> From: "LEMAY,KEVIN (A-Roseville,ex1)" To: "'Julian Satran'" , ips@ece.cmu.edu Subject: RE: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Date: Fri, 30 Nov 2001 08:40:33 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I support professor Russell on this one. This requires more code to handle special case that is not needed. I put my vote in for removing 0 as a special case. Kevin Lemay Agilent Technologies -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Friday, November 30, 2001 5:26 AM To: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Bob, It was claimed that a value of 0 - used uniformly independent of the selection rule (min or max) would be easier to handle. I personally don't care - but would like to avoid another raging war on defaults. If you find some other supporters I am ready to remove the don't care altogether. Regards, Julo "Robert D. Russell" 11/30/2001 01:17 PM To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Julian: I would like to request that using the value of 0 as a "don't care" in numeric negotiations be removed completely. You said it is there to "simplify exchanges", but I do not see how it simplifies them because it makes a special case where none is needed. If the numerical choice function is "min", then to indicate "don't care" simply offer the maximum value in the allowed range (which clearly the offering side must be able to deal with if it really doesn't care). The responder they replies with any number it wants to choose from the allowed range, because by definition that will be the "min" value. When the numerical choice function is "max", the offering side simply offers the minimum value in the allowed range, etc. By making 0 a special case, extra code is necessary on the receiving side, and extra wording is needed in the standard, yet there is no gain in functionality. This is why I would like to have it removed -- it is an unnecessary special case. Thanks, Bob Russell On Fri, 30 Nov 2001, Julian Satran wrote: > Robert, > > Sorry for the confusion. > The value 0 was on purpose changed from its "unlimited" meaning in mode > pages to a syntactic "don't care" > in some negotiations to simplify exchanges where one of the parties does > not care. > In that case the "burden" of the decision is upon the responder. > > > Julo > > > > > "Robert D. Russell" > 11/29/2001 12:30 PM > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest > > > > Julian: > > If I understand correctly what Tow was asking for, and your response to > him, this means that the statement: > > "A value of 0 MAY be used as a "don't care" offer in negotiations." > > should be removed from the description in Appendix D of draft 9 for > the 6 keys: > > MaxRecvPDULength > MaxBurstSize > FirstBurstSize > LogoutLoginMaxTime > LogoutLoginMinTime > MaxOutstandingR2T > > > The statement in section 2.2.4 on page 31 should also be removed: > > "For numerical negotiations, the value 0 MAY be specified by the > offering party as a "don't care"/"unlimited" value for parameters > that explicitly allow it; in this case, the responder may choose any > legal value for the parameter." > > > Thanks, > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > > On Thu, 29 Nov 2001, Julian Satran wrote: > > > 0 as a stand in for "whatever" is out of version 09. > > We don't see any need for it. It was originally provide for > compatibility > > with the T10 docs when we had values "residing" in mode pages accessible > > by SCSI commands. > > > > Julo > > > > > > > > > > Tow Wang > > Sent by: owner-ips@ece.cmu.edu > > 28-11-01 01:55 > > > > > > To: "Robert D. Russell" , > ips@ece.cmu.edu > > cc: > > Subject: Value of "unlimited". Was Re: iSCSI: current UNH > Plugfest > > > > > > > > Hi all. > > > > I fully agree with the suggestions for issue 4. If we really need to > > reserve a > > value to mean "unlimited" or "infinite", could we use a value of all > bits > > set to > > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be > great > > for > > most 32-bit processors and above.) This would make arithmetic > comparisons > > more > > intuitive, IMHO. > > > > "Robert D. Russell" wrote: > > > > > Attached are the new issues that arose during the iSCSI plugfest > > > at UNH on Wednesday 31-Oct-2001. > > > > > > (Note: these issues do not take into account any modifications or > > > clarifications that occured in the standard due to the issues raised > > > on Monday or Tuesday.) > > > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > > now allow: "A value of 0 indicates no limit." > > > > > > Is this useful? Does it buy anything? > > > > > > The difficulties implementers are having with this are: > > > > > > 1) It is a special case. > > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > > 3) It violates the min/max function normally used for the key. > > > 4) There is always a limit anyway. > > > > > > Consider FirstBurstSize, which can have a value that is described > > > as "<0|number-64-2**24>", and for which the minimum of the 2 > numbers > > > is selected. > > > > > > I one side offers 0 to mean unlimited, and the other side > > > has a limit, it will reply with that limit, say 128 Kbytes. > > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > > The statement in the standard that "the minimum of the 2 numbers is > > > selected" is therefore wrong when one of the numbers is 0. > > > > > > Furthermore, when an initiator or target receives an offer for one > > > of these keys, it cannot simply check that the offered value is > > > legal by testing it against some minimum and maximum. It must > first > > > check for 0 and then only if that check shows the value is non-zero > > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > > > Finally, there is always a limit. If nothing else it is the > > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > > requesting the transfer. It is useless to specify a FirstBurstSize > > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > > the largest possible DataSegmentLength in any PDU that can use > > > this value is 2**24-1. > > > > > > The suggestion is to just eliminate this special case of 0 and > > require > > > that the range 64-to-(2**24-1) be used instead -- it has exactly > the > > > same effect in all cases, it is easier to describe in the standard > > > because it avoids all the extra words, and it is easier to code > > > because it avoids all the special cases. > > > > > > NOTE: the standard should specify the limit in the ranges for > > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 > instead > > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > > DataSegmentLength field and therefore can never be used. > > > > -- > > Tow Wang > > Adaptec Software Engineer Telephone: 408 957 4838 > > Mail stop 46 800 869 8883 x4838 > > 691 South Milpitas Boulevard Fax: 530 686 8023 > > Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Fri Nov 30 12:38:58 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA23641 for ; Fri, 30 Nov 2001 12:38:57 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUGgfT23198 for ips-outgoing; Fri, 30 Nov 2001 11:42:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUGgel23193 for ; Fri, 30 Nov 2001 11:42:40 -0500 (EST) Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1.cos.agilent.com (Postfix) with ESMTP id 329DF341A; Fri, 30 Nov 2001 09:42:39 -0700 (MST) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 05F8E63F; Fri, 30 Nov 2001 09:42:39 -0700 (MST) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 30 Nov 2001 09:42:38 -0700 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 30 Nov 2001 09:42:38 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0092B2888@axcs04.cos.agilent.com> From: pat_thaler@agilent.com To: Julian_Satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Date: Fri, 30 Nov 2001 09:42:36 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, I am in favor of removing the don't care. Putting the maximum value in accomplishes "don't care" for a numeric negotiation. Regards, Pat -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Friday, November 30, 2001 5:26 AM To: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Bob, It was claimed that a value of 0 - used uniformly independent of the selection rule (min or max) would be easier to handle. I personally don't care - but would like to avoid another raging war on defaults. If you find some other supporters I am ready to remove the don't care altogether. Regards, Julo "Robert D. Russell" 11/30/2001 01:17 PM To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Julian: I would like to request that using the value of 0 as a "don't care" in numeric negotiations be removed completely. You said it is there to "simplify exchanges", but I do not see how it simplifies them because it makes a special case where none is needed. If the numerical choice function is "min", then to indicate "don't care" simply offer the maximum value in the allowed range (which clearly the offering side must be able to deal with if it really doesn't care). The responder they replies with any number it wants to choose from the allowed range, because by definition that will be the "min" value. When the numerical choice function is "max", the offering side simply offers the minimum value in the allowed range, etc. By making 0 a special case, extra code is necessary on the receiving side, and extra wording is needed in the standard, yet there is no gain in functionality. This is why I would like to have it removed -- it is an unnecessary special case. Thanks, Bob Russell On Fri, 30 Nov 2001, Julian Satran wrote: > Robert, > > Sorry for the confusion. > The value 0 was on purpose changed from its "unlimited" meaning in mode > pages to a syntactic "don't care" > in some negotiations to simplify exchanges where one of the parties does > not care. > In that case the "burden" of the decision is upon the responder. > > > Julo > > > > > "Robert D. Russell" > 11/29/2001 12:30 PM > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest > > > > Julian: > > If I understand correctly what Tow was asking for, and your response to > him, this means that the statement: > > "A value of 0 MAY be used as a "don't care" offer in negotiations." > > should be removed from the description in Appendix D of draft 9 for > the 6 keys: > > MaxRecvPDULength > MaxBurstSize > FirstBurstSize > LogoutLoginMaxTime > LogoutLoginMinTime > MaxOutstandingR2T > > > The statement in section 2.2.4 on page 31 should also be removed: > > "For numerical negotiations, the value 0 MAY be specified by the > offering party as a "don't care"/"unlimited" value for parameters > that explicitly allow it; in this case, the responder may choose any > legal value for the parameter." > > > Thanks, > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > > On Thu, 29 Nov 2001, Julian Satran wrote: > > > 0 as a stand in for "whatever" is out of version 09. > > We don't see any need for it. It was originally provide for > compatibility > > with the T10 docs when we had values "residing" in mode pages accessible > > by SCSI commands. > > > > Julo > > > > > > > > > > Tow Wang > > Sent by: owner-ips@ece.cmu.edu > > 28-11-01 01:55 > > > > > > To: "Robert D. Russell" , > ips@ece.cmu.edu > > cc: > > Subject: Value of "unlimited". Was Re: iSCSI: current UNH > Plugfest > > > > > > > > Hi all. > > > > I fully agree with the suggestions for issue 4. If we really need to > > reserve a > > value to mean "unlimited" or "infinite", could we use a value of all > bits > > set to > > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be > great > > for > > most 32-bit processors and above.) This would make arithmetic > comparisons > > more > > intuitive, IMHO. > > > > "Robert D. Russell" wrote: > > > > > Attached are the new issues that arose during the iSCSI plugfest > > > at UNH on Wednesday 31-Oct-2001. > > > > > > (Note: these issues do not take into account any modifications or > > > clarifications that occured in the standard due to the issues raised > > > on Monday or Tuesday.) > > > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > > now allow: "A value of 0 indicates no limit." > > > > > > Is this useful? Does it buy anything? > > > > > > The difficulties implementers are having with this are: > > > > > > 1) It is a special case. > > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > > 3) It violates the min/max function normally used for the key. > > > 4) There is always a limit anyway. > > > > > > Consider FirstBurstSize, which can have a value that is described > > > as "<0|number-64-2**24>", and for which the minimum of the 2 > numbers > > > is selected. > > > > > > I one side offers 0 to mean unlimited, and the other side > > > has a limit, it will reply with that limit, say 128 Kbytes. > > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > > The statement in the standard that "the minimum of the 2 numbers is > > > selected" is therefore wrong when one of the numbers is 0. > > > > > > Furthermore, when an initiator or target receives an offer for one > > > of these keys, it cannot simply check that the offered value is > > > legal by testing it against some minimum and maximum. It must > first > > > check for 0 and then only if that check shows the value is non-zero > > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > > > Finally, there is always a limit. If nothing else it is the > > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > > requesting the transfer. It is useless to specify a FirstBurstSize > > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > > the largest possible DataSegmentLength in any PDU that can use > > > this value is 2**24-1. > > > > > > The suggestion is to just eliminate this special case of 0 and > > require > > > that the range 64-to-(2**24-1) be used instead -- it has exactly > the > > > same effect in all cases, it is easier to describe in the standard > > > because it avoids all the extra words, and it is easier to code > > > because it avoids all the special cases. > > > > > > NOTE: the standard should specify the limit in the ranges for > > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 > instead > > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > > DataSegmentLength field and therefore can never be used. > > > > -- > > Tow Wang > > Adaptec Software Engineer Telephone: 408 957 4838 > > Mail stop 46 800 869 8883 x4838 > > 691 South Milpitas Boulevard Fax: 530 686 8023 > > Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Fri Nov 30 13:36:05 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26809 for ; Fri, 30 Nov 2001 13:36:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUI1XA29654 for ips-outgoing; Fri, 30 Nov 2001 13:01:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUI1Wl29650 for ; Fri, 30 Nov 2001 13:01:32 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id C4B94E51; Fri, 30 Nov 2001 13:01:30 -0500 (EST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id KAA22677; Fri, 30 Nov 2001 10:02:59 -0800 (PST) Message-Id: <200111301802.KAA22677@core.rose.hp.com> Date: Fri, 30 Nov 2001 10:14:18 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: "Robert D. Russell" , ips Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian and Bob, Sometime ago, in my private email conversations with Julian, I leaned towards retaining "0" for the reasons Julian suggested. But as Bob suggests, a value of all-1's or a value of 0 would be more straightforward than a special case value. So, I also recommend removing the "no limit" special value. Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com "Robert D. Russell" wrote: > > Julian: > > I would like to request that using the value of 0 as a "don't care" > in numeric negotiations be removed completely. You said it is there > to "simplify exchanges", but I do not see how it simplifies them > because it makes a special case where none is needed. > > If the numerical choice function is "min", then to indicate "don't care" > simply offer the maximum value in the allowed range (which clearly the > offering side must be able to deal with if it really doesn't care). > The responder they replies with any number it wants to choose from the > allowed range, because by definition that will be the "min" value. > When the numerical choice function is "max", the offering side > simply offers the minimum value in the allowed range, etc. > > By making 0 a special case, extra code is necessary on the receiving side, > and extra wording is needed in the standard, yet there is no gain in > functionality. This is why I would like to have it removed -- it is > an unnecessary special case. > > Thanks, > Bob Russell > > On Fri, 30 Nov 2001, Julian Satran wrote: > > > Robert, > > > > Sorry for the confusion. > > The value 0 was on purpose changed from its "unlimited" meaning in mode > > pages to a syntactic "don't care" > > in some negotiations to simplify exchanges where one of the parties does > > not care. > > In that case the "burden" of the decision is upon the responder. > > > > > > Julo > > > > > > > > > > "Robert D. Russell" > > 11/29/2001 12:30 PM > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu > > Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest > > > > > > > > Julian: > > > > If I understand correctly what Tow was asking for, and your response to > > him, this means that the statement: > > > > "A value of 0 MAY be used as a "don't care" offer in negotiations." > > > > should be removed from the description in Appendix D of draft 9 for > > the 6 keys: > > > > MaxRecvPDULength > > MaxBurstSize > > FirstBurstSize > > LogoutLoginMaxTime > > LogoutLoginMinTime > > MaxOutstandingR2T > > > > > > The statement in section 2.2.4 on page 31 should also be removed: > > > > "For numerical negotiations, the value 0 MAY be specified by the > > offering party as a "don't care"/"unlimited" value for parameters > > that explicitly allow it; in this case, the responder may choose any > > legal value for the parameter." > > > > > > Thanks, > > > > Bob Russell > > InterOperability Lab > > University of New Hampshire > > rdr@iol.unh.edu > > 603-862-3774 > > > > > > > > On Thu, 29 Nov 2001, Julian Satran wrote: > > > > > 0 as a stand in for "whatever" is out of version 09. > > > We don't see any need for it. It was originally provide for > > compatibility > > > with the T10 docs when we had values "residing" in mode pages accessible > > > by SCSI commands. > > > > > > Julo > > > > > > > > > > > > > > > Tow Wang > > > Sent by: owner-ips@ece.cmu.edu > > > 28-11-01 01:55 > > > > > > > > > To: "Robert D. Russell" , > > ips@ece.cmu.edu > > > cc: > > > Subject: Value of "unlimited". Was Re: iSCSI: current UNH > > Plugfest > > > > > > > > > > > > Hi all. > > > > > > I fully agree with the suggestions for issue 4. If we really need to > > > reserve a > > > value to mean "unlimited" or "infinite", could we use a value of all > > bits > > > set to > > > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be > > great > > > for > > > most 32-bit processors and above.) This would make arithmetic > > comparisons > > > more > > > intuitive, IMHO. > > > > > > "Robert D. Russell" wrote: > > > > > > > Attached are the new issues that arose during the iSCSI plugfest > > > > at UNH on Wednesday 31-Oct-2001. > > > > > > > > (Note: these issues do not take into account any modifications or > > > > clarifications that occured in the standard due to the issues raised > > > > on Monday or Tuesday.) > > > > > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > > > now allow: "A value of 0 indicates no limit." > > > > > > > > Is this useful? Does it buy anything? > > > > > > > > The difficulties implementers are having with this are: > > > > > > > > 1) It is a special case. > > > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > > > 3) It violates the min/max function normally used for the key. > > > > 4) There is always a limit anyway. > > > > > > > > Consider FirstBurstSize, which can have a value that is described > > > > as "<0|number-64-2**24>", and for which the minimum of the 2 > > numbers > > > > is selected. > > > > > > > > I one side offers 0 to mean unlimited, and the other side > > > > has a limit, it will reply with that limit, say 128 Kbytes. > > > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > > > The statement in the standard that "the minimum of the 2 numbers is > > > > selected" is therefore wrong when one of the numbers is 0. > > > > > > > > Furthermore, when an initiator or target receives an offer for one > > > > of these keys, it cannot simply check that the offered value is > > > > legal by testing it against some minimum and maximum. It must > > first > > > > check for 0 and then only if that check shows the value is non-zero > > > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > > > > > Finally, there is always a limit. If nothing else it is the > > > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > > > requesting the transfer. It is useless to specify a FirstBurstSize > > > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > > > the largest possible DataSegmentLength in any PDU that can use > > > > this value is 2**24-1. > > > > > > > > The suggestion is to just eliminate this special case of 0 and > > > require > > > > that the range 64-to-(2**24-1) be used instead -- it has exactly > > the > > > > same effect in all cases, it is easier to describe in the standard > > > > because it avoids all the extra words, and it is easier to code > > > > because it avoids all the special cases. > > > > > > > > NOTE: the standard should specify the limit in the ranges for > > > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 > > instead > > > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > > > DataSegmentLength field and therefore can never be used. > > > > > > -- > > > Tow Wang > > > Adaptec Software Engineer Telephone: 408 957 4838 > > > Mail stop 46 800 869 8883 x4838 > > > 691 South Milpitas Boulevard Fax: 530 686 8023 > > > Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Fri Nov 30 13:39:51 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26980 for ; Fri, 30 Nov 2001 13:39:51 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUHcNl27736 for ips-outgoing; Fri, 30 Nov 2001 12:38:23 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lucy.trebia.com ([65.192.191.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUGtvl24213 for ; Fri, 30 Nov 2001 11:55:57 -0500 (EST) Received: from trebia.com (trebia-dhcp-232.trebia.com [65.192.191.232]) by lucy.trebia.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id X6ZMS2WB; Fri, 30 Nov 2001 11:54:10 -0500 Message-ID: <3C07B9FD.F80DCF50@trebia.com> Date: Fri, 30 Nov 2001 11:55:25 -0500 From: Anshul Chadda Organization: Trebia Networks X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.4.2-2 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest References: <9F8400020EC5D311AF62009027C3961605F26730@axcs09.cos.agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit I support removing 0 as a special case exactly for the reasons stated here(posted earlier in this thread). Anshul > > > > > The suggestion is to just eliminate this special case of 0 and > > > require > > > > that the range 64-to-(2**24-1) be used instead -- it has exactly > > the > > > > same effect in all cases, it is easier to describe in the standard > > > > because it avoids all the extra words, and it is easier to code > > > > because it avoids all the special cases. > > > > From owner-ips@ece.cmu.edu Fri Nov 30 14:55:59 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA02513 for ; Fri, 30 Nov 2001 14:55:59 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUIZ2h02342 for ips-outgoing; Fri, 30 Nov 2001 13:35:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fAUIZ0l02337 for ; Fri, 30 Nov 2001 13:35:00 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id TAA61634 for ; Fri, 30 Nov 2001 19:34:54 +0100 Received: from d10hubm1.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.11.1m3/NCO v5.01) with ESMTP id fAUIZAb111104 for ; Fri, 30 Nov 2001 19:35:10 +0100 Subject: RE: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: "Julian Satran" Date: Fri, 30 Nov 2001 20:22:09 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.8 |June 18, 2001) at 30/11/2001 20:35:01 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I think I've heard enough strong voices to remove. And so I will. Thanks, Julo pat_thaler@agi lent.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu Sent by: cc: owner-ips@ece. Subject: RE: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest cmu.edu 11/30/2001 06:42 PM Julo, I am in favor of removing the don't care. Putting the maximum value in accomplishes "don't care" for a numeric negotiation. Regards, Pat -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Friday, November 30, 2001 5:26 AM To: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Bob, It was claimed that a value of 0 - used uniformly independent of the selection rule (min or max) would be easier to handle. I personally don't care - but would like to avoid another raging war on defaults. If you find some other supporters I am ready to remove the don't care altogether. Regards, Julo "Robert D. Russell" 11/30/2001 01:17 PM To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest Julian: I would like to request that using the value of 0 as a "don't care" in numeric negotiations be removed completely. You said it is there to "simplify exchanges", but I do not see how it simplifies them because it makes a special case where none is needed. If the numerical choice function is "min", then to indicate "don't care" simply offer the maximum value in the allowed range (which clearly the offering side must be able to deal with if it really doesn't care). The responder they replies with any number it wants to choose from the allowed range, because by definition that will be the "min" value. When the numerical choice function is "max", the offering side simply offers the minimum value in the allowed range, etc. By making 0 a special case, extra code is necessary on the receiving side, and extra wording is needed in the standard, yet there is no gain in functionality. This is why I would like to have it removed -- it is an unnecessary special case. Thanks, Bob Russell On Fri, 30 Nov 2001, Julian Satran wrote: > Robert, > > Sorry for the confusion. > The value 0 was on purpose changed from its "unlimited" meaning in mode > pages to a syntactic "don't care" > in some negotiations to simplify exchanges where one of the parties does > not care. > In that case the "burden" of the decision is upon the responder. > > > Julo > > > > > "Robert D. Russell" > 11/29/2001 12:30 PM > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: Value of "unlimited". Was Re: iSCSI: current UNH Plugfest > > > > Julian: > > If I understand correctly what Tow was asking for, and your response to > him, this means that the statement: > > "A value of 0 MAY be used as a "don't care" offer in negotiations." > > should be removed from the description in Appendix D of draft 9 for > the 6 keys: > > MaxRecvPDULength > MaxBurstSize > FirstBurstSize > LogoutLoginMaxTime > LogoutLoginMinTime > MaxOutstandingR2T > > > The statement in section 2.2.4 on page 31 should also be removed: > > "For numerical negotiations, the value 0 MAY be specified by the > offering party as a "don't care"/"unlimited" value for parameters > that explicitly allow it; in this case, the responder may choose any > legal value for the parameter." > > > Thanks, > > Bob Russell > InterOperability Lab > University of New Hampshire > rdr@iol.unh.edu > 603-862-3774 > > > > On Thu, 29 Nov 2001, Julian Satran wrote: > > > 0 as a stand in for "whatever" is out of version 09. > > We don't see any need for it. It was originally provide for > compatibility > > with the T10 docs when we had values "residing" in mode pages accessible > > by SCSI commands. > > > > Julo > > > > > > > > > > Tow Wang > > Sent by: owner-ips@ece.cmu.edu > > 28-11-01 01:55 > > > > > > To: "Robert D. Russell" , > ips@ece.cmu.edu > > cc: > > Subject: Value of "unlimited". Was Re: iSCSI: current UNH > Plugfest > > > > > > > > Hi all. > > > > I fully agree with the suggestions for issue 4. If we really need to > > reserve a > > value to mean "unlimited" or "infinite", could we use a value of all > bits > > set to > > one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be > great > > for > > most 32-bit processors and above.) This would make arithmetic > comparisons > > more > > intuitive, IMHO. > > > > "Robert D. Russell" wrote: > > > > > Attached are the new issues that arose during the iSCSI plugfest > > > at UNH on Wednesday 31-Oct-2001. > > > > > > (Note: these issues do not take into account any modifications or > > > clarifications that occured in the standard due to the issues raised > > > on Monday or Tuesday.) > > > > > > 4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize) > > > now allow: "A value of 0 indicates no limit." > > > > > > Is this useful? Does it buy anything? > > > > > > The difficulties implementers are having with this are: > > > > > > 1) It is a special case. > > > 2) It causes discontinuous ranges (for example, [0,64..2**24]) > > > 3) It violates the min/max function normally used for the key. > > > 4) There is always a limit anyway. > > > > > > Consider FirstBurstSize, which can have a value that is described > > > as "<0|number-64-2**24>", and for which the minimum of the 2 > numbers > > > is selected. > > > > > > I one side offers 0 to mean unlimited, and the other side > > > has a limit, it will reply with that limit, say 128 Kbytes. > > > Therefore, the result is not min(0, 128K) but rather max(0, 128K). > > > The statement in the standard that "the minimum of the 2 numbers is > > > selected" is therefore wrong when one of the numbers is 0. > > > > > > Furthermore, when an initiator or target receives an offer for one > > > of these keys, it cannot simply check that the offered value is > > > legal by testing it against some minimum and maximum. It must > first > > > check for 0 and then only if that check shows the value is non-zero > > > can it do the min/max range check for legality (i.e., 64-2**24). > > > > > > Finally, there is always a limit. If nothing else it is the > > > limit imposed by the 24-bit DataSegmentLength field of the PDU > > > requesting the transfer. It is useless to specify a FirstBurstSize > > > (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because > > > the largest possible DataSegmentLength in any PDU that can use > > > this value is 2**24-1. > > > > > > The suggestion is to just eliminate this special case of 0 and > > require > > > that the range 64-to-(2**24-1) be used instead -- it has exactly > the > > > same effect in all cases, it is easier to describe in the standard > > > because it avoids all the extra words, and it is easier to code > > > because it avoids all the special cases. > > > > > > NOTE: the standard should specify the limit in the ranges for > > > MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 > instead > > > of 2**24. The number 2**24 cannot be represented in the 24-bit > > > DataSegmentLength field and therefore can never be used. > > > > -- > > Tow Wang > > Adaptec Software Engineer Telephone: 408 957 4838 > > Mail stop 46 800 869 8883 x4838 > > 691 South Milpitas Boulevard Fax: 530 686 8023 > > Milpitas, California 95035 E-mail: Tow_Wang@adaptec.com > > > > > > > > > > > > > > From owner-ips@ece.cmu.edu Fri Nov 30 19:33:24 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA17680 for ; Fri, 30 Nov 2001 19:33:24 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fAUNacY26564 for ips-outgoing; Fri, 30 Nov 2001 18:36:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.11.0/8.10.2) with SMTP id fAUNaZl26555 for ; Fri, 30 Nov 2001 18:36:35 -0500 (EST) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by crufty; Fri Nov 30 18:31:47 EST 2001 Received: from aura.research.bell-labs.com (aura.research.bell-labs.com [135.104.46.10]) by grubby.research.bell-labs.com (8.11.6/8.11.6) with ESMTP id fAUNaIo37392 for ; Fri, 30 Nov 2001 18:36:18 -0500 (EST) Received: from research.bell-labs.com (IDENT:sandeepj@sandeepj-pcmh.research.bell-labs.com [135.104.47.90]) by aura.research.bell-labs.com (8.9.1/8.9.1) with ESMTP id SAA13979 for ; Fri, 30 Nov 2001 18:36:18 -0500 (EST) Message-ID: <3C0817F2.88647D65@research.bell-labs.com> Date: Fri, 30 Nov 2001 18:36:18 -0500 From: Sandeep Joshi X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: unsolicited data References: <000f01c17843$70943e70$eea0f40f@rose.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, Perhaps this thread was missed..could you please elaborate on this deadlock issue..when exactly does it occur? Note that I am not asking for out-of-order data! There was one more question - regarding the allowance given to the initiator to *not* send unsolicited PDUs right after the command PDUs, which raises persistent questions. See http://www.pdl.cmu.edu/mailinglists/ips/mail/msg07930.html for the latest question on this issue. IIRC, the reasoning here was that this helps the target in "positioning" operations - since it gets a few commands first, and then the data. By positioning, do you imply disk/tape seeks or something else ? Note that the draft must atleast mention the above rationale if we wish initiators to behave as such, and also targets to optimize for it. Regards, -Sandeep "Mallikarjun C." wrote: > I however could not really see a deadlock per se - my comment was to > understand > what it is. More comments would certainly help. > > ----- Original Message ----- > From: "Julian Satran" > > Mallikarjun, > > > > The deadlock free requirement here goes a bit deeper than your comment may > > suggest. > > Unsolicited data needs are very hard to predict and ordering them may > > remove the need to do so to a large extent. From owner-ips@ece.cmu.edu Fri Nov 30 20:08:00 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19106 for ; Fri, 30 Nov 2001 20:08:00 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fB10Fmi29117 for ips-outgoing; Fri, 30 Nov 2001 19:15:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel12.hp.com (palrel12.hp.com [156.153.255.237]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fB10Fkl29112 for ; Fri, 30 Nov 2001 19:15:47 -0500 (EST) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel12.hp.com (Postfix) with ESMTP id AB0AC1FB23 for ; Fri, 30 Nov 2001 16:15:28 -0800 (PST) Received: from rose.hp.com (hpsil34.rose.hp.com [15.43.211.225]) by core.rose.hp.com with ESMTP (8.9.3 (PHNE_22672)/8.8.6 SMKit7.02) id QAA22437 for ; Fri, 30 Nov 2001 16:16:58 -0800 (PST) Message-Id: <200112010016.QAA22437@core.rose.hp.com> Date: Fri, 30 Nov 2001 16:28:17 -0800 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard, Roseville X-Mailer: Mozilla 4.78 [en] (X11; U; HP-UX B.10.20 9000/715) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: Re: iSCSI:Clear Task Set References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: 7bit Julian, I tend to agree with John on this one. Passing the 'clear task set' onto target SCSI layer once the current session is dealt with is the right choice. That avoids iSCSI having to say anything on the relative order across sessions, and also addresses the issue John raised - that of differing LUN mappings for the same LU. At a minimum, the second sentence in the current wording "All tasks associated with the specified LUN and initiator. For all other initiators all tasks at LUN with no regard to order." should be dropped since it implies that 'clear task set' affects all other initiators. In fact, this determination can not be made until the TST bit is checked in the SCSI control mode page - which is completely in the SCSI realm. That of course requires the task management function to be propagated up. Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Organization MS 5668 Hewlett-Packard, Roseville. cbm@rose.hp.com Julian Satran wrote: > > John, > > That looks more like in T10 territory. > T10 defines differently Abort Task Set and Clear Task Set. > We could either: > > decide not to implement clear task set (T10 allows that but "per target" > not "per transport") > enable clear task set - in which case we have to say something about the > relative order of the task management request with regard to the task > comming from other initiators - and that is what I attempted to say in 9.4 > > Julo > > John Hufferd/San Jose/IBM@IBMUS > Sent by: owner-ips@ece.cmu.edu > 23-11-01 23:30 > > > To: Julian Satran/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu > Subject: Re: iSCSI:Clear Task Set > > > > Julian, and list, > The question now becomes, if we have all that carefully thought out > processing that is defined in 9.4 in how to handle the other > Tasks/Commands > that are "In Flight", and not yet given to SCSI, how does that apply to > the > other Sessions with other initiators? > > That is, at the iSCSI Target layer we do not have the LU Number to LU > mapping on any Session with any Initiator, so how do we cause the careful > processing, which is defined in 9.4, to occur on the other sessions that > may have and association to the subject LU? Especially since all we know > is > a LU Number on the Clearing Session. > > Perhaps we do not care about the 9.4 processing on the other Session and > Other Initiators and just let SCSI layer do its thing, and at the iSCSI > layer we pay no attention to the other Sessions. Do you think this is > correct? > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com > > Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 11/23/2001 07:56:36 AM > > Sent by: owner-ips@ece.cmu.edu > > To: ips@ece.cmu.edu > cc: > Subject: Re: iSCSI:Clear Task Set > > John, > > The LUN is just a mistake there - in all three instances it should be LU. > The definitions are in accordance with SAM . There are two task management > modes - tasks sets-per-initiator at each LU or common for all initiators. > The mode is a SCSI issue controlled by a field in the Control-Mode page. > Clear task set MAY clear all the tasks in the task set - even if common to > all initiators if that is the way the task set is managed. That is also > the difference between clear-task-set and abort task set. > > Julo > > John Hufferd@IBMUS > 23-11-01 11:29 > > To: Julian Satran/Haifa/IBM@IBMIL@IBMDE > cc: ips@ece.cmu.edu > From: John Hufferd/San Jose/IBM@IBMUS > Subject: iSCSI:Clear Task Set > > Julian, and List (using v 0.9) > In point 9.4, just before 9.5 the Table entry associated with Clear Task > Set applies to: > > "All tasks associated with the specified LUN and initiator. For all other > initiators all tasks at LUN with no regard to order." > > Perhaps we mean LU here, but I know that the iSCSI layer does not have > information about LU, only about the LU Number (LUN) in the command. We > can not tell, at the iSCSI layer, if the LU represented by a LUN on > Session 1, has any relationship to any LUN on any other session. > > This is because each initiator may have their own numbering for LUs. > Therefore, do we just pass the Clear Task Set to the SCSI layer and hope > for the best, or does the iSCSI layer also suppose to apply the Clear Task > Set to all the sessions that it has coming into the iSCSI (SCSI) Target > Port? If the latter, again how will that work when the iSCSI layer has no > idea what LU an Initiator's LUN will map to? > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSG San Jose Ca > Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Home Office (408) 997-6136, Cell: (408) 499-9702 > Internet address: hufferd@us.ibm.com From owner-ips@ece.cmu.edu Fri Nov 30 21:01:03 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA20756 for ; Fri, 30 Nov 2001 21:01:02 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fB11PEX03240 for ips-outgoing; Fri, 30 Nov 2001 20:25:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fB11PDl03235 for ; Fri, 30 Nov 2001 20:25:13 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 30 Nov 2001 20:27:38 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293722A@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: IPS SLC Preliminary Agenda Date: Fri, 30 Nov 2001 20:14:30 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Comments to me, by Monday if possible. Thanks, --David IETF IP Storage (IPS) Working Group Salt Lake City IETF Meeting, December 9-14, 2001 ------------------------------------------- CHAIRS: David L. Black Elizabeth Rodriguez AGENDA: SUBJECT TO CHANGE INTERIM MEETING: February 6-8, Huntington Beach, CA, USA General meeting covering all WG work items. An announcement with location and additional details will be posted to the list. Some WG Last Calls are expected during January - among the purposes of this meeting is to address issues raised during WG Last Call. ---- Monday, December 10, 0900-1130 ---- - Agenda Bashing and Administrivia (5 min) - BLUE SHEETS - NOTE WELL - Interim Meeting (see above) - Charter and Milestone revisions (coming soon) - Separation of normative and non-normative references in drafts - Security (30 min) draft-ietf-ips-security-07.txt Certificate issues and encapsulation mode (transport,tunnel) requirements - FC Encapsulation (5 min) draft-ietf-ips-fcencapsulation-04.txt Status update - iFCP MIB (5 min) draft-ietf-ips-ifcp-mib-00.txt Status update - iFCP (30 min) draft-ietf-ips-ifcp-07.txt Includes FC broadcast and multiple iFCP instances at same IP address. - iSNS (20 min) draft-ietf-ips-isns-06.txt Security of iSNS and iSNS as used with iFCP and iSCSI, including security policy distribution. - iSNS MIB (5 min) draft-ietf-ips-isns-mib-01.txt Status update - FCIP (45 min) draft-ietf-ips-fcovertcipip-07.txt Includes WWN short frame security - SLP and FCIP (5 min) draft-ietf-ips-fcip-slp-01.txt Status update. ---- Tuesday, December 11, 0900-1130 ---- - Agenda Re-Bashing and Administrivia (5 min) - SCSI, FC Management and FCIP MIBs (10 min) draft-ietf-ips-fcip-mib-00.txt Status update. No drafts available for SCSI and FC Management MIBs. ***May be taken up on Monday if time is available** - iSCSI Plugfest Report and Discussion (20 min) - iSCSI SRP Intellectual Property Rights (15 min) Report on known status of IPR issues for SRP. NOTE: Progress is being made on these issues. - iSCSI Framing Mechanism Requirements (15 min) draft-ietf-tsvwg-tcp-ulp-frame-01.txt NOTE: The contents of this draft will be discussed in TSVWG. This IPS WG time is to discuss iSCSI requirements for use of these mechanisms. - iSCSI (40 min) draft-ietf-ips-iscsi-09.txt Includes Out of Order Operation - iSCSI Boot (5 min) draft-ietf-ips-iscsi-boot-04.txt draft-sarkar-dhc-iscsi-boot-00.txt Status update. - iSCSI Naming and Discovery Requirements (20 min) draft-ietf-ips-iscsi-name-disc-03.txt Includes ISID changes. - iSCSI stringprep (5 min) draft-bakke-iscsi-stringprep-00.txt The author would like the WG to adopt this as an official WG work item. - Use of SLP and iSNS with ISCSI (10 min) draft-ietf-ips-iscsi-slp-02.txt draft-ietf-ips-isns-06.txt Status updates - iSCSI MIB (5 min) draft-ietf-ips-iscsi-mib-03.txt Status update and relationship to SCSI MIB. Table structure diagram available at: ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-mib-structure-03.pdf DESCRIPTION: The IP Storage WG works on using IP-based networks to transport block storage traffic. See http://www.ietf.org/html.charters/ips-charter.html --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Nov 30 21:32:13 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA21465 for ; Fri, 30 Nov 2001 21:32:12 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fB11Gap02717 for ips-outgoing; Fri, 30 Nov 2001 20:16:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic2.us.dg.com (mxic2.us.dg.com [128.221.31.40]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fB11GZl02712 for ; Fri, 30 Nov 2001 20:16:35 -0500 (EST) Received: by mxic2.us.dg.com with Internet Mail Service (5.5.2653.19) id ; Fri, 30 Nov 2001 20:12:34 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E02937229@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: IPS Internet-Draft status Date: Fri, 30 Nov 2001 20:05:51 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is what I believe the status of the Internet-Drafts in the IPS WG is for the Salt Lake City meeting. At the moment, I believe that only the FCIP SLP draft is still in the works at the secretariat. Please send any updates/corrections/comments to me and the list. Thanks, --David -- Security for all IPS protocols draft-ietf-ips-security-06.txt To be a proposed standard RFC. Getting close to done - open issues in certificate usage (things to do in practice to increase the likelihood of certificates working) and encapsulation mode (tunnel, transport) requirements for IPsec SAs. Note that the SLPv2 security text will be extensively revised in -07. Hope to close any remaining issues at December 2001 IETF meeting. -- iSCSI draft-ietf-ips-iscsi-09.txt To become a proposed standard RFC. Large portions are stable, major security and login issues have been resolved for the most part. Major editorial revision along with document restructure and reorganization is expected in -10 version (after December meeting). Hope to close any remaining issues at December 2001 IETF meeting draft-ietf-ips-iscsi-boot-04.txt To become an informational RFC, after the main iSCSI document. No major open issues, possible minor issues in DHCP usage. Hope to close any remaining issues at December 2001 IETF meeting. The related draft, draft-sarkar-dhc-iscsi-boot-00.txt describes use of DHCP to distribute iSCSI root volume location information for boot purposes. Current plans are to advance that draft through the DHC WG. draft-ietf-ips-iscsi-reqmts-05.txt To become an informational RFC. Has completed WG Last Call, currently being considered by the IESG, IETF Last Call has not been issued. draft-ietf-ips-iscsi-name-disc-03.txt To become an informational RFC. No major open issues, hope to close any remaining issues at December 2001 IETF meeting. Plan to issue WG Last Call for this with main iSCSI draft. draft-ietf-ips-iscsi-slp-02.txt To become a proposed standard RFC. Close to done. Issues in this general area seem to come up in the context of the iSCSI naming and discovery draft (iscsi-name-disc-03) rather than this draft. WG Last Call will be after the naming and discovery draft. draft-bakke-iscsi-stringprep-00.txt New draft to become a proposed standard RFC. Not expected to be controversial. December meeting will be asked to adopt this as an official WG draft. draft-ietf-ips-iscsi-mib-03.txt To be a proposed standard RFC. Close to done. WG Last Call will follow main iSCSI draft. Structure diagram available at: ftp://ftpeng.cisco.com/mbakke/iscsi/Visio-ietf-iscsi-mib-structure-03.pdf -- FC Common Encapsulation draft-ietf-ips-fcencapsulation-04.txt To be a proposed standard RFC. No known open issues. Will accompany FCIP and/or iFCP drafts in Last Call, probably after December 2001 IETF. -- FCIP draft-ietf-ips-fcovertcpip-07.txt To be a proposed standard RFC. Close to done - only major open issue is WWN short frame security. Hope to close that and any remaining issues at December 2001 IETF meeting. draft-ietf-ips-fcip-slp-01.txt To be a proposed standard RFC. Close to done. Hope to close any remaining issues at December 2001 IETF meeting. draft-ietf-ips-fcip-mib-00.txt To be a proposed standard RFC. Will undergo major changes as a consequence of restructuring of the FC Management MIB (see below). -- iFCP draft-ietf-ips-ifcp-07.txt To be a proposed standard RFC. Note that changes will be made in -08 in the areas of implementation of FC broadcasts and allowing multiple iFCP instances at a single IP address. Otherwise believed to be close to done. Hope to close any remaining issues at December 2001 IETF meeting. draft-ietf-ips-ifcp-mib-00.txt To be a proposed standard RFC. Likely timeframe for closing issues is March 2002 IETF meeting. -- iSNS draft-ietf-ips-isns-06.txt To be a proposed standard RFC. WG Last Call to follow iSCSI and iFCP drafts. Note that changes will be necessary to support forthcoming changes in the -08 version of the iFCP draft. Hope to close any remaining issues at December 2001 IETF meeting. draft-ietf-ips-isns-mib-01.txt To be a proposed standard RFC. Likely timeframe for closing issues is March 2002 IETF meeting. -- Framework (draft-ietf-ips-framework-00.txt) has been abandoned. -- SCSI MIB Design team is formulating an initial draft which is not yet available. -- Fibre Channel Management MIB This has been transferred to the IPS WG from the IPFC WG. Major restructuring and content changes are underway from the last ipfc version (draft-ietf-ipfc-fcmgmt-int-mib-07.txt), but a revised draft is not yet available. Work on this MIB may expand to incorporate some or all of the contents of the FC Fabric Element MIB (RFC 2837); the WG will determine whether to do this and to what extent after the first revised draft of the FC Management MIB is available. ----- Document Completion Guesstimates ------ A rough guess is 3 waves of documents: (1) Main protocol standards. Close technical issues at December 2001 IETF (or before), WG Last Call in early 2002, submit to IESG before March 2002 IETF Meeting. - iSCSI - iSCSI Naming/Discovery (and stringprep) - FC Encapsulation - FCIP - iFCP - IPS Security Last Call scheduling will be an issue, as there are at least two, and possibly more 4 week WG Last Calls required for the above that probably should not overlap with each other. WG Last Call for iSCSI will have to wait for -10 version of that draft to receive a reasonable level of review within the WG. (2) Supporting Protocol Documents. Close technical issues at December IETF if possible (but above documents have priority). WG Last Call to follow above, probably in March-April 2002. - iSCSI Boot - iSCSI SLP - FCIP SLP - iSNS iSNS Last Call should not overlap iSCSI Last Calls, hence at least two will be needed, but 2 week WG Last Call periods should suffice for these documents. (3) MIBs To follow supporting documents, probably in multiple groups over time, as the MIB maturities vary from the iSCSI MIB (close to done) to the SCSI MIB (initial work underway, draft to appear soon). - iSCSI MIB - FCIP MIB - iFCP MIB - iSNS MIB - SCSI MIB - FC Management MIB Probably need multiple Last Call periods, but again, 2 weeks each should be sufficient. May move Last Call schedule up for MIBs that are ready to go earlier (e.g., after December 2001 IETF meeting). The waves will overlap in practice, but each main protocol document needs to make it through WG Last Call before any WG Last Calls are issued for documents that depend on it. So for any protocol, wave (1) has to come before (2) and (3), but (2) and (3) can run in parallel, and the different protocols may run on different schedules. From owner-ips@ece.cmu.edu Fri Nov 30 21:42:47 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA22708 for ; Fri, 30 Nov 2001 21:42:47 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fB121WP05462 for ips-outgoing; Fri, 30 Nov 2001 21:01:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxic1.isus.emc.com ([168.159.129.100]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fB121Vl05458 for ; Fri, 30 Nov 2001 21:01:31 -0500 (EST) Received: by mxic1.isus.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 30 Nov 2001 21:03:56 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0293722B@CORPMX14> From: Black_David@emc.com To: ips@ece.cmu.edu Subject: SLC Presenters - Please note Date: Fri, 30 Nov 2001 20:50:48 -0500 Importance: high X-Priority: 1 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk (1) Do not use valuable WG meeting time for presenting the contents of your draft. Assume that the audience has read your draft and knows what's in it (if they haven't, you don't have enough time to explain it to them). WG meeting time is for issue resolution. At most one slide summarizing changes/status is acceptable. Everything else should be devoted to raising open issues and controversial topics ... and remember that brevity is a virtue on slides. Also please refrain from using multicolored corporate slide templates, especially those with large logos. Black text on white background is always acceptable. (2) There are a bunch of agenda items assigned 5 min (or 5 min per draft) described as a status update. Taken together, these items consume a lot of time to convey status, and hence we'd like to optimize them, and in particular take out one of the two laptop to projector cable swaps. - Best choice: Please do your status update with no slides, just get up and talk. Your hardworking WG chairs will appreciate having one less slide set to deal with in assembling the minutes (it can take more of our time to get a set of status update slides incorporated into the minutes than is spent talking to them). Anyone who plans to do their status update without slides should send an email to Elizabeth and me (egrodriguez@lucent.com, black_david@emc.com). - Second choice: Use at most 2 slides, do not use a separate title slide. Send the slides to Elizabeth and me (egrodriguez@lucent.com, black_david@emc.com) and we'll get them onto our laptops. In addition if you can get the presentation onto an adjoining presenter's laptop in advance that would be appreciated (e.g., there are opportunities where we've placed a protocol and its MIB next to each other in the agenda for this reason). (3) Would presenters for items other than the status updates please show up early for your session so that we can pass floppies to get status update presentations onto laptops for the adjoining agenda items. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ece.cmu.edu Fri Nov 30 21:44:04 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA22838 for ; Fri, 30 Nov 2001 21:44:04 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fB12GoW06407 for ips-outgoing; Fri, 30 Nov 2001 21:16:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fB12Gml06402 for ; Fri, 30 Nov 2001 21:16:48 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA16405 for ; Fri, 30 Nov 2001 18:15:25 -0800 (PST) Received: from aimexc07.corp.adaptec.com (aimexc07.corp.adaptec.com [162.62.62.47]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA26490 for ; Fri, 30 Nov 2001 18:00:18 -0800 (PST) Received: by aimexc07.corp.adaptec.com with Internet Mail Service (5.5.2653.19) id ; Fri, 30 Nov 2001 18:15:23 -0800 Message-ID: From: "Mukund, Shridhar" To: ips@ece.cmu.edu Cc: "Mukund, Shridhar" Subject: Choice of ESP alg. for IPS/IPSec - 3DES-CBC vs. 3DES-CBC-I Date: Fri, 30 Nov 2001 18:15:15 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello, Re: Choice of ESP alg. in http://www.ietf.org/internet-drafts/draft-ietf-ips-security-06.txt Question: As noted, we need an algorithm implementable in hardware at speeds of up to 10Gbps, as well as being efficient for implementation in software at speeds of 100Mbps or slower. AES-CTR is an excellent solution. But then it will take time to get approved and further time to get "time tested" before being adopted. Even after adotion of AES-CTR, 3DES-CBC will need to co-exist for many years to come. 3DES-CBC does not gracefully scale to 10Gbps for two reasons: 1. Frequent rekeying at 10Gbps: This issue is discussed in depth in the draft. Although very inconvenient, state-of-art IKE stacks (esp. when running on off-load processor) can deal with it. 2. Lack of pipeline-ability: The feedback loop dictated by CBC prohibits pipelined high-speed VLSI implementation of the 3DES-CBC engine. The ANSI standard X9.52-1998 which specifies 3DES-CBC(TCBC) also specifies an equally standard variant called TCBC-I(say 3DES-CBC-Interleaved) with same security properties. The effort required to enhance existing software and VLSI implementations of 3DES-CBC to 3DES-CBC-I is "minor". 3DES-CBC can be realized simply thru' a degenerate usage of the 3DES-CBC-I module. On the positive side, it brings "substantial" savings in multi-gig VLSI implementation. Was the candidate ESP algorithm 3DES-CBC-I (superset of 3DES-CBC) considered for the SHOULD implement option? Eventually something like AES-CTR will pervade, but for the interm this is indeed a low-cost option to get to speeds up to 10Gbps. Comments on the VLSI implementation: A 3DES(not 3DES-CBC) engine by itself is highly pipeline-able and can pump 10Gbps even on an FPGA. However for 3DES-CBC, one has to wait for 3DES to be completed on a given 64-bit symbol before commencing 3DES on the next symbol. As a result, a "single" 3DES-CBC engine max throughput is somewhere above 1Gbps, depending on the process technology. As usual, there is a brute-force solution to the problem which requires use of multiple 3DES-CBC units. These engines take up significant silicon real estate. The implementation complexity is not just due to the multiplicity of 3DES-CBC units but more so due to all the "incidental" kitchen-sinks and bath-tubs that get thrown into the cauldron to support the multiplicity: scheduler, buffers per engine(think jumbo frames), keeping track of contexts (10Gbps traffic could all belong to the one connection or multiple connections), latency, power, ... 3DES-CBC-I partitions the symbol stream into three sub-streams so that a single engine with three pipeline stages can pump 3X throughput and hence bring about a 3X reduction in the kitchen-sink count and complexity. Further more: At the time 3DES-CBC-I was conceived multi-gig throughput at the network end-point was probably not anticipated(my guess). As a result, they stopped at tri-partitioning or 3-levels of interleaving(my guess). After all it is only the IP Storage application that is pioneering multi-gig IPSec throughput at the end point. If we used 8-levels of interleaving we can pump all 10Gbps of throughput through a single engine using current process technologies. No kitchen-sinks, no bath-tubs! Thoughts, Comments, Concerns ? -Shridhar Mukund From owner-ips@ece.cmu.edu Fri Nov 30 23:37:49 2001 Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.136.200]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA27308 for ; Fri, 30 Nov 2001 23:37:48 -0500 (EST) Received: (from majordom@localhost) by ece.cmu.edu (8.11.0/8.10.2) id fB13kMZ11489 for ips-outgoing; Fri, 30 Nov 2001 22:46:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lucy.trebia.com ([65.192.191.151]) by ece.cmu.edu (8.11.0/8.10.2) with ESMTP id fB13kKl11482 for ; Fri, 30 Nov 2001 22:46:20 -0500 (EST) Received: by lucy with Internet Mail Service (5.5.2653.19) id ; Fri, 30 Nov 2001 22:44:32 -0500 Message-ID: <63616B261CEFD411834D00D0B7A86CF75760CA@lucy> From: Sudhir Srinivasan To: "'Murali Rajagopal'" , Sudhir Srinivasan , ips@ece.cmu.edu Cc: Sudhir Srinivasan Subject: RE: FCIP: Echoing a Changed Short Frame? Date: Fri, 30 Nov 2001 22:44:31 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk It is my opinion that having each side persist with Short Frames till an echo happens with no change is not much more elaborate than the current scheme. (but seeing as even the simple suggestion of flipping two bits in the pFlags field was "shot down in flames", I'm not holding my breath on this one... ;-) ) -S ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trebia Networks, Inc. Sudhir.Srinivasan@trebia.com 978-929-0830 x139 www.trebia.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > -----Original Message----- > From: Murali Rajagopal [mailto:muralir@lightsand.com] > Sent: Thursday, November 29, 2001 12:12 PM > To: Sudhir Srinivasan; ips@ece.cmu.edu > Cc: sudhir.srinivasan@trebia.com > Subject: RE: FCIP: Echoing a Changed Short Frame? > > > Sudhir: > > I think your guess about the purpose and subsequent behavior > of the changed > CH bit is correct. This behavior keeps the protocol simple > without getting into an ellaborate negotiation cycle. > > -Murali > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Sudhir Srinivasan > Sent: Wednesday, November 28, 2001 9:27 PM > To: ips@ece.cmu.edu > Cc: sudhir.srinivasan@trebia.com > Subject: FCIP: Echoing a Changed Short Frame? > > > > Section 8.1.2 of v0.7 of the FCIP draft states in the last > three paras on page 25 that the echoed Short Frame must > match exactly the one that was originated; otherwise the > connection SHALL be closed. > > Section 8.1.5 on pages 28-29 describes how the receiver of > the Short Frame can change the contents of the frame and > then SHALL echo it back to the originator (with the CH bit set). > > What's the purpose of the Change bit and echoing a changed Short > Frame if the originator will simply terminate the connection? > Is the expectation that the originator can capture the changed info > and try to set up the connection again with the new information? > If so, would it not be easier to just have the originator respond with > another Short Frame with the new information and repeat this > back-and-forth until the echo'ed SF has no changes or either side > decides to give up? > > Thanks, > - Sudhir > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Trebia Networks, Inc. Sudhir.Srinivasan@trebia.com > 978-929-0830 x139 www.trebia.com > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >