Subject: RE: iscsi : iscsi parameter default values

From: "Julian Satran" <[email-masked]>
Date: Sat, 29 Sep 2001 08:51:59 +0300

Somesh,

There is a lot of confusion between immediate and non-solicited in general.
There is no need to change the transport at all.
See my previous answer to Robert Snively.

Julo

                    "Somesh Gupta"                                                                           
                    <somesh_gupta@silverbacksy       To:     "Julian Satran" <[email-masked]>,     
                    stems.com>                        <[email-masked]>                                      
                                                     cc:                                                     
                    28-09-01 23:44                   Subject:     RE: iscsi : iscsi parameter default values 
                    Please respond to                                                                        
                    somesh_gupta                                                                             

Is there any target today that will handle ImmediateData
without a significant change to the transport API? Also
is there any target that does use out of order R2Ts?
(Just curious)

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Julian Satran
> Sent: Thursday, September 27, 2001 9:50 AM
> To: [email-masked]
> Subject: RE: iscsi : iscsi parameter default values
>
>
>
> The one that I have trouble living with is ImmediateData. This is
> important
> for low-end desktops and hardly matters for large boxes.
> As such I would suggest it stays as yes.
>
> Julo
>
>
>
>
>                     "Eddy
>
>                     Quicksall"            To:     "'Santosh Rao'"
> <[email-masked]>,
>                     <EQuicksall@med        <[email-masked]>
>
>                     iaone.net>            cc:
>
>                     Sent by:              Subject:     RE: iscsi
> : iscsi parameter default values
>                     [email-masked]
>
>                     mu.edu
>
>
>
>
>
>                     27-09-01 17:22
>
>                     Please respond
>
>                     to "Eddy
>
>                     Quicksall"
>
>
>
>
>
>
>
>
> I like your defaults below.
>
> But, section 5 says:
>
>  The initial Login request MUST include the InitiatorName and
>  SessionType key=value pairs.
>
> Since SessionType is REQUIRED, naming a default would imply a
> possible typo
> in the spec.
>
> Eddy
>
> -----Original Message-----
> From: Santosh Rao [mailto:[email-masked]]
> Sent: Wednesday, September 26, 2001 10:29 PM
> To: [email-masked]
> Subject: iscsi : iscsi parameter default values
>
>
> All,
>
> With the issue of mode page vs. login keys having [almost] drawn to a
> close, I would like to
> raise the below issues again, on the subject of default values for login
> keys and other iscsi
> parameters :
>
>
>    * In keeping with traditional use of scsi mode pages, iscsi should
> not specify any default
>      settings for any mode pages that continue to exist for iscsi.
> "Default values" for mode
>      pages are target specific and should not be bound to the protocol
> draft.
>
>    * MORE IMPORTANTLY, I read the default for EMDP as being set to 1
> :-<  This implies that
>      targets must be always prepared to deal with out of order data and
> initiators must be
>      prepared to deal with out of order data, unless the initiator
> performs a mode select to
>      disable it. [which defeats all the previous advantages gained thru
> mandating use of login
>      keys for other negotiations.]. A default, if it were to exist,
> should be 0. (in-order, by
>      default).
>
>    * Conservative specification of defaults for login keys along the
> following lines :
>                             MaxConnections = 1
>                             FMarker = "no"
>                             InitialR2T = "yes"
>                             BidiInitialR2T = "yes"
>                             ImmediateData = "no"
>                             DataPDULength = 16
>                             MaxOutstandingR2T = 1
>                             DataPDUInOrder = "yes"
>                             ErrorRecoveryLevel = 0
>                             SessionType = "normal"
>
>    * Should the iscsi protocol require a "Lun Control Mode Page"? IOW,
> is an EnableCRN bit
>      required at the transport layer ? If the device server capability
> is to be negotiated , I
>      suggest this bit be moved to a SCSI ULP Mode Page such as the
> "Control Mode Page", through a
>      T10 change as a part of the SCSI changes being driven by iscsi.
>
> Comments ?
>
> Thanks,
> Santosh
>
>
> Santosh Rao wrote:
>
> > There are the separate issues of :
> >
> >    * iscsi's specification of defaults for its mode pages which is not
> in line with mode page
> >      usage. This impacts the target's ability to enforce values other
> than the protocol
> >      specified default, if the initiator were to not use mode
> sense/select.
> >
> >    * default settings for login keys.
> >
> >    * Is there a need for the "LUN Control Mode Page" and whether
> "Enable CRN" should be in a
> >      transport specific mode page ?
> >
> > which need to be driven to closure as well.
> >
> > Regards,
> > Santosh
>
>
>
>
>
>
>
>

---

From: "Julian Satran" <[email-masked]>
Date: Sat, 29 Sep 2001 08:29:07 +0300

Robert,

Unlike FCP - iSCSI has two forms of unsolicited "immediate" and "separate
unsolicited". They can be used separately or together.
Immediate data is a single PDU comming together with the command while the
separate unsolicited may come after.

FCP has only the second type.

With separate unsolicited the target has to have the buffers for the burst
and find the "control blocks" by indexing based on ITT.

With Immediate data the target has to deal with a single PDU (not the
entire burst). Indexing is not done twice (it is done as a check for the
Control block that is being built).

This is why Immediate Data is considered far less  invasive than separate
unsolicited (a single buffer, and no indexing) and give the performance
boost it gives we are going to see it probably on every target.

Julo

                    Robert Snively                                                               
                    <rsnively@broc       To:     Julian Satran/Haifa/IBM@IBMIL, [email-masked]  
                    ade.com>             cc:                                                     
                                         Subject:     RE: iscsi : iscsi parameter default values 
                    29-09-01 01:31                                                               
                    Please respond                                                               
                    to Robert                                                                    
                    Snively                                                                      

Julian,

For SCSI Write operations, ImmediateData = yes is
the most demanding mode for the target.  If I understand
it correctly, it essentially over-rides the R2T state,
allowing the first burst to be included as immediate data.
In SCSI, except for special cases like this, the target
is the device in charge of data transfers.

This mode requires the target to have a guaranteed collection of
received buffer space adequate to receive all possible
outbound queued operations from all possible initiators
through all possible target sessions (which may sum to
1000s of output operations), regardless of what other
buffer-intensive operations may be going on in the device.

It then requires the device to provide association with each
of those commands instead of just the commands it has elected
to activate.  Without immediate data, the command buffer can be
separated and separately managed from the data buffer,
limiting buffer requirements.

It then requires the device to operate in a non-zero-copy
mode (which raises buffer utilization and increases latency)
while the device analyzes the command to determine what should
be done with the data.  It further requires the subsequent
data (if there is more than one PDU to be transmitted) to
be separately managed, and perhaps actually stored in a
separate operation if the buffer must be cleared to make
space available for it.  It further requires the target to
analyze the data for completeness before going on to determine
what to do with it.

Sure, it is easy for the initiator, but so is everything else
except out-of-order reads.
It is the target you are stressing with this.

For local sub-LANs, and depending on the actual buffering
available in the target, the performance may actually be lower with
ImmediateData allowed, because zero copy behavior of the
target to non-volatile media is not available, which raises
buffer utilization.

Have I missed something that would change my mind?

Bob

> From: Julian Satran [mailto:[email-masked]]
> Sent: Friday, September 28, 2001 10:38 AM
> To: [email-masked]
> Robert,
> I disagree that Immediate is "the most demanding" mode.  That
> is not my
> experience and not what I heard from most of the developers.  On the
> contrary immediate data do not require a separate indexing
> operation on the
> target (as non-immediate unsolicited do) and that was the base for the
> consensus to have them negotiated separately.
>
> For the software initiator it is the most inexpensive way to
> perform short
> data transfers (4-8k) typical for major applications like
> Database access
> and so it is for lightweight target.
>
> Julo
>
>
>
>
>
>                     Robert Snively
>
>                     <rsnively@broc       To:     John
> Hufferd/San Jose/IBM@IBMUS, Julian
>                     ade.com>
> Satran/Haifa/IBM@IBMIL
>                                          cc:
> [email-masked]
>                     28-09-01 17:25       Subject:     RE:
> iscsi : iscsi parameter default values
>                     Please respond
>
>                     to Robert
>
>                     Snively
>
>
>
>
>
>
>
>
> I vote no as the default value on ImmediateData.
>
> A default value of yes on ImmediateData requires implementation
> of the most complex and demanding mode of operation as the
> default.  SCSI has traditionally made its default behavior the
> simplest and most encompassing, setting more sophisticated
> behavior by subsequent agreement.  While it may be your earnest
> desire to encourage the implementation of this function, it
> is not appropriate to demand that as the default behavior
> of all devices.  In particular, there is no special benefit
> to providing ImmediateData in low-cost local sub-lans.
>
> If you want to encourage it in a profile, fine, but demanding
> it as the default in the core standard is not appropriate.
>
> Note that the behavior of SCSI is traditionally managed
> entirely by the target.  As such, there has never before now
> been a requirement for the target to, as a default, accept
> any PDU except a command or task management function
> that was not explicitly solicited.  That is one of the mechanisms
> that assists SCSI in achieving a low-overhead zero copy
> capability while operating with a large number of initiators
> and with deep command queues.
>
> Bob Snively                        e-mail:    [email-masked]
> Brocade Communications Systems     phone:  408 487 8135
> 1745 Technology Drive
> San Jose, CA 95110
>
>
>
> > From: John Hufferd [mailto:[email-masked]]
> > Sent: Friday, September 28, 2001 1:33 AM
> > To: Julian Satran
> > Cc: [email-masked]
> > I also agree with this.  It should be yes.
> >
> > .
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/SSG San Jose Ca
> > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > Home Office (408) 997-6136
> > Internet address: [email-masked]
> >
> >
> > Julian Satran/Haifa/IBM@[email-masked] on 09/27/2001 09:50:21 AM
> >
> > Sent by:  [email-masked]
> >
> >
> > To:   [email-masked]
> > cc:
> > The one that I have trouble living with is ImmediateData.
> > This is important
> >                     Sent by:              Subject:     RE:
> > iscsi : iscsi
> > parameter default values

---

From: "Eddy Quicksall" <[email-masked]>
Date: Fri, 28 Sep 2001 11:27:19 -0400

My apologies, I did not notice that the "reject" I am referring to below
only applies to list negotiations ... so, I retract this EMAIL.

Eddy

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Friday, September 28, 2001 11:14 AM
To: [email-masked]'
Subject: RE: iscsi : iscsi parameter default values

BTW,

Since 2.2.4 says:

 If a target is not supporting, or not allowed to use with a specific
 initiator, any of the offered options, it may use the value "reject".

I assume the below would be legal:

I -> ImmediateData=yes
T <- ImmediateData=reject

The target "is not supporting" immediate data so it is now telling the
initiator "no".

Comments?

I think this should be spelled out very clearly. Something like:

For any leading negotiation, the responding party MUST respond. If the
response is reject, <I don't know what to suggest here because it would
depend on the key and you can't say "take the default" because the above
would require the target to disconnect (unless we make the default be
"no")>.

Eddy

---

From: Robert Snively <[email-masked]>
Date: Fri, 28 Sep 2001 16:31:12 -0700

Julian,

Bob

---

From: Robert Snively <[email-masked]>
Date: Fri, 28 Sep 2001 08:25:45 -0700

---

From: "John Hufferd" <[email-masked]>
Date: Sun, 30 Sep 2001 02:28:58 -0700

Robert,
You only have to perform moves, if you have not coded your buffer manager
to manage Chained Segments.  Since this is a very old and highly used
technique in buffer management, especially involving Telecommunications, I
am surprised that you have not considered it.  Perhaps I am missing
something in what you are trying to say but I have used it often to avoid
moving data and obtaining zero-copy.  It is also works well with devices
that support Scatter Gather, which is the usual case with Disk.

Immediate data is simple, perhaps the simplest of all our functions to
code.  You are correct, that it means that there may have to be reasonably
sized buffers.  This argument was valid years ago, but it is now just too
hard not to have large buffers in a storage controller (in fact now small
cost more).

Yes, I know there are limits to what I said above, but that is why the
ImmediateData=No can be used.  But your focus and arguments are old
arguments, for devices that thought memory was too expensive to use a great
deal of it.

Example: since most uses of ImmediateData will probably average less then
4k (example Database pages),  four thousand connections means 16 Meg, I do
not think you can even get a decent price on memories that small.  If you
have a device that supports 4 thousand host connections, I suggest that 16
meg is not an issue.  Why would we even discuss it?

.
.
.
John L. Hufferd

Robert Snively <[email-masked]>@ece.cmu.edu on 09/28/2001 04:31:12 PM

To:   Julian Satran/Haifa/IBM@IBMIL, [email-masked]
cc:
Subject:  RE: iscsi : iscsi parameter default values

Julian,

Bob

---

From: "Julian Satran" <[email-masked]>
Date: Sun, 30 Sep 2001 14:43:35 +0300

Robert,

Immediate data is governed by the PDU-size. In the extrem 1 PDU can be as 
large as First-Burst-SIze.
In your example the total buffer space needed is 500 MBytes - and 
certainly a 50 LU controller with queues that go 50 deep can afford that.

In any case  the default value is (in this case) more an encouragement to 
implement than anything else as resetting
the value to no involves no extra exchange (even a minimal login can 
achieve it).

Regards,
Julo

Robert Snively <[email-masked]>
30-09-01 07:48
Please respond to Robert Snively

Julian,

Are not both immediate data and "unsolicited" data outside the
command PDU both governed by the first-burst size?  If so,
the problem remains the same.  If you have 50 initiators through
each of 10 ports to 50 logical units with 50 commands queued in
each with immediate data of 4K bytes, that adds up to a lot of
bytes which have to have buffers pre-reserved to allow normal
operation.  It is the buffer management that is invasive, not
the indexing of the buffer contents.  It is the dual-copy requirement
that is invasive, significantly increasing buffer utilization. 

For these problems, immediate data and unsolicited data are
equivalent.  Frankly, I see no functional difference between
sending data in the same PDU and sending data in an immediately
following PDU.  In FCP, we found that immediate data caused
sufficient complexity, both in parsing command structures and
in performing error recovery, that we chose not to allow it
since there was no performance benefit anyway.

> Sent: Friday, September 28, 2001 10:29 PM
> To: [email-masked]

---

From: Robert Snively <[email-masked]>
Date: Sat, 29 Sep 2001 22:48:05 -0700

Julian,

---

From: "Sanjeev Bhagat (TRIPACE/Zoetermeer)" <[email-masked]>
Date: Sun, 30 Sep 2001 21:50:59 +0200

All,

I guess this is a very important input from robert.

"In FCP, we found that immediate data caused
sufficient complexity, both in parsing command structures and
in performing error recovery, that we chose not to allow it
since there was no performance benefit anyway."

Considering this i have a question is immediate data really needed at all?
and definitely the default value should be no in this case.

Sanjeev

-----Original Message-----
From: Robert Snively [mailto:[email-masked]]
Sent: Sunday, September 30, 2001 7:48 AM
To: 'Julian Satran'; [email-masked]
Subject: RE: iscsi : iscsi parameter default values

Julian,

---

From: "John Hufferd" <[email-masked]>
Date: Sun, 30 Sep 2001 13:42:09 -0700

Sanjeev,
On the contrary, Immediate Data is simple, it permits  important
performance improvements, and the memory arguments are all old arguments.

Perhaps they were valid in the time that FC was being created, but the
memory arguments are not now valid.  (see my, and Julian's previous notes
on this.)

Almost every one I know is able to easily code chained Buffer management,
which permits the zero copy capability, and with that the rest of Immediate
Data handling is very easy since you have the associated command with the
data.  Using Immediate Data is the simple path, the conservative path.

ImmediateData=yes should be the default.  We should not be bound by the
limitations of previous memory technology.

.
.
.
John L. Hufferd

"Sanjeev Bhagat (TRIPACE/Zoetermeer)" <[email-masked]>@ece.cmu.edu on
09/30/2001 12:50:59 PM

To:   "'Robert Snively'" <[email-masked]>, Julian
      Satran/Haifa/IBM@IBMIL, [email-masked]
cc:
Subject:  RE: iscsi : iscsi parameter default values

All,

Sanjeev

-----Original Message-----
From: Robert Snively [mailto:[email-masked]]
Sent: Sunday, September 30, 2001 7:48 AM
To: 'Julian Satran'; [email-masked]
Subject: RE: iscsi : iscsi parameter default values

Julian,