Subject: Re: iSCSI - bookmarks change

From: "Julian Satran" <[email-masked]>
Date: Thu, 6 Sep 2001 08:12:03 +0300

Mark,

Comments in text - Julo

Mark Bakke <[email-masked]>@cisco.com on 05-09-2001 22:21:31

Please respond to Mark Bakke <[email-masked]>

Sent by:  [email-masked]

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked]
Subject:  Re: iSCSI - bookmarks change

The timeouts are the reason several of us had wanted something
other than bookmarks in the first place; that's when we had
proposed the "option 5" scheme that allowed an initiator to
control the amount of data to be sent, and the target to not
have to keep any state for text commands.  Since the initiator
has to keep state anyway (while waiting for responses), we did
not want to also have to keep state on the target.

+++ I understand that. I went through this line of though, and saw that the
the target will anyhow have to implement a time-out as it can't completely
rely aon the initiator for more than a single request/response (and even
there not).
It was preferable then to let thing be as simple as possible - 1
response/request. +++

Anyway, the bookmarks will work fine; I just want to see
this not get too complicated, and not have too many options.

If we are going to use bookmarks, I would at least like to
see only one outstanding text command at a time, regardless
of what we do with the ITT.  I really don't see any difference
between looking up an ITT or just looking at some value in
a connection structure; I'll have the latter anyway.

+++ Just regularity - the ITT mechanism is there and will be used for every
PDU
and for some recovery+++

I don't see the difficulty.  Anyway, leaving ITT in does not
hurt anything in the target, if we can at least change the
SHOULD to MUST, the target can do the lookup either way.

If we simply have to have a way for a special implementation
to do multiple outstanding text commands, we could say:

    An initiator SHOULD have only one outstanding command on
    a connection at any given time.

    A target receiving a text command while another text command
    is in progress on the same connection MAY reject either or
    both commands.
+++ OK +++
Standard implementations would then have only one outstanding
command at a time; vendor-specific implementations could have
multiples if they wish (providing that both initiator and target
support them).

--
Mark

Julian Satran wrote:
>
> Mark,
>
> I gave it some tought.  Relating it to the connection and task makes for
a
> more difficult implementation.
> You have to make available structures per connection and you can never
then
> relax the one-text-command per connection (now it is only a resource
> statement more than a logical constraint).
>
> The ITT allegiance is simple and it lets you store the local bookmark in
a
> data structure that will time-out.
> I assume that a simple implementation will have one such
> structure/initiator.
>
> Regards,
> Julo
>
> Mark Bakke <[email-masked]>@cisco.com on 05-09-2001 18:29:49
>
> Please respond to Mark Bakke <[email-masked]>
>
> Sent by:  [email-masked]
>
> To:   Julian Satran/Haifa/IBM@IBMIL
> cc:   [email-masked]
> Subject:  Re: iSCSI - bookmarks change
>
> Julian-
>
> This is getting a lot simpler.  We can go even further by turning
> the SHOULD below into a MUST.
>
> If we say:
>
>    A connection MUST have only one outstanding text command at
>    any given time.
>
> This means that the bookmark flag references the connection on
> which the text command is done, and makes the Initiator Task
> Tag unnecessary in both the text command and response.  We can
> turn these into reserve fields, and reference the connection
> instead of the ITT when discussing bookmarked commands.
>
> --
> Mark
>
> Julian Satran wrote:
> >
> > Now that text commands are out of login (and we don't have to care too
> much
> > about the immediate version) we can rationalize the bookmarks.
> >
> > Here is my suggestion for the new text request/response:
> >
> > 1.1  Text Command
> >
> >    The Text Command is provided to allow the exchange of information
and
> >    for future extensions. It permits the initiator to inform a target
of
> >    its capabilities or to request some special operations.
> >
> >    A connection SHOULD have only one outstanding text command at any
> given
> >    time.
> >
> >    Byte /    0       |       1       |       2       |       3       |
> >       /              |               |               |               |
> >      |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|
> >      +---------------+---------------+---------------+---------------+
> >     0|X|I| 0x04      |F|B| Rsvd      | Reserved                      |
> >      +---------------+---------------+---------------+---------------+
> >     4| Reserved      | DataSegmentLength                             |
> >      +---------------+---------------+---------------+---------------+
> >     8| Reserved                                                      |
> >      +                                                               +
> >    12|                                                               |
> >      +---------------+---------------+---------------+---------------+
> >    16| Initiator Task Tag                                            |
> >      +---------------+---------------+---------------+---------------+
> >    20| Reserved                                                      |
> >      +---------------+---------------+---------------+---------------+
> >    24| CmdSN                                                         |
> >      +---------------+---------------+---------------+---------------+
> >    28| ExpStatSN                                                     |
> >      +---------------+---------------+---------------+---------------+
> >    32/ Reserved                                                      /
> >     +/                                                               /
> >      +---------------+---------------+---------------+---------------+
> >    48| Digests if any...                                             |
> >      +---------------+---------------+---------------+---------------+
> >      / DataSegment (Text)                                            /
> >     +/                                                               /
> >      +---------------+---------------+---------------+---------------+
> >
> > 1.1.1     F (Final) Bit
> >
> >    When set to 1 it indicates that this is the last or only text
command
> in
> >    a sequence of commands; otherwise it indicates that more commands
will
> >    follow.
> >
> > 1.1.2     I - Immediate
> >
> >    During the Login Phase (the current stage part of CNxSG is either
> >    SecurityNegotiation or LoginOperationalNegotiation), the Text
command
> >    SHOULD be issued as an immediate command (I=1).
> >
> > 1.1.3     B - bookmark bit
> >
> >    The bookmark bit set to 1 tells the target to continue from its last
> >    bookmark for the specific Initiator Task Tag and thus allows a
target
> to
> >    transfer a large amount of textual data over a sequence of
> >    text-command/text-response exchanges.
> >
> >    The bookmark bit set to 0 tells the target that this is a new Text
> >    request; the target should reset any bookmark it holds on the
specific
> >    Initiator Task Tag.
> >
> >    A target MAY reject an old Bookmark.
> >
> >    Bookmark generation at target is detailed in 2.11.2.
> >
> >    Long text responses are handled as in the following example:
> >
> >       I->T Text SendTargets=all (F=1,B=0)
> >       T->I Text <part 1> (F=0,B=1)
> >       I->T Text <empty> (F=1,B=1)
> >       T->I Text <part 2> (F=0,B=1)
> >       I->T Text <empty> (F=1,B=1)
> >       ...
> >       T->I Text <part n> (F=1,B=0)
> >
> > 1.1.4     Initiator Task Tag
> >
> >    The initiator assigned identifier for this Text Command.
> >    If the command is sent as part of a sequence of commands (e.g., the
> >    Login Phase or a sequence of Text commands) the Initiator Task Tag
> MUST
> >    be the same for all the commands within the sequence (similar to
> linked
> >    SCSI commands).
> >
> > 1.1.5     Text
> >
> >    The initiator sends the target a set of key=value or key=list pairs
> >    encoded in UTF-8 Unicode. All the text keys and text values
specified
> in
> >    this document are to be presented and interpreted in the case they
> >    appear in this document (they are case sensitive). The key and value
> are
> >    separated by a '=' (0x3d) delimiter. Every key=value pair (including
> the
> >    last or only pair) MUST be followed by null (0x00) delimiter.  A
list
> is
> >    a set of values separated by comma (0x2c). Large binary items can be
> >    encoded using their hexadecimal representation (e.g., 8190 is
0x1ffe)
> or
> >    decimal representation. The maximum length of an individual value
(not
> >    its string representation) is 255 bytes.
> >
> >    The data lengths of a text command or response MUST NOT exceed 4096
> >    bytes.  Key names MUST NOT exceed 63 bytes. Key values MUST NOT
exceed
> >    255 bytes.
> >
> >    Character strings are represented as plain text. Numeric and binary
> >    values are represented using either decimal numbers or the
hexadecimal
> >    0xffff notation. Upper and lower case letters may be used
> >    interchangeably in hexadecimal notation (i.e., 0x1aBc, 0x1AbC and
> 0x1ABC
> >    are equivalent).
> >
> >    The target responds by sending its response back to the initiator.
The
> >    response text format is similar to the request text format.
> >
> >    Some basic key=value pairs are described in Appendix A and Appendix
D.
> >    All keys in Appendix D, except for the X- extension format, MUST be
> >    supported by iSCSI initiators and targets. Keys in Appendix A MUST
be
> >    supported only when the function they refer to is mandatory to
> >    implement.
> >
> >    Manufacturers may introduce new keys by prefixing them with X-
> followed
> >    by their (reversed) domain name, for example the company owning the
> >    domain acme.com can issue:
> >
> >       X-com.acme.bar.foo.do_something=0000000000000003
> >
> >    Any other key not understood by the target may be ignored by the
> target
> >    without affecting basic function. However the Text Response for a
key
> >    that was not understood MUST be key=NotUnderstood.
> >
> >    Text operations are usually meant for parameter setting/negotiations
> but
> >    can be used also to perform some long lasting operations.
> >
> >    Text operations that will take a long time should be placed in their
> own
> >    Text command.
> >
> > 1.2
> >
> > Text Response
> >
> >    The Text Response PDU contains the target's responses to the
> initiator's
> >    Text Command. The format of the Text field matches that of the Text
> >    Command.
> >
> >    Byte /    0       |       1       |       2       |       3       |
> >       /              |               |               |               |
> >      |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|
> >      +---------------+---------------+---------------+---------------+
> >     0|1|1| 0x24      |F|B| Rsvd      | Reserved                      |
> >      +---------------+---------------+---------------+---------------+
> >     4| Reserved      | DataSegmentLength                             |
> >      +---------------+---------------+---------------+---------------+
> >     8| Reserved                                                      |
> >      +                                                               +
> >    12|                                                               |
> >      +---------------+---------------+---------------+---------------+
> >    16| Initiator Task Tag                                            |
> >      +---------------+---------------+---------------+---------------+
> >    20| Reserved                                                      |
> >      +---------------+---------------+---------------+---------------+
> >    24| StatSN                                                        |
> >      +---------------+---------------+---------------+---------------+
> >    28| ExpCmdSN                                                      |
> >      +---------------+---------------+---------------+---------------+
> >    32| MaxCmdSN                                                      |
> >      +---------------+---------------+---------------+---------------+
> >    36/ Reserved                                                      /
> >     +/                                                               /
> >      +---------------+---------------+---------------+---------------+
> >    48| Digests if any...                                             |
> >      +---------------+---------------+---------------+---------------+
> >      / DataSegment (Text)                                            /
> >     +/                                                               /
> >      +---------------+---------------+---------------+---------------+
> >
> > 1.2.1     F (Final) Bit
> >
> >    When set to 1 in response to a text command with the Final bit set
to
> 1
> >    the F bit indicates that the target has finished the current stage
of
> >    the operation or the whole operation.  Otherwise if set to 0 in
> response
> >    to a text command with the Final Bit set to 1 it indicates that the
> >    target has more work to do (invites a follow-on text command).  A
text
> >    response with the F bit set to 1 in response to a text command with
> the
> >    F bit set to 0 is a protocol error.
> >
> >    A text response with a F bit set to 1 MUST NOT contain key=value
pairs
> >    that may require additional answers from the initiator.
> >
> > 1.2.2     B - bookmark Bit
> >
> >    Whenever a target can't transfer all the remaining text data in a
> single
> >    Text response, it attempts to set an internal bookmark. If
successful,
> >    the target associates the bookmark with the Initiator Task Tag.  The
> >    target indicates that it holds a bookmark for the specific Initiator
> >    Task Tag by setting the bookmark (B) bit to 1 in the Text response.
> The
> >    target resets the internal bookmark associated with a given
Initiator
> >    Task Tag if it receives a Text request with the specified Initiator
> Task
> >    Tag with the bookmark bit set to 0.  When a target can't transfer
all
> >    the text data in a single text response and it cannot set an
internal
> >    bookmark it rejects the Text request with an appropriate Reject
code.
> A
> >    target may reset its internal bookmark(s) after some time in order
to
> >    reclaim resources associated with the bookmark and reject subsequent
> >    Text requests with the bookmark bit set to 1.
> >
> >    When all the text data fit in a single Text response the bookmark
bit
> of
> >    the response is set to 0 and the bookmark associated with the
> Initiator
> >    Task Tag is reset.
> >
> > 1.2.3     Initiator Task Tag
> >
> >    The Initiator Task Tag matches the tag used in the initial Text
> Command
> >    or the Login Initiator Task Tag.
> >
> > 1.2.4     Text Response Data
> >
> >    The Text Response Data Segment contains responses in the same
> key=value
> >    format as the Text Command and with the same length and coding
> >    constraints. Appendix A and Appendix D lists some basic Text
Commands
> >    and their Responses.
> >
> >    Although the initiator is the requesting party and controls the
> >    request-response initiation and termination the target can offer
> >    key=value pairs of its own as part of a sequence and not only in
> >    response to an identical key=value pair offered by the initiator.
> >
> >    A Key=value pair must be confined to a given text response even in
the
> >    presence of bookmark - i.e., it must start and end within one Text
> >    Response.
> >
> > The Reject Reasons table looks like:
> >
> > 1.1.1     Reason
> >
> >    The reject Reason is coded as follows:
> >
> >
+------+-----------------------------------------+------------------+
> >    | Code | Explanation                             | Can the original
|
> >    | (hex)|                                         | PDU be re-sent?
|
+------+-----------------------------------------+------------------+
> >    | 0x01 | Format Error                            | no
|
> >    |      |                                         |
|
> >    | 0x02 | Data (payload) Digest Error             | yes  (Note 1)
|
> >    |      |                                         |
|
> >    | 0x03 | Data-SNACK Reject                       | yes
|
> >    |      |                                         |
|
> >    | 0x04 | Protocol Error (e.g., SNACK request for | no
|
> >    |      | a status that was already acknowledged) |
|
> >    |      |                                         |
|
> >    | 0x05 | Command not supported in this session   | no
|
> >    |      | type                                    |
|
> >    |      |                                         |
|
> >    | 0x06 | Immediate Command Reject - too many     | yes
|
> >    |      | immediate commands                      |
|
> >    |      |                                         |
|
> >    | 0x07 | Bookmark Reject - No bookmark for this  | no
|
> >    |      | Initiator Task Tag                      |
|
> >    |      |                                         |
|
> >    | 0x08 | Bookmark Reject - Can't generate        | yes
|
> >    |      | bookmark - out of resources             |
|
> >    |      |                                         |
|
> >    | 0x0f | Full Feature Phase Command before login | no
|
+------+-----------------------------------------+------------------+
> >  Comments?
> >  Julo
> --
> Mark A. Bakke
> Cisco Systems
> [email-masked]
> 763.398.1054

--
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

---

From: "Eddy Quicksall" <[email-masked]>
Date: Thu, 6 Sep 2001 09:43:55 -0400

If the ITT is going to be kept, why not also have the TTT? The same argument
used to justify the ITT for the initiator could be used to justify the TTT
for the target.

Eddy

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Thursday, September 06, 2001 1:12 AM
To: [email-masked]
Subject: Re: iSCSI - bookmarks change

Mark,

Comments in text - Julo

+++ I understand that. I went through this line of though, and saw that
the
the target will anyhow have to implement a time-out as it can't
completely
rely aon the initiator for more than a single request/response (and even
there not).
It was preferable then to let thing be as simple as possible - 1
response/request. +++

+++ Just regularity - the ITT mechanism is there and will be used for
every
PDU
and for some recovery+++

--

---

From: Santosh Rao <[email-masked]>
Date: Thu, 06 Sep 2001 12:08:34 -0700

Eddy,

I am also in favor of keeping the TTT in text command/responses and had
proposed the same about 4 months ago.
(See : http://www.pdl.cmu.edu/mailinglists/ips/mail/msg04527.html).

In any multi PDU exchange, such as a WRITE scsi cmd, or a multi-pdu text
command, the target should be allowed to use a TTT that allows it to perform a
lookup for its state information for that exchange.

This is the simplest scheme and prevents any special case code in targets to
perform a lookup of their state information for an exchange.

Regards,
Santosh