Subject: RE: iSCSI - bookmarks change

From: "Julian Satran" <[email-masked]>
Date: Wed, 5 Sep 2001 16:59:37 +0300

Ooops - thanks ... I thought I caught them all
I had a serialization problem. I also talk about a field that is not in the
figure anymore :-)

Thanks- Julo

"Eddy Quicksall" <[email-masked]>@ece.cmu.edu on 05-09-2001
16:36:34

Please respond to <[email-masked]>

Sent by:  [email-masked]

To:   <[email-masked]>
cc:
Subject:  RE: iSCSI - bookmarks change

Section "4. Login Phase" of your previous EMAIL (iSCSI - Login changes and
the latest agreements) says:

     The login phase is implemented via login request and responses only.

But below you say:

     During the Login Phase ... the Text command
   SHOULD be issued as an immediate command (I=1).

These seem contradictory.

Eddy

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Tuesday, September 04, 2001 5:19 AM
To: [email-masked]
Subject: iSCSI - bookmarks change

Now that text commands are out of login (and we don't have to care too much
about the immediate version) we can rationalize the bookmarks.

Here is my suggestion for the new text request/response:

1.1  Text Command

   The Text Command is provided to allow the exchange of information and
   for future extensions. It permits the initiator to inform a target of
   its capabilities or to request some special operations.

   A connection SHOULD have only one outstanding text command at any given
   time.

   Byte /    0       |       1       |       2       |       3       |
      /              |               |               |               |
     |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|
     +---------------+---------------+---------------+---------------+
    0|X|I| 0x04      |F|B| Rsvd      | Reserved                      |
     +---------------+---------------+---------------+---------------+
    4| Reserved      | DataSegmentLength                             |
     +---------------+---------------+---------------+---------------+
    8| Reserved                                                      |
     +                                                               +
   12|                                                               |
     +---------------+---------------+---------------+---------------+
   16| Initiator Task Tag                                            |
     +---------------+---------------+---------------+---------------+
   20| Reserved                                                      |
     +---------------+---------------+---------------+---------------+
   24| CmdSN                                                         |
     +---------------+---------------+---------------+---------------+
   28| ExpStatSN                                                     |
     +---------------+---------------+---------------+---------------+
   32/ Reserved                                                      /
    +/                                                               /
     +---------------+---------------+---------------+---------------+
   48| Digests if any...                                             |
     +---------------+---------------+---------------+---------------+
     / DataSegment (Text)                                            /
    +/                                                               /
     +---------------+---------------+---------------+---------------+

1.1.1     F (Final) Bit

   When set to 1 it indicates that this is the last or only text command in
   a sequence of commands; otherwise it indicates that more commands will
   follow.

1.1.2     I - Immediate

   During the Login Phase (the current stage part of CNxSG is either
   SecurityNegotiation or LoginOperationalNegotiation), the Text command
   SHOULD be issued as an immediate command (I=1).

1.1.3     B - bookmark bit

   The bookmark bit set to 1 tells the target to continue from its last
   bookmark for the specific Initiator Task Tag and thus allows a target to
   transfer a large amount of textual data over a sequence of
   text-command/text-response exchanges.

   The bookmark bit set to 0 tells the target that this is a new Text
   request; the target should reset any bookmark it holds on the specific
   Initiator Task Tag.

   A target MAY reject an old Bookmark.

   Bookmark generation at target is detailed in 2.11.2.

   Long text responses are handled as in the following example:

      I->T Text SendTargets=all (F=1,B=0)
      T->I Text <part 1> (F=0,B=1)
      I->T Text <empty> (F=1,B=1)
      T->I Text <part 2> (F=0,B=1)
      I->T Text <empty> (F=1,B=1)
      ...
      T->I Text <part n> (F=1,B=0)

1.1.4     Initiator Task Tag

   The initiator assigned identifier for this Text Command.
   If the command is sent as part of a sequence of commands (e.g., the
   Login Phase or a sequence of Text commands) the Initiator Task Tag MUST
   be the same for all the commands within the sequence (similar to linked
   SCSI commands).

1.1.5     Text

   The initiator sends the target a set of key=value or key=list pairs
   encoded in UTF-8 Unicode. All the text keys and text values specified in
   this document are to be presented and interpreted in the case they
   appear in this document (they are case sensitive). The key and value are
   separated by a '=' (0x3d) delimiter. Every key=value pair (including the
   last or only pair) MUST be followed by null (0x00) delimiter.  A list is
   a set of values separated by comma (0x2c). Large binary items can be
   encoded using their hexadecimal representation (e.g., 8190 is 0x1ffe) or
   decimal representation. The maximum length of an individual value (not
   its string representation) is 255 bytes.

   The data lengths of a text command or response MUST NOT exceed 4096
   bytes.  Key names MUST NOT exceed 63 bytes. Key values MUST NOT exceed
   255 bytes.

   Character strings are represented as plain text. Numeric and binary
   values are represented using either decimal numbers or the hexadecimal
   0xffff notation. Upper and lower case letters may be used
   interchangeably in hexadecimal notation (i.e., 0x1aBc, 0x1AbC and 0x1ABC
   are equivalent).

   The target responds by sending its response back to the initiator. The
   response text format is similar to the request text format.

   Some basic key=value pairs are described in Appendix A and Appendix D.
   All keys in Appendix D, except for the X- extension format, MUST be
   supported by iSCSI initiators and targets. Keys in Appendix A MUST be
   supported only when the function they refer to is mandatory to
   implement.

   Manufacturers may introduce new keys by prefixing them with X- followed
   by their (reversed) domain name, for example the company owning the
   domain acme.com can issue:

      X-com.acme.bar.foo.do_something=0000000000000003

   Any other key not understood by the target may be ignored by the target
   without affecting basic function. However the Text Response for a key
   that was not understood MUST be key=NotUnderstood.

   Text operations are usually meant for parameter setting/negotiations but
   can be used also to perform some long lasting operations.

   Text operations that will take a long time should be placed in their own
   Text command.

1.2

Text Response

   The Text Response PDU contains the target's responses to the initiator's
   Text Command. The format of the Text field matches that of the Text
   Command.

   Byte /    0       |       1       |       2       |       3       |
      /              |               |               |               |
     |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|
     +---------------+---------------+---------------+---------------+
    0|1|1| 0x24      |F|B| Rsvd      | Reserved                      |
     +---------------+---------------+---------------+---------------+
    4| Reserved      | DataSegmentLength                             |
     +---------------+---------------+---------------+---------------+
    8| Reserved                                                      |
     +                                                               +
   12|                                                               |
     +---------------+---------------+---------------+---------------+
   16| Initiator Task Tag                                            |
     +---------------+---------------+---------------+---------------+
   20| Reserved                                                      |
     +---------------+---------------+---------------+---------------+
   24| StatSN                                                        |
     +---------------+---------------+---------------+---------------+
   28| ExpCmdSN                                                      |
     +---------------+---------------+---------------+---------------+
   32| MaxCmdSN                                                      |
     +---------------+---------------+---------------+---------------+
   36/ Reserved                                                      /
    +/                                                               /
     +---------------+---------------+---------------+---------------+
   48| Digests if any...                                             |
     +---------------+---------------+---------------+---------------+
     / DataSegment (Text)                                            /
    +/                                                               /
     +---------------+---------------+---------------+---------------+

1.2.1     F (Final) Bit

   When set to 1 in response to a text command with the Final bit set to 1
   the F bit indicates that the target has finished the current stage of
   the operation or the whole operation.  Otherwise if set to 0 in response
   to a text command with the Final Bit set to 1 it indicates that the
   target has more work to do (invites a follow-on text command).  A text
   response with the F bit set to 1 in response to a text command with the
   F bit set to 0 is a protocol error.

   A text response with a F bit set to 1 MUST NOT contain key=value pairs
   that may require additional answers from the initiator.

1.2.2     B - bookmark Bit

   Whenever a target can't transfer all the remaining text data in a single
   Text response, it attempts to set an internal bookmark. If successful,
   the target associates the bookmark with the Initiator Task Tag.  The
   target indicates that it holds a bookmark for the specific Initiator
   Task Tag by setting the bookmark (B) bit to 1 in the Text response. The
   target resets the internal bookmark associated with a given Initiator
   Task Tag if it receives a Text request with the specified Initiator Task
   Tag with the bookmark bit set to 0.  When a target can't transfer all
   the text data in a single text response and it cannot set an internal
   bookmark it rejects the Text request with an appropriate Reject code. A
   target may reset its internal bookmark(s) after some time in order to
   reclaim resources associated with the bookmark and reject subsequent
   Text requests with the bookmark bit set to 1.

   When all the text data fit in a single Text response the bookmark bit of
   the response is set to 0 and the bookmark associated with the Initiator
   Task Tag is reset.

1.2.3     Initiator Task Tag

   The Initiator Task Tag matches the tag used in the initial Text Command
   or the Login Initiator Task Tag.

1.2.4     Text Response Data

   The Text Response Data Segment contains responses in the same key=value
   format as the Text Command and with the same length and coding
   constraints. Appendix A and Appendix D lists some basic Text Commands
   and their Responses.

   Although the initiator is the requesting party and controls the
   request-response initiation and termination the target can offer
   key=value pairs of its own as part of a sequence and not only in
   response to an identical key=value pair offered by the initiator.

   A Key=value pair must be confined to a given text response even in the
   presence of bookmark - i.e., it must start and end within one Text
   Response.

The Reject Reasons table looks like:

1.1.1     Reason

   The reject Reason is coded as follows:

   +------+-----------------------------------------+------------------+
   | Code | Explanation                             | Can the original |
   | (hex)|                                         | PDU be re-sent?  |
   +------+-----------------------------------------+------------------+
   | 0x01 | Format Error                            | no               |
   |      |                                         |                  |
   | 0x02 | Data (payload) Digest Error             | yes  (Note 1)    |
   |      |                                         |                  |
   | 0x03 | Data-SNACK Reject                       | yes              |
   |      |                                         |                  |
   | 0x04 | Protocol Error (e.g., SNACK request for | no               |
   |      | a status that was already acknowledged) |                  |
   |      |                                         |                  |
   | 0x05 | Command not supported in this session   | no               |
   |      | type                                    |                  |
   |      |                                         |                  |
   | 0x06 | Immediate Command Reject - too many     | yes              |
   |      | immediate commands                      |                  |
   |      |                                         |                  |
   | 0x07 | Bookmark Reject - No bookmark for this  | no               |
   |      | Initiator Task Tag                      |                  |
   |      |                                         |                  |
   | 0x08 | Bookmark Reject - Can't generate        | yes              |
   |      | bookmark - out of resources             |                  |
   |      |                                         |                  |
   | 0x0f | Full Feature Phase Command before login | no               |
   +------+-----------------------------------------+------------------+

 Comments?

 Julo

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 6 Sep 2001 08:20:33 +0300

We could live with this too.  The SHOULD is strong enough to dissuade any
initiator to issue more than one but MUST is good too.  The important thing
is not to have this as a hidden assumption for other things so that if we
feel (now or in the future that we want to remove the restriction we can do
so).

Julo

[email-masked] on 05-09-2001 23:04:46

Please respond to [email-masked]

To:   [email-masked], Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked]
Subject:  RE: iSCSI - bookmarks change

> If we simply have to have a way for a special implementation
> to do multiple outstanding text commands, we could say:
>
>    An initiator SHOULD have only one outstanding command on
>    a connection at any given time.
>
>    A target receiving a text command while another text command
>    is in progress on the same connection MAY reject either or
>    both commands.
>
> Standard implementations would then have only one outstanding
> command at a time; vendor-specific implementations could have
> multiples if they wish (providing that both initiator and target
> support them).

Sorry to butt in on this, but ...

That sort of compromise leads to interoperability problems.
Either support for multiple outstanding commands is REQUIRED,
or there is REQUIRED behavior when an implementation does
this *and the Target detects it* (it's possible to overlap
commands in a fashion that the target won't detect), or
support for overlapping commands is negotiated on login
leading to one of the previous two possibilities.

The quoted text is in invitation to mismatched assumptions
between Initiator and Target and resulting interoperability
difficulties - the truly dangerous piece of Mark's text is
the unpredictable ability to reject a new command
based on existence of an old one, as it's entirely plausible
to create a situation in which no forward progress is possible.

I think Mark's original suggestion of turning the SHOULD into
a MUST (only one outstanding Text Command on a connection) is
the better course of action, possibly with an additional sentence
requiring Task Abort to work on Text commands (i.e., to abort
a text command that's using bookmarks, fire a Task Abort at
it, after that succeeds, the next Text command will also).
A target that receives a text command in the presence of an
outstanding one then rejects the second as a protocol error.

Thanks,
--David
---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------