Subject: Re: [storm] iSCSI drafts - T10 and T11 references

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Sun, 23 Sep 2012 05:33:52 +0000

Hi David,

I have noticed this comment as I am working through all Last Call comments.

The references to SAM-3 and SAM-4 are keyed to this paragraph in Section 13.9:

"[SAM2] and [SAM3] specifications note in their informative text that TPGT value should be non-zero, note that it is incorrect.  A zero value is allowed as a legal value for TPGT. This discrepancy currently stands corrected in [SAM4]."

Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes that it defines, in Section 11.14.5 of the draft.

So, I believe references to all three SAM specs would continue to be appropriate.

The current draft references FC-FS in discussing the NAA format in a few places, so the current reference at the end is intentionally the same. Is the older reference a problem? If so, I suppose we can replace all references to FC-FS-4, but that seems to be still in early stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3?

Thanks.

Mallikarjun

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, July 26, 2012 6:47 AM
To: [email-masked]
Subject: [storm] iSCSI drafts - T10 and T11 references

In making up the lists of T10 and T11 documents that are currently referenced, some of those references don't look right to me, for example:

- T10: I'm not sure whether we need to reference SAM-2, -3 and -4.
- T11: The FC-FS reference is out of date.

As part of dealing with the comments from IETF Last Call, we'll need to go over all of the T10 and T11 references (and should check all the references) to make sure that they're up to date and necessary.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

_______________________________________________
storm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/storm

---

From: "Knight, Frederick" <[email-masked]>
Date: Sun, 23 Sep 2012 10:38:18 +0000

Interesting, I just came across a question in a related area (only for SAM-2).

RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn from SAM-2, and some from SAM-3.

We have references to I_T NEXUS LOSS events in RFC3720.  The I_T Nexus loss concept does not exist in SAM-2.  SAM-2 does include references to RESERVE and RELEASE commands.  SAM-3 includes I_T Nexus loss, but it has no reference to RESERVE state.  There are allusions in RFC3720 about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit.  RFC3720 references other documents (SAM2, SAM3, SBC, SPC3), none of which address the situation.

Consider text such as the following:
4.4.3.1. I_T Nexus State

  Certain nexus relationships contain an explicit state (e.g.,
  initiator-specific mode pages) that may need to be preserved by
  the device server [SAM2] in a logical unit through changes or
  failures in the iSCSI layer (e.g., session failures). In order for
  that state to be restored, the iSCSI initiator should reestablish
  its session (re-login) to the same Target Portal Group using the
  previous ISID. That is, it should perform session recovery as
  described in Chapter 6. This is because the SCSI initiator port
  identifier and the SCSI target port identifier (or relative target
  port) form the datum that the SCSI logical unit device server uses
  to identify the I_T nexus.

Is RESERVE state - explicit state?  When an I_T nexus is reestablished, initiator-specific mode pages are preserved.  That implies that an I_T nexus loss should be treated in the way that SAM-2 describes via CLEAR TASK SET (where mode pages and RESERVE state are preserved).  If a session reestablishment caused saved mode pages to be restored rather than the established initiator-specific mode pages to be restored, then it would be like a LU reset (where the RESERVE state is lost).  Is RESERVE state explicit state that is to be preserved?

Appendix E covers clearing effects during various events, but it never mentions RESERVE state.

Section E.2 contains the following text:

  The only iSCSI protocol action that can effect clearing actions on
  SCSI objects is the "I_T nexus loss" notification (Section 6.3.5.1
  Loss of Nexus notification). [SPC3] describes the clearing effects
  of this notification on a variety of SCSI attributes. In addition,
  SCSI standards documents (such as [SAM2] and [SBC]) define
  additional clearing actions that may take place for several SCSI
  objects on SCSI events such as LU resets and power-on resets.

I could not find any text in SPC3 that describes these clearing effects (I don't know if this SPC3 reference is incorrect, but at a minimum, it is ineffective).  The SAM2 text for LU resets and power-on resets is easily found, and clearly says that any RESERVE is released in those cases.  As an aside, the reference to section 6.3.5.1 is a circular reference).  The text in E.2 goes on:

  Since iSCSI defines a target cold reset as a protocol-equivalent
  to a target power-cycle, the iSCSI target cold reset must also be
  considered as the power-on reset event in interpreting the actions
  defined in the SCSI standards.

  When the iSCSI session is reconstructed (between the same SCSI
  ports with the same nexus identifier) reestablishing the same I_T
  nexus, all SCSI objects that are defined to not clear on the "I_T
  nexus loss" notification event, such as persistent reservations,
  are automatically associated to this new session.

So, it is clear what happens to PERSISTENT RESERVE state, but not so clear about RESERVE state.  It also states that all SCSI objects that are defined to not clear on the I_T nexus loss are retained.  But again, the only list I can find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction between RESERVE and I_T Nexus loss.

T10 has never had to deal with this because they removed RESERVE before they added I_T Nexus loss.  However, iSCSI did not do that; we discuss BOTH, but we fail to describe how they interact.  iSCSI also has the feature of reestablishment of a session that does not exist in SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't exist).  So again, if session reestablishment is examined in the light of parallel SCSI, the intent would be to retain the RESERVE state.  Here is another section on the topic:

6.3.5.1. Loss of Nexus Notification

  The iSCSI layer provides the SCSI layer with the "I_T nexus loss"
  notification when any one of the following events happens:

     Successful completion of session reinstatement.
     Session closure event.
     Session timeout event.

  Certain SCSI object clearing actions may result due to the
  notification in the SCSI end nodes, as documented in Appendix F. -
  Clearing Effects of Various Events on Targets.

FWIW - the reference above is incorrect - there is no Appendix F in the consolidated draft - this should be Appendix E (and also, BTW - this is a circular reference).

So anyway, it seems like the intent was that session reinstatement would preserve the RESERVE state across the I_T Nexus loss event.  

So aside from the incorrect reference (and circular nature of it), any thoughts on whether we should get explicit about this situation?  Should we remain silent; should we tweak Appendix E along these lines (or other lines that others might suggest):

4.4.3.1. I_T Nexus State

  Certain nexus relationships contain an explicit state (e.g.,
  initiator-specific mode pages, and RESERVE state) that may need to be preserved by
  the device server [SAM2] in a logical unit through changes or
  failures in the iSCSI layer (e.g., session failures)....

	Fred Knight

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mallikarjun Chadalapaka
Sent: Sunday, September 23, 2012 1:34 AM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSCSI drafts - T10 and T11 references

Hi David,

Thanks.

Mallikarjun

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, July 26, 2012 6:47 AM
To: [email-masked]
Subject: [storm] iSCSI drafts - T10 and T11 references

Thanks,
--David

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Sun, 23 Sep 2012 22:34:36 +0000

Hi Fred,

SPC-3 discusses clearing effects of I-T nexus loss in several places.

Section 5.8.2.9 for ALUA
Section 5.5.3.2 for foreground self-test
Section 5.5.3.3 for background self-test
Section 5.6.1 on persistent reservations
Section 5.13 on timestamps
......

Thanks for pointing out the broken cross-reference - Appendix F instead of E. I have just fixed it - and reviewed all other Appendix references elsewhere in the draft. However, I do not see a circular reference problem per se - Appendix E is correctly referencing Section 6.3.5 for definition of terms, e.g. session continuation. And Section 6.3.5.1 is correctly deferring to Appendix E when discussing clearing effects. Let me know if I'm overlooking something here.

Now, the reserve/release management state, :) : I agree the formal I_T nexus loss event didn't exist in SAM-2, however the notion of a new I_T nexus state did co-exist with a session-oriented I_T nexus well before iSCSI - FCP process login (PRLO) comes to mind here. So I'd be curious about traditional FCP handling of reserve/release management state. And I'm also checking internally here at MS. From what I know now though, it does *appear* that reserve/release management state should not be cleared with I_T nexus loss.

Having said all that, note that reserve/release management state is squarely in the SCSI territory. iSCSI spec has so far by design deferred to SCSI standards in specifying clearing effects on SCSI objects - as Appendix E.2 makes it clear. What we do specify is the clearing effects of SCSI actions, e.g. LU reset, on iSCSI objects.

Thanks.

Mallikarjun

-----Original Message-----
From: Knight, Frederick [mailto:[email-masked]] 
Sent: Sunday, September 23, 2012 3:38 AM
To: Mallikarjun Chadalapaka; [email-masked]; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

4.4.3.1. I_T Nexus State

	Fred Knight

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mallikarjun Chadalapaka
Sent: Sunday, September 23, 2012 1:34 AM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSCSI drafts - T10 and T11 references

Hi David,

Thanks.

Mallikarjun

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, July 26, 2012 6:47 AM
To: [email-masked]
Subject: [storm] iSCSI drafts - T10 and T11 references

Thanks,
--David

---

From: "Knight, Frederick" <[email-masked]>
Date: Mon, 24 Sep 2012 12:57:49 +0000

FCP is also not explicit (it defers to the SCSI specs - which do not say).  The closest statements I can find still requires extrapolation:

Here are some key points (from SAM 6.3.4 I_T nexus loss):

1) I_T nexus loss clears all ACA conditions;
2) established a UA - one of the following is suggested:

"If the logical unit retains state information for the I_T nexus that is lost, its response to the subsequent I_T nexus
re-establishment for the logical unit should include establishing a unit attention with an additional sense code set
to I_T NEXUS LOSS OCCURRED.

If the logical unit does not retain state information for the I_T nexus that is lost, it shall consider the subsequent
I_T nexus re-establishment, if any, as the formation of a new I_T nexus for which there is no past history (e.g.,
establish a unit attention with an additional sense code set to POWER ON OCCURRED)."

But, there is no definition of what it means to "retain state".  The conclusion I could jump to based on #1 above, is that RESERVE state is also cleared (this action lines up more with LU RESET than CLEAR TASK SET); and based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on conditions) I could also assume that the RESERVE state is cleared; but based on the UA of 29/07 (retain state), I could jump to the conclusion that RESERVE state is retained.

But these are all conclusions I've jump to with very limited basis in the text.

Based on what RFC3720 says about explicit state (e.g., mode pages), I would assume that hosts would believe that every command they explicitly send (like MODE SELECT or RESERVE) would have created explicit state and therefore the host would assume RESERVE state is retained.  It doesn't make sense to have mode page values retained, but have RESERVE state lost.

      Certain nexus relationships contain an explicit state (e.g.,
      initiator-specific mode pages) that may need to be preserved by
      the device server [SAM2] in a logical unit through changes or
      failures in the iSCSI layer (e.g., session failures).

It will be interesting to hear what a real host assumes about this situation.

	Fred

-----Original Message-----
From: Mallikarjun Chadalapaka [mailto:[email-masked]] 
Sent: Sunday, September 23, 2012 6:35 PM
To: Knight, Frederick; [email-masked]; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

Hi Fred,

Section 5.8.2.9 for ALUA
Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background self-test Section 5.6.1 on persistent reservations Section 5.13 on timestamps ......

Thanks.

Mallikarjun

-----Original Message-----
From: Knight, Frederick [mailto:[email-masked]]
Sent: Sunday, September 23, 2012 3:38 AM
To: Mallikarjun Chadalapaka; [email-masked]; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

4.4.3.1. I_T Nexus State

	Fred Knight

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mallikarjun Chadalapaka
Sent: Sunday, September 23, 2012 1:34 AM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSCSI drafts - T10 and T11 references

Hi David,

Thanks.

Mallikarjun

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, July 26, 2012 6:47 AM
To: [email-masked]
Subject: [storm] iSCSI drafts - T10 and T11 references

Thanks,
--David

---

From: "Black, David" <[email-masked]>
Date: Mon, 24 Sep 2012 11:36:41 -0400

Fred,

Thank you for the discussion, particularly around the dependence on SAM-3.

In light of that, and a few other things (e.g., a strong preference that
the iSCSI consolidated draft not reference work in progress), my
recommendations on the T10 and T11 references are:

--- T10 (SCSI) ---

1) [SBC2] and [SPC3] are the current published standards; those should be the
correct references for the iSCSI consolidated draft.  [SPC3] is clearly
normative, but I think [SBC2] is (and ought to be) informative - [SBC2] is
only cited once in what looks like an informative fashion in Appendix E.2:

	In addition, SCSI standards documents (such as [SAM2] and [SBC2])
	define additional clearing actions that may take place for several
	SCSI objects on SCSI events such as LU resets and power-on resets.

Moreover, there is no requirement to implement the SBC command set for all
iSCSI implementations, whereas the SPC command set is a requirement for
obvious reasons (P = Primary).

2) [SAM2] and [SAM3] ought to be normative references, in that we're drawing
some concepts from [SAM-3].  OTOH, [SAM-4] ought to be an informative
reference, as the TPGT text that is referred to in SAM-4:

> "[SAM2] and [SAM3] specifications note in their informative text that TPGT
> value should be non-zero, note that it is incorrect.  A zero value is allowed
> as a legal value for TPGT. This discrepancy currently stands corrected in
> [SAM4]."

is an informative table footnote in an annex.  Further,

> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes
> that it defines, in Section 11.14.5 of the draft.

That new ASC/Q code's name does not occur in the body of SAM-4, so I would 
remove that second citation of SAM-4.

3) The [SAS] reference needs to be expanded ;-).  The current informative
[SAS] reference needs to be replaced by references to SAS-2.1, SPL and
SPL Amendment 1, see:

	http://www.t10.org/drafts.htm#SCSI3_SAS

4) OTOH, I recommend removing the [SRP] reference and related material,
'namely the definition of "SCSI RDMA Protocol" and the alternative expansion
of the SRP acronym.  The SRP-2 standards project to update SRP was abandoned
by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us with
a vintage-2002 SRP document.

--- T11 (Fibre Channel) ---

The FC-FS reference ought to be replaced by a reference to the current
published FS-FS-3 standard.

Thanks,
--David

> Sent: Sunday, September 23, 2012 6:38 AM
> To: Mallikarjun Chadalapaka; Black, David; [email-masked]
> We have references to I_T NEXUS LOSS events in RFC3720.  The I_T Nexus loss
> concept does not exist in SAM-2.  SAM-2 does include references to RESERVE and
> RELEASE commands.  SAM-3 includes I_T Nexus loss, but it has no reference to
> RESERVE state.  There are allusions in RFC3720 about the intersection of
> RESERVE/RELEASE and I_T NEXUS LOSS, but there is nothing explicit.  RFC3720
> references other documents (SAM2, SAM3, SBC, SPC3), none of which address the
> situation.
> 
> Is RESERVE state - explicit state?  When an I_T nexus is reestablished,
> initiator-specific mode pages are preserved.  That implies that an I_T nexus
> loss should be treated in the way that SAM-2 describes via CLEAR TASK SET
> (where mode pages and RESERVE state are preserved).  If a session
> reestablishment caused saved mode pages to be restored rather than the
> established initiator-specific mode pages to be restored, then it would be
> like a LU reset (where the RESERVE state is lost).  Is RESERVE state explicit
> state that is to be preserved?
> 
> I could not find any text in SPC3 that describes these clearing effects (I
> don't know if this SPC3 reference is incorrect, but at a minimum, it is
> ineffective).  The SAM2 text for LU resets and power-on resets is easily
> found, and clearly says that any RESERVE is released in those cases.  As an
> aside, the reference to section 6.3.5.1 is a circular reference).  The text in
> E.2 goes on:
> 
> So, it is clear what happens to PERSISTENT RESERVE state, but not so clear
> about RESERVE state.  It also states that all SCSI objects that are defined to
> not clear on the I_T nexus loss are retained.  But again, the only list I can
> find of what is cleared is here in this Appendix, since SAM-2, SAM-3, SAM-4
> are all silent on the interaction between RESERVE and I_T Nexus loss.
> 
> T10 has never had to deal with this because they removed RESERVE before they
> added I_T Nexus loss.  However, iSCSI did not do that; we discuss BOTH, but we
> fail to describe how they interact.  iSCSI also has the feature of
> reestablishment of a session that does not exist in SAM-2 (SAM-2 was still
> parallel SCSI focused, where connections didn't exist).  So again, if session
> reestablishment is examined in the light of parallel SCSI, the intent would be
> to retain the RESERVE state.  Here is another section on the topic:
> 
> FWIW - the reference above is incorrect - there is no Appendix F in the
> consolidated draft - this should be Appendix E (and also, BTW - this is a
> circular reference).
> 
> So anyway, it seems like the intent was that session reinstatement would
> preserve the RESERVE state across the I_T Nexus loss event.
> 
> So aside from the incorrect reference (and circular nature of it), any
> thoughts on whether we should get explicit about this situation?  Should we
> remain silent; should we tweak Appendix E along these lines (or other lines
> that others might suggest):
> 
> 4.4.3.1. I_T Nexus State
> 
> "[SAM2] and [SAM3] specifications note in their informative text that TPGT
> value should be non-zero, note that it is incorrect.  A zero value is allowed
> as a legal value for TPGT. This discrepancy currently stands corrected in
> [SAM4]."
> 
> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ codes
> that it defines, in Section 11.14.5 of the draft.
> 
> The current draft references FC-FS in discussing the NAA format in a few
> places, so the current reference at the end is intentionally the same. Is the
> older reference a problem? If so, I suppose we can replace all references to
> FC-FS-4, but that seems to be still in early stages(0.1 spec posted on April
> 2012), so perhaps we should use FC-FS3?
> 
> Thanks.
> 
> Mallikarjun
> 
> 
> 
> 
> In making up the lists of T10 and T11 documents that are currently referenced,
> some of those references don't look right to me, for example:
> 
> As part of dealing with the comments from IETF Last Call, we'll need to go
> over all of the T10 and T11 references (and should check all the references)
> to make sure that they're up to date and necessary.
> 
> Thanks,
> --David

---

From: "Black, David" <[email-masked]>
Date: Mon, 24 Sep 2012 12:42:07 -0400

Fred,

This is interesting ...

> "If the logical unit retains state information for the I_T nexus that
> is lost, its response to the subsequent I_T nexus
> re-establishment for the logical unit should include establishing a
> unit attention with an additional sense code set
> If the logical unit does not retain state information for the I_T nexus
> that is lost, it shall consider the subsequent
> I_T nexus re-establishment, if any, as the formation of a new I_T nexus
> for which there is no past history (e.g.,

That "state information" has to identify the endpoints of the I_T Nexus
- for iSCSI that means at least the ISID and TPGT.  If that's gone, I
think the RESERVE state is gone as well, in contrast to PERSISTENT
RESERVE state.  On RESERVE state, SPC-2 has this to say in 5.5.2 "The
Reserve/Release management method":

        The reserve/release management method commands, RESERVE(6),
        RESERVE(10), RELEASE(6), and RELEASE(10) are used among multiple
        initiators that do not require operations to be protected across
        initiator failures (and subsequent hard resets).

I think it's fair to regard I_T Nexus loss as an initiator failure.  I
also don't see any good way to do a 3rd-party release on a RESERVE
reservation that persists after I_T Nexus loss.  I also believe that
initiator software that uses Reserve/Release does not expect
reservations to persist across I_T Nexus loss.

As to what needs to be stated in the iSCSI consolidated draft - the SPC
reference has been updated to SPC-3 in which the Reserve/Release
management method is obsolete, so there isn't any absolute requirement
to cover that.  OTOH, it might be reasonable to add a sentence to say
that Reserve/Release state is implicit (e.g., as there's no way for an
initiator to figure out which I_T Nexus holds a reservation), and that
treating Reserve/Release state as indefinitely persistent after I_T
Nexus Loss is a bad idea ("should not", but lower case) as it may
require the initiator to reset the target device in order to clear out
that state (issuing RELEASE commands isn't enough).  This sort of text
would require a reference to SPC-2.

Thanks,
--David

> Sent: Monday, September 24, 2012 8:58 AM
> FCP is also not explicit (it defers to the SCSI specs - which do not say).
> The closest statements I can find still requires extrapolation:
>
> re-establishment for the logical unit should include establishing a unit
> attention with an additional sense code set
> If the logical unit does not retain state information for the I_T nexus that
> is lost, it shall consider the subsequent
> But, there is no definition of what it means to "retain state".  The
> conclusion I could jump to based on #1 above, is that RESERVE state is also
> cleared (this action lines up more with LU RESET than CLEAR TASK SET); and
> based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both are power on
> conditions) I could also assume that the RESERVE state is cleared; but based
> on the UA of 29/07 (retain state), I could jump to the conclusion that RESERVE
> state is retained.
>
> Based on what RFC3720 says about explicit state (e.g., mode pages), I would
> assume that hosts would believe that every command they explicitly send (like
> MODE SELECT or RESERVE) would have created explicit state and therefore the
> host would assume RESERVE state is retained.  It doesn't make sense to have
> mode page values retained, but have RESERVE state lost.
>
> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for background self-
> test Section 5.6.1 on persistent reservations Section 5.13 on timestamps
> ......
>
> Thanks for pointing out the broken cross-reference - Appendix F instead of E.
> I have just fixed it - and reviewed all other Appendix references elsewhere in
> the draft. However, I do not see a circular reference problem per se -
> Appendix E is correctly referencing Section 6.3.5 for definition of terms,
> e.g. session continuation. And Section 6.3.5.1 is correctly deferring to
> Appendix E when discussing clearing effects. Let me know if I'm overlooking
> something here.
>
> Now, the reserve/release management state, :) : I  agree the formal I_T nexus
> loss event didn't exist in SAM-2, however the notion of a new I_T nexus state
> did co-exist with a session-oriented I_T nexus well before iSCSI - FCP process
> login (PRLO) comes to mind here. So I'd be curious about traditional FCP
> handling of reserve/release management state. And I'm also checking internally
> here at MS. From what I know now though, it does *appear* that reserve/release
> management state should not be cleared with I_T nexus loss.
>
> Having said all that, note that reserve/release management state is squarely
> in the SCSI territory. iSCSI spec has so far by design deferred to SCSI
> standards in specifying clearing effects on SCSI objects - as Appendix E.2
> makes it clear. What we do specify is the clearing effects of SCSI actions,
> e.g. LU reset, on iSCSI objects.
>
> Thanks.
>
> Mallikarjun
>
>
>
>
>

---

From: Ralph Weber <[email-masked]>
Date: Mon, 24 Sep 2012 12:28:32 -0500

Because there is no practical way to say anything about this subject
in the modern SCSI standards (i.e., RESERVE/RELEASE has been obsolete
for years), I support adding a few well-chosen words to the consolidated
iSCSI draft/RFC.

Thanks,

Ralph Weber

---

From: Roger Cummings <[email-masked]>
Date: Mon, 24 Sep 2012 13:56:34 -0400

David, Fred,

I'm not actually sure if this is relevant to the point under
discussion, but thought I'd raise it just in case.

While the Reserve/Release method is obsolete in SCSI-3 as you say, the
are actually some words (penned by yours truly) in SPC-4 section 5.13
called "Exceptions to SPC-2 RESERVE and RELEASE behavior" which
handles the case when Reserve/Release and persistent reservations are
mixed. This happens quite a bit in the field, and as an app vendor we
were getting tired of targets doing brain-dead things in that
circumstance like rejecting one command or other with a
hitherto-unseen ASC/ASCQ, and so we drafted the text that essentially
makes Reserve or Release a nop (and thus no reservation state is
created) when those commands are received from an initiator that has
access under a existing persistent reservation.

I wonder if it's worth a note in the iSCSI consolidated draft that
says that reservation state isn't created or cleared when an enabling
persistent reservation is in place? I think us app vendors would thank
you for it!

FWIW,

Roger Cummings

---

From: "Black, David" <[email-masked]>
Date: Mon, 24 Sep 2012 14:26:12 -0400

Roger,

IMHO, that makes sense to add, but at a somewhat more abstract level with
an important initial caveat:

- The Reserve/Release method is obsolete in the SCSI standards; Persistent
        Reservations should be used instead (lower case "should"). [SPC3]
- When Persistent Reservations are used, Reserve/Release may not behave
        as expected - see the discussion of "Exceptions to SPC-2 RESERVE and
        RELEASE behavior" in [SPC4] (informative reference, subclause number
        deliberately omitted as it may change).

I'd prefer to leave the explanation of when RESERVE and RELEASE are no-ops
to SPC-4 ... where it belongs.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Roger Cummings
> Sent: Monday, September 24, 2012 1:57 PM
> To: Black, David; Knight, Frederick
> Cc: Mallikarjun Chadalapaka; [email-masked]
> Subject: Re: [storm] RESERVE state - clearing effects
>
> David, Fred,
>
> >> "If the logical unit retains state information for the I_T nexus that is
> lost,
> >> its response to the subsequent I_T nexus
> that
> >> RFC3720 was written in between SAM-2 and SAM-3; some concepts were drawn
> from
> >> SAM-2, and some from SAM-3.
> >>

---

From: "Knight, Frederick" <[email-masked]>
Date: Mon, 24 Sep 2012 18:30:56 +0000

I would say this is already covered in SPC (which we reference), and we don't need to repeat it.  SCSI stuff in SCSI documents, and transport stuff in transport documents.

Maybe saying that "the interaction between RESERVE state and PERSISTENT RESERVE state is defined in SPC-x" would be OK.

But, since you're an application, what can you share about your recovery actions for 29/07?  Does an I_T Nexus loss cause you to fully reestablish state, or do you assume some of that state was retained?  Do you treat 29/07 any different from 29/00?

	Fred

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Roger Cummings
Sent: Monday, September 24, 2012 1:57 PM
To: Black, David; Knight, Frederick
Cc: Mallikarjun Chadalapaka; [email-masked]
Subject: Re: [storm] RESERVE state - clearing effects

David, Fred,

I'm not actually sure if this is relevant to the point under discussion, but thought I'd raise it just in case.

While the Reserve/Release method is obsolete in SCSI-3 as you say, the are actually some words (penned by yours truly) in SPC-4 section 5.13 called "Exceptions to SPC-2 RESERVE and RELEASE behavior" which handles the case when Reserve/Release and persistent reservations are mixed. This happens quite a bit in the field, and as an app vendor we were getting tired of targets doing brain-dead things in that circumstance like rejecting one command or other with a hitherto-unseen ASC/ASCQ, and so we drafted the text that essentially makes Reserve or Release a nop (and thus no reservation state is
created) when those commands are received from an initiator that has access under a existing persistent reservation.

I wonder if it's worth a note in the iSCSI consolidated draft that says that reservation state isn't created or cleared when an enabling persistent reservation is in place? I think us app vendors would thank you for it!

FWIW,

Roger Cummings

On Mon, Sep 24, 2012 at 12:42 PM, Black, David <[email-masked]> wrote:
>> is lost, its response to the subsequent I_T nexus re-establishment 
>> for the logical unit should include establishing a unit attention 
>> with an additional sense code set to I_T NEXUS LOSS OCCURRED.
>>
>> If the logical unit does not retain state information for the I_T 
>> nexus that is lost, it shall consider the subsequent I_T nexus 
>> re-establishment, if any, as the formation of a new I_T nexus for 
>> which there is no past history (e.g., establish a unit attention with 
>> an additional sense code set to POWER ON OCCURRED)."
>
> As to what needs to be stated in the iSCSI consolidated draft - the 
> SPC reference has been updated to SPC-3 in which the Reserve/Release 
>> is lost, its response to the subsequent I_T nexus re-establishment 
>> for the logical unit should include establishing a unit attention 
>> with an additional sense code set to I_T NEXUS LOSS OCCURRED.
>>
>> If the logical unit does not retain state information for the I_T 
>> nexus that is lost, it shall consider the subsequent I_T nexus 
>> re-establishment, if any, as the formation of a new I_T nexus for 
>> which there is no past history (e.g., establish a unit attention with 
>> an additional sense code set to POWER ON OCCURRED)."
>>
>> conclusion I could jump to based on #1 above, is that RESERVE state 
>> is also cleared (this action lines up more with LU RESET than CLEAR 
>> TASK SET); and based on the POWER ON CONDITION UA of 29/01 (or 29/00 
>> - both are power on
>> conditions) I could also assume that the RESERVE state is cleared; 
>> but based on the UA of 29/07 (retain state), I could jump to the 
>> conclusion that RESERVE state is retained.
>>
>> Based on what RFC3720 says about explicit state (e.g., mode pages), I 
>> would assume that hosts would believe that every command they 
>> explicitly send (like MODE SELECT or RESERVE) would have created 
>> explicit state and therefore the host would assume RESERVE state is 
>> retained.  It doesn't make sense to have mode page values retained, but have RESERVE state lost.
>>
>> Section 5.5.3.2 for foreground self-test Section 5.5.3.3 for 
>> background self- test Section 5.6.1 on persistent reservations 
>> Section 5.13 on timestamps ......
>>
>> I have just fixed it - and reviewed all other Appendix references 
>> elsewhere in the draft. However, I do not see a circular reference 
>> problem per se - Appendix E is correctly referencing Section 6.3.5 
>> for definition of terms, e.g. session continuation. And Section 
>> 6.3.5.1 is correctly deferring to Appendix E when discussing clearing 
>> effects. Let me know if I'm overlooking something here.
>>
>> Now, the reserve/release management state, :) : I  agree the formal 
>> I_T nexus loss event didn't exist in SAM-2, however the notion of a 
>> new I_T nexus state did co-exist with a session-oriented I_T nexus 
>> well before iSCSI - FCP process login (PRLO) comes to mind here. So 
>> I'd be curious about traditional FCP handling of reserve/release 
>> management state. And I'm also checking internally here at MS. From 
>> what I know now though, it does *appear* that reserve/release management state should not be cleared with I_T nexus loss.
>>
>> Having said all that, note that reserve/release management state is 
>> squarely in the SCSI territory. iSCSI spec has so far by design 
>> deferred to SCSI standards in specifying clearing effects on SCSI 
>> objects - as Appendix E.2 makes it clear. What we do specify is the 
>> clearing effects of SCSI actions, e.g. LU reset, on iSCSI objects.
>>
>> Thanks.
>>
>> Mallikarjun
>>
>>
>>
>>
>>
>> We have references to I_T NEXUS LOSS events in RFC3720.  The I_T 
>> Nexus loss concept does not exist in SAM-2.  SAM-2 does include 
>> references to RESERVE and RELEASE commands.  SAM-3 includes I_T Nexus 
>> loss, but it has no reference to RESERVE state.  There are allusions 
>> in RFC3720 about the intersection of RESERVE/RELEASE and I_T NEXUS 
>> LOSS, but there is nothing explicit.  RFC3720 references other 
>> documents (SAM2, SAM3, SBC, SPC3), none of which address the situation.
>>
>> Is RESERVE state - explicit state?  When an I_T nexus is 
>> reestablished, initiator-specific mode pages are preserved.  That 
>> implies that an I_T nexus loss should be treated in the way that 
>> SAM-2 describes via CLEAR TASK SET (where mode pages and RESERVE 
>> state are preserved).  If a session reestablishment caused saved mode 
>> pages to be restored rather than the established initiator-specific 
>> mode pages to be restored, then it would be like a LU reset (where 
>> the RESERVE state is lost).  Is RESERVE state explicit state that is to be preserved?
>>
>> I could not find any text in SPC3 that describes these clearing 
>> effects (I don't know if this SPC3 reference is incorrect, but at a 
>> minimum, it is ineffective).  The SAM2 text for LU resets and 
>> power-on resets is easily found, and clearly says that any RESERVE is 
>> released in those cases.  As an aside, the reference to section 
>> 6.3.5.1 is a circular reference).  The text in
>> E.2 goes on:
>>
>> So, it is clear what happens to PERSISTENT RESERVE state, but not so 
>> clear about RESERVE state.  It also states that all SCSI objects that 
>> are defined to not clear on the I_T nexus loss are retained.  But 
>> again, the only list I can find of what is cleared is here in this 
>> Appendix, since SAM-2, SAM-3, SAM-4 are all silent on the interaction between RESERVE and I_T Nexus loss.
>>
>> T10 has never had to deal with this because they removed RESERVE 
>> before they added I_T Nexus loss.  However, iSCSI did not do that; we 
>> discuss BOTH, but we fail to describe how they interact.  iSCSI also 
>> has the feature of reestablishment of a session that does not exist 
>> in SAM-2 (SAM-2 was still parallel SCSI focused, where connections 
>> didn't exist).  So again, if session reestablishment is examined in 
>> the light of parallel SCSI, the intent would be to retain the RESERVE state.  Here is another section on the topic:
>>
>> FWIW - the reference above is incorrect - there is no Appendix F in 
>> the consolidated draft - this should be Appendix E (and also, BTW - 
>> this is a circular reference).
>>
>> So anyway, it seems like the intent was that session reinstatement 
>> would preserve the RESERVE state across the I_T Nexus loss event.
>>
>> So aside from the incorrect reference (and circular nature of it), 
>> any thoughts on whether we should get explicit about this situation?  
>> Should we remain silent; should we tweak Appendix E along these lines 
>> (or other lines that others might suggest):
>>
>> 4.4.3.1. I_T Nexus State
>>
>> "[SAM2] and [SAM3] specifications note in their informative text that 
>> TPGT value should be non-zero, note that it is incorrect.  A zero 
>> value is allowed as a legal value for TPGT. This discrepancy 
>> currently stands corrected in [SAM4]."
>>
>> Additionally, SAM4 reference is also used in citing the new ASC/ASCQ 
>> codes that it defines, in Section 11.14.5 of the draft.
>>
>> The current draft references FC-FS in discussing the NAA format in a 
>> few places, so the current reference at the end is intentionally the 
>> same. Is the older reference a problem? If so, I suppose we can 
>> replace all references to FC-FS-4, but that seems to be still in 
>> early stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3?
>>
>> Thanks.
>>
>> Mallikarjun
>>
>>
>>
>>
>> In making up the lists of T10 and T11 documents that are currently 
>> referenced, some of those references don't look right to me, for example:
>>
>> As part of dealing with the comments from IETF Last Call, we'll need 
>> to go over all of the T10 and T11 references (and should check all 
>> the references) to make sure that they're up to date and necessary.
>>
>> Thanks,
>> --David
>> David L. Black, Distinguished Engineer EMC Corporation, 176 South 
>> St., Hopkinton, MA  01748

---

From: "Knight, Frederick" <[email-masked]>
Date: Mon, 24 Sep 2012 18:56:37 +0000

I agree that I_T Nexus loss might be an initiator failure - it might just as easily be a wire/Ethernet switch failure (between the initiator and target), and the initiator is perfectly fine - which, is the whole point of logging back in using the same ISID; and, the point of the failure case the specific paragraph in RFC3720 references - "a failure in the iSCSI layer".

RFC3720 classifies MODE SELECT commands (that set port specific mode parameters) as explicit.  There is also no way for a third party to impact those port specific mode parameters.  RFC3270 is specific in this case, that this state IS retained across the I_T Nexus loss (and SAM-2 is specific that CLEAR TASK SET also retains that state, and that LU RESET eliminates that state).  I assume a command from a host is something the host explicitly asked for, and not something that implicitly happens.

Should an initiator that does a relogin using the same ISID, just be able to continue where it left off, or must it assume state is lost and reestablish state before it can resume?  If it is to assume state is lost, then why does RFC3710 require that I_T specific mode parameters be retained?  If the host is to assume state is lost, then what is the difference between I_T Nexus loss and Power On?  Is the host to assume only some state is retained?  How does the host know which state is retained and which is lost?

>       failures in the iSCSI layer (e.g., session failures).

SAM-2 was written in the context of parallel SCSI, so I would think that SAM-2 (while silent on the subject), would have expected a parallel SCSI RESERVE command to retain state - if we could have ever figured out what an I_T Nexus loss might have look like in the parallel SCSI context.

I suppose that's enough questions for now.

	Fred

-----Original Message-----
From: Black, David [mailto:[email-masked]] 
Sent: Monday, September 24, 2012 12:42 PM
To: Knight, Frederick; Mallikarjun Chadalapaka; [email-masked]
Subject: RESERVE state - clearing effects

Fred,

This is interesting ...

> is lost, its response to the subsequent I_T nexus re-establishment for 
> the logical unit should include establishing a unit attention with an 
> additional sense code set to I_T NEXUS LOSS OCCURRED.
>

That "state information" has to identify the endpoints of the I_T Nexus
- for iSCSI that means at least the ISID and TPGT.  If that's gone, I think the RESERVE state is gone as well, in contrast to PERSISTENT RESERVE state.  On RESERVE state, SPC-2 has this to say in 5.5.2 "The Reserve/Release management method":

I think it's fair to regard I_T Nexus loss as an initiator failure.  I also don't see any good way to do a 3rd-party release on a RESERVE reservation that persists after I_T Nexus loss.  I also believe that initiator software that uses Reserve/Release does not expect reservations to persist across I_T Nexus loss.

As to what needs to be stated in the iSCSI consolidated draft - the SPC reference has been updated to SPC-3 in which the Reserve/Release management method is obsolete, so there isn't any absolute requirement to cover that.  OTOH, it might be reasonable to add a sentence to say that Reserve/Release state is implicit (e.g., as there's no way for an initiator to figure out which I_T Nexus holds a reservation), and that treating Reserve/Release state as indefinitely persistent after I_T Nexus Loss is a bad idea ("should not", but lower case) as it may require the initiator to reset the target device in order to clear out that state (issuing RELEASE commands isn't enough).  This sort of text would require a reference to SPC-2.

Thanks,
--David

> is lost, its response to the subsequent I_T nexus re-establishment for 
> the logical unit should include establishing a unit attention with an 
> additional sense code set to I_T NEXUS LOSS OCCURRED.
>
> conclusion I could jump to based on #1 above, is that RESERVE state is 
> also cleared (this action lines up more with LU RESET than CLEAR TASK 
> SET); and based on the POWER ON CONDITION UA of 29/01 (or 29/00 - both 
> are power on
> conditions) I could also assume that the RESERVE state is cleared; but 
> based on the UA of 29/07 (retain state), I could jump to the 
> background self- test Section 5.6.1 on persistent reservations Section 
> 5.13 on timestamps ......
>
> problem per se - Appendix E is correctly referencing Section 6.3.5 for 
> definition of terms, e.g. session continuation. And Section 6.3.5.1 is 
> correctly deferring to Appendix E when discussing clearing effects. 
> Let me know if I'm overlooking something here.
>
> We have references to I_T NEXUS LOSS events in RFC3720.  The I_T Nexus 
> loss concept does not exist in SAM-2.  SAM-2 does include references 
> to RESERVE and RELEASE commands.  SAM-3 includes I_T Nexus loss, but 
> it has no reference to RESERVE state.  There are allusions in RFC3720 
> about the intersection of RESERVE/RELEASE and I_T NEXUS LOSS, but 
> there is nothing explicit.  RFC3720 references other documents (SAM2, 
> SAM3, SBC, SPC3), none of which address the situation.
>
> implies that an I_T nexus loss should be treated in the way that SAM-2 
> describes via CLEAR TASK SET (where mode pages and RESERVE state are 
> preserved).  If a session reestablishment caused saved mode pages to 
> be restored rather than the established initiator-specific mode pages 
> to be restored, then it would be like a LU reset (where the RESERVE 
> state is lost).  Is RESERVE state explicit state that is to be preserved?
>
> minimum, it is ineffective).  The SAM2 text for LU resets and power-on 
> resets is easily found, and clearly says that any RESERVE is released 
> in those cases.  As an aside, the reference to section 6.3.5.1 is a 
> circular reference).  The text in
> E.2 goes on:
>
> has the feature of reestablishment of a session that does not exist in 
> SAM-2 (SAM-2 was still parallel SCSI focused, where connections didn't 
> exist).  So again, if session reestablishment is examined in the light 
> of parallel SCSI, the intent would be to retain the RESERVE state.  Here is another section on the topic:
>
> thoughts on whether we should get explicit about this situation?  
> value is allowed as a legal value for TPGT. This discrepancy currently 
> stands corrected in [SAM4]."
>
> replace all references to FC-FS-4, but that seems to be still in early 
> stages(0.1 spec posted on April 2012), so perhaps we should use FC-FS3?
>
> Thanks.
>
> Mallikarjun
>
>
>
>
> to go over all of the T10 and T11 references (and should check all the 
> references) to make sure that they're up to date and necessary.
>
> Thanks,
> --David
> David L. Black, Distinguished Engineer EMC Corporation, 176 South St., 
> Hopkinton, MA  01748

---

From: "Black, David" <[email-masked]>
Date: Mon, 24 Sep 2012 17:57:21 -0400

Fred,

> I agree that I_T Nexus loss might be an initiator failure - it might just as
> easily be a wire/Ethernet switch failure (between the initiator and target),
> and the initiator is perfectly fine - which, is the whole point of logging
> back in using the same ISID; and, the point of the failure case the specific
> paragraph in RFC3720 references - "a failure in the iSCSI layer".

So far so good - the approach I'm favoring allows for both scenarios.  IMHO,
it'd be ok for the target to retain RESERVE state for a while, but retaining
it indefinitely in a situation where the initiator cannot log back in with
the same ISID requires a "hard reset" to recover the target (as stated in SPC-2).

[... snip ...]

> Should an initiator that does a relogin using the same ISID, just be able to
> continue where it left off, or must it assume state is lost and reestablish
> state before it can resume?

Aha, I think I see where we're talking past each other ;-).  It looks like
you're concerned with the initiator successfully logging back in and being
able to pick up where it left off, whereas I'm concerned with what happens
when the initiator can't or doesn't log back in and specifically the
side effects on other I_T Nexuses in that scenario.

For an initiator that successfully logs back in with the same ISID, the
question of whether it can continue is answered in the text that you quoted
from SAM on which UA condition is established for such a re-login.  In that
situation, I'm fine with linking survival of the RESERVE state with survival
of I_T Nexus-specific mode state (and both would be indicated by which UA
occurs).

My concern is what happens when the I_T Nexus is not re-established - if the
RESERVATION CONFLICT side effects on other I_T Nexuses persist indefinitely,
the only means of recovery is a "hard reset" via some other I_T Nexus, and
I really don't want to add a new requirement for that.

Thanks,
--David

> Sent: Monday, September 24, 2012 2:57 PM
> To: Black, David; Mallikarjun Chadalapaka; [email-masked]
> Subject: RE: RESERVE state - clearing effects
>
> I agree that I_T Nexus loss might be an initiator failure - it might just as
> easily be a wire/Ethernet switch failure (between the initiator and target),
> and the initiator is perfectly fine - which, is the whole point of logging
> back in using the same ISID; and, the point of the failure case the specific
> paragraph in RFC3720 references - "a failure in the iSCSI layer".
>
> RFC3720 classifies MODE SELECT commands (that set port specific mode
> parameters) as explicit.  There is also no way for a third party to impact
> those port specific mode parameters.  RFC3270 is specific in this case, that
> this state IS retained across the I_T Nexus loss (and SAM-2 is specific that
> CLEAR TASK SET also retains that state, and that LU RESET eliminates that
> state).  I assume a command from a host is something the host explicitly asked
> for, and not something that implicitly happens.
>
> Should an initiator that does a relogin using the same ISID, just be able to
> continue where it left off, or must it assume state is lost and reestablish
> state before it can resume?  If it is to assume state is lost, then why does
> RFC3710 require that I_T specific mode parameters be retained?  If the host is
> to assume state is lost, then what is the difference between I_T Nexus loss
> and Power On?  Is the host to assume only some state is retained?  How does
> the host know which state is retained and which is lost?
>
> SAM-2 was written in the context of parallel SCSI, so I would think that SAM-2
> (while silent on the subject), would have expected a parallel SCSI RESERVE
> command to retain state - if we could have ever figured out what an I_T Nexus
> loss might have look like in the parallel SCSI context.
>
> - for iSCSI that means at least the ISID and TPGT.  If that's gone, I think
> the RESERVE state is gone as well, in contrast to PERSISTENT RESERVE state.
> On RESERVE state, SPC-2 has this to say in 5.5.2 "The Reserve/Release
> management method":
>
> I think it's fair to regard I_T Nexus loss as an initiator failure.  I also
> don't see any good way to do a 3rd-party release on a RESERVE reservation that
> persists after I_T Nexus loss.  I also believe that initiator software that
> uses Reserve/Release does not expect reservations to persist across I_T Nexus
> loss.
>
> reference has been updated to SPC-3 in which the Reserve/Release management
> method is obsolete, so there isn't any absolute requirement to cover that.
> OTOH, it might be reasonable to add a sentence to say that Reserve/Release
> state is implicit (e.g., as there's no way for an initiator to figure out
> which I_T Nexus holds a reservation), and that treating Reserve/Release state
> as indefinitely persistent after I_T Nexus Loss is a bad idea ("should not",
> but lower case) as it may require the initiator to reset the target device in
> order to clear out that state (issuing RELEASE commands isn't enough).  This
> sort of text would require a reference to SPC-2.
>
> Thanks,
> --David
>
> > what I know now though, it does *appear* that reserve/release management
> state should not be cleared with I_T nexus loss.
> >

---

From: Julian Satran <[email-masked]>
Date: Tue, 25 Sep 2012 06:55:03 +0200

Did you consideir clusters or migrating vms? With those usually one expects targets to retain some state accross IT nexus loss. To avoid third party reservation cleanup SCSI (t10) may want to introduce a timeout mechanism associated with nexus loss (state is preserved for a while after nexus loss and the timeout values can be set).

julo

Sent from my iPad

---

From: Ralph Weber <[email-masked]>
Date: Tue, 25 Sep 2012 07:32:30 -0500

Persistent Reservations were invented because Reserve/Release failed
to provide the quality of reservations services demanded by clusters.
Clusters that depend on Reserve/Release reservations get exactly
what they deserve.

As for Open VMS, my recollection is that it had only one minor
dependence on reservations. The most I remember needing from
reservations was a way to ensure that disks stopped processing
I/Os from an expelled cluster member. As long as all commands are
aborted at the same time a reservation is released, the VMS I
remember will not suffer any adverse effects.

In any case, VMS has had two decades to migrate to Persistent
Reservations for whatever they needed or developed a dependence
upon. Even for OS engineers, that should be long enough for
them to have matched their software to the available services.

All the best,

.Ralph

P.S. David and Fred, I am not venturing into the topics you are
discussing because I see nothing upon which I can provide knowledgeable
additional comment. Whatever agreement you all reach will be okay ...
as far as I can tell.

---

From: Julian Satran <[email-masked]>
Date: Tue, 25 Sep 2012 14:50:58 +0200

Ralph,

I understand. I missed the fact that the discussion was exclusively abour reserve/release. And i was referring to VMs (even I am too young to remeber VMS - that was during the trojan war wasn't it?).

Julo

Sent from my iPad

---

From: "Black, David" <[email-masked]>
Date: Tue, 25 Sep 2012 09:15:10 -0400

To put the final "nail in the coffin" of clusters and preservation
of reserve/release reservations, I know of one reasonably widely
deployed cluster that did use reserve/release reservations ...
... and that cluster used periodic loss of a reservation (via
TARGET RESET) as a heartbeat mechanism ... making this discussion
about reservation preservation basically irrelevant ... at least to
that cluster.

That sort of (ab)use of TARGET RESET (or the current LOGICAL UNIT
RESET) is among the reasons why persistent reservations are the
strongly preferred approach.  VM migration varies by hypervisor -
persistent reservations would be used if a reservation is needed
on the VM's storage but not all hypervisors are implemented that
way.  I know of an implementation that uses reserve/release to
create short critical sections (for which COMPARE AND WRITE is
the preferred replacement), but it's safe to abort the critical
sections in progress on I_T Nexus loss.

I will start a new thread with some proposed text.  A crucial
consideration will be that we cannot introduce new requirements
into the consolidated iSCSI draft that existing implementations
might not satisfy.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Julian Satran
> Sent: Tuesday, September 25, 2012 8:51 AM
> To: Ralph Weber
> Cc: [email-masked]
> I understand. I missed the fact that the discussion was exclusively abour
> reserve/release. And i was referring to VMs (even I am too young to remeber
> VMS - that was during the trojan war wasn't it?).
>
> Julo
>
> Sent from my iPad
>
> >> Did you consideir clusters or migrating vms? With those usually one expects
> targets to retain some state accross IT nexus loss. To avoid third party
> reservation cleanup SCSI (t10) may want to introduce a timeout mechanism
> associated with nexus loss (state is preserved for a while after nexus loss
> and the timeout values can be set).
> >>
> >> julo
> >>
> >> Sent from my iPad
> >>
> also
> based
> would
> elsewhere in
> squarely
> loss
> clear
> TPGT
> codes
> referenced,

---

From: Ralph Weber <[email-masked]>
Date: Tue, 25 Sep 2012 09:13:14 -0500

Julian,

Do you know what really grates on the nerves? ...

You know you are old when you visit a computer museum and discover
that the last half of its contents represent the story of your life.

That is what happened to me about ten years ago.

All the best,

.Ralph

On 9/25/2012 7:50 AM, Julian Satran wrote:
>>>>> Mallikarjun Chadalapaka
>>>>> [email-masked]

---

From: Milton Scritsmier <[email-masked]>
Date: Mon, 24 Sep 2012 09:54:49 -0600

> We have references to I_T NEXUS LOSS events in RFC3720.  The I_T Nexus loss concept does not exist in SAM-2.  SAM-2 does include references to RESERVE and RELEASE commands.  SAM-3 includes I_T Nexu…

The concepts of I_T NEXUS and RESERVE/RELEASE go all the way back to
SCSI-2 in the mid-80s, on which I did a little work (see the spec at
http://www.mit.edu/afs/sipb.mit.edu/contrib/doc/specs/protocol/scsi-2/s2-r10l.txt
). In SCSI-2, the I_T NEXUS concept was created to deal with
disconnect/reconnect on the parallel SCSI bus and the idea that a
command could be active on both the host and target even though the bus
was disconnected or the target was talking to another initiator. The
RESERVE condition was meant to survive anything except a bus reset or a
device power cycle (back then it was not assumed that devices had flash
memory or any means of storing RESERVE/RELEASE status between power
cycles or after a bus reset).

So for example, if the initiator aborted a command, or persistent parity
errors on the bus prevented communications, any RESERVE condition was
still in effect. If a target was reset or power cycled, it would notify
the host via CHECK CONDITION of the reset on the next command, which
would implicitly tell the host that any RESERVE condition was no longer
in effect. Section 9.2.12.1 of the SCSI-2 spec gives the conditions
which clear a RESERVE condition, and it's clear that these are rather
"catastrophic" conditions. Thus it was always the intent that the
RESERVE condition was meant to survive an I_T NEXUS loss, although that
concept is not well-defined in SCSI-2.

I don't know if these concepts were carried through beyond parallel
buses in SCSI-3, but that was the original intent.

---

From: Ralph Weber <[email-masked]>
Date: Tue, 25 Sep 2012 10:54:44 -0500

All of this is more-or-less true, but the problem at hand is
I_T NEXUS LOSS ... a concept created long after SCSI-2 had
graduated to SCSI-3. The first serial protocol with any
widespread usage was Fibre Channel Protocol (FCP), and that
protocol included a Login/Logout concept, which has a much
longer lifespan that Disconnect/Reconnect.

One frequent mapping of I_T NEXUS LOSS is to an FCP Logout
(either intentional or forced from the outside). This mapping
is common, but far from universal.

Thus, the question becomes where to install I_T NEXUS LOSS (a
concept never envisioned by SCSI-2 because it is nonsense on
the parallel bus) in the iSCSI pantheon.

The global-effects ladder usually looks like this, in order
of increasingly bigger hammers.

...
{old} CLEAR TASK SET
[new] I_T NEXUS LOSS
{old} LOGICAL UNIT RESET
...

CLEAR TASK SET preserves Reserve/Release reservations.
LOGICAL UNIT RESET clears them.

David and Fred are discussing which way I_T NEXUS LOSS should
swing on this pendulum.

As noted earlier today, where these two gentlemen are is quite
clear. To one who lacks sufficient real-world knowledge of the
applicable use cases, the only option is a coin toss.

Fortunately, Fred and David are better equipped than this author.

All the best,

.Ralph

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Tue, 25 Sep 2012 17:34:28 +0000

David,

Thanks for the suggestions. I am largely in agreement.

A couple of questions:

>2) [SAM2] and [SAM3] ought to be normative references

Don't know the related T10 history here, but I no longer see SAM-3 spec on the T10 web site. Any insights?

>[SAM-4] ought to be an informative reference

Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM-4? (I can't tell because I don't see SAM-3... If the notification were introduced in SAM-4, that could make SAM-4 a normative reference.

SAM-2 does not formally discuss I_T nexus loss notification, other than an oblique reference to the related check condition in clause 5.3.2, status precedence discussion.

Thanks.

Mallikarjun

-----Original Message-----
From: Black, David [mailto:[email-masked]] 
Sent: Monday, September 24, 2012 8:37 AM
To: Knight, Frederick; Mallikarjun Chadalapaka; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

Fred,

In light of that, and a few other things (e.g., a strong preference that the iSCSI consolidated draft not reference work in progress), my recommendations on the T10 and T11 references are:

--- T10 (SCSI) ---

1) [SBC2] and [SPC3] are the current published standards; those should be the correct references for the iSCSI consolidated draft.  [SPC3] is clearly normative, but I think [SBC2] is (and ought to be) informative - [SBC2] is only cited once in what looks like an informative fashion in Appendix E.2:

Moreover, there is no requirement to implement the SBC command set for all iSCSI implementations, whereas the SPC command set is a requirement for obvious reasons (P = Primary).

> codes that it defines, in Section 11.14.5 of the draft.

That new ASC/Q code's name does not occur in the body of SAM-4, so I would remove that second citation of SAM-4.

3) The [SAS] reference needs to be expanded ;-).  The current informative [SAS] reference needs to be replaced by references to SAS-2.1, SPL and SPL Amendment 1, see:

4) OTOH, I recommend removing the [SRP] reference and related material, 'namely the definition of "SCSI RDMA Protocol" and the alternative expansion of the SRP acronym.  The SRP-2 standards project to update SRP was abandoned by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us with a vintage-2002 SRP document.

--- T11 (Fibre Channel) ---

The FC-FS reference ought to be replaced by a reference to the current published FS-FS-3 standard.

Thanks,
--David

---

From: "Black, David" <[email-masked]>
Date: Tue, 25 Sep 2012 14:28:49 -0400

> Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM-4?
> (I can't tell because I don't see SAM-3... If the notification were introduced
> in SAM-4, that could make SAM-4 a normative reference.

It's a defined term in SAM-3 (3.1.44) and specified in clause 6 of SAM-3.

Thanks,
--David

> Sent: Tuesday, September 25, 2012 1:34 PM
> To: Black, David; Knight, Frederick; [email-masked]
> Was the concept of I_T nexus loss notification introduced in SAM-3 or SAM-4?
> (I can't tell because I don't see SAM-3... If the notification were introduced
> in SAM-4, that could make SAM-4 a normative reference.
> 
> SAM-2 does not formally discuss I_T nexus loss notification, other than an
> oblique reference to the related check condition in clause 5.3.2, status
> precedence discussion.
> 
> Thanks.
> 
> Mallikarjun
> 
> 
> 
> 
> In light of that, and a few other things (e.g., a strong preference that the
> iSCSI consolidated draft not reference work in progress), my recommendations
> on the T10 and T11 references are:
> 
> --- T10 (SCSI) ---
> 
> [SAS] reference needs to be replaced by references to SAS-2.1, SPL and SPL
> Amendment 1, see:
> 
> by T10 in 2005 due to lack of interest (see T10/05-097r0), leaving us with a
> vintage-2002 SRP document.
>

---

From: "Knight, Frederick" <[email-masked]>
Date: Tue, 25 Sep 2012 19:18:46 +0000

FK> INLINE

-----Original Message-----
From: Mallikarjun Chadalapaka [mailto:[email-masked]] 
Sent: Tuesday, September 25, 2012 1:34 PM
To: Black, David; Knight, Frederick; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

David,

FK> Look down under the withdrawn standards.  BTW - should we really normatively referencing something that INCITS/T10 withdrew?

FK> It was introduced in SAM-3.

	Fred

Thanks.

Mallikarjun

-----Original Message-----
From: Black, David [mailto:[email-masked]]
Sent: Monday, September 24, 2012 8:37 AM
To: Knight, Frederick; Mallikarjun Chadalapaka; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

Fred,

--- T11 (Fibre Channel) ---

Thanks,
--David

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Tue, 25 Sep 2012 19:25:03 +0000

Yes, thanks Fred for pointing this out, I too had realized that SAM-3 was withdrawn after I sent my previous note.

IMO, it does not make sense to normatively reference a withdrawn T10 standard in a new RFC. So my default plan now is to replace all SAM-3 references to SAM-4, and leave the latter as a normative reference. David?

Thanks.

Mallikarjun

-----Original Message-----
From: Knight, Frederick [mailto:[email-masked]] 
Sent: Tuesday, September 25, 2012 12:19 PM
To: Mallikarjun Chadalapaka; Black, David; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

FK> INLINE

-----Original Message-----
From: Mallikarjun Chadalapaka [mailto:[email-masked]]
Sent: Tuesday, September 25, 2012 1:34 PM
To: Black, David; Knight, Frederick; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

David,

	Fred

Thanks.

Mallikarjun

-----Original Message-----
From: Black, David [mailto:[email-masked]]
Sent: Monday, September 24, 2012 8:37 AM
To: Knight, Frederick; Mallikarjun Chadalapaka; [email-masked]
Subject: RE: iSCSI drafts - T10 and T11 references

Fred,

--- T11 (Fibre Channel) ---

Thanks,
--David

---

From: "Black, David" <[email-masked]>
Date: Tue, 25 Sep 2012 15:36:52 -0400

Yes, that's the right answer, as I had also missed T10's withdrawal of SAM-3.

Thanks,
--David

> Sent: Tuesday, September 25, 2012 3:25 PM
> To: Knight, Frederick; Black, David; [email-masked]
> IMO, it does not make sense to normatively reference a withdrawn T10 standard
> in a new RFC. So my default plan now is to replace all SAM-3 references to
> SAM-4, and leave the latter as a normative reference. David?
> 
> Thanks.
> 
> Mallikarjun
> 
> 
> FK> Look down under the withdrawn standards.  BTW - should we really
> normatively referencing something that INCITS/T10 withdrew?
>

---

From: "brown, david (eng)" <[email-masked]>
Date: Tue, 25 Sep 2012 11:34:16 -0400

An I_T nexus in SCSI-2 is different from an I_T nexus in SCSI-3.  In parallel SCSI, as you noted, an I_T nexus is created when a command is received and ends when that command completes.
 
In SCSI-3, the nexus created by an inbound command is called an I_T_L nexus, not a I_T nexus, since each command must be addressed to a specific LUN.  In SCSI-3, an I_T nexus is created by a port login.  That nexus persists even in the absence of commands.

In short, a SCSI-2 RESERVE command creates a condition that outlives a SCSI-2 I_T nexus. (Since the I_T nexus ends when the target sends status for the RESERVE command, the reservation had better survive past the end of that nexus, or else it would be pretty useless.)  The question before us is whether the reservation should survive after the end of a SCSI-3 I_T nexus.  Since SCSI-2 had nothing resembling a login, there is plenty of room to debate what happens to a SCSI-2 reservation in a SCSI-3 situation.

The closest analogy to a SCSI-3 I_T NEXUS LOSS in SCSI-2 might be a reselection timeout.  When that happens, the target discovers that the initiator is no longer responding.  The target can either retry later or reset the bus.  If he chooses the latter, the reservation ends.

DJ Brown