Subject: [storm] SPC-2 reserve/release text - version 6

From: "Black, David" <[email-masked]>
Date: Wed, 26 Sep 2012 18:25:14 -0400

Here's where I think we've arrived (I've made a few minor editorial
tweaks to what's been discussed).  Despite all the detailed discussion,
I think this version 6 comes close to Julian's suggestion that:

> i would mention only that the state that must be maintained while
> the session is maintained includes the reserve/release and leave it to that.

Although we can't actually say "must" as that would be a new requirement,
and implementations differ in what they do about reserve/release
reservation state.  The current language characterizes retention of
that state as the "preferred implementation approach".

Beyond that, I think the two additional caveats about reserve/release
reservations (possible need for a reset, potentially unexpected
interaction with persistent reservations, see below) are useful to include.

-----------------------------------------

[1] For clarity, the following change should be made in
4.4.3.1:

OLD
  That is, it should perform session recovery as
  described in Chapter 6.
NEW
  That is, it should reinstate the session via iSCSI session
  reinstatement (Section 6.3.5) or continue via session
  continuation (Section 6.3.6).
END

[2] Add the following sentence to the end of 4.4.3.1:

     The specific Unit Attention condition that results from session
     reestablishment indicates whether nexus state was preserved,
     see Section 6.3.4 of [SAM4].

[3] NEW TEXT:

4.4.3.2. Reservations

  There are two reservation management methods defined in the SCSI
  standards, reserve/release reservations, based on the RESERVE and
  RELEASE commands [SPC2], and persistent reservations, based on the
  PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3].
  Reserve/release reservations are obsolete [SPC3] and SHOULD NOT be
  used; persistent reservations SHOULD be used instead.

  State for persistent reservations is required to persist
  through changes and failures at the iSCSI layer that result in
  I_T Nexus failures, see [SPC3] for details and specific requirements.

  In contrast, [SPC2] does not specify detailed persistence
  requirements for reserve/release reservation state after an I_T
  Nexus failure.  Nonetheless, when reserve/release reservations are
  supported by an iSCSI target, the preferred implementation approach
  is to preserve reserve/release reservation state for iSCSI session
  reinstatement (see Section 6.3.5) or session continuation
  (see Section 6.3.6).

  Two additional caveats apply to reserve/release reservations:

  - When connection failure causes the iSCSI session to fail, and
      the session is not reinstated or continued, target retention
      of that session's reserve/release reservation state may
      require an initiator to issue a reset (e.g., LOGICAL UNIT RESET,
      see section 11.5) in order to remove that reservation state.

  - Reserve/release reservations may not behave as expected when
      persistent reservations are also used on the same logical unit;
      see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE
      behavior" in [SPC4].

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Julian Satran <[email-masked]>
Date: Thu, 27 Sep 2012 04:25:22 +0200

David,

The text in 4.4.3 states the obvious (there is no other speced method for continuation or reinstatement). Moreover if the the session is reinstated after time2wait you create a need for a UA (or at least suggest one) and that was not there. Just stating that creation/continuation of a session done according 6.x includes preservation of reserve/release state -that was already there implied by conformance to spc2 and the relation between it nexus and session.

The new text you suggest to 4.4 is merely a best practices recomendations and will need additions once the new "locking" through compare-and-swap get in widespread use.
If instead we refrain from stating anything specific and state that target should maintain whatever state is required by SPC while a session is continued and may discard state when a session is terminated and may indicate that it discarded state through a ua you are on safer ground and do not create new requirement.

Sent from my iPad

On 27 בספט 2012, at 00:25, "Black, David" <[email-masked]> wrote:

> _______________________________________________
> storm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/storm

---

From: "Black, David" <[email-masked]>
Date: Thu, 27 Sep 2012 10:46:45 -0400

Julian,

> The text in 4.4.3 states the obvious (there is no other speced method for
> continuation or reinstatement). Moreover if the session is reinstated
> after time2wait you create a need for a UA (or at least suggest one) and that
> was not there.

The UAs are courtesy of SAM-3 (and SAM-4).  To encompass systems that may
not deliver these UAs and situations in which they don't occur (e.g., the
target doesn't recognize what happened as nexus reestablishment,
independent of what the initiator thinks it's doing) I think the text
should be rephrased as:

     If a Unit Attention condition results from session reestablishment,
     the specific Unit Attention condition indicates whether nexus state
     was preserved, see Section 6.3.4 of [SAM4].

> Just stating that creation/continuation of a session done
> according 6.x includes preservation of reserve/release state -that was already
> there implied by conformance to spc2 and the relation between it nexus and
> session.

Unfortunately, SPC-2 is sufficiently imprecise on this subject that such an
implication is at best a "preferred implementation approach".  It's not a
requirement that can be safely stated in any stronger fashion without the
risk of rendering existing implementations non-compliant.

> The new text you suggest to 4.4 is merely a best practices recommendations and
> will need additions once the new "locking" through compare-and-swap get in
> widespread use.

It's definitely best practices recommendations - no argument there.

For context, what Julian is referring to is use of the COMPARE AND WRITE command
that performs an atomic operation at the target instead of using RESERVE and RELEASE
to create a critical section at the initiator based on read and write commands.

COMPARE AND WRITE is in use - a sentence mentioning that command could be added,
although this best practices text is aimed primarily at uses of reserve/release
for long-term reservations.  I think there's an implicit "when a reservation is
the desired behavior" implication to the "persistent reservations SHOULD be used
instead" text that I don't think needs to be made explicit.  OTOH, that "SHOULD"
could be changed to a "should", or it might be even better to rewrite that text
as: 

	persistent reservations are suggested as an alternative, see Annex B
	of [SPC4].

In contrast, "SHOULD NOT be used" is clearly justified for reserve/release
reservations by [SPC3].

Thanks,
--David

> -----Original Message-----
> From: Julian Satran [mailto:[email-masked]]
> Sent: Wednesday, September 26, 2012 10:25 PM
> To: Black, David
> Cc: [email-masked]
> Subject: Re: [storm] SPC-2 reserve/release text - version 6
> 
> David,
> 
> The text in 4.4.3 states the obvious (there is no other speced method for
> continuation or reinstatement). Moreover if the the session is reinstated
> after time2wait you create a need for a UA (or at least suggest one) and that
> was not there. Just stating that creation/continuation of a session done
> according 6.x includes preservation of reserve/release state -that was already
> there implied by conformance to spc2 and the relation between it nexus and
> session.
> 
> The new text you suggest to 4.4 is merely a best practices recomendations and
> will need additions once the new "locking" through compare-and-swap get in
> widespread use.
> If instead we refrain from stating anything specific and state that target
> should maintain whatever state is required by SPC while a session is continued
> and may discard state when a session is terminated and may indicate that it
> discarded state through a ua you are on safer ground and do not create new
> requirement.
> 
> Sent from my iPad
>

---

From: "Knight, Frederick" <[email-masked]>
Date: Thu, 27 Sep 2012 15:11:00 +0000

I afraid I have to contest a little on this - if I understand this discussion correctly.

If a SCSI device loses any state and does not raise a UNIT ATTENTION, then it is BROKEN.  It is not a matter of anything in iSCSI (3720); I think SAM and SPC are pretty clear on this.  If there is no UNIT ATTENTION, then state SHALL be preserved; otherwise, all sorts of stuff will never work (mode page settings, RESERVE commands, all sorts of stuff).

Is someone really suggesting that some devices will drop state and NOT tell the host, and somehow that it is OK to do that?

The suggest rewording says this is possible. 

	Fred

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Black, David
Sent: Thursday, September 27, 2012 10:47 AM
To: Julian Satran
Cc: [email-masked]
Subject: Re: [storm] SPC-2 reserve/release text - version 6

Julian,

> The text in 4.4.3 states the obvious (there is no other speced method 
> for continuation or reinstatement). Moreover if the session is 
> reinstated after time2wait you create a need for a UA (or at least 
> suggest one) and that was not there.

The UAs are courtesy of SAM-3 (and SAM-4).  To encompass systems that may not deliver these UAs and situations in which they don't occur (e.g., the target doesn't recognize what happened as nexus reestablishment, independent of what the initiator thinks it's doing) I think the text should be rephrased as:

> Just stating that creation/continuation of a session done according 
> 6.x includes preservation of reserve/release state -that was already 
> there implied by conformance to spc2 and the relation between it nexus 
> and session.

Unfortunately, SPC-2 is sufficiently imprecise on this subject that such an implication is at best a "preferred implementation approach".  It's not a requirement that can be safely stated in any stronger fashion without the risk of rendering existing implementations non-compliant.

> The new text you suggest to 4.4 is merely a best practices 
> recommendations and will need additions once the new "locking" through 
> compare-and-swap get in widespread use.

For context, what Julian is referring to is use of the COMPARE AND WRITE command that performs an atomic operation at the target instead of using RESERVE and RELEASE to create a critical section at the initiator based on read and write commands.

COMPARE AND WRITE is in use - a sentence mentioning that command could be added, although this best practices text is aimed primarily at uses of reserve/release for long-term reservations.  I think there's an implicit "when a reservation is the desired behavior" implication to the "persistent reservations SHOULD be used instead" text that I don't think needs to be made explicit.  OTOH, that "SHOULD"
could be changed to a "should", or it might be even better to rewrite that text
as: 

In contrast, "SHOULD NOT be used" is clearly justified for reserve/release reservations by [SPC3].

Thanks,
--David

> The text in 4.4.3 states the obvious (there is no other speced method 
> for continuation or reinstatement). Moreover if the the session is 
> reinstated after time2wait you create a need for a UA (or at least 
> suggest one) and that was not there. Just stating that 
> creation/continuation of a session done according 6.x includes 
> preservation of reserve/release state -that was already there implied 
> by conformance to spc2 and the relation between it nexus and session.
> 
> The new text you suggest to 4.4 is merely a best practices 
> recomendations and will need additions once the new "locking" through 
> compare-and-swap get in widespread use.
> If instead we refrain from stating anything specific and state that 
> target should maintain whatever state is required by SPC while a 
> session is continued and may discard state when a session is 
> terminated and may indicate that it discarded state through a ua you 
> are on safer ground and do not create new requirement.
> 
> Sent from my iPad
> 
> > tweaks to what's been discussed).  Despite all the detailed 
> > discussion, I think this version 6 comes close to Julian's suggestion that:
> >
> >> the session is maintained includes the reserve/release and leave it 
> >> to
> that.
> >
> > Although we can't actually say "must" as that would be a new 
> > requirement, and implementations differ in what they do about 
> > reserve/release reservation state.  The current language 
> > characterizes retention of that state as the "preferred implementation approach".
> >
> > Beyond that, I think the two additional caveats about 
> > reserve/release reservations (possible need for a reset, potentially 
> > unexpected interaction with persistent reservations, see below) are useful to include.
> >
> >  That is, it should perform session recovery as  described in 
> > Chapter 6.
> > NEW
> > reinstatement (Section 6.3.5) or continue via session  continuation 
> > (Section 6.3.6).
> > END
> >
> >  State for persistent reservations is required to persist  through 
> > changes and failures at the iSCSI layer that result in  I_T Nexus 
> > failures, see [SPC3] for details and specific requirements.
> >
> > reinstatement (see Section 6.3.5) or session continuation  (see 
> > Section 6.3.6).
> >
> > David L. Black, Distinguished Engineer EMC Corporation, 176 South 
> > St., Hopkinton, MA  01748

---

From: "Black, David" <[email-masked]>
Date: Thu, 27 Sep 2012 11:55:40 -0400

Fred,

The problem is not that the device does not raise a UA, it's that the UA
may never be delivered.

A summary of how this can happen is that the new connection with the same
ISID may not be recognized as reinstating or continuing the session at the
target if the timeouts have expired - the UA gets established, but nobody
shows up to pick it up in time, so the UA is discarded along with the rest
of the session state and the target fails to recognize the initiator's
subsequent ISID reuse even though the initiator believes that it's re-
establishing the session.

This is further complicated by the UA that indicates nexus state preservation
not being specified in SAM-2.

Thanks,
--David

> From: Knight, Frederick [mailto:[email-masked]]
> Sent: Thursday, September 27, 2012 11:11 AM
> To: Black, David; Julian Satran
> Cc: [email-masked]
> Subject: RE: [storm] SPC-2 reserve/release text - version 6
> 
> If a SCSI device loses any state and does not raise a UNIT ATTENTION, then it
> is BROKEN.  It is not a matter of anything in iSCSI (3720); I think SAM and
> SPC are pretty clear on this.  If there is no UNIT ATTENTION, then state SHALL
> be preserved; otherwise, all sorts of stuff will never work (mode page
> settings, RESERVE commands, all sorts of stuff).
> 
> Is someone really suggesting that some devices will drop state and NOT tell
> the host, and somehow that it is OK to do that?
> 
> The UAs are courtesy of SAM-3 (and SAM-4).  To encompass systems that may not
> deliver these UAs and situations in which they don't occur (e.g., the target
> doesn't recognize what happened as nexus reestablishment, independent of what
> the initiator thinks it's doing) I think the text should be rephrased as:
> 
> requirement that can be safely stated in any stronger fashion without the risk
> of rendering existing implementations non-compliant.
> 
> For context, what Julian is referring to is use of the COMPARE AND WRITE
> command that performs an atomic operation at the target instead of using
> RESERVE and RELEASE to create a critical section at the initiator based on
> read and write commands.
> 
> COMPARE AND WRITE is in use - a sentence mentioning that command could be
> added, although this best practices text is aimed primarily at uses of
> reserve/release for long-term reservations.  I think there's an implicit "when
> a reservation is the desired behavior" implication to the "persistent
> reservations SHOULD be used instead" text that I don't think needs to be made
> explicit.  OTOH, that "SHOULD"

---

From: "Knight, Frederick" <[email-masked]>
Date: Thu, 27 Sep 2012 22:29:30 +0000

I would think if the ISID is not recognized as an old one, then it must be recognized as a new one.  If it is a new one, then it will get a UA: 29/00 reflecting the first access after power on from the "new" initiator.

Is there something other than new or old?  Is there such a thing as a middle-aged initiator?

	Fred

-----Original Message-----
From: Black, David [mailto:[email-masked]] 
Sent: Thursday, September 27, 2012 11:56 AM
To: Knight, Frederick
Cc: [email-masked]
Subject: RE: [storm] SPC-2 reserve/release text - version 6

Fred,

The problem is not that the device does not raise a UA, it's that the UA may never be delivered.

A summary of how this can happen is that the new connection with the same ISID may not be recognized as reinstating or continuing the session at the target if the timeouts have expired - the UA gets established, but nobody shows up to pick it up in time, so the UA is discarded along with the rest of the session state and the target fails to recognize the initiator's subsequent ISID reuse even though the initiator believes that it's re- establishing the session.

This is further complicated by the UA that indicates nexus state preservation not being specified in SAM-2.

Thanks,
--David

> If a SCSI device loses any state and does not raise a UNIT ATTENTION, 
> then it is BROKEN.  It is not a matter of anything in iSCSI (3720); I 
> think SAM and SPC are pretty clear on this.  If there is no UNIT 
> ATTENTION, then state SHALL be preserved; otherwise, all sorts of 
> stuff will never work (mode page settings, RESERVE commands, all sorts of stuff).
> 
> Is someone really suggesting that some devices will drop state and NOT 
> tell the host, and somehow that it is OK to do that?
> 
> > The text in 4.4.3 states the obvious (there is no other speced 
> > method for continuation or reinstatement). Moreover if the session 
> > is reinstated after time2wait you create a need for a UA (or at 
> > least suggest one) and that was not there.
> 
> The UAs are courtesy of SAM-3 (and SAM-4).  To encompass systems that 
> may not deliver these UAs and situations in which they don't occur 
> (e.g., the target doesn't recognize what happened as nexus 
> reestablishment, independent of what the initiator thinks it's doing) I think the text should be rephrased as:
> 
> > there implied by conformance to spc2 and the relation between it 
> > nexus and session.
> 
> Unfortunately, SPC-2 is sufficiently imprecise on this subject that 
> such an implication is at best a "preferred implementation approach".  
> It's not a requirement that can be safely stated in any stronger 
> fashion without the risk of rendering existing implementations non-compliant.
> 
> > recommendations and will need additions once the new "locking" 
> > through compare-and-swap get in widespread use.
> 
> For context, what Julian is referring to is use of the COMPARE AND 
> WRITE command that performs an atomic operation at the target instead 
> of using RESERVE and RELEASE to create a critical section at the 
> initiator based on read and write commands.
> 
> COMPARE AND WRITE is in use - a sentence mentioning that command could 
> be added, although this best practices text is aimed primarily at uses 
> of reserve/release for long-term reservations.  I think there's an 
> implicit "when a reservation is the desired behavior" implication to 
> the "persistent reservations SHOULD be used instead" text that I don't 
> think needs to be made explicit.  OTOH, that "SHOULD"
> > The text in 4.4.3 states the obvious (there is no other speced 
> > method for continuation or reinstatement). Moreover if the the 
> > session is reinstated after time2wait you create a need for a UA (or 
> > at least suggest one) and that was not there. Just stating that 
> > preservation of reserve/release state -that was already there 
> > implied by conformance to spc2 and the relation between it nexus and session.
> >
> > recomendations and will need additions once the new "locking" 
> > through compare-and-swap get in widespread use.
> > > Here's where I think we've arrived (I've made a few minor 
> > > editorial tweaks to what's been discussed).  Despite all the 
> > > detailed discussion, I think this version 6 comes close to 
> > > Julian's suggestion
> that:
> > >
> > >> the session is maintained includes the reserve/release and leave 
> > >> it to
> > that.
> > >
> > > characterizes retention of that state as the "preferred 
> > > implementation
> approach".
> > >
> > > reserve/release reservations (possible need for a reset, 
> > > potentially unexpected interaction with persistent reservations, 
> > > see below) are useful
> to include.
> > >
> > >  Reserve/release reservations are obsolete [SPC3] and SHOULD NOT 
> > > be used; persistent reservations SHOULD be used instead.
> > >
> > > supported by an iSCSI target, the preferred implementation 
> > > approach is to preserve reserve/release reservation state for 
> > > iSCSI session reinstatement (see Section 6.3.5) or session 
> > > continuation  (see Section 6.3.6).
> > >

---

From: Julian Satran <[email-masked]>
Date: Fri, 28 Sep 2012 08:03:21 +0200

There is no such thing. And the reason the timeout is there is to limit the amount of state the targets have to keep. A target may choose to accept very large timeouts and then do what you wanted done but beyond that only the power-up ua is required.

julo

Sent from my iPad

---

From: "Knight, Frederick" <[email-masked]>
Date: Fri, 28 Sep 2012 12:37:43 +0000

OK, that's what I thought.

If state is retained (it's an old "I" reconnecting), then which UA is defined by SAM-3.

If state is not retained (it's a new "I"), then the UA is defined by SAM-3.

Since both old "I"s and new "I"s result in the "IF" condition being true, what is the condition that would make the "IF" false?

That "IF" language allows a target to loose state and not deliver any UA.

	Fred

-----Original Message-----
From: Julian Satran [mailto:[email-masked]] 
Sent: Friday, September 28, 2012 2:03 AM
To: Knight, Frederick
Cc: Black, David; [email-masked]
Subject: Re: [storm] SPC-2 reserve/release text - version 6

julo

Sent from my iPad

On 28 בספט 2012, at 00:29, "Knight, Frederick" <[email-masked]> wrote:

>> Is someone really suggesting that some devices will drop state and 
>> NOT tell the host, and somehow that it is OK to do that?
>> 
>> COMPARE AND WRITE is in use - a sentence mentioning that command 
>> could be added, although this best practices text is aimed primarily 
>> at uses of reserve/release for long-term reservations.  I think 
>> there's an implicit "when a reservation is the desired behavior" 
>> implication to the "persistent reservations SHOULD be used instead" 
>> text that I don't think needs to be made explicit.  OTOH, that "SHOULD"
>>>> discussion, I think this version 6 comes close to Julian's 
>>>> suggestion
>> that:
>>>>

---

From: "Black, David" <[email-masked]>
Date: Fri, 28 Sep 2012 09:51:04 -0400

Fred,

> Since both old "I"s and new "I"s result in the "IF" condition being true, what
> is the condition that would make the "IF" false?

You almost answered your own question ;-).

The answer is .... (drum roll) ... a target implementation based on RFC 3720
SAM-2, although I would hope that such implementations at least picked
up RFC 5048's changes.  

The I_T Nexus Loss concept on which this UA discussion is based is part of
the SCSI events and event notifications model that was introduced in SAM-3,
but neither RFC 3720 nor RFC 5048 references SAM-3.

There were definitely implementations of iSCSI based on SAM-2 - I agree with
Julian that we should *not* be introducing a UA requirement in this draft
that might break such implementations.

Thanks,
--David

> Sent: Friday, September 28, 2012 8:38 AM
> To: Julian Satran
> Since both old "I"s and new "I"s result in the "IF" condition being true, what
> is the condition that would make the "IF" false?
> 
> There is no such thing. And the reason the timeout is there is to limit the
> amount of state the targets have to keep. A target may choose to accept very
> large timeouts and then do what you wanted done but beyond that only the
> power-up ua is required.
> 
> julo
> 
> Sent from my iPad
> 
> > I would think if the ISID is not recognized as an old one, then it must be
> recognized as a new one.  If it is a new one, then it will get a UA: 29/00
> reflecting the first access after power on from the "new" initiator.
> >
> shows up to pick it up in time, so the UA is discarded along with the rest of
> the session state and the target fails to recognize the initiator's subsequent
> ISID reuse even though the initiator believes that it's re- establishing the
> session.
> >
> > This is further complicated by the UA that indicates nexus state
> preservation not being specified in SAM-2.
> >
> > Thanks,
> > --David
> >
> >

---

From: "Knight, Frederick" <[email-masked]>
Date: Fri, 28 Sep 2012 14:20:50 +0000

But we also shouldn't codify BROKEN designs in a brand new RFC.  It would better (in my opinion) to leave it alone than sanction and describe how to build a broken implementation.  Adding the "IF" is wrong.

We've fixed the issues raised by RFC5048 by incorporating it into the new RFC.

	Fred

-----Original Message-----
From: Black, David [mailto:[email-masked]] 
Sent: Friday, September 28, 2012 9:51 AM
To: Knight, Frederick
Cc: [email-masked]
Subject: RE: [storm] SPC-2 reserve/release text - version 6

Fred,

> Since both old "I"s and new "I"s result in the "IF" condition being 
> true, what is the condition that would make the "IF" false?

The answer is .... (drum roll) ... a target implementation based on RFC 3720 SAM-2, although I would hope that such implementations at least picked up RFC 5048's changes.  

The I_T Nexus Loss concept on which this UA discussion is based is part of the SCSI events and event notifications model that was introduced in SAM-3, but neither RFC 3720 nor RFC 5048 references SAM-3.

There were definitely implementations of iSCSI based on SAM-2 - I agree with Julian that we should *not* be introducing a UA requirement in this draft that might break such implementations.

Thanks,
--David

> Since both old "I"s and new "I"s result in the "IF" condition being 
> true, what is the condition that would make the "IF" false?
> 
> There is no such thing. And the reason the timeout is there is to 
> limit the amount of state the targets have to keep. A target may 
> choose to accept very large timeouts and then do what you wanted done 
> but beyond that only the power-up ua is required.
> 
> julo
> 
> Sent from my iPad
> 
> On 28 בספט 2012, at 00:29, "Knight, Frederick" 
> <[email-masked]>
> wrote:
> 
> recognized as a new one.  If it is a new one, then it will get a UA: 
> 29/00 reflecting the first access after power on from the "new" initiator.
> >
> > Is there something other than new or old?  Is there such a thing as 
> > a
> middle-aged initiator?
> >
> >    Fred
> >
> > The problem is not that the device does not raise a UA, it's that 
> > the UA may
> never be delivered.
> >
> ISID may not be recognized as reinstating or continuing the session at 
> the target if the timeouts have expired - the UA gets established, but 
> nobody shows up to pick it up in time, so the UA is discarded along 
> with the rest of the session state and the target fails to recognize 
> the initiator's subsequent ISID reuse even though the initiator 
> believes that it's re- establishing the session.
> >
> >> If a SCSI device loses any state and does not raise a UNIT 
> >> ATTENTION, then it is BROKEN.  It is not a matter of anything in 
> >> iSCSI (3720); I think SAM and SPC are pretty clear on this.  If 
> >> there is no UNIT ATTENTION, then state SHALL be preserved; 
> >> otherwise, all sorts of stuff will never work (mode page settings, 
> >> RESERVE commands, all sorts of
> stuff).
> >>
> >> The UAs are courtesy of SAM-3 (and SAM-4).  To encompass systems 
> >> that may not deliver these UAs and situations in which they don't 
> >> occur (e.g., the target doesn't recognize what happened as nexus 
> >> reestablishment, independent of what the initiator thinks it's 
> >> doing) I
> think the text should be rephrased as:
> >>
> >>> according 6.x includes preservation of reserve/release state -that 
> >>> was already there implied by conformance to spc2 and the relation 
> >>> between it nexus and session.
> >>
> >> WRITE command that performs an atomic operation at the target 
> >> instead of using RESERVE and RELEASE to create a critical section 
> >> at the initiator based on read and write commands.
> >>
> >> could be added, although this best practices text is aimed 
> >> primarily at uses of reserve/release for long-term reservations.  I 
> >> think there's an implicit "when a reservation is the desired behavior"
> >>> session is reinstated after time2wait you create a need for a UA 
> >>> (or at least suggest one) and that was not there. Just stating 
> >>> that creation/continuation of a session done according 6.x 
> >>> includes preservation of reserve/release state -that was already 
> >>> nexus and
> >>> If instead we refrain from stating anything specific and state 
> >>> that target should maintain whatever state is required by SPC 
> >>> while a session is continued and may discard state when a session 
> >>> is terminated and may indicate that it discarded state through a 
> >>> ua you are on safer ground and do not create new requirement.
> >>>
> >>> Sent from my iPad
> >>>
> >>>>> i would mention only that the state that must be maintained 
> >>>>> while the session is maintained includes the reserve/release and 
> >>>>> leave it to
> >>> that.
> >>>>
> >>>> RELEASE commands [SPC2], and persistent reservations, based on 
> >>>> the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3].
> >>>> Nexus failure.  Nonetheless, when reserve/release reservations 
> >>>> are supported by an iSCSI target, the preferred implementation

---

From: "Black, David" <[email-masked]>
Date: Fri, 28 Sep 2012 10:34:24 -0400

I'm not sure what's BROKEN, but a statement requiring the UAs would
be a post-SAM-2 requirements change that would have to go in the
-sam- draft so that it's subject to the negotiation of the text key
in that draft.  I have no problem with a blanket statement that
negotiating that text key to a value that promises the functionality
in the -sam- draft requires that the implementation adhere to
everything in SAM-4 that may differ from prior versions of SAM,
including I_T Nexus loss UA behavior, but there will need to be
some language in there about what happens after Time2Retain expires
before the ISID is reused.

OTOH, I am strongly opposed to revising the baseline requirements
in the consolidated draft in a fashion that may break "running code"
that's working fine, especially when the change is based on a point
of principle that does not exist in the specifications against
which that "running code" was developed (i.e., SAM-2).

Thanks,
--David

> Sent: Friday, September 28, 2012 10:21 AM
> To: Black, David
> Cc: [email-masked]
> But we also shouldn't codify BROKEN designs in a brand new RFC.  It would
> better (in my opinion) to leave it alone than sanction and describe how to
> build a broken implementation.  Adding the "IF" is wrong.
>
> SAM-2, although I would hope that such implementations at least picked up RFC
> 5048's changes.
>
> The I_T Nexus Loss concept on which this UA discussion is based is part of the
> SCSI events and event notifications model that was introduced in SAM-3, but
> neither RFC 3720 nor RFC 5048 references SAM-3.
>
> Julian that we should *not* be introducing a UA requirement in this draft that
> might break such implementations.
>
> Thanks,
> --David
>

---

From: "Knight, Frederick" <[email-masked]>
Date: Fri, 28 Sep 2012 15:08:56 +0000

> > >>     was preserved, see Section 6.3.4 of [SAM4].

Is what is broken.

IF the target decides to establish a UA, then it must follow SAM4 otherwise, if the target decides not to establish a UA, then everything is fine.  Consider the following sequence:

Initiator A - RESERVE command (GOOD status)
Initiator B - WRITE (RESERVATION CONFLICT status)
Initiator A gets disconnected/reconnected and RESERVE state is lost
Initiator B - WRITE (GOOD status)

The target decided not to establish a UA as a result of the session reestablishment, so everything just proceeds on its merry way?  Here's a suggestion that removes the "IF" and doesn't add any new must/should requirements:

Unit Attention conditions that result from session reestablishment
indicate whether nexus state was preserved, see Section 6.3.4 of [SAM4].

	Fred

-----Original Message-----
From: Black, David [mailto:[email-masked]] 
Sent: Friday, September 28, 2012 10:34 AM
To: Knight, Frederick
Cc: [email-masked]
Subject: RE: [storm] SPC-2 reserve/release text - version 6

I'm not sure what's BROKEN, but a statement requiring the UAs would be a post-SAM-2 requirements change that would have to go in the
-sam- draft so that it's subject to the negotiation of the text key in that draft.  I have no problem with a blanket statement that negotiating that text key to a value that promises the functionality in the -sam- draft requires that the implementation adhere to everything in SAM-4 that may differ from prior versions of SAM, including I_T Nexus loss UA behavior, but there will need to be some language in there about what happens after Time2Retain expires before the ISID is reused.

OTOH, I am strongly opposed to revising the baseline requirements in the consolidated draft in a fashion that may break "running code"
that's working fine, especially when the change is based on a point of principle that does not exist in the specifications against which that "running code" was developed (i.e., SAM-2).

Thanks,
--David

> But we also shouldn't codify BROKEN designs in a brand new RFC.  It 
> would better (in my opinion) to leave it alone than sanction and 
> describe how to build a broken implementation.  Adding the "IF" is wrong.
>
> The answer is .... (drum roll) ... a target implementation based on 
> RFC 3720 SAM-2, although I would hope that such implementations at 
> least picked up RFC 5048's changes.
>
> The I_T Nexus Loss concept on which this UA discussion is based is 
> part of the SCSI events and event notifications model that was 
> introduced in SAM-3, but neither RFC 3720 nor RFC 5048 references SAM-3.
>
> There were definitely implementations of iSCSI based on SAM-2 - I 
> agree with Julian that we should *not* be introducing a UA requirement 
> in this draft that might break such implementations.
>
> Thanks,
> --David
>
> > choose to accept very large timeouts and then do what you wanted 
> > done but beyond that only the power-up ua is required.
> >
> > julo
> >
> > Sent from my iPad
> >
> > > Is there something other than new or old?  Is there such a thing 
> > > as a
> > middle-aged initiator?
> > >
> > >    Fred
> > >
> > ISID may not be recognized as reinstating or continuing the session 
> > at the target if the timeouts have expired - the UA gets 
> > established, but nobody shows up to pick it up in time, so the UA is 
> > discarded along with the rest of the session state and the target 
> > fails to recognize the initiator's subsequent ISID reuse even though 
> > the initiator believes that it's re- establishing the session.
> > >
> > >> otherwise, all sorts of stuff will never work (mode page 
> > >> Is someone really suggesting that some devices will drop state 
> > >> and NOT tell the host, and somehow that it is OK to do that?
> > >>
> > >>> method for continuation or reinstatement). Moreover if the 
> > >>> (or at least suggest one) and that was not there.
> > >>
> > >>> according 6.x includes preservation of reserve/release state 
> > >>> -that was already there implied by conformance to spc2 and the 
> > >>> relation between it nexus and session.
> > >>
> > >> Unfortunately, SPC-2 is sufficiently imprecise on this subject 
> > >> that such an implication is at best a "preferred implementation approach".
> > >> fashion without the risk of rendering existing implementations 
> > >> non-
> > compliant.
> > >>
> > >> For context, what Julian is referring to is use of the COMPARE 
> > >> AND WRITE command that performs an atomic operation at the target 
> > >> primarily at uses of reserve/release for long-term reservations.  
> > >> I think there's an implicit "when a reservation is the desired behavior"
> > >>> while a session is continued and may discard state when a 
> > >>> session is terminated and may indicate that it discarded state 
> > >>> through a ua you are on safer ground and do not create new requirement.
> > >>>
> > >>> Sent from my iPad
> > >>>
> > >>>>> while the session is maintained includes the reserve/release 
> > >>>>> and leave it to
> > >>> that.
> > >>>>
> > >>>> potentially unexpected interaction with persistent 
> > >>>> reservations, see below) are useful
> > >> to include.
> > >>>>
> > >>>> There are two reservation management methods defined in the 
> > >>>> SCSI standards, reserve/release reservations, based on the 
> > >>>> RESERVE and RELEASE commands [SPC2], and persistent 
> > >>>> reservations, based on the PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT commands [SPC3].
> > >>>> David L. Black, Distinguished Engineer EMC Corporation, 176 
> > >>>> South St., Hopkinton, MA  01748

---

From: "Black, David" <[email-masked]>
Date: Fri, 28 Sep 2012 11:43:48 -0400

Fred,

> Here's a suggestion that removes the "IF" and doesn't add any new must/should
> requirements:
>

That text is acceptable to me.

I don't think we need anything beyond that text in the draft, but here's
the rest of the explanation ...

An important aspect of what's going on here is that with respect to the
use of the term "re-establishment" for an I_T Nexus in Section 6.3.4 of
[SAM4], after Time2Retain expires, an iSCSI target is entitled to consider
ISID re-use to not be a "re-establishment" of the I_T Nexus.  This is
because when Time2Retain expires, the target is allowed to discard all
of its state for that nexus, including the target's record of the ISID
value.

Avoidance of the problematic scenario involving a reservation loss is a

Thanks,
--David

> Sent: Friday, September 28, 2012 11:09 AM
> To: Black, David
> Cc: [email-masked]
> IF the target decides to establish a UA, then it must follow SAM4 otherwise,
> if the target decides not to establish a UA, then everything is fine.
> Consider the following sequence:
>
> The target decided not to establish a UA as a result of the session
> reestablishment, so everything just proceeds on its merry way?  Here's a
> suggestion that removes the "IF" and doesn't add any new must/should
> requirements:
>
> I'm not sure what's BROKEN, but a statement requiring the UAs would be a post-
> SAM-2 requirements change that would have to go in the
> -sam- draft so that it's subject to the negotiation of the text key in that
> draft.  I have no problem with a blanket statement that negotiating that text
> key to a value that promises the functionality in the -sam- draft requires
> that the implementation adhere to everything in SAM-4 that may differ from
> prior versions of SAM, including I_T Nexus loss UA behavior, but there will
> need to be some language in there about what happens after Time2Retain expires
> OTOH, I am strongly opposed to revising the baseline requirements in the
> consolidated draft in a fashion that may break "running code"
> that's working fine, especially when the change is based on a point of
> principle that does not exist in the specifications against which that
> "running code" was developed (i.e., SAM-2).
>
> Thanks,
> --David
>

---

From: "Knight, Frederick" <[email-masked]>
Date: Fri, 28 Sep 2012 16:40:10 +0000

> Initiator B - WRITE (GOOD status)

Is in violation of the text in SPC-2 that says:

"Reserve/release managed reservations are retained by
the device server until released or until reset by mechanisms specified in this standard."

Since - disconnect/reconnect is NOT specified in SPC-2, some form of reset is the only model left (and resets establish UAs).

Anyway, I'm glad we could settle on some text.

	Fred

-----Original Message-----
From: Black, David [mailto:[email-masked]] 
Sent: Friday, September 28, 2012 11:44 AM
To: Knight, Frederick
Cc: [email-masked]
Subject: RE: [storm] SPC-2 reserve/release text - version 6

Fred,

Thanks,
--David