Subject: Re: [storm] iSER - one last issue

From: Michael Ko <[email-masked]>
Date: Thu, 19 Jan 2012 09:38:30 -0800

For all RCaP layers, the Send message (or any variant) MUST be used, as 
opposed to using RDMA Read or Write to accomplish the task.  So a SHOULD for 
RCaP other than iWARP is incorrect.  RFC 5040 can mandate that all RDMAP 
messages be supported, but it is still up to implementation to decide what 
messages to use in any situation.  This iSER update is meant to reflect 
running code and to generalize the support of iSER over any RCaP, not just 
iWARP.  So the question is are there any running code that uses SendSE and 
SendInvSE for iSER.  Conversely, if the running code can get by using Send 
messages to accomplish the task, what is the point of mandating a MUST at 
this point for SendSE and SendInvSE?  Note that the receiver cannot 
completely rely on the sender for the Invalidate feature anyway.  The iSER 
layer at the initiator is required to explicitly invalidate the STag for 
bidirectional commands and abnormal completion of a command.  Even when 
automatic invalidation is used, the iSER layer is required to sanity 
checking the automatic invalidation.  In other words, the iSER spec puts the 
burden on the receiver for doing the right thing in handling the Send 
messages, and not relying on the sender for the mere convenience.

Mike
----- Original Message ----- 
From: Hemal Shah
To: Tom Talpey ; [email-masked] ; [email-masked] ; 
[email-masked]
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

You are welcome! You are right about 7.3.4 as the only section mandating 
Send message.

I think your concern about specifying the requirement for the other RCaP 
implementation is valid. I suggest we should word the requirements as below:

"If the RCaP layer is as specified in [RFC5040] and [RFC5041], the SendSE 
message MUST be used. For any other RCaP layer, the Send message SHOULD be 
used. "

Hemal

From: Tom Talpey [mailto:[email-masked]]
Sent: Thursday, January 19, 2012 8:09 AM
To: Hemal Shah; [email-masked]; [email-masked]; 
[email-masked]
Subject: RE: iSER - one last issue

Thanks for pointing out that sentence in RFC5046 7.3.4. I believe it is the 
only such stipulation.

I'm ok with stating the requirement as RFC5046-over-RFC5040 compliance. But 
this new draft opens the door to alternative behaviors, therefore it needs 
to be stated more clearly. What's the requirement when SendSE is not 
supported? And, how does the receiver know what to expect?

In other words, if it's proposed that the statement be:

                "When the RCaP layer is as specified in [RFC5040] and 
[RFC5041], the SendSE message MUST be used."

. then what MUST be used when the RCaP is not? That statement needs to be 
made, too.

From: Hemal Shah [mailto:[email-masked]]
Sent: Thursday, January 19, 2012 10:38 AM
To: Tom Talpey; [email-masked]; [email-masked]; 
[email-masked]
Subject: RE: iSER - one last issue

Tom,

RFC5046 is very clear about the use of Send Message Type for iSCSI 
control-type PDU. For specific iSCSI PDUs, Section 7 mandates the use of 
Send, SendSE, or SendSEInv. Section 7 of RFC5046 clearly specifies the use 
of specific Send Message Type for each iSCSI control-type PDUs (Login, 
Logout, Text request, Text response.). Section 7.3.4 mandates the use of 
plain-old Send also as needed. For example, see text below from 7.3.4.

For unsolicited data, if the F bit is set to 0 in a SCSI Data-out

PDU, the iSER layer at the initiator MUST use a Send Message to send

the SCSI Data-out PDU.

Regarding your comment in the second paragraph below, RFC5040 expectation is 
all the RDMAP messages specified in RFC5040 are supported (if that is not 
the case then we cannot count on any of the other RDMAP messages to be 
supported). So, an implementation that does not support SendSE at the sender 
is not compliant with RFC5040. So, your second point does not apply.

I hope that addresses your comments.

Hemal

From: Tom Talpey [mailto:[email-masked]]
Sent: Thursday, January 19, 2012 5:51 AM
To: Hemal Shah; [email-masked]; [email-masked]; 
[email-masked]
Subject: RE: iSER - one last issue

If using SendSE is so important here, when MUST a plain-old Send be used? 
There is no mention of any distinction in RFC5046 section 1.4.2 nor in this 
draft's section 2.4.2.

Even with a MUST on the SendSE, the behavior is indeterminate, since it 
would still be contingent on support for SendSE at the *sender*. This 
critical exception voids any normative requirement, even over iWARP. The 
strongest statement that can be made is SHOULD.

Tom.

From: [email-masked] [mailto:[email-masked]] On Behalf Of 
Hemal Shah
Sent: Wednesday, January 18, 2012 6:56 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue

David and Mike,

I believe that the requirements should not be weakened and we should stick 
to the requirement in RFC5046.

RFC5046 required the use of specific Send Message Type for valid reasons.

In the case of requiring SendSE message use, the intent was to give a hint 
to the remote side to generate an event upon processing of SendSE message. 
For example, the SendSE is specifically carrying SCSI information that needs 
attention on the remote side e.g. SCSI command, Logout.

In the case of requiring SendInvSE Message, the intent was to invalidate the 
STag used in the data transfer as well as inform the remote side to generate 
an event. For example, mandating the use of SendInvSE for the SCSI response 
PDU for a SCSI Read limits the exposure of STag used in SCSI Read data 
transfer and avoids the explicit invalidation of the STag at the initiator. 
Plus, it allows the target to generate an event on the initiator upon the 
completion of SCSI Read command.

By relaxing RFC5046 requirements in the latest iSER draft, we will not only 
impact all iWARP based iSER implementations that rely on the use of specific 
Send Message Type for SCSI data transfers but also change the intended 
behavior of initiators and targets.

For iWARP, I strongly suggest that the iSER draft does not relax the 
requirements specified in RFC5046.

I hope that helps.

Hemal

From: [email-masked] [mailto:[email-masked]] On Behalf Of 
[email-masked]
Sent: Wednesday, January 18, 2012 3:10 PM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue

Mike,

Thank you for the explanation, but I don't believe you've correctly stated

the intent of RFC 5046.  Here are some examples from RFC 5046's text:

1.5.  SCSI Read Overview

   The iSER Message is transferred to the target using a SendSE Message.

   The iSER layer at the target uses a SendInvSE Message to transfer the

   SCSI Response PDU back to the iSER layer at the initiator.

Similar language occurs in 1.6 SCSI Write Overview.

5.2.1.  Normal Connection Termination at the Initiator

   The iSER layer at the initiator MUST

   use a SendSE Message to send the Logout Request PDU to the target.

Similar language occurs in 5.2.2 for target side normal connection 
termination,

and more importantly in 7.3.1 SCSI Command:

   The iSER layer at the initiator MUST send the SCSI command in a

   SendSE Message to the target.

None of the quoted text permits use of Send instead of SendSE or SendInv

instead of SendInvSE.

What you propose to do effectively changes "MUST" to "should" (lower case,

weaker than "SHOULD"), and that sure looks like a change to RFC 5046.

Are there good implementation-based reasons to weaken these requirements 
now?

Does anyone else have a viewpoint on this topic?

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]
Sent: Wednesday, January 18, 2012 2:22 PM
To: Black, David; [email-masked]
Subject: Re: iSER - one last issue

David,

Here is my rationale for using the lower case "should".

The intent of RFC5046 is that the Send Message type must be used instead of 
RDMA Reads or Writes.  The Solicited Event feature of the Send Message is 
provided as a convenience.  The receiver must still do the right thing in 
handling the Send Message regardless of whether the SE feature is used.  In 
other words, the sender is responsible for using the right format for the 
message (Send vs RDMA) but the receiver must not rely on the sender to 
determine how to handle the received message.  The same rationale goes for 
the Invalidate feature.

Mike

----- Original Message ----- 

From: [email-masked]

To: [email-masked] ; [email-masked]

Sent: Monday, January 16, 2012 2:06 PM

Subject: iSER - one last issue

Mike,

Thanks for getting the -08 version of the iSER draft posted.  I think that

draft addresses all of the open issues, but I have a question for the WG

about how to express the SendSE and SendInvSE requirements from RFC 5046.

The -08 version of the iSER draft expresses requirements for the SendSE

and SendInvSE messages (this is primarily for iSER over RDDP/iWARP) as:

The SendSE Message should be used if

supported by the RCaP layer (e.g., iWARP).

My reading of RFC 5046 is that its requirements are tighter - to accurately

reflect RFC 5046, I would replace "should" with "MUST" in the above text,

at least for iWARP.

In the alternative, if the "should"s remain, an explanatory item

needs to be added to the Appendix A list of changes from RFC 5046.

What do others think the right course of action is here, use "MUST" or

explain weakening of requirement to "should" ?

Thanks,
--David

---

From: Tom Talpey <[email-masked]>
Date: Thu, 19 Jan 2012 20:32:31 +0000

More precisely, are there any iSER *initiators* that *depend* on receiving solicited events over the iWARP RDMA transport? I believe this to be a very short list, if any. But I'd be ok with being proven wrong.

From: Michael Ko [mailto:[email-masked]]
Sent: Thursday, January 19, 2012 12:39 PM
To: Hemal Shah; Tom Talpey; [email-masked]; [email-masked]
Subject: Re: iSER - one last issue

For all RCaP layers, the Send message (or any variant) MUST be used, as opposed to using RDMA Read or Write to accomplish the task.  So a SHOULD for RCaP other than iWARP is incorrect.  RFC 5040 can mandate that all RDMAP messages be supported, but it is still up to implementation to decide what messages to use in any situation.  This iSER update is meant to reflect running code and to generalize the support of iSER over any RCaP, not just iWARP.  So the question is are there any running code that uses SendSE and SendInvSE for iSER.  Conversely, if the running code can get by using Send messages to accomplish the task, what is the point of mandating a MUST at this point for SendSE and SendInvSE?  Note that the receiver cannot completely rely on the sender for the Invalidate feature anyway.  The iSER layer at the initiator is required to explicitly invalidate the STag for bidirectional commands and abnormal completion of a command.  Even when automatic invalidation is used, the iSER layer is required to sanity checking the automatic invalidation.  In other words, the iSER spec puts the burden on the receiver for doing the right thing in handling the Send messages, and not relying on the sender for the mere convenience.

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

You are welcome! You are right about 7.3.4 as the only section mandating Send message.

I think your concern about specifying the requirement for the other RCaP implementation is valid. I suggest we should word the requirements as below:

"If the RCaP layer is as specified in [RFC5040] and [RFC5041], the SendSE message MUST be used. For any other RCaP layer, the Send message SHOULD be used. "

Hemal

From: Tom Talpey [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 8:09 AM
To: Hemal Shah; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: iSER - one last issue

Thanks for pointing out that sentence in RFC5046 7.3.4. I believe it is the only such stipulation.

I'm ok with stating the requirement as RFC5046-over-RFC5040 compliance. But this new draft opens the door to alternative behaviors, therefore it needs to be stated more clearly. What's the requirement when SendSE is not supported? And, how does the receiver know what to expect?

In other words, if it's proposed that the statement be:
                "When the RCaP layer is as specified in [RFC5040] and [RFC5041], the SendSE message MUST be used."

... then what MUST be used when the RCaP is not? That statement needs to be made, too.

From: Hemal Shah [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 10:38 AM
To: Tom Talpey; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: iSER - one last issue

Tom,

RFC5046 is very clear about the use of Send Message Type for iSCSI control-type PDU. For specific iSCSI PDUs, Section 7 mandates the use of Send, SendSE, or SendSEInv. Section 7 of RFC5046 clearly specifies the use of specific Send Message Type for each iSCSI control-type PDUs (Login, Logout, Text request, Text response...). Section 7.3.4 mandates the use of plain-old Send also as needed. For example, see text below from 7.3.4.

Regarding your comment in the second paragraph below, RFC5040 expectation is all the RDMAP messages specified in RFC5040 are supported (if that is not the case then we cannot count on any of the other RDMAP messages to be supported). So, an implementation that does not support SendSE at the sender is not compliant with RFC5040. So, your second point does not apply.

Hemal

From: Tom Talpey [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 5:51 AM
To: Hemal Shah; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: iSER - one last issue

If using SendSE is so important here, when MUST a plain-old Send be used? There is no mention of any distinction in RFC5046 section 1.4.2 nor in this draft's section 2.4.2.

Even with a MUST on the SendSE, the behavior is indeterminate, since it would still be contingent on support for SendSE at the *sender*. This critical exception voids any normative requirement, even over iWARP. The strongest statement that can be made is SHOULD.

Tom.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Hemal Shah
Sent: Wednesday, January 18, 2012 6:56 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - one last issue

David and Mike,

I believe that the requirements should not be weakened and we should stick to the requirement in RFC5046.

In the case of requiring SendSE message use, the intent was to give a hint to the remote side to generate an event upon processing of SendSE message. For example, the SendSE is specifically carrying SCSI information that needs attention on the remote side e.g. SCSI command, Logout...

In the case of requiring SendInvSE Message, the intent was to invalidate the STag used in the data transfer as well as inform the remote side to generate an event. For example, mandating the use of SendInvSE for the SCSI response PDU for a SCSI Read limits the exposure of STag used in SCSI Read data transfer and avoids the explicit invalidation of the STag at the initiator. Plus, it allows the target to generate an event on the initiator upon the completion of SCSI Read command.

By relaxing RFC5046 requirements in the latest iSER draft, we will not only impact all iWARP based iSER implementations that rely on the use of specific Send Message Type for SCSI data transfers but also change the intended behavior of initiators and targets.

For iWARP, I strongly suggest that the iSER draft does not relax the requirements specified in RFC5046.

I hope that helps.

Hemal

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, January 18, 2012 3:10 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - one last issue

Mike,

1.5.  SCSI Read Overview

Similar language occurs in 5.2.2 for target side normal connection termination,
and more importantly in 7.3.1 SCSI Command:

Are there good implementation-based reasons to weaken these requirements now?

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Wednesday, January 18, 2012 2:22 PM
To: Black, David; [email-masked]<mailto:[email-masked]>
Subject: Re: iSER - one last issue

David,

The intent of RFC5046 is that the Send Message type must be used instead of RDMA Reads or Writes.  The Solicited Event feature of the Send Message is provided as a convenience.  The receiver must still do the right thing in handling the Send Message regardless of whether the SE feature is used.  In other words, the sender is responsible for using the right format for the message (Send vs RDMA) but the receiver must not rely on the sender to determine how to handle the received message.  The same rationale goes for the Invalidate feature.

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David

---

From: <[email-masked]>
Date: Thu, 19 Jan 2012 19:03:29 -0500

[WG co chair hat on]

I think the implementation aspect of this discussion is useful.

I also want to remind everyone that the following text from the storm WG charter
(http://datatracker.ietf.org/wg/storm/charter/) is highly relevant to the
message usage topic under discussion:

  Stability is critical to the usage of these protocols, so backwards
  compatibility with existing implementations will be a requirement
  imposed on for all protocol changes and additions. Note that this is a
  requirement for implementation compatibility - if it is the case that
  all implementations of a protocol have done something different than
  what the RFC specifies, it is appropriate for a new RFC to document what
  the "running code" actually does and deprecate the unused original behavior.

I think it's ok for iSER's message usage requirements to vary between iWARP
and InfiniBand - such a difference could make it more difficult to run iSER
over an RCaP gateway between iWARP and InfiniBand, but I don't think there's
much interest in doing that.  OTOH, all changes to RFC5046 need justification
under the "all implementations of a protocol have done something different
than what the RFC specifies" condition quoted above.

Thanks,
--David (as WG co-chair)

From: Tom Talpey [mailto:[email-masked]]
Sent: Thursday, January 19, 2012 3:33 PM
To: Michael Ko; Hemal Shah; Black, David; [email-masked]
Subject: RE: iSER - one last issue

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

Hemal

Tom,

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal

Mike,

1.5.  SCSI Read Overview

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David

---

From: <[email-masked]>
Date: Mon, 23 Jan 2012 20:04:55 -0500

[WG chair hat off]

This is a message intended for discussion and comments.

Looking for a way forward, this observation from Mike seems like a useful place to start:

> This iSER update is meant to reflect running code and to generalize the support of iSER over any RCaP, not just iWARP.

RFC 5046 (iSER) was written on the assumption that the RCaP layer supports Solicited
Event functionality.  I also read RFC 5040 as requiring RDMAP implementations (part of
iWARP) to support Solicited Event functionality.

This first step seems easy:

[1] There are RCaPs that don't support Solicited Event functionality; those RCaPs have
no choice - they MUST use Send (not SendSE) and SendInv (not SendInvSE).

I would prefer to write iSER requirements in terms of whether the RCaP supports
Solicited Event functionality as opposed to calling out iWARP explicitly as having
different requirements.  OTOH, I'm prepared to listen to arguments to the contrary.

Beyond that, I think there are two primary choices:

[2a] Leave things alone - if the RCaP supports Solicited Events, then the "MUST"
requirements for SendSE and SendInvSE apply as they did in RFC 5046.

[2b] In the alternative, if there are iSER implementations for iWARP that aren't using
SendSE and SendInvSE in accordance with RFC 5046's requirements, then we need to change
those requirements to match the implementations.

I would hope we can agree on [1], and I believe that'll cover the InfiniBand case, right?

For choosing between [2a] and [2b] it would help a lot to konw what iSER over iWARP
implementations are actually doing (SendSE/SendInvSE vs. Send/SendInv).

*** Who is familiar with the iSER over iWARP implementation(s) ??? ***

In general, what do folks think ought to be done here - [2a], [2b] or something else?

[WG chair hat on]

We do need to resolve this issue and reflect that resolution in a -09 iSER draft before
RFC publication can be requested.

Thanks,
--David

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

Hemal

Tom,

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal

Mike,

1.5.  SCSI Read Overview

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David

---

From: Tom Talpey <[email-masked]>
Date: Tue, 24 Jan 2012 13:48:30 +0000

[WG co-chair hat off]

> requirements for SendSE and SendInvSE apply as they did in RFC 5046.

The core issue here is the statement "if the RCaP supports Solicited Events". Both the iWARP and Infiniband protocols support them. But Infiniband does not require all adapters support them - the architecture requires it for HCAs but leaves it optional for TCAs (Infiniband architecture v1.2.1 table 319 on page 1045). Because this latter property cannot be detected by the remote upper layer, it does not appear to be possible to make an interoperable decision for all protocols and all implementations.

That said, in the absence of any existing iSER-over-iWARP SendSE implementations, I agree the entire discussion is effectively moot.

Tom.

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, January 23, 2012 8:05 PM
To: [email-masked]
Subject: [storm] iSER - one last issue: update

[WG chair hat off]

[WG chair hat on]

Thanks,
--David

From: Tom Talpey [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 3:33 PM
To: Michael Ko; Hemal Shah; Black, David; [email-masked]<mailto:[email-masked]>
Subject: RE: iSER - one last issue

From: Michael Ko [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 12:39 PM
To: Hemal Shah; Tom Talpey; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: iSER - one last issue

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

Hemal

Tom,

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal

Mike,

1.5.  SCSI Read Overview

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Tue, 24 Jan 2012 16:25:21 -0800

I think we should go with [1] and [2a] that David outlined.  As Hemal said,
there were good reasons why RFC 5046 stipulated SendInvSE and SendSE, where
it did.  I would rather not weaken it, at least for the iWARP RCaP
ecosystem.

Mallikarjun

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Tom Talpey
Sent: Tuesday, January 24, 2012 5:49 AM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue: update

[WG co-chair hat off]

> requirements for SendSE and SendInvSE apply as they did in RFC 5046.

The core issue here is the statement "if the RCaP supports Solicited
Events". Both the iWARP and Infiniband protocols support them. But
Infiniband does not require all adapters support them - the architecture
requires it for HCAs but leaves it optional for TCAs (Infiniband
architecture v1.2.1 table 319 on page 1045). Because this latter property
cannot be detected by the remote upper layer, it does not appear to be
possible to make an interoperable decision for all protocols and all
implementations.

That said, in the absence of any existing iSER-over-iWARP SendSE
implementations, I agree the entire discussion is effectively moot.

Tom.

[WG chair hat off]

Looking for a way forward, this observation from Mike seems like a useful
place to start:

> This iSER update is meant to reflect running code and to generalize the
support of iSER over any RCaP, not just iWARP. 

RFC 5046 (iSER) was written on the assumption that the RCaP layer supports
Solicited

Event functionality.  I also read RFC 5040 as requiring RDMAP
implementations (part of

[1] There are RCaPs that don't support Solicited Event functionality; those
RCaPs have

I would prefer to write iSER requirements in terms of whether the RCaP
supports

Solicited Event functionality as opposed to calling out iWARP explicitly as
having

different requirements.  OTOH, I'm prepared to listen to arguments to the
contrary.

[2a] Leave things alone - if the RCaP supports Solicited Events, then the
"MUST"

[2b] In the alternative, if there are iSER implementations for iWARP that
aren't using

SendSE and SendInvSE in accordance with RFC 5046's requirements, then we
need to change

I would hope we can agree on [1], and I believe that'll cover the InfiniBand
case, right?

For choosing between [2a] and [2b] it would help a lot to konw what iSER
over iWARP

In general, what do folks think ought to be done here - [2a], [2b] or
something else?

[WG chair hat on]

We do need to resolve this issue and reflect that resolution in a -09 iSER
draft before

Thanks,
--David

More precisely, are there any iSER *initiators* that *depend* on receiving
solicited events over the iWARP RDMA transport? I believe this to be a very
short list, if any. But I'd be ok with being proven wrong.

Mike

----- Original Message ----- 

From: Hemal Shah <mailto:[email-masked]>  

To: Tom Talpey <mailto:[email-masked]>  ; [email-masked] ;
[email-masked] ; [email-masked] 

Tom,

Hemal

 

Tom,

the SCSI Data-out PDU.

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal    

Mike,

1.5.  SCSI Read Overview

instead of SendInvSE.

Thanks,
--David

David,

Mike

----- Original Message ----- 

Mike,

at least for iWARP.

Thanks,
--David

---

From: <[email-masked]>
Date: Tue, 24 Jan 2012 19:41:56 -0500

[WG chair hat off]

Proposed slight refinement: If the SendSE and SendInvSE requirements are applied to RCaPs that "require all implementations to support Solicited Events", that set of RCaPs includes iWARP, but not InfiniBand.

> If using SendSE is so important here, when MUST a plain-old Send be used?

There are at least a couple of approaches to this one:

[3a] SendSE/SendInvSE MUST NOT be used with other RCaPs.

[3b] Explain how a sender figures out whether to use SendSE/SendInvSE or not (aka "negotiation").

[3a] would be the simplest.  The "running code" matters here - does the IB implementation of iSER use SendSE/SendInvSE?  If not, [3a] should be fine.

Thanks,
--David

From: Mallikarjun Chadalapaka [mailto:[email-masked]]
Sent: Tuesday, January 24, 2012 7:25 PM
To: Tom Talpey; Black, David; [email-masked]
Subject: RE: [storm] iSER - one last issue: update

I think we should go with [1] and [2a] that David outlined.  As Hemal said, there were good reasons why RFC 5046 stipulated SendInvSE and SendSE, where it did.  I would rather not weaken it, at least for the iWARP RCaP ecosystem.

Mallikarjun

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Tuesday, January 24, 2012 5:49 AM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue: update

[WG co-chair hat off]

Tom.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Monday, January 23, 2012 8:05 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [storm] iSER - one last issue: update

[WG chair hat off]

[WG chair hat on]

Thanks,
--David

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

Hemal

Tom,

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal

Mike,

1.5.  SCSI Read Overview

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David

---

From: Tom Talpey <[email-masked]>
Date: Wed, 25 Jan 2012 03:50:11 +0000

To take [3a], making SendSE mandatory for iWARP and mandatory-to-not for all others, would be very inconsistent and somewhat contradictory. For example, Infiniband currently runs at speeds approximately triple that of Ethernet, and up. Surely a performance-oriented tweak such as SendSE matters more for IB than iWARP, if it matters at all. And, such a decision makes it more difficult to design a common upper layer implementation.

So, just to throw another log on this fire, [3c], I suggest we ask the Infiniband community whether they're ok with SendSE/SendInvSE being a MUST for solicited replies exactly as specified in RFC5046. It means that for Infiniband, all HCAs, but only those TCAs which support solicited events, can be used with the protocol. And it means the sending side of the existing draft implementations may have to change to send them where required.

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, January 24, 2012 7:42 PM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue: update

[WG chair hat off]

Thanks,
--David

From: Mallikarjun Chadalapaka [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Tuesday, January 24, 2012 7:25 PM
To: Tom Talpey; Black, David; [email-masked]<mailto:[email-masked]>
Subject: RE: [storm] iSER - one last issue: update

Mallikarjun

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Tom Talpey
Sent: Tuesday, January 24, 2012 5:49 AM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - one last issue: update

[WG co-chair hat off]

Tom.

[WG chair hat off]

[WG chair hat on]

Thanks,
--David

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

Hemal

Tom,

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal

Mike,

1.5.  SCSI Read Overview

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David

---

From: "Hemal Shah" <[email-masked]>
Date: Wed, 25 Jan 2012 23:47:31 +0000

I think we should investigate [3c]. From the consistency standpoint, [3c] makes iSER consistent across different RCaPs.

Can someone from InfiniBand side comment?

Hemal

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Tuesday, January 24, 2012 7:50 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue: update

[WG chair hat off]

Thanks,
--David

Mallikarjun

[WG co-chair hat off]

Tom.

[WG chair hat off]

[WG chair hat on]

Thanks,
--David

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

Hemal

Tom,

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal

Mike,

1.5.  SCSI Read Overview

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David

---

From: "Hemal Shah" <[email-masked]>
Date: Wed, 25 Jan 2012 23:42:36 +0000

I'm also think we should go with [1] and [2a].

Hemal

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mallikarjun Chadalapaka
Sent: Tuesday, January 24, 2012 4:25 PM
To: Tom Talpey; [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue: update

Mallikarjun

[WG co-chair hat off]

Tom.

[WG chair hat off]

[WG chair hat on]

Thanks,
--David

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

Hemal

Tom,

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal

Mike,

1.5.  SCSI Read Overview

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David