Subject: [storm] iSER - now what?

From: <[email-masked]>
Date: Mon, 9 Apr 2012 11:16:22 -0400

My understanding of the state of the state of the open issue around iSER is that we have resolved what the IB implementation does (no use of SendSE, SendInv, or SendInvSE), and that implementation also supports iWARP, but it’s unclear whether there are other iWARP implementations that use those primitives.

Does anyone have any additional information to contribute?

I’d like to figure out what we ought to do about this sometime soon ... but as Tom notes, we only have one implementation to base our decision on so far.

Thanks,
--David (as WG co-chair)
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

From: Tom Talpey [mailto:[email-masked]]
Sent: Thursday, March 08, 2012 5:19 PM
To: Hemal Shah; Black, David; [email-masked]; [email-masked]
Subject: RE: [storm] FW: iSER - one last issue: update

That’s not what I’m arguing. I am arguing that we have had exactly one implementation state any support for iSER, and it has not implemented solicited events at all. What is the purpose of preserving a requirement that no actual implementation complies with, and as Alexander states below, even disagrees with?

As I said earlier in this thread, I would be happy to be proven wrong on existing implementations’ use of these. But today it appears to be a cast of one. Our goal should be to encourage more to join, of course!

Tom.

From: Hemal Shah [mailto:[email-masked]]
Sent: Thursday, March 08, 2012 4:29 PM
To: Tom Talpey; [email-masked]; [email-masked]; [email-masked]
Subject: RE: [storm] FW: iSER - one last issue: update

Relying on a single existing implementation for a specific operating environment should not be the basis for changing the originally defined iSER requirements for iWARP.

I agree with David that the interoperability goal should be within a single RCaP. Requiring iSER to take the lowest common denominator capabilities of RCaPs (InfiniBand and iWARP here) should not be the goal.

Hemal

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Tom Talpey
Sent: Thursday, March 08, 2012 10:13 AM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] FW: iSER - one last issue: update

<Also with WG chair hat off>

I am not sure I draw the same conclusion. Alexander refers to an existing implementation of iSER which neither sends nor processes solicited events. Presumably this applies equally to whether it is running over Infiniband or iWARP. We should take this into consideration as “running code” feedback.

In the absence of any iSER providers acknowledging that they both send and require solicited events, what exact basis are we using to place a MUST on this for iWARP, and only for iWARP, going forward?

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Thursday, March 08, 2012 12:52 AM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] FW: iSER - one last issue: update
Importance: High

Alexander,

<WG chair hat off>

Thank you for adding to the discussion - I think this is the key additional information:

> First of all the current implementation of iSER (which is in Linux, comprising a low-level scsi kernel module on the
> initiator side and user-space plug-in to STGT target) does not use SendSE / SendInvalidate / SendInvalidateSE, on either side.

Returning to the options, for InfiniBand, this looks like something close to [1]:

[1] There are RCaPs that don’t support Solicited Event functionality; those RCaPs have
no choice - they MUST use Send (not SendSE) and SendInv (not SendInvSE).

And it looks like we have rough consensus on [2a] for iWARP (no change to RFC 5046 requirements):

[2a] Leave things alone - if the RCaP supports Solicited Events, then the “MUST”
requirements for SendSE and SendInvSE apply as they did in RFC 5046.

In an attempt to keep it simple, can we stick with just documenting the “running code”, as iSER
implementations only exist for InfiniBand and iWARP?

If so, the requirements could be:
- InfiniBand: MUST NOT use SendSE, SendInv or SendInvSE.
- iWARP: MUST use SendSE and SendInvSE.

That could be accompanied by text to say that a (hypothetical) future RCaP would have to pick
one of these approaches (or something similar) and lay down the requirements to ensure
interoperability, but interoperability is only a goal within a single RCaP, not across RCaPs
(e.g., as running iSER across RCaPs will generally require I/O termination and redrive at the
gateway between RCaPs, as RDMA data transfer is unlikely to cross that gateway transparently).

Everyone else - Thoughts?  Comments?

I’d really like to get this issue closed ...

Thanks,
--David

From: Alexander Nezhinsky [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Sunday, March 04, 2012 3:05 PM
To: [email-masked]<mailto:[email-masked]>; Black, David; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Mike Ko; Tom Talpey
Subject: Re: FW: [storm] iSER - one last issue: update

Hi

Back after being virtually absent from the list for quite a time (unfortunately)...

I've read the last discussion on iSER and tried to summarize my experience and the current viewpoint here. But anyway, i'm hardly representing "the" IB community :)

First of all the current implementation of iSER (which is in Linux, comprising a low-level scsi kernel module on the initiator side and user-space plug-in to STGT target) does not use SendSE / SendInvalidate / SendInvalidateSE, on either side.

Neither initiator nor target require this functionality for the correct operation and, to my opinion, they would hardly ever require it. I'll try to explain my reasoning.

If I understand correctly, the main rationale behind using SendSE is to try to decrease the number of times interrupts are armed on the hardware level. Let's see how interrupt arming is done in practice.

Infiniband based applications may be implemented either in kernel or in user space. In kernel a completion notification results in an interrupt which is served in ISR. In user space the interrupt events (at least in linux) are delivered to the application using a file descriptor which becomes readable. This mechanism, especially in user space (where kernel-user interaction is added to interrupt latency), is quite inefficient, relative to the high speeds of transfer being used.

Thus the common practice is to re-arm the notifications only after a period of absence of completions and to drain the CQ repetitively after receiving such notification using poll_cq mechanism, for as long as possible. In protocols like iSER under heavy load conditions it is possible to receive most of the completions without ever arming an interrupt at least during the bursts of traffic. In the extreme cases some applications may decide to work exclusively in polling mode to ensure minimal latency and increase throughput when the packets are expected to be small.

Thus the target implementations would almost uniformly rely on a mix of interrupt-driven notifications and CQ polling. So the initiator software has no way to know if the target will ever request notifications for solicited events only. Moreover it can safely bet on a target not using it or not relying on it. Initiators, usually being more "general-purpose" would use polling less aggressively, but i believe they will also tend to employ some "arm/poll/re-arm" scheme which does not depend on SendSE to ensure correctness.

Another concern here is the utility of SendSE in iWarp and, if any, its relevance to Infiniband settings. The only case where SendSE is used with iWarp is Unsolicied Data-OUTs, in cases when more than a single Data-OUT PDU is sent. By the way, this means that the initiators don't rely on SendSE at all, doesn't it?

The current iSER target implementation opts for Solicited-only mode of operation, refusing to accept Unsolicited Data-OUTs through negotiation. This was done initially because of certain technical limitations of STGT design, but a few experiments have shown that relying upon Unsolicited Data-OUTs for small data sizes does not improve performance and that mixing Unsolicited and Solicited Data-OUTs sometimes even decreases it (I can put forward some theories about why this happens, if anybody is interested, i can share with you in a separate mail). This is only an unreliable observation coming from a single implementation but I guess that even when Unsolicited data is enabled, it is unwise to use many small Data-OUTs. This will lead to unnecessary overhead in processing interrupts, completions and protocol packet headers. Thus using only immediate data or a single relatively large Data-OUT (by setting FirstBurstLength = MaxRecvDataSegmentLength) is always preferred, performance-wise.

To summarize, those very scenarios that justified using SendSE in iWarp design are grossly irrelevant to the reality of the implementations based on Infiniband. And, to tell the truth, I can't see why iWarp setup should be significantly different from Infiniband one in this aspect.

I don't know much about TCA types and unaware of other applications to inform, but if my previous arguments are acceptable, these questions are less relevant.

Now, regarding SendInvalidate. I think that being unable to use it with bidirectional commands alone makes this quite impractical even in iWarp. Coupled with Mike's observation that iSER layer is required to check invalidation anyway, makes the requirement to use SendInvalidate quite strange, unless there are some implicit reasons, which i am unaware of. In this case those reasons are better made explicit.

Alexander.
On Mon, Jan 30, 2012 at 3:53 PM, Tom Talpey <[email-masked]<mailto:[email-masked]>> wrote:
Alexander, have you been watching the IETF/STORM discussion on iSER? There is a critical issue on the table which requires input from Infiniband iSER implementors.

iSER is considering maintaining the RFC5046 requirement that the use of SendSE and SendInvSE be mandatory for certain target sends, however the IB spec does not require all TCAs to implement these operations. We seek to know if:

1)      The IB community agrees or disagrees with this

2)      Whether any IB/iSER initiator implementations require receiving solicited events for correct operation (i.e. do they arm for solicited or all?)

3)      Whether there are in fact any relevant TCAs which fail to support solicited events

4)      What other implementations we may need to inform of this issue.

You should make your opinion known to the list, so we may advance the specification.

From: Hemal Shah [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, January 25, 2012 6:48 PM
To: Tom Talpey; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

Subject: RE: [storm] iSER - one last issue: update

I think we should investigate [3c]. From the consistency standpoint, [3c] makes iSER consistent across different RCaPs.

Can someone from InfiniBand side comment?

Hemal

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Tom Talpey
Sent: Tuesday, January 24, 2012 7:50 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - one last issue: update

To take [3a], making SendSE mandatory for iWARP and mandatory-to-not for all others, would be very inconsistent and somewhat contradictory. For example, Infiniband currently runs at speeds approximately triple that of Ethernet, and up. Surely a performance-oriented tweak such as SendSE matters more for IB than iWARP, if it matters at all. And, such a decision makes it more difficult to design a common upper layer implementation.

So, just to throw another log on this fire, [3c], I suggest we ask the Infiniband community whether they’re ok with SendSE/SendInvSE being a MUST for solicited replies exactly as specified in RFC5046. It means that for Infiniband, all HCAs, but only those TCAs which support solicited events, can be used with the protocol. And it means the sending side of the existing draft implementations may have to change to send them where required.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Tuesday, January 24, 2012 7:42 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - one last issue: update

[WG chair hat off]

Proposed slight refinement: If the SendSE and SendInvSE requirements are applied to RCaPs that “require all implementations to support Solicited Events”, that set of RCaPs includes iWARP, but not InfiniBand.

That refinement creates a need to figure out the complementary requirement that applies to InfiniBand, which Tom more or less asked about earlier in this thread:

> If using SendSE is so important here, when MUST a plain-old Send be used?

There are at least a couple of approaches to this one:

[3a] SendSE/SendInvSE MUST NOT be used with other RCaPs.

[3b] Explain how a sender figures out whether to use SendSE/SendInvSE or not (aka “negotiation”).

[3a] would be the simplest.  The “running code” matters here - does the IB implementation of iSER use SendSE/SendInvSE?  If not, [3a] should be fine.

Thanks,
--David

From: Mallikarjun Chadalapaka [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Tuesday, January 24, 2012 7:25 PM
To: Tom Talpey; Black, David; [email-masked]<mailto:[email-masked]>
Subject: RE: [storm] iSER - one last issue: update

I think we should go with [1] and [2a] that David outlined.  As Hemal said, there were good reasons why RFC 5046 stipulated SendInvSE and SendSE, where it did.  I would rather not weaken it, at least for the iWARP RCaP ecosystem.

Mallikarjun

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Tom Talpey
Sent: Tuesday, January 24, 2012 5:49 AM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - one last issue: update

[WG co-chair hat off]

> [2a] Leave things alone - if the RCaP supports Solicited Events, then the “MUST”
> requirements for SendSE and SendInvSE apply as they did in RFC 5046.

The core issue here is the statement “if the RCaP supports Solicited Events”. Both the iWARP and Infiniband protocols support them. But Infiniband does not require all adapters support them – the architecture requires it for HCAs but leaves it optional for TCAs (Infiniband architecture v1.2.1 table 319 on page 1045). Because this latter property cannot be detected by the remote upper layer, it does not appear to be possible to make an interoperable decision for all protocols and all implementations.

That said, in the absence of any existing iSER-over-iWARP SendSE implementations, I agree the entire discussion is effectively moot.

Tom.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Monday, January 23, 2012 8:05 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [storm] iSER - one last issue: update

[WG chair hat off]

This is a message intended for discussion and comments.

Looking for a way forward, this observation from Mike seems like a useful place to start:

> This iSER update is meant to reflect running code and to generalize the support of iSER over any RCaP, not just iWARP.

RFC 5046 (iSER) was written on the assumption that the RCaP layer supports Solicited
Event functionality.  I also read RFC 5040 as requiring RDMAP implementations (part of
iWARP) to support Solicited Event functionality.

This first step seems easy:

[1] There are RCaPs that don’t support Solicited Event functionality; those RCaPs have
no choice - they MUST use Send (not SendSE) and SendInv (not SendInvSE).

I would prefer to write iSER requirements in terms of whether the RCaP supports
Solicited Event functionality as opposed to calling out iWARP explicitly as having
different requirements.  OTOH, I’m prepared to listen to arguments to the contrary.

Beyond that, I think there are two primary choices:

[2a] Leave things alone - if the RCaP supports Solicited Events, then the “MUST”
requirements for SendSE and SendInvSE apply as they did in RFC 5046.

[2b] In the alternative, if there are iSER implementations for iWARP that aren’t using
SendSE and SendInvSE in accordance with RFC 5046’s requirements, then we need to change
those requirements to match the implementations.

I would hope we can agree on [1], and I believe that’ll cover the InfiniBand case, right?

For choosing between [2a] and [2b] it would help a lot to konw what iSER over iWARP
implementations are actually doing (SendSE/SendInvSE vs. Send/SendInv).

*** Who is familiar with the iSER over iWARP implementation(s) ??? ***

In general, what do folks think ought to be done here - [2a], [2b] or something else?

[WG chair hat on]

We do need to resolve this issue and reflect that resolution in a -09 iSER draft before
RFC publication can be requested.

Thanks,
--David

From: Tom Talpey [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 3:33 PM
To: Michael Ko; Hemal Shah; Black, David; [email-masked]<mailto:[email-masked]>
Subject: RE: iSER - one last issue

More precisely, are there any iSER *initiators* that *depend* on receiving solicited events over the iWARP RDMA transport? I believe this to be a very short list, if any. But I’d be ok with being proven wrong.

From: Michael Ko [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 12:39 PM
To: Hemal Shah; Tom Talpey; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: iSER - one last issue

For all RCaP layers, the Send message (or any variant) MUST be used, as opposed to using RDMA Read or Write to accomplish the task.  So a SHOULD for RCaP other than iWARP is incorrect.  RFC 5040 can mandate that all RDMAP messages be supported, but it is still up to implementation to decide what messages to use in any situation.  This iSER update is meant to reflect running code and to generalize the support of iSER over any RCaP, not just iWARP.  So the question is are there any running code that uses SendSE and SendInvSE for iSER.  Conversely, if the running code can get by using Send messages to accomplish the task, what is the point of mandating a MUST at this point for SendSE and SendInvSE?  Note that the receiver cannot completely rely on the sender for the Invalidate feature anyway.  The iSER layer at the initiator is required to explicitly invalidate the STag for bidirectional commands and abnormal completion of a command.  Even when automatic invalidation is used, the iSER layer is required to sanity checking the automatic invalidation.  In other words, the iSER spec puts the burden on the receiver for doing the right thing in handling the Send messages, and not relying on the sender for the mere convenience.

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

You are welcome! You are right about 7.3.4 as the only section mandating Send message.

I think your concern about specifying the requirement for the other RCaP implementation is valid. I suggest we should word the requirements as below:

“If the RCaP layer is as specified in [RFC5040] and [RFC5041], the SendSE message MUST be used. For any other RCaP layer, the Send message SHOULD be used. ”

Hemal

From: Tom Talpey [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 8:09 AM
To: Hemal Shah; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: iSER - one last issue

Thanks for pointing out that sentence in RFC5046 7.3.4. I believe it is the only such stipulation.

I’m ok with stating the requirement as RFC5046-over-RFC5040 compliance. But this new draft opens the door to alternative behaviors, therefore it needs to be stated more clearly. What’s the requirement when SendSE is not supported? And, how does the receiver know what to expect?

In other words, if it’s proposed that the statement be:
                “When the RCaP layer is as specified in [RFC5040] and [RFC5041], the SendSE message MUST be used.”

… then what MUST be used when the RCaP is not? That statement needs to be made, too.

From: Hemal Shah [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 10:38 AM
To: Tom Talpey; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: iSER - one last issue

Tom,

RFC5046 is very clear about the use of Send Message Type for iSCSI control-type PDU. For specific iSCSI PDUs, Section 7 mandates the use of Send, SendSE, or SendSEInv. Section 7 of RFC5046 clearly specifies the use of specific Send Message Type for each iSCSI control-type PDUs (Login, Logout, Text request, Text response…). Section 7.3.4 mandates the use of plain-old Send also as needed. For example, see text below from 7.3.4.

For unsolicited data, if the F bit is set to 0 in a SCSI Data-out
PDU, the iSER layer at the initiator MUST use a Send Message to send
the SCSI Data-out PDU.

Regarding your comment in the second paragraph below, RFC5040 expectation is all the RDMAP messages specified in RFC5040 are supported (if that is not the case then we cannot count on any of the other RDMAP messages to be supported). So, an implementation that does not support SendSE at the sender is not compliant with RFC5040. So, your second point does not apply.

I hope that addresses your comments.

Hemal

From: Tom Talpey [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, January 19, 2012 5:51 AM
To: Hemal Shah; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: iSER - one last issue

If using SendSE is so important here, when MUST a plain-old Send be used? There is no mention of any distinction in RFC5046 section 1.4.2 nor in this draft’s section 2.4.2.

Even with a MUST on the SendSE, the behavior is indeterminate, since it would still be contingent on support for SendSE at the *sender*. This critical exception voids any normative requirement, even over iWARP. The strongest statement that can be made is SHOULD.

Tom.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Hemal Shah
Sent: Wednesday, January 18, 2012 6:56 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - one last issue

David and Mike,

I believe that the requirements should not be weakened and we should stick to the requirement in RFC5046.

RFC5046 required the use of specific Send Message Type for valid reasons.

In the case of requiring SendSE message use, the intent was to give a hint to the remote side to generate an event upon processing of SendSE message. For example, the SendSE is specifically carrying SCSI information that needs attention on the remote side e.g. SCSI command, Logout…

In the case of requiring SendInvSE Message, the intent was to invalidate the STag used in the data transfer as well as inform the remote side to generate an event. For example, mandating the use of SendInvSE for the SCSI response PDU for a SCSI Read limits the exposure of STag used in SCSI Read data transfer and avoids the explicit invalidation of the STag at the initiator. Plus, it allows the target to generate an event on the initiator upon the completion of SCSI Read command.

By relaxing RFC5046 requirements in the latest iSER draft, we will not only impact all iWARP based iSER implementations that rely on the use of specific Send Message Type for SCSI data transfers but also change the intended behavior of initiators and targets.

For iWARP, I strongly suggest that the iSER draft does not relax the requirements specified in RFC5046.

I hope that helps.

Hemal

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, January 18, 2012 3:10 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - one last issue

Mike,

Thank you for the explanation, but I don’t believe you’ve correctly stated
the intent of RFC 5046.  Here are some examples from RFC 5046’s text:

1.5.  SCSI Read Overview

   The iSER Message is transferred to the target using a SendSE Message.

   The iSER layer at the target uses a SendInvSE Message to transfer the
   SCSI Response PDU back to the iSER layer at the initiator.

Similar language occurs in 1.6 SCSI Write Overview.

5.2.1.  Normal Connection Termination at the Initiator

   The iSER layer at the initiator MUST
   use a SendSE Message to send the Logout Request PDU to the target.

Similar language occurs in 5.2.2 for target side normal connection termination,
and more importantly in 7.3.1 SCSI Command:

   The iSER layer at the initiator MUST send the SCSI command in a
   SendSE Message to the target.

None of the quoted text permits use of Send instead of SendSE or SendInv
instead of SendInvSE.

What you propose to do effectively changes “MUST” to “should” (lower case,
weaker than “SHOULD”), and that sure looks like a change to RFC 5046.

Are there good implementation-based reasons to weaken these requirements now?

Does anyone else have a viewpoint on this topic?

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Wednesday, January 18, 2012 2:22 PM
To: Black, David; [email-masked]<mailto:[email-masked]>
Subject: Re: iSER - one last issue

David,

Here is my rationale for using the lower case "should".

The intent of RFC5046 is that the Send Message type must be used instead of RDMA Reads or Writes.  The Solicited Event feature of the Send Message is provided as a convenience.  The receiver must still do the right thing in handling the Send Message regardless of whether the SE feature is used.  In other words, the sender is responsible for using the right format for the message (Send vs RDMA) but the receiver must not rely on the sender to determine how to handle the received message.  The same rationale goes for the Invalidate feature.

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks for getting the -08 version of the iSER draft posted.  I think that
draft addresses all of the open issues, but I have a question for the WG
about how to express the SendSE and SendInvSE requirements from RFC 5046.

The -08 version of the iSER draft expresses requirements for the SendSE
and SendInvSE messages (this is primarily for iSER over RDDP/iWARP) as:

The SendSE Message should be used if
supported by the RCaP layer (e.g., iWARP).

My reading of RFC 5046 is that its requirements are tighter - to accurately
reflect RFC 5046, I would replace “should” with “MUST” in the above text,
at least for iWARP.

In the alternative, if the “should”s remain, an explanatory item
needs to be added to the Appendix A list of changes from RFC 5046.

What do others think the right course of action is here, use “MUST” or
explain weakening of requirement to “should” ?

Thanks,
--David

---

From: "Hemal Shah" <[email-masked]>
Date: Mon, 16 Apr 2012 21:24:40 +0000

David,

Since we have only one implementation to base our decision so far, I suggest we stick to option 2a suggested originally.

Hemal

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, April 09, 2012 8:16 AM
To: [email-masked]; [email-masked]
Subject: [storm] iSER - now what?
Importance: High

Thanks,
--David (as WG co-chair)

Tom.

Hemal

Alexander,

<WG chair hat off>

Thanks,
--David

Hi

Alexander.

Hemal

[WG chair hat off]

Thanks,
--David

Mallikarjun

[WG co-chair hat off]

Tom.

[WG chair hat off]

[WG chair hat on]

Thanks,
--David

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue

Tom,

Hemal

Tom,

Hemal

Tom.

David and Mike,

I hope that helps.

Hemal

Mike,

1.5.  SCSI Read Overview

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue

Mike,

Thanks,
--David

---

From: Michael Ko <[email-masked]>
Date: Tue, 17 Apr 2012 21:12:56 -0700

Since this iSER update is intended to document the “running code”, then the
implementation currrently in existence should be the model upon which the
iSER update should be based on.  That means the "MUST" requirement applies
to Send, and not SendInv, or SendSE, or SendInvSE.  RFC 5046 already
indicates that it is is the responsibility of the receiver to do the right
thing for processing Send messages, and not relying on which flavor of Send
messages the sender uses.  To dictate that the iSER implementation supports
different Send requirements would mean that it has to know what RDMA
transport it is running on.  As long as we are talking hypothetical
situation here since iWARP implementation does not exist today, consider
the hypothetical situation where an iSER initiator running on Infiniband
talks to an iSER target running on iWARP, what flavors of Send messages are
required?  IMHO, the usage model for SendInv, SendInvSE and SendSE as
described in the latest iSER draft (not RFC 5046) is adequate and no
changes are necessary; laying out extraneous requirements based on the
underlying RDMA transport can lead to potential interoperability problems
and open up a can of worms.

Mike

On Mon, Apr 16, 2012 at 2:24 PM, Hemal Shah <[email-masked]> wrote:

>  *David,*
>
> * *
>
> *Since we have only one implementation to base our decision so far, I
> suggest we stick to option 2a suggested originally.*
>
> * *
>
> *Hemal *
>
> * *
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of [email-masked]
> *Sent:* Monday, April 09, 2012 8:16 AM
> *To:* [email-masked]; [email-masked]
> *Subject:* [storm] iSER - now what?
> *Importance:* High****
>
> ** **
>
> My understanding of the state of the state of the open issue around iSER
> is that we have resolved what the IB implementation does (no use of SendSE,
> SendInv, or SendInvSE), and that implementation also supports iWARP, but
> it’s unclear whether there are other iWARP implementations that use those
> primitives.****
>
> ** **
>
> Does anyone have any additional information to contribute?****
>
> ** **
>
> I’d like to figure out what we ought to do about this sometime soon ...
> but as Tom notes, we only have one implementation to base our decision on
> so far.****
>
> ** **
>
> Thanks,
> --David (as WG co-chair)
> ----------------------------------------------------****
>
> ** **
>
> *From:* Tom Talpey [mailto:[email-masked]]
> *Sent:* Thursday, March 08, 2012 5:19 PM
> *To:* Hemal Shah; Black, David; [email-masked]; [email-masked]
> *Subject:* RE: [storm] FW: iSER - one last issue: update****
>
> ** **
>
> That’s not what I’m arguing. I am arguing that we have had exactly one
> implementation state any support for iSER, and it has not implemented
> solicited events at all. What is the purpose of preserving a requirement
> that no actual implementation complies with, and as Alexander states below,
> even disagrees with?****
>
> ** **
>
> As I said earlier in this thread, I would be happy to be proven wrong on
> existing implementations’ use of these. But today it appears to be a cast
> of one. Our goal should be to encourage more to join, of course!****
>
> ** **
>
> Tom.****
>
> ** **
>
> *From:* Hemal Shah [mailto:[email-masked]]
> *Sent:* Thursday, March 08, 2012 4:29 PM
> *To:* Tom Talpey; [email-masked]; [email-masked]; [email-masked]
> *Subject:* RE: [storm] FW: iSER - one last issue: update****
>
> ** **
>
> *Relying on a single existing implementation for a specific operating
> environment should not be the basis for changing the originally defined
> iSER requirements for iWARP.*
>
> * *
>
> *I agree with David that the interoperability goal should be within a
> single RCaP. Requiring iSER to take the lowest common denominator
> capabilities of RCaPs (InfiniBand and iWARP here) should not be the goal.*
>
> * *
>
> *Hemal *
>
> * *
>
> * *
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Tom Talpey
> *Sent:* Thursday, March 08, 2012 10:13 AM
> *To:* [email-masked]; [email-masked]; [email-masked]
> *Subject:* Re: [storm] FW: iSER - one last issue: update****
>
> ** **
>
> <Also with WG chair hat off>****
>
> ** **
>
> I am not sure I draw the same conclusion. Alexander refers to an existing
> implementation of iSER which neither sends nor processes solicited events.
> Presumably this applies equally to whether it is running over Infiniband or
> iWARP. We should take this into consideration as “running code” feedback.*
> ***
>
> ** **
>
> In the absence of any iSER providers acknowledging that they both send and
> require solicited events, what exact basis are we using to place a MUST on
> this for iWARP, and only for iWARP, going forward? ****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of [email-masked]
> *Sent:* Thursday, March 08, 2012 12:52 AM
> *To:* [email-masked]; [email-masked]
> *Subject:* Re: [storm] FW: iSER - one last issue: update
> *Importance:* High****
>
> ** **
>
> Alexander,****
>
> ** **
>
> <WG chair hat off>****
>
> ** **
>
> Thank you for adding to the discussion - I think this is the key
> additional information:****
>
> ** **
>
> > First of all the current implementation of iSER (which is in Linux,
> comprising a low-level scsi kernel module on the****
>
> > initiator side and user-space plug-in to STGT target) does not use
> SendSE / SendInvalidate / SendInvalidateSE, on either side.****
>
> ** **
>
> Returning to the options, for InfiniBand, this looks like something close
> to [1]:****
>
> ** **
>
> [1] There are RCaPs that don’t support Solicited Event functionality;
> those RCaPs have****
>
> no choice - they MUST use Send (not SendSE) and SendInv (not SendInvSE).**
> **
>
> ** **
>
> And it looks like we have rough consensus on [2a] for iWARP (no change to
> RFC 5046 requirements):****
>
> ** **
>
> [2a] Leave things alone - if the RCaP supports Solicited Events, then the
> “MUST”****
>
> requirements for SendSE and SendInvSE apply as they did in RFC 5046.****
>
> ** **
>
> In an attempt to keep it simple, can we stick with just documenting the
> “running code”, as iSER****
>
> implementations only exist for InfiniBand and iWARP?  ****
>
> ** **
>
> If so, the requirements could be:****
>
> - InfiniBand: MUST NOT use SendSE, SendInv or SendInvSE.****
>
> - iWARP: MUST use SendSE and SendInvSE.****
>
> ** **
>
> That could be accompanied by text to say that a (hypothetical) future RCaP
> would have to pick****
>
> one of these approaches (or something similar) and lay down the
> requirements to ensure****
>
> interoperability, but interoperability is only a goal within a single
> RCaP, not across RCaPs****
>
> (e.g., as running iSER across RCaPs will generally require I/O termination
> and redrive at the****
>
> gateway between RCaPs, as RDMA data transfer is unlikely to cross that
> gateway transparently).****
>
> ** **
>
> Everyone else - Thoughts?  Comments?****
>
> ** **
>
> I’d really like to get this issue closed ...****
>
> ** **
>
> Thanks,
> --David****
>
> ** **
>
> *From:* Alexander Nezhinsky [mailto:[email-masked]]
> *Sent:* Sunday, March 04, 2012 3:05 PM
> *To:* [email-masked]; Black, David; [email-masked];
> [email-masked]; Mike Ko; Tom Talpey
> *Subject:* Re: FW: [storm] iSER - one last issue: update****
>
> ** **
>
> Hi
>
> I've read the last discussion on iSER and tried to summarize my experience
> and the current viewpoint here. But anyway, i'm hardly representing "the"
> IB community :)
>
> First of all the current implementation of iSER (which is in Linux,
> comprising a low-level scsi kernel module on the initiator side and
> user-space plug-in to STGT target) does not use SendSE / SendInvalidate /
> SendInvalidateSE, on either side.
>
> Neither initiator nor target require this functionality for the correct
> operation and, to my opinion, they would hardly ever require it. I'll try
> to explain my reasoning.
>
> If I understand correctly, the main rationale behind using SendSE is to
> try to decrease the number of times interrupts are armed on the hardware
> level. Let's see how interrupt arming is done in practice.
>
> Infiniband based applications may be implemented either in kernel or in
> user space. In kernel a completion notification results in an interrupt
> which is served in ISR. In user space the interrupt events (at least in
> linux) are delivered to the application using a file descriptor which
> becomes readable. This mechanism, especially in user space (where
> kernel-user interaction is added to interrupt latency), is quite
> inefficient, relative to the high speeds of transfer being used.
>
> Thus the common practice is to re-arm the notifications only after a
> period of absence of completions and to drain the CQ repetitively after
> receiving such notification using poll_cq mechanism, for as long as
> possible. In protocols like iSER under heavy load conditions it is possible
> to receive most of the completions without ever arming an interrupt at
> least during the bursts of traffic. In the extreme cases some
> applications may decide to work exclusively in polling mode to ensure
> minimal latency and increase throughput when the packets are expected to be
> small.
>
> Thus the target implementations would almost uniformly rely on a mix of
> interrupt-driven notifications and CQ polling. So the initiator software
> has no way to know if the target will ever request notifications for
> solicited events only. Moreover it can safely bet on a target not using it
> or not relying on it. Initiators, usually being more "general-purpose"
> would use polling less aggressively, but i believe they will also tend to
> employ some "arm/poll/re-arm" scheme which does not depend on SendSE to
> ensure correctness.
>
> Another concern here is the utility of SendSE in iWarp and, if any, its
> relevance to Infiniband settings. The only case where SendSE is used with
> iWarp is Unsolicied Data-OUTs, in cases when more than a single Data-OUT
> PDU is sent. By the way, this means that the initiators don't rely on
> SendSE at all, doesn't it?
>
> The current iSER target implementation opts for Solicited-only mode of
> operation, refusing to accept Unsolicited Data-OUTs through negotiation.
> This was done initially because of certain technical limitations of STGT
> design, but a few experiments have shown that relying upon Unsolicited
> Data-OUTs for small data sizes does not improve performance and that mixing
> Unsolicited and Solicited Data-OUTs sometimes even decreases it (I can put
> forward some theories about why this happens, if anybody is interested, i
> can share with you in a separate mail). This is only an unreliable
> observation coming from a single implementation but I guess that even when
> Unsolicited data is enabled, it is unwise to use many small Data-OUTs. This
> will lead to unnecessary overhead in processing interrupts, completions and
> protocol packet headers. Thus using only immediate data or a single
> relatively large Data-OUT (by setting FirstBurstLength =
> MaxRecvDataSegmentLength) is always preferred, performance-wise.
>
> To summarize, those very scenarios that justified using SendSE in iWarp
> design are grossly irrelevant to the reality of the implementations based
> on Infiniband. And, to tell the truth, I can't see why iWarp setup should
> be significantly different from Infiniband one in this aspect.
>
> I don't know much about TCA types and unaware of other applications to
> inform, but if my previous arguments are acceptable, these questions are
> less relevant.
>
> Now, regarding SendInvalidate. I think that being unable to use it with
> bidirectional commands alone makes this quite impractical even in iWarp.
> Coupled with Mike's observation that iSER layer is required to check
> invalidation anyway, makes the requirement to use SendInvalidate quite
> strange, unless there are some implicit reasons, which i am unaware of. In
> this case those reasons are better made explicit.
>
>
> Alexander.****
>
> On Mon, Jan 30, 2012 at 3:53 PM, Tom Talpey <[email-masked]> wrote:
> ****
>
> Alexander, have you been watching the IETF/STORM discussion on iSER? There
> is a critical issue on the table which requires input from Infiniband iSER
> implementors.****
>
>  ****
>
> iSER is considering maintaining the RFC5046 requirement that the use of
> SendSE and SendInvSE be mandatory for certain target sends, however the IB
> spec does not require all TCAs to implement these operations. We seek to
> know if:****
>
> 1)      The IB community agrees or disagrees with this****
>
> 2)      Whether any IB/iSER initiator implementations require receiving
> solicited events for correct operation (i.e. do they arm for solicited or
> all?)****
>
> 3)      Whether there are in fact any relevant TCAs which fail to support
> solicited events****
>
> 4)      What other implementations we may need to inform of this issue.***
> *
>
>  ****
>
> You should make your opinion known to the list, so we may advance the
> specification.****
>
>  ****
>
> *From:* Hemal Shah [mailto:[email-masked]]
> *Sent:* Wednesday, January 25, 2012 6:48 PM
> *To:* Tom Talpey; [email-masked]; [email-masked]; [email-masked]
> ****
>
>
> *Subject:* RE: [storm] iSER - one last issue: update****
>
>  ****
>
> *I think we should investigate [3c]. From the consistency standpoint,
> [3c] makes iSER consistent across different RCaPs.*****
>
> * *****
>
> *Can someone from InfiniBand side comment?*****
>
> * *****
>
> *Hemal*****
>
> * *****
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Tom Talpey
> *Sent:* Tuesday, January 24, 2012 7:50 PM
> *To:* [email-masked]; [email-masked]; [email-masked]
> *Subject:* Re: [storm] iSER - one last issue: update****
>
>  ****
>
> To take [3a], making SendSE mandatory for iWARP and mandatory-to-not for
> all others, would be very inconsistent and somewhat contradictory. For
> example, Infiniband currently runs at speeds approximately triple that of
> Ethernet, and up. Surely a performance-oriented tweak such as SendSE
> matters more for IB than iWARP, if it matters at all. And, such a decision
> makes it more difficult to design a common upper layer implementation.****
>
>  ****
>
> So, just to throw another log on this fire, [3c], I suggest we ask the
> Infiniband community whether they’re ok with SendSE/SendInvSE being a MUST
> for solicited replies exactly as specified in RFC5046. It means that for
> Infiniband, all HCAs, but only those TCAs which support solicited events,
> can be used with the protocol. And it means the sending side of the
> existing draft implementations may have to change to send them where
> required.****
>
>  ****
>
>  ****
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of [email-masked]
> *Sent:* Tuesday, January 24, 2012 7:42 PM
> *To:* [email-masked]; [email-masked]
> *Subject:* Re: [storm] iSER - one last issue: update****
>
>  ****
>
> [WG chair hat off]****
>
>  ****
>
> Proposed slight refinement: If the SendSE and SendInvSE requirements are
> applied to RCaPs that “require all implementations to support Solicited
> Events”, that set of RCaPs includes iWARP, but not InfiniBand.****
>
>  ****
>
> That refinement creates a need to figure out the complementary requirement
> that applies to InfiniBand, which Tom more or less asked about earlier in
> this thread:****
>
>  ****
>
> > If using SendSE is so important here, when MUST a plain-old Send be
> used?****
>
>  ****
>
> There are at least a couple of approaches to this one:****
>
>  ****
>
> [3a] SendSE/SendInvSE MUST NOT be used with other RCaPs.****
>
>  ****
>
> [3b] Explain how a sender figures out whether to use SendSE/SendInvSE or
> not (aka “negotiation”).****
>
>  ****
>
> [3a] would be the simplest.  The “running code” matters here - does the IB
> implementation of iSER use SendSE/SendInvSE?  If not, [3a] should be fine.
> ****
>
>  ****
>
> Thanks,
> --David****
>
>  ****
>
> *From:* Mallikarjun Chadalapaka [mailto:[email-masked]]
> *Sent:* Tuesday, January 24, 2012 7:25 PM
> *To:* Tom Talpey; Black, David; [email-masked]
> *Subject:* RE: [storm] iSER - one last issue: update****
>
>  ****
>
> I think we should go with [1] and [2a] that David outlined.  As Hemal
> said, there were good reasons why RFC 5046 stipulated SendInvSE and SendSE,
> where it did.  I would rather not weaken it, at least for the iWARP RCaP
> ecosystem.****
>
>  ****
>
> Mallikarjun****
>
>  ****
>
>  ****
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Tom Talpey
> *Sent:* Tuesday, January 24, 2012 5:49 AM
> *To:* [email-masked]; [email-masked]
> *Subject:* Re: [storm] iSER - one last issue: update****
>
>  ****
>
> [WG co-chair hat off]****
>
>  ****
>
> > [2a] Leave things alone - if the RCaP supports Solicited Events, then
> the “MUST”****
>
> > requirements for SendSE and SendInvSE apply as they did in RFC 5046.****
>
>  ****
>
> The core issue here is the statement “if the RCaP supports Solicited
> Events”. Both the iWARP and Infiniband protocols support them. But
> Infiniband does not require all adapters support them – the architecture
> requires it for HCAs but leaves it optional for TCAs (Infiniband
> architecture v1.2.1 table 319 on page 1045). Because this latter property
> cannot be detected by the remote upper layer, it does not appear to be
> possible to make an interoperable decision for all protocols and all
> implementations.****
>
>  ****
>
> That said, in the absence of any existing iSER-over-iWARP SendSE
> implementations, I agree the entire discussion is effectively moot.****
>
>  ****
>
> Tom.****
>
>  ****
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of [email-masked]
> *Sent:* Monday, January 23, 2012 8:05 PM
> *To:* [email-masked]
> *Subject:* [storm] iSER - one last issue: update****
>
>  ****
>
> [WG chair hat off]****
>
>  ****
>
> This is a message intended for discussion and comments.****
>
>  ****
>
> Looking for a way forward, this observation from Mike seems like a useful
> place to start:****
>
>  ****
>
> > This iSER update is meant to reflect running code and to generalize the
> support of iSER over any RCaP, not just iWARP. ****
>
>  ****
>
> RFC 5046 (iSER) was written on the assumption that the RCaP layer supports
> Solicited****
>
> Event functionality.  I also read RFC 5040 as requiring RDMAP
> implementations (part of****
>
> iWARP) to support Solicited Event functionality.****
>
>  ****
>
> This first step seems easy:****
>
>  ****
>
> [1] There are RCaPs that don’t support Solicited Event functionality;
> those RCaPs have****
>
> ****
> ****
> I would prefer to write iSER requirements in terms of whether the RCaP
> supports****
>
> Solicited Event functionality as opposed to calling out iWARP explicitly
> as having****
>
> different requirements.  OTOH, I’m prepared to listen to arguments to the
> contrary.****
>
>  ****
>
> Beyond that, I think there are two primary choices:****
>
>  ****
>
> [2a] Leave things alone - if the RCaP supports Solicited Events, then the
> “MUST”****
>
> requirements for SendSE and SendInvSE apply as they did in RFC 5046.****
>
>  ****
>
> [2b] In the alternative, if there are iSER implementations for iWARP that
> aren’t using****
>
> SendSE and SendInvSE in accordance with RFC 5046’s requirements, then we
> need to change****
>
> those requirements to match the implementations.****
>
>  ****
>
> I would hope we can agree on [1], and I believe that’ll cover the
> InfiniBand case, right?****
>
>  ****
>
> For choosing between [2a] and [2b] it would help a lot to konw what iSER
> over iWARP****
>
> implementations are actually doing (SendSE/SendInvSE vs. Send/SendInv).***
> *
>
>  ****
>
> *** Who is familiar with the iSER over iWARP implementation(s) ??? *******
>
>  ****
>
> In general, what do folks think ought to be done here - [2a], [2b] or
> something else?****
>
>  ****
>
> [WG chair hat on]****
>
>  ****
>
> We do need to resolve this issue and reflect that resolution in a -09 iSER
> draft before****
>
> RFC publication can be requested.****
>
>  ****
>
> Thanks,
> --David****
>
>  ****
>
> *From:* Tom Talpey [mailto:[email-masked]]
> *Sent:* Thursday, January 19, 2012 3:33 PM
> *To:* Michael Ko; Hemal Shah; Black, David; [email-masked]
> *Subject:* RE: iSER - one last issue****
>
>  ****
>
> More precisely, are there any iSER **initiators** that **depend** on
> receiving solicited events over the iWARP RDMA transport? I believe this to
> be a very short list, if any. But I’d be ok with being proven wrong.****
>
>  ****
>
>  ****
>
>  ****
>
> *From:* Michael Ko [mailto:[email-masked]]
> *Sent:* Thursday, January 19, 2012 12:39 PM
> *To:* Hemal Shah; Tom Talpey; [email-masked]; [email-masked]
> *Subject:* Re: iSER - one last issue****
>
>  ****
>
> For all RCaP layers, the Send message (or any variant) MUST be used, as
> opposed to using RDMA Read or Write to accomplish the task.  So a SHOULD
> for RCaP other than iWARP is incorrect.  RFC 5040 can mandate that all
> RDMAP messages be supported, but it is still up to implementation to decide
> what messages to use in any situation.  This iSER update is meant to
> reflect running code and to generalize the support of iSER over any RCaP,
> not just iWARP.  So the question is are there any running code that uses
> SendSE and SendInvSE for iSER.  Conversely, if the running code can get by
> using Send messages to accomplish the task, what is the point of mandating
> a MUST at this point for SendSE and SendInvSE?  Note that the receiver
> cannot completely rely on the sender for the Invalidate feature
> anyway.  The iSER layer at the initiator is required to explicitly
> invalidate the STag for bidirectional commands and abnormal completion of a
> command.  Even when automatic invalidation is used, the iSER layer is
> required to sanity checking the automatic invalidation.  In other words,
> the iSER spec puts the burden on the receiver for doing the right thing in
> handling the Send messages, and not relying on the sender for the mere
> convenience. ****
>
>  ****
>
> Mike****
>
> ----- Original Message ----- ****
>
> *From:* Hemal Shah <[email-masked]> ****
>
> *To:* Tom Talpey <[email-masked]> ; [email-masked] ;
> [email-masked] ; [email-masked] ****
>
> *Sent:* Thursday, January 19, 2012 8:44 AM****
>
> *Subject:* RE: iSER - one last issue****
>
>  ****
>
> *Tom,*****
>
> * *****
>
> *You are welcome! You are right about 7.3.4 as the only section mandating
> Send message.*****
>
> * *****
>
> *I think your concern about specifying the requirement for the other RCaP
> implementation is valid. I suggest we should word the requirements as below:
> *****
>
> * *****
>
> “If the RCaP layer is as specified in [RFC5040] and [RFC5041], the SendSE
> message MUST be used. For any other RCaP layer, the Send message SHOULD be
> used. ”****
>
>  ****
>
> *Hemal*****
>
> * *****
>
> *From:* Tom Talpey [mailto:[email-masked]]
> *Sent:* Thursday, January 19, 2012 8:09 AM
> *To:* Hemal Shah; [email-masked]; [email-masked];
> [email-masked]
> *Subject:* RE: iSER - one last issue****
>
>  ****
>
> Thanks for pointing out that sentence in RFC5046 7.3.4. I believe it is
> the only such stipulation.****
>
>  ****
>
> I’m ok with stating the requirement as RFC5046-over-RFC5040 compliance.
> But this new draft opens the door to alternative behaviors, therefore it
> needs to be stated more clearly. What’s the requirement when SendSE is not
> supported? And, how does the receiver know what to expect?****
>
>  ****
>
> In other words, if it’s proposed that the statement be:****
>
>                 “When the RCaP layer is as specified in [RFC5040] and
> [RFC5041], the SendSE message MUST be used.”****
>
>  ****
>
> … then what MUST be used when the RCaP is not? That statement needs to be
> made, too.****
>
>  ****
>
>  ****
>
> *From:* Hemal Shah [mailto:[email-masked]]
> *Sent:* Thursday, January 19, 2012 10:38 AM
> *To:* Tom Talpey; [email-masked]; [email-masked];
> [email-masked]
> *Subject:* RE: iSER - one last issue****
>
>  ****
>
> *Tom,*****
>
> * *****
>
> *RFC5046 is very clear about the use of Send Message Type for iSCSI
> control-type PDU. For specific iSCSI PDUs, Section 7 mandates the use of
> Send, SendSE, or SendSEInv. Section 7 of RFC5046 clearly specifies the use
> of specific Send Message Type for each iSCSI control-type PDUs (Login,
> Logout, Text request, Text response…). Section 7.3.4 mandates the use of
> plain-old Send also as needed. For example, see text below from 7.3.4.****
> *
>
> * *****
>
> For unsolicited data, if the F bit is set to 0 in a SCSI Data-out****
>
> PDU, the iSER layer at the initiator MUST use a Send Message to send****
>
> the SCSI Data-out PDU.****
>
>  ****
>
> *Regarding your comment in the second paragraph below, RFC5040
> expectation is all the RDMAP messages specified in RFC5040 are supported
> (if that is not the case then we cannot count on any of the other RDMAP
> messages to be supported). So, an implementation that does not support
> SendSE at the sender is not compliant with RFC5040. So, your second point
> does not apply.*****
>
> * *****
>
> *I hope that addresses your comments.*****
>
> * *****
>
> *Hemal*****
>
> * *****
>
> *From:* Tom Talpey [mailto:[email-masked]]
> *Sent:* Thursday, January 19, 2012 5:51 AM
> *To:* Hemal Shah; [email-masked]; [email-masked];
> [email-masked]
> *Subject:* RE: iSER - one last issue****
>
>  ****
>
> There is no mention of any distinction in RFC5046 section 1.4.2 nor in this
> draft’s section 2.4.2.****
>
>  ****
>
> Even with a MUST on the SendSE, the behavior is indeterminate, since it
> would still be contingent on support for SendSE at the **sender**. This
> critical exception voids any normative requirement, even over iWARP. The
> strongest statement that can be made is SHOULD.****
>
>  ****
>
> Tom.****
>
>  ****
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Hemal Shah
> *Sent:* Wednesday, January 18, 2012 6:56 PM
> *To:* [email-masked]; [email-masked]; [email-masked]
> *Subject:* Re: [storm] iSER - one last issue****
>
>  ****
>
> *David and Mike,*****
>
> * *****
>
> *I believe that the requirements should not be weakened and we should
> stick to the requirement in RFC5046.*****
>
> * *****
>
> *RFC5046 required the use of specific Send Message Type for valid reasons.
> *****
>
> * *****
>
> *In the case of requiring SendSE message use, the intent was to give a
> hint to the remote side to generate an event upon processing of SendSE
> message. For example, the SendSE is specifically carrying SCSI information
> that needs attention on the remote side e.g. SCSI command, Logout…*****
>
> * *****
>
> *In the case of requiring SendInvSE Message, the intent was to invalidate
> the STag used in the data transfer as well as inform the remote side to
> generate an event. For example, mandating the use of SendInvSE for the SCSI
> response PDU for a SCSI Read limits the exposure of STag used in SCSI Read
> data transfer and avoids the explicit invalidation of the STag at the
> initiator. Plus, it allows the target to generate an event on the initiator
> upon the completion of SCSI Read command.*****
>
> * *****
>
> *By relaxing RFC5046 requirements in the latest iSER draft, we will not
> only impact all iWARP based iSER implementations that rely on the use of
> specific Send Message Type for SCSI data transfers but also change the
> intended behavior of initiators and targets.*****
>
> * *****
>
> *For iWARP, I strongly suggest that the iSER draft does not relax the
> requirements specified in RFC5046.*****
>
> * *****
>
> *I hope that helps.*****
>
> * *****
>
> *Hemal    *****
>
> * *****
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of [email-masked]
> *Sent:* Wednesday, January 18, 2012 3:10 PM
> *To:* [email-masked]; [email-masked]
> *Subject:* Re: [storm] iSER - one last issue****
>
>  ****
>
> Mike,****
>
>  ****
>
> ****
> the intent of RFC 5046.  Here are some examples from RFC 5046’s text:****
>
>  ****
>
> 1.5.  SCSI Read Overview****
>
>  ****
>
>    The iSER Message is transferred to the target using a SendSE Message.**
> **
>
>  ****
>
>    The iSER layer at the target uses a SendInvSE Message to transfer the**
> **
>
>    SCSI Response PDU back to the iSER layer at the initiator.****
>
>  ****
>
> Similar language occurs in 1.6 SCSI Write Overview.****
>
>  ****
>
> 5.2.1.  Normal Connection Termination at the Initiator****
>
>  ****
>
>    The iSER layer at the initiator MUST****
>
>    use a SendSE Message to send the Logout Request PDU to the target.****
>
>  ****
>
> Similar language occurs in 5.2.2 for target side normal connection
> termination,****
>
> and more importantly in 7.3.1 SCSI Command:****
>
>  ****
>
>    The iSER layer at the initiator MUST send the SCSI command in a****
>
>    SendSE Message to the target.****
>
>  ****
>
> None of the quoted text permits use of Send instead of SendSE or SendInv**
> **
>
> instead of SendInvSE.****
>
>  ****
>
> ****
> weaker than “SHOULD”), and that sure looks like a change to RFC 5046.****
>
>  ****
>
> Are there good implementation-based reasons to weaken these requirements
> now?****
>
>  ****
>
> Does anyone else have a viewpoint on this topic?****
>
>  ****
>
> Thanks,
> --David****
>
>  ****
>
> *From:* Michael Ko [mailto:[email-masked]]
> *Sent:* Wednesday, January 18, 2012 2:22 PM
> *To:* Black, David; [email-masked]
> *Subject:* Re: iSER - one last issue****
>
>  ****
>
> David,****
>
>  ****
>
> Here is my rationale for using the lower case "should".****
>
>  ****
>
> The intent of RFC5046 is that the Send Message type must be used instead
> of RDMA Reads or Writes.  The Solicited Event feature of the Send Message
> is provided as a convenience.  The receiver must still do the right thing
> in handling the Send Message regardless of whether the SE feature is used.
> In other words, the sender is responsible for using the right format for
> the message (Send vs RDMA) but the receiver must not rely on the sender to
> determine how to handle the received message.  The same rationale goes for
> the Invalidate feature.  ****
>
>  ****
>
> Mike****
>
> ----- Original Message ----- ****
>
> *From:* [email-masked] ****
>
> *To:* [email-masked] ; [email-masked] ****
>
> *Sent:* Monday, January 16, 2012 2:06 PM****
>
> *Subject:* iSER - one last issue****
>
>  ****
>
> Mike,****
>
>  ****
>
> ****
> draft addresses all of the open issues, but I have a question for the WG**
> **
>
> about how to express the SendSE and SendInvSE requirements from RFC 5046.*
> ***
>
>  ****
>
> The -08 version of the iSER draft expresses requirements for the SendSE***
> *
>
> and SendInvSE messages (this is primarily for iSER over RDDP/iWARP) as:***
> *
>
>  ****
>
> The SendSE Message should be used if****
>
> supported by the RCaP layer (e.g., iWARP).****
>
>  ****
>
> ****
> reflect RFC 5046, I would replace “should” with “MUST” in the above text,*
> ***
>
> at least for iWARP.****
>
>  ****
>
> In the alternative, if the “should”s remain, an explanatory item****
>
> needs to be added to the Appendix A list of changes from RFC 5046.****
>
>  ****
>
> What do others think the right course of action is here, use “MUST” or****
>
> explain weakening of requirement to “should” ?****
>
>  ****
>
> Thanks,
> --David****
>
> ** **
>
> _______________________________________________
> storm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/storm
>
>

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Wed, 18 Apr 2012 14:20:56 -0700

A “hypothetical situation where an iSER initiator running on Infiniband talks to an iSER target running on iWARP “ scenario without an RDMA-aware gateway does sound like a true hypothetical situation to me, ☺

I am of the school that David summarized well before:
>interoperability is only a goal within a single RCaP, not across RCaPs

To Mike's point:
> To dictate that the iSER implementation supports different Send requirements would mean that it has to know what RDMA transport it is running on

This is not a major consideration. It should be fairly straightforward to abstract the RCaP semantic differences via implementation/platform APIs, as exposed to a local iSER layer.

After reading the thread again, I still think the right approach here is  a) leave related RFC 5046 MUST requirements unchanged for iWARP RCaP, b) specify the related SHOULD/MUST requirements for IB RCaP, c) allow other RCaPs in future to adopt reasonable/feasible approach on their own. This builds off David’s option-2a.

My 0.02.

Mallikarjun

From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Ko
Sent: Tuesday, April 17, 2012 9:13 PM
To: Hemal Shah
Cc: [email-masked]
Subject: Re: [storm] iSER - now what?

Since this iSER update is intended to document the “running code”, then the implementation currrently in existence should be the model upon which the iSER update should be based on.  That means the "MUST" requirement applies to Send, and not SendInv, or SendSE, or SendInvSE.  RFC 5046 already indicates that it is is the responsibility of the receiver to do the right thing for processing Send messages, and not relying on which flavor of Send messages the sender uses.  To dictate that the iSER implementation supports different Send requirements would mean that it has to know what RDMA transport it is running on.  As long as we are talking hypothetical situation here since iWARP implementation does not exist today, consider the hypothetical situation where an iSER initiator running on Infiniband talks to an iSER target running on iWARP, what flavors of Send messages are required?  IMHO, the usage model for SendInv, SendInvSE and SendSE as described in the latest iSER draft (not RFC 5046) is adequate and no changes are necessary; laying out extraneous requirements based on the underlying RDMA transport can lead to potential interoperability problems and open up a can of worms.  

Mike

Hemal 

 

Thanks,
--David (as WG co-chair)

 
 
Tom.

 
 
Hemal 

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Thursday, March 08, 2012 10:13 AM
To: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [storm] FW: iSER - one last issue: update

 
 
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, March 08, 2012 12:52 AM
To: [email-masked]; [email-masked]
Subject: Re: [storm] FW: iSER - one last issue: update
Importance: High
 
Alexander,
 
<WG chair hat off>

 

 

        

 
Thanks,
--David
 
From: Alexander Nezhinsky [mailto:[email-masked]] 
Sent: Sunday, March 04, 2012 3:05 PM
To: [email-masked]; Black, David; [email-masked]; [email-masked]; Mike Ko; Tom Talpey
Subject: Re: FW: [storm] iSER - one last issue: update
 
Hi

Alexander.

 
From: Hemal Shah [mailto:[email-masked]] 
Sent: Wednesday, January 25, 2012 6:48 PM
To: Tom Talpey; [email-masked]; [email-masked]; [email-masked]

 
 
Hemal
 
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Tuesday, January 24, 2012 7:50 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue: update

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, January 24, 2012 7:42 PM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue: update
 
[WG chair hat off]

 

 

Thanks,
--David
 
From: Mallikarjun Chadalapaka [mailto:[email-masked]] 
Sent: Tuesday, January 24, 2012 7:25 PM
To: Tom Talpey; Black, David; [email-masked]
Subject: RE: [storm] iSER - one last issue: update

Mallikarjun

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Tuesday, January 24, 2012 5:49 AM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue: update
 
[WG co-chair hat off]

 
 
Tom.
 
To: [email-masked]
 
[WG chair hat off]

 

 

 

 

 
[WG chair hat on]

Thanks,
--David
 
From: Tom Talpey [mailto:[email-masked]] 
Sent: Thursday, January 19, 2012 3:33 PM
To: Michael Ko; Hemal Shah; Black, David; [email-masked]
Subject: RE: iSER - one last issue

 
From: Michael Ko [mailto:[email-masked]] 
Sent: Thursday, January 19, 2012 12:39 PM
To: Hemal Shah; Tom Talpey; [email-masked]; [email-masked]
Subject: Re: iSER - one last issue

Mike
----- Original Message ----- 
From: Hemal Shah 
To: Tom Talpey ; [email-masked] ; [email-masked] ; [email-masked] 
Sent: Thursday, January 19, 2012 8:44 AM
Subject: RE: iSER - one last issue
 
Tom,

 
 
Hemal
 
From: Tom Talpey [mailto:[email-masked]] 
Sent: Thursday, January 19, 2012 8:09 AM
To: Hemal Shah; [email-masked]; [email-masked]; [email-masked]
Subject: RE: iSER - one last issue

 

From: Hemal Shah [mailto:[email-masked]] 
Sent: Thursday, January 19, 2012 10:38 AM
To: Tom Talpey; [email-masked]; [email-masked]; [email-masked]
Subject: RE: iSER - one last issue
 
Tom,

 

Hemal
 
From: Tom Talpey [mailto:[email-masked]] 
Sent: Thursday, January 19, 2012 5:51 AM
To: Hemal Shah; [email-masked]; [email-masked]; [email-masked]
Subject: RE: iSER - one last issue

 
Tom.
 
From: [email-masked] [mailto:[email-masked]] On Behalf Of Hemal Shah
Sent: Wednesday, January 18, 2012 6:56 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue
 
David and Mike,

 

 
 
I hope that helps.
 
Hemal    
 
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Wednesday, January 18, 2012 3:10 PM
To: [email-masked]; [email-masked]
Subject: Re: [storm] iSER - one last issue
 
Mike,

1.5.  SCSI Read Overview

 

 

 

 
Thanks,
--David
 
From: Michael Ko [mailto:[email-masked]] 
Sent: Wednesday, January 18, 2012 2:22 PM
To: Black, David; [email-masked]
Subject: Re: iSER - one last issue
 
David,

 
Mike
----- Original Message ----- 
From: [email-masked] 
To: [email-masked] ; [email-masked] 
Sent: Monday, January 16, 2012 2:06 PM
Subject: iSER - one last issue
 
Mike,

 

 
 
Thanks,
--David

---

From: Michael Ko <[email-masked]>
Date: Wed, 18 Apr 2012 14:24:46 -0700

Mike