Subject: [storm] iSER - another try

From: <[email-masked]>
Date: Mon, 23 Apr 2012 09:02:24 -0400

We don't seem to be making progress on definitive requirements that
vary by RCaP, so here's another suggestion (would be [4] if I were
numbering it ...)

It's clear that at least one implementation has diverged from the
requirements in RFC 5046 for use of SendSE, SendInv and SendInvSE.
I suggest documenting this and applying the IETF approach of "be
conservative in what you send, be liberal in what you accept",
roughly as follows:

i) Some iSER implementations have not followed RFC 5046's strict requirements
	for use of SendSE, SendInv and SendInvSE; they use Send instead.
ii) For interoperability, iSER implementations SHOULD accept and correctly
	process SendSE, SendInv and SendInvSE messages.
iii) SendSE, SendInv and SendInvSE should be regarded as optimizations or
	enhancements to the basic Send message, and their support varies by
	RCaP.  If these messages are used, the implementation SHOULD be
	capable of reverting to use of Send in order to work with a receiver
	that does not support these messages (lack of receiver support MUST
	result in an error send back to the sender).  These messages
	SHOULD NOT be used with the InfiniBand RCaP because InfiniBand does
	not require support for their functionality.
iv) New iSER implementations SHOULD use Send (and not these three additional
	messages) unless there are compelling reasons for doing otherwise
	(latter is implicit in use of "SHOULD", but worth saying explicitly,
	IMHO).
v) Some new text will be needed (including Security Considerations) to
	make it clear that STag invalidation is necessary for security
	reasons, and has to be performed independent of whether an
	Invalidate version of Send was used or not.

Is this workable?

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Michael Ko <[email-masked]>
Date: Mon, 23 Apr 2012 08:55:23 -0700

David,

I agree with this position.

Mike

On Mon, Apr 23, 2012 at 6:02 AM, <[email-masked]> wrote:

> _______________________________________________
> storm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/storm
>

---

From: Tom Talpey <[email-masked]>
Date: Mon, 23 Apr 2012 21:31:29 +0000

We need to separate the issues of Solicited Events from Invalidations.

I don't believe that the use of SendSE is somehow detectable with this approach. The conclusion from item ii) is that the iSER initiator SHOULD NOT arm for solicited events - it MUST be prepared for any event, whether solicited or not. This in turn implies that senders (targets) SHOULD NOT use SendSE at all - unless it's certain that *both* the RCaP and the initiator support them. This feature can neither be detected, nor is it negotiated by iSER.

The "invalidate" form is, however, sensible to allow after carefully documenting the requirement. The initiator MUST verify that the appropriate memory region was invalidated, if any. However, whether it MUST or SHOULD perform some additional action is going to be a tough choice. A MUST could be too strong of a requirement, if the implementation chooses to cache or share mappings. On the other hand, SHOULD can introduce security vulnerabilities. Thoughts?

Also,  isn't the following a new requirement on the RDMA layer?

>(lack of receiver support MUST result in an error send back to the sender).

From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Ko
Sent: Monday, April 23, 2012 11:55 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [storm] iSER - another try

David,

Mike
On Mon, Apr 23, 2012 at 6:02 AM, <[email-masked]<mailto:[email-masked]>> wrote:
We don't seem to be making progress on definitive requirements that
vary by RCaP, so here's another suggestion (would be [4] if I were
numbering it ...)

Is this workable?

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953<tel:%2B1%20%28508%29%20293-7953>             FAX: +1 (508) 293-7786<tel:%2B1%20%28508%29%20293-7786>
[email-masked]<mailto:[email-masked]>        Mobile: +1 (978) 394-7754<tel:%2B1%20%28978%29%20394-7754>
----------------------------------------------------

_______________________________________________
storm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/storm

---

From: <[email-masked]>
Date: Mon, 23 Apr 2012 17:52:16 -0400

> >(lack of receiver support MUST result in an error send back to the sender

For iWARP that should be an Unexpected Opcode error.  For other RCaPs, it should
be that or a "that won't work" error at the sender if the API is invocable.

Thanks,
--David

From: Tom Talpey [mailto:[email-masked]]
Sent: Monday, April 23, 2012 5:31 PM
To: Michael Ko; Black, David
Cc: [email-masked]
Subject: RE: [storm] iSER - another try

David,

Mike

Is this workable?

Thanks,
--David

---

From: <[email-masked]>
Date: Mon, 23 Apr 2012 17:58:59 -0400

Tom,

> The "invalidate" form is, however, sensible to allow after carefully documenting the requirement.
> The initiator MUST verify that the appropriate memory region was invalidated, if any. However,
> whether it MUST or SHOULD perform some additional action is going to be a tough choice. A MUST
> could be too strong of a requirement, if the implementation chooses to cache or share mappings.
> On the other hand, SHOULD can introduce security vulnerabilities. Thoughts?

I think a SHOULD requirement would be ok if text is added to the security considerations section
to discuss exposure of initiator memory via STags, the initiator's responsibility to manage that
exposure via STag invalidation, and the potential security consequences of lazy invalidation that
an initiator implementation may choose to trade off against performance benefits.  This would be
in addition to new security considerations text that warns that an initiator MUST NOT rely upon
responder use of "invalidate" forms of Send (your "MUST verify" stated as a "MUST NOT" requirement).

Thanks,
--David

David,

Mike

Is this workable?

Thanks,
--David

---

From: Tom Talpey <[email-masked]>
Date: Tue, 24 Apr 2012 14:02:55 +0000

Sure, but the error being proposed is generated by the receiver, and therefore if detected it will be a fatal error (closing the connection). That's a rather inconvenient way to negotiate the facility.

Also, my reading of the Infiniband spec section 9.2.3 is that the responder is free to ignore the Solicited Event bit, which means IB may not support the "that won't work" detection. In other words, IB allows the responder to translate SE to ordinary Send.

I would agree with the statement if it read "(lack of receiver support MAY result in an error sent back to the sender)". But would it be useful?

From: [email-masked] [mailto:[email-masked]]
Sent: Monday, April 23, 2012 5:52 PM
To: Tom Talpey
Cc: [email-masked]
Subject: Error behavior (RE: [storm] iSER - another try)

Thanks,
--David

From: Tom Talpey [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Monday, April 23, 2012 5:31 PM
To: Michael Ko; Black, David
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [storm] iSER - another try

>(lack of receiver support MUST result in an error send back to the sender).

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Michael Ko
Sent: Monday, April 23, 2012 11:55 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER - another try

David,

Mike

Is this workable?

Thanks,
--David

---

From: Tom Talpey <[email-masked]>
Date: Tue, 24 Apr 2012 14:08:32 +0000

SHOULD sounds good to me, with the proposed discussion.

The main point is that the invalidation behavior is optional for the sender (and even the RCaP), and therefore must be checked by the receiver. Fortunately, it is readily detectable and the requirements are well-understood.

From: [email-masked] [mailto:[email-masked]]
Sent: Monday, April 23, 2012 5:59 PM
To: Tom Talpey
Cc: [email-masked]
Subject: Invalidation (was RE: [storm] iSER - another try)

Tom,

Thanks,
--David

David,

Mike

Is this workable?

Thanks,
--David

---

From: <[email-masked]>
Date: Tue, 24 Apr 2012 10:31:21 -0400

> Sure, but the error being proposed is generated by the receiver, and therefore if detected it will
> be a fatal error (closing the connection). That's a rather inconvenient way to negotiate the facility.

Unfortunately, that's all we have.  If iWARP RDMAP implementations support the invalidate versions of
Send, I'd prefer to confine the "SHOULD NOT use" recommendation for SendInv to IB (we'd still need a
MUST requirement on initiators to ensure appropriate STag invalidation).

> Also, my reading of the Infiniband spec section 9.2.3 is that the responder is free to ignore
> the Solicited Event bit, which means IB may not support the "that won't work" detection. In
> other words, IB allows the responder to translate SE to ordinary Send.

Now that's interesting ... RDMAP should return an error in that case.  Based on the IB behavior,
I'm inclined to agree with your "SHOULD NOT use" suggestion for the two solicited event versions
of send (SendSE and SendInvSE).

> I would agree with the statement if it read "(lack of receiver support MAY result in an error
> sent back to the sender)". But would it be useful?

Ah, I see the point now ... I think you're right that explaining the possible error behavior (MAY)
without requiring it is the better approach.

Thanks,
--David

From: Tom Talpey [mailto:[email-masked]]
Sent: Tuesday, April 24, 2012 10:03 AM
To: Black, David
Cc: [email-masked]
Subject: RE: Error behavior (RE: [storm] iSER - another try)

Thanks,
--David

David,

Mike

Is this workable?

Thanks,
--David