Subject: [storm] RDMAP opcodes - proposal

From: <[email-masked]>
Date: Tue, 10 Jan 2012 20:54:26 -0500

After thinking about the RDMAP opcode situation over the weekend, I have a proposed course of action.  Please comment, as I would like to move on this fairly quickly in order to get both the MPA Peer Connect and RDMAP registries drafts approved by the IESG.

There are 8 currently unused RDMAP opcodes.  The RDMAP extensions draft will use 4 of them, and setting aside 1 more for test/experimentation leaves 3.  I'm not comfortable committing to a course of action that consumes more than half of the unused opcodes without taking a good hard look at the alternatives and consequences.  Hence I propose to do the following:

(1) Defer allocation of text/experimentation RDMAP opcode to RDMAP extensions draft, which is already going to allocate RDMAP opcodes.

(2) As part of discussion of the RDMAP extensions draft, consider the 2 reserved (and currently unused) bits that are adjacent to the RDMAP opcode field.  Although any use of those bits requires modifying RFC 5040 (RDMAP), that's a reasonable thing for the RDMAP extensions draft to do *if* it won't cause serious implementation problems.

(3) As part of the discussion of the reserved bits, the following questions should be addressed:
- Does the RDMAP extensions draft need 4 opcodes?  For example, would 2 opcodes
	plus use of one of the reserved bits as a direction bit (Request/Response)
	for those two opcodes be a better approach?
- Should 4 or 8 of the currently unused 4-bit RDMAP opcodes be turned into 6-bit
	opcodes by including the two reserved bits?

Implementation concerns matter to this discussion, as they may constrain practical use of those 2 reserved bits.  By deferring allocation of the test/experimentation opcode, we keep our options open for this discussion.

Does this sound like a reasonable plan?

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: "Hemal Shah" <[email-masked]>
Date: Thu, 12 Jan 2012 21:59:44 +0000

David,

I'm not in agreement with your proposal below. Even though we have 4 opcodes remaining, I'm not comfortable with making protocol changes in the anticipation of running out of opcodes in the future. I would rather make protocol changes when we run out of opcodes.

Regarding (1), I do not think we should defer the allocation of RDMA opcodes for RDMA protocol extensions draft. I see RDMA protocol extensions as enhancements to RFC5040. RDMA opcodes for RDMAP extensions should be viewed as an integral part of RDMAP.

Regarding (2), the use of reserved bits for RDMAP extensions draft will not make RDMA protocol extensions draft compatible with RFC5040. It will cause implementation problems. For example, an RFC5040 compliant implementation (that will ignore reserved bits on receive) can misinterpret new opcodes defined in RDMA protocol extensions draft as one of the valid opcodes defined in RFC5040.

Regarding (3), RDMA protocol extensions draft needs 4 opcodes. Atomic Request/Response having separate opcodes for request and response is consistent with RDMA messages and opcodes definition in RFC5040. We have separate message opcodes for RDMA read request and RDMA read response in RFC5040. I would like to keep the opcodes for request/response separate and consistent in both RFC5040 and RDMA protocol extensions draft.

When we run out of opcodes in the future, we can revise the protocol version and increase the size of RDMA opcode field as needed. Doing it now is not necessary and can introduce implementation problems.

I also think we should go ahead and request allocation of RDMAP opcodes for the RDMAP extensions.

Regards,

Hemal    

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, January 10, 2012 5:54 PM
To: [email-masked]
Subject: [storm] RDMAP opcodes - proposal
Importance: High

Thanks,
--David

_______________________________________________
storm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/storm

---

From: <[email-masked]>
Date: Thu, 12 Jan 2012 17:50:11 -0500

Hemal,

Thanks for the response - your viewpoints are helpful, especially the implementation-based comment that extending the opcode field beyond the 4-bits specified in RFC 5040 would be problematic.

I do have one clarification:

> Regarding (1), I do not think we should defer the allocation of RDMA opcodes for RDMA protocol
> extensions draft.

> I also think we should go ahead and request allocation of RDMAP opcodes for the RDMAP extensions.

That proposed course of action, adding RDMAP extensions opcodes to the RDDP registries draft, will result in a normative reference to the RDMAP extensions draft, and that reference will delay RFC publication of not only the RDDP registries draft, but also the MPA peer connect draft until the RDMAP extensions draft catches up with them.

I don't think imposing that delay is a good idea, as the RDMAP extensions draft can allocate and register its own opcodes 0x8-0xB - the IANA Considerations text in Section 10 of the -02 version of that draft will do that.  In practice, these four opcodes are implicitly reserved for the RDMAP extension operations, as there are no other proposals to use those opcodes, and any such proposal would need to come through the storm WG.

> Regarding (2), the use of reserved bits for RDMAP extensions draft will not make RDMA protocol
> extensions draft compatible with RFC5040. It will cause implementation problems.

> Regarding (3), RDMA protocol extensions draft needs 4 opcodes. Atomic Request/Response having separate
> opcodes for request and response is consistent with RDMA messages and opcodes definition in RFC5040.

> When we run out of opcodes in the future, we can revise the protocol version and increase the size of
> RDMA opcode field as needed. Doing it now is not necessary and can introduce implementation problems.

This would lead to a plan to use the RDDP registries draft to register opcode 0xF for test and experimentation now, and to use the RDMAP extensions draft to register 0x8-0xB for atomic and immediate operations when it is approved, leaving 3 unassigned RDMAP opcodes, 0xC-0xE, for possible future use.

I'll wait until early next week to see if anyone else has comments before submitting a revised version of the RDDP registries draft that allocates RDMAP opcode 0xF for test and experimentation, but I would like to get the revised RDDP registries draft submitted soon, as this opcode issue is the last item that is blocking IESG approval of the MPA peer connect draft.

Thanks,
--David

> From: Hemal Shah [mailto:[email-masked]]
> Sent: Thursday, January 12, 2012 5:00 PM
> To: Black, David; [email-masked]
> Subject: RE: RDMAP opcodes - proposal
> 
> David,
> 
> I'm not in agreement with your proposal below. Even though we have 4 opcodes remaining, I'm not
> comfortable with making protocol changes in the anticipation of running out of opcodes in the future.
> I would rather make protocol changes when we run out of opcodes.
> 
> Regarding (1), I do not think we should defer the allocation of RDMA opcodes for RDMA protocol
> extensions draft. I see RDMA protocol extensions as enhancements to RFC5040. RDMA opcodes for RDMAP
> extensions should be viewed as an integral part of RDMAP.
> 
> Regarding (2), the use of reserved bits for RDMAP extensions draft will not make RDMA protocol
> extensions draft compatible with RFC5040. It will cause implementation problems. For example, an
> RFC5040 compliant implementation (that will ignore reserved bits on receive) can misinterpret new
> opcodes defined in RDMA protocol extensions draft as one of the valid opcodes defined in RFC5040.
> 
> Regarding (3), RDMA protocol extensions draft needs 4 opcodes. Atomic Request/Response having separate
> opcodes for request and response is consistent with RDMA messages and opcodes definition in RFC5040.
> We have separate message opcodes for RDMA read request and RDMA read response in RFC5040. I would like
> to keep the opcodes for request/response separate and consistent in both RFC5040 and RDMA protocol
> extensions draft.
> 
> When we run out of opcodes in the future, we can revise the protocol version and increase the size of
> RDMA opcode field as needed. Doing it now is not necessary and can introduce implementation problems.
> 
> After thinking about the RDMAP opcode situation over the weekend, I have a proposed course of action.
> Please comment, as I would like to move on this fairly quickly in order to get both the MPA Peer
> Connect and RDMAP registries drafts approved by the IESG.
> 
> There are 8 currently unused RDMAP opcodes.  The RDMAP extensions draft will use 4 of them, and
> setting aside 1 more for test/experimentation leaves 3.  I'm not comfortable committing to a course of
> action that consumes more than half of the unused opcodes without taking a good hard look at the
> alternatives and consequences.  Hence I propose to do the following:
> 
> (2) As part of discussion of the RDMAP extensions draft, consider the 2 reserved (and currently
> unused) bits that are adjacent to the RDMAP opcode field.  Although any use of those bits requires
> modifying RFC 5040 (RDMAP), that's a reasonable thing for the RDMAP extensions draft to do *if* it
> won't cause serious implementation problems.
> 
> Implementation concerns matter to this discussion, as they may constrain practical use of those 2
> reserved bits.  By deferring allocation of the test/experimentation opcode, we keep our options open
> for this discussion.
>

---

From: "Hemal Shah" <[email-masked]>
Date: Fri, 13 Jan 2012 18:02:14 +0000

David,

Thanks for the clarification! I'm in agreement with you for not delaying RDDP registries draft for the RFC publication of RDMAP extensions draft. The RDMAP extensions draft -02 already has text in IANA considerations section requesting its own RDMAP op codes.

I'm also fine with reserving 0xF opcode in RDDP registries draft for test and experimentation.

Regards,

Hemal 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, January 12, 2012 2:50 PM
To: Hemal Shah; [email-masked]
Subject: RE: RDMAP opcodes - proposal

Hemal,

Thanks,
--David