Subject: [storm] comments on http://tools.ietf.org/html/draft-ietf-storm-rdmap-ext-00

From: arkady kanevsky <[email-masked]>
Date: Tue, 3 May 2011 12:42:24 -0400

Here are a few comments on the draft:
In Glossary it is stated:

Atomic Operation - is an operation that results in an execution of a
   64-bit operation at a specific address on a remote node. The
   consumer can use atomic operations to read, modify and write at the
   destination address while at the same time guarantee that no other
   read or write operation will occur across any other RDMAP/DDP
   Streams on an RNIC at the Data Sink.

later in section 5 it is stated:
The
   operations atomically read, modify and write back the contents of
   the destination address and guarantee that atomic operations on this
   address by other Queue Pairs (QPs) on the same RNIC do not occur
   between the read and the write.

in
5.3. Atomicity Guarantees   Atomicity of the RMW on the responder's
node by the Atomic Operation
   SHALL be assured in the presence of concurrent atomic accesses by
   other QPs on the same RNIC.

The three statements are very different. The cases that need to be handled are:
other operation of the same QP, the same RNIC, other devices, local ULP access.

2. It is stated:

The discovery of whether the
   atomic operations are implemented or not is outside the scope of
   this specification and it should be handled by the ULPs or
   applications.

Is this really the best way? Should the connection setup extension draft
which I submitted be extended to exchange this info also? That is which
version of RDMA protocol the remote side supports or requests to support.
The reason I am asking it it not clear to me what option does ULP have if
atomic or immediate data is not supported.

3. What is the Endian(ness) format of Add Data for fetchadd? ditto for
cmpswp.

4. Is there a separate memory registration (Stag creation) for atomic
operation? Or any registered memory can be used for atomic op?

5. How does the requester knows that Stag+remote tagged offset does
corresponds to  a naturally aligned buffer address?

6. "The Swap atomic operation result is unknown when the buffer address is
not naturally aligned." feels a bit loose. I would assume that RNIC will at
least not change any data outside memory specified by Stag.
Also in 5.2.1 #4 states:

At the Remote Peer, when an invalid Atomic Operation Request
      Message is delivered to the Remote Peer's RDMAP layer, an error
      is surfaced.

It sounds like breaking natural alignment is not an error. That is it will
break the connection.

Also:

6. At the Remote Peer, when an invalid Atomic Operation Response
      Message is delivered to the Remote Peer's RDMAP layer, an error
      is surfaced.

Does the error carries the request identifier?

7. Any guidance for sizing ORD/IRD for atomic ops? Does this doc requires my
doc with ORD/IRD negotiation at connect time?

8. If the same memory has two Stag (overlapping memory regions) is atomic op
ordering only preserved for a single QP/connection? What happens if natural
alignment is less then 64-bit? Is there any order guaranteed per Stag (if
two QPs use the same Stag for atomic or RDMA ops)?

9. It is strange that for atomic op we are using 64-bit terminology but for
immediate data we use 8-byte terminology.

10.Suggest to specify which queues and buffers immediate data message uses?

Arkady

-- 
Cheers,
Arkady Kanevsky

---

From: "Sharp, Robert O" <[email-masked]>
Date: Wed, 4 May 2011 15:06:18 -0700

Hi Arkady,

Thanks for the comments!  Responses below...

Thanks,
Bob

> -----Original Message-----
> From: arkady kanevsky [mailto:[email-masked]]
> Sent: Tuesday, May 03, 2011 11:42 AM
> To: [email-masked]; Sharp, Robert O
> Subject: comments on http://tools.ietf.org/html/draft-ietf-storm-rdmap-
> ext-00
> 
> 5.3. Atomicity Guarantees
>    Atomicity of the RMW on the responder's node by the Atomic Operation

The intention of the draft is that atomicity is guaranteed on the same or 
different RDMAP/DDP Streams on a single RNIC.  The atomicity guarantees 
apply to local or remote ULP access as long as both are strictly accessing 
the memory referenced by the atomic operation through the same RNIC.
Since this is a wire protocol specification, the ULP access description will
be considered out of scope. There is clearly an inconsistency in terminology 
here when QP was used.  One clarification that we will make is to replace 
the QP term with RDMAP/DDP Streams throughout the document.  I believe that
the sections in the draft are consistent with this description, but if you
see something specific that could be made more clear, we'll be happy to 
address further input.  We can certainly add a statement that says that
there are no guarantees between different RNICs or other devices if that 
would help.

> Is this really the best way? Should the connection setup extension
> draft which I submitted be extended to exchange this info also? That is
> which version of RDMA protocol the remote side supports or requests to
> support. The reason I am asking it it not clear to me what option does
> ULP have if atomic or immediate data is not supported.
> 

For this draft the statement is appropriate, however we certainly would
be interested in building better support into the MPA to exchange
the enhanced capabilities between peers at connection setup.  I assume
that this would be future revision to the MPA draft at this point in
time.  Do you agree?

> 3. What is the Endian(ness) format of Add Data for fetchadd? ditto for

The wire format (Big Endian) and the conversion to the appropriate 
Endian(ness) for the host is specified in sections 5.1.1 and 5.1.3.

> operation? Or any registered memory can be used for atomic op?

In practice, applications will likely use separate STags for atomic
operations, but there is not a requirement to do so.  Any registered
memory can be used for atomic operations.

> corresponds to  a naturally aligned buffer address?

Applications must exchange STags and tagged offsets in order to expose
the target of an atomic operation in a application specific way.  It is 
the responsibility of the application that is exposing the memory to 
ensure that the buffer is naturally aligned.  The requestor must trust
the peer to have properly aligned an exposed target of an atomic 
operation. If the target exposed an improperly aligned STag and tagged 
offset, an error will be surfaced by the RNIC.

> 6. "The Swap atomic operation result is unknown when the buffer address
> is not naturally aligned." feels a bit loose. I would assume that RNIC
> will at least not change any data outside memory specified by Stag.
> Also in 5.2.1 #4 states:
> It sounds like breaking natural alignment is not an error. That is it
> will break the connection.
> 

There is room for improvement here as you have pointed out.  We will add
a better description that indicates that the responder RNIC will prevent 
atomic operation access to a memory location that is not naturally aligned.  
This will be an error that is always surfaced.  The result will be a 
terminate message back to the requester.  We will add a description of
the terminate semantics to the draft.

> Does the error carries the request identifier?

The terminate message generated will carry the RDMAP header of the 
offending atomic operation request which includes the request identifier.
This will be called out in the new description of the terminate semantics.

> 
> 7. Any guidance for sizing ORD/IRD for atomic ops? Does this doc
> requires my doc with ORD/IRD negotiation at connect time?
> 

Sizing ORD/IRD is out of scope for this wire format level of draft since
the application is responsible for determining the acceptable requirements.
As with the new capabilities related to this draft, it would be good to work
this in with new MPA enhancements.

> 8. If the same memory has two Stag (overlapping memory regions) is
> atomic op ordering only preserved for a single QP/connection? 

STags do not have an impact on the atomic guarantees. Atomicity is
guaranteed within the scope of an RNIC across all QPs supported by the
RNIC without respect to the STag used to reference a memory location 
accessed by the atomic operation.

> What
> happens if natural alignment is less then 64-bit? Is there any order
> guaranteed per Stag (if two QPs use the same Stag for atomic or RDMA
> ops)?
> 

Operations that violate the natural alignment assumption will be terminated
by the responder before the operation is performed per the changes 
discussed above.

> 9. It is strange that for atomic op we are using 64-bit terminology but
> for immediate data we use 8-byte terminology.
> 

We intentionally used the separate terminology.  We used the "byte"
terminology to discuss data payload and the "bit" terminology to discuss
fields within the data payload.  I believe that we were consistent but if
you see something that should be clarified, we'll be happy to make the
correction.

> 10.Suggest to specify which queues and buffers immediate data message

The queue number (qn=0) and data buffers (untagged buffers) are specified
in section 6.3.  If additional clarifications would help, please let us know.