Subject: Re: [storm] iSER draft

From: Tom Talpey <[email-masked]>
Date: Wed, 13 Jul 2011 15:12:56 +0000

Yes, understood on the ZBVA/Infiniband issue. Regarding the VA term’s presence in an earlier draft, I did not see it in RFC5046 and I did not review the expired draft, so consider these comments to be on the overall change, not the last revision.

Let me try a different approach to perhaps make myself clearer.

First, it seems to me that a “virtual address” has no real meaning to the network protocol. It begins, perhaps, as some value in an address space on the initiator, but it’s not meaningful on the target in this way, it’s only used to request that the remote RNIC perform a transfer to that originally registered remote address. So calling it a Virtual Address as an iSER protocol object seems, to me, an artificial and somewhat leading convention.

Second, the Virtual Address goes out from the initiator to the target in a Control PDU, but where does it come back? The RDMA Read or Write as depicted in (xx) shows only a Tagged Offset. So, it’s not clear what its protocol meaning is.

Third, I don’t ever see a Tagged Buffer described by a fully qualified four-tuple. I see it appearing as either { Stag, TO, length } or { Stag, VA, length }, depending on the addressing mode.

I think the non-ZBVA mode is really just a special case of the existing one, but where the meaning of TO has changed from a small offset to some other token, generated and managed only at the initiator. So, it seems artificial to define it as distinct, and document it as possessing some new properties. Isn’t it just a Target Offset, still?

Tom.

From: Alexander Nezhinsky [mailto:[email-masked]]
Sent: Thursday, June 30, 2011 2:08 PM
To: Michael Ko
Cc: Tom Talpey; [email-masked]
Subject: Re: [storm] iSER draft

On Thu, Jun 30, 2011 at 12:26 AM, Michael Ko <[email-masked]<mailto:[email-masked]>> wrote:
Tom,

As originally written, the iSER spec always assumes ZBVA (zero-based virtual address).  But when iSER is used in conjunction with transports other than iWARP, there is a need to include the virtual address for the tagged buffer.  Such is the case for Infiniband.  Accordingly, the Virtual Address field has been added to the iSCSI Control Type PDUs (sec. 9.2).  Alex pointed out that the tagged buffer should now be identified by the 4-tuple (STag, VA, TO, and length) instead.  This should be updated everywhere where the term "tagged buffer" is mentioned in the spec.  For example, in sec. 1.1 under Advertisement, the sentence "A typical method would be for the iSER Layer to embed the Tagged Buffer's STag, TO, and buffer length in a Send Message destined for the remote iSER Layer" should be modified to "A typical method would be for the iSER Layer to embed the Tagged Buffer's STag, VA, TO, and buffer length in a Send Message destined for the remote iSER Layer."  BTW, this sentence also answers your question as to whether VA is passed on the wire.

Agree, using VA was in the draft already, I only suggested to clarify its use (and modes of use) from the level of definitions and up,
so that it won't "suddenly" appear in the header format description section.

The Virtual Address is literally the virtual address of the tagged buffer.  The tagged offset is the displacement relative to the beginning of the buffer starting at the virtual address.

We could rephrase the definition of Virtual Address in sec. 1.1 as follows:

"Virtual Address (VA) - This is the virtual address of the Tagged Buffer on a Node.  When set to 0, it indicates that a virtual address is not needed to locate the Tagged Buffer."

Read and write buffers are identified by the combination of STag and VA.

OK. I tried to make the same point in the following:
Both fields should be required for any transport. This requirement does not preclude
using ZBVA where possible. When VA is not used (as in iWARP) it is set to 0.

Below is the actually used format:

struct iser_hdr {
  uint8_t   flags;
  uint8_t   rsvd[3];
  uint32_t  write_stag; /* write rkey in IB */
  uint64_t  write_va;
  uint32_t  read_stag;  /* read rkey in IB */
  uint64_t  read_va;
}
I would like to add that InfiniBand does have ZBVA mode. It supposedly provides less than optimal performance with IB.
And it is not exported through the current rdma-cm API, which supports RDMA/Ethernet as well, and is the only option
to use RDMA in user space linux apps, as of today.

So you could look at the additional fields as a kind of "reserved" in case of standard ZBVA implementation,
used to practically support certain APIs and get enhanced performance in some cases.

Alex

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 13 Jul 2011 21:49:32 -0700

> I think the non-ZBVA mode is really just a special case of the existing one, but where the meaning of TO has changed from a small offset to some other token, generated and managed only at the initi…

I agree, so far we have not seen a protocol justification for the need to add "Virtual Address" to the glossary as something distinct from Target Offset
for the purpose of defining an IETF protocol.

I am sympathetic to the interoperability issues raised, but I don't think those can justify something that has *no* justification in the protocol.

If someone could site a class of implementation where there is a real need for this distinction in an iSER adapter, but as far as I can see the adapters have
to be able to translate to TO *anyway* in order to use an RDMA Write or RDMA Read.

Local Interface compatibility with IB can make a *lot* of sense, but why does it have to make its way into the *wire* protocol?

--
Caitlin Bestler
[email-masked]
http://www.asomi.com/CaitlinBestlerResume.html

---

From: Michael Ko <[email-masked]>
Date: Thu, 14 Jul 2011 11:38:36 -0700

When "Virtual Address" is not used, as in iWARP, the initiator side needs to translate the "Tagged Offset" which is just the offset into the buffer, into a usable address by adding it to the starting base address of the buffer.  If the starting base address is communicated to the target side, as is done in some RCaPs, then the "Tagged Offset" is the usable address itself at the initiator where data is to be fetched or stored.  Alex suggested that perhaps we can change the name "Virtual Address" to "Steering Address".  Then it can be defined in the glossary without tripping over the common term "virtual address".

Mike

---

From: Tom Talpey <[email-masked]>
Date: Thu, 14 Jul 2011 20:16:15 +0000

Changing the name doesn't address the core issues here. There are no processing rules defined in the draft to support the arithmetic you describe below, or even to use the Steering/Virtual Address.

For example, section 7.3.5 states the target MUST use the Buffer Offset of the Data-In PDU as the Tagged Offset of an RDMA Write. Where does the new Steering Address in the control PDU header get folded into the Tagged Offset? And what protocol state would trigger the target to do so, including perform the arithmetic? Same question for several other sections.

From: Michael Ko [mailto:[email-masked]]
Sent: Thursday, July 14, 2011 2:39 PM
To: Caitlin Bestler; Tom Talpey
Cc: Alexander Nezhinsky; [email-masked]
Subject: Re: [storm] iSER draft

Mike
----- Original Message -----
From: Caitlin Bestler<mailto:[email-masked]>
To: Tom Talpey<mailto:[email-masked]>
Cc: Alexander Nezhinsky<mailto:[email-masked]> ; Michael Ko<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Wednesday, July 13, 2011 9:49 PM
Subject: Re: [storm] iSER draft

On Jul 13, 2011, at 8:12 AM, Tom Talpey wrote:

> Yes, understood on the ZBVA/Infiniband issue. Regarding the VA term's presence in an earlier draft, I did not see it in RFC5046 and I did not review the expired draft, so consider these comments to be on the overall change, not the last revision.
>
> First, it seems to me that a "virtual address" has no real meaning to the network protocol. It begins, perhaps, as some value in an address space on the initiator, but it's not meaningful on the target in this way, it's only used to request that the remote RNIC perform a transfer to that originally registered remote address. So calling it a Virtual Address as an iSER protocol object seems, to me, an artificial and somewhat leading convention.
>
> Second, the Virtual Address goes out from the initiator to the target in a Control PDU, but where does it come back? The RDMA Read or Write as depicted in (xx) shows only a Tagged Offset. So, it's not clear what its protocol meaning is.
>
> Third, I don't ever see a Tagged Buffer described by a fully qualified four-tuple. I see it appearing as either { Stag, TO, length } or { Stag, VA, length }, depending on the addressing mode.
>
> I think the non-ZBVA mode is really just a special case of the existing one, but where the meaning of TO has changed from a small offset to some other token, generated and managed only at the initiator. So, it seems artificial to define it as distinct, and document it as possessing some new properties. Isn't it just a Target Offset, still?
>
> Tom.
>
>

--
Caitlin Bestler
[email-masked]<mailto:[email-masked]>
http://www.asomi.com/CaitlinBestlerResume.html

---

From: Michael Ko <[email-masked]>
Date: Thu, 14 Jul 2011 13:53:08 -0700

For clarity, I need to update in the next revision of the spec everywhere where "Tagged Offset", "Steering Tag", etc. are mentioned to specify how the Tagged Offset is computed from the "Steering Address" and the buffer offset. 

So for example, in the example you cited in section 7.3.5, the target computes the Tagged Offset using the Steering Address and the Buffer Offset.  For iWARP, the Steering Address is zero, and so the Tagged Offset is the Buffer Offset in the SCSI Data-in PDU as defined in RFC 5046.

Mike
  ----- Original Message ----- 
  From: Tom Talpey 
  To: Michael Ko ; Caitlin Bestler 
  Cc: Alexander Nezhinsky ; [email-masked] 
  Sent: Thursday, July 14, 2011 1:16 PM
  Subject: RE: [storm] iSER draft

   

  Mike

    ----- Original Message ----- 

    From: Caitlin Bestler 

    To: Tom Talpey

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Thu, 14 Jul 2011 18:26:15 -0700

I suggest we have to be explicit about the responsibilities of each protocol
layer because the phrase “initiator side needs to translate the Tagged
Offset" sounds vague to me.  Also, it may be worth formalizing the notion of
"if Virtual Address was used for this command" via a negotiated connection
attribute. 

If I followed this thread right, it seems like we are talking roughly about
the following:

1) A shared iSCSI/iSER-level connection attribute (connection-scoped text
key?) that indicates whether Steering Address ("Address") or Tagged Offset
("Offset") is required by the RCaP in the RDMA requests on the wire for that
connection.  How the RCaP API exposes this capability to its local iSER
layer is implementation-dependent, and we do not need spec it.

2) If the connection attribute = "Offset:, 
	a) the semantic requirement on the RCaP layer on the initiator is
that it treats the incoming RDMA memory references as offsets, and does the
appropriate base+offset local translation for RDMA Reads and RDMA Writes.   
	b) The initiator iSER layer must only use ZB-VA, and must set the
Steering Address to 0 in appropriate control-type PDUs. 
	c) The target iSER layer must assume that zero TO for the advertised
STag points to the beginning of the initiator I/O Buffer in all the RDMA
Writes and RDMA Reads that it issues (same as now).

3) If the connection attribute = "Address":
	a)  the semantic requirement on the RCaP layer on the initiator is
that it treats incoming RDMA memory references as complete Steering
Addresses in RDMA Reads and RDMA Writes.  
	b) The initiator iSER layer must use non-ZB-VA, and must set the
Steering Address to the RCaP-API-returned Virtual Address in appropriate
control-type PDUs.
	c) The target iSER layer must assume that the Steering Address for
the advertised STag points to the beginning of the initiator I/O Buffer, and
compute the Tagged Offset in all the RDMA Writes and RDMA Reads that it
issues by adding the Steering Address to the Buffer Offset received from the
Data_Descriptor of Datamover API.

Mallikarjun

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Michael Ko
Sent: Thursday, July 14, 2011 1:53 PM
To: Tom Talpey; Caitlin Bestler
Cc: [email-masked]
Subject: Re: [storm] iSER draft

For clarity, I need to update in the next revision of the spec everywhere
where "Tagged Offset", "Steering Tag", etc. are mentioned to specify how the
Tagged Offset is computed from the "Steering Address" and the buffer
offset. 
 
So for example, in the example you cited in section 7.3.5, the target
computes the Tagged Offset using the Steering Address and the Buffer
Offset.  For iWARP, the Steering Address is zero, and so the Tagged Offset
is the Buffer Offset in the SCSI Data-in PDU as defined in RFC 5046.
 
Mike
----- Original Message ----- 
From: Tom Talpey 
To: Michael Ko ; Caitlin Bestler 
Cc: Alexander Nezhinsky ; [email-masked] 
Sent: Thursday, July 14, 2011 1:16 PM
Subject: RE: [storm] iSER draft

Changing the name doesn’t address the core issues here. There are no
processing rules defined in the draft to support the arithmetic you describe
below, or even to use the Steering/Virtual Address.

For example, section 7.3.5 states the target MUST use the Buffer Offset of
the Data-In PDU as the Tagged Offset of an RDMA Write. Where does the new
Steering Address in the control PDU header get folded into the Tagged
Offset? And what protocol state would trigger the target to do so, including
perform the arithmetic? Same question for several other sections.

When "Virtual Address" is not used, as in iWARP, the initiator side needs to
translate the "Tagged Offset" which is just the offset into the buffer, into
a usable address by adding it to the starting base address of the buffer. 
If the starting base address is communicated to the target side, as is done
in some RCaPs, then the "Tagged Offset" is the usable address itself at the
initiator where data is to be fetched or stored.  Alex suggested that
perhaps we can change the name "Virtual Address" to "Steering Address". 
Then it can be defined in the glossary without tripping over the common term
"virtual address".
 
Mike
----- Original Message ----- 
From: Caitlin Bestler 
To: Tom Talpey 
Cc: Alexander Nezhinsky ; Michael Ko ; [email-masked] 
Sent: Wednesday, July 13, 2011 9:49 PM
Subject: Re: [storm] iSER draft

On Jul 13, 2011, at 8:12 AM, Tom Talpey wrote:

> Yes, understood on the ZBVA/Infiniband issue. Regarding the VA term’s
presence in an earlier draft, I did not see it in RFC5046 and I did not
review the expired draft, so consider these comments to be on the overall
change, not the last revision.
> First, it seems to me that a “virtual address” has no real meaning to the
network protocol. It begins, perhaps, as some value in an address space on
the initiator, but it’s not meaningful on the target in this way, it’s only
used to request that the remote RNIC perform a transfer to that originally
registered remote address. So calling it a Virtual Address as an iSER
protocol object seems, to me, an artificial and somewhat leading convention.
>  
> Second, the Virtual Address goes out from the initiator to the target in a
Control PDU, but where does it come back? The RDMA Read or Write as depicted
in (xx) shows only a Tagged Offset. So, it’s not clear what its protocol
meaning is.
>  
> Third, I don’t ever see a Tagged Buffer described by a fully qualified
four-tuple. I see it appearing as either { Stag, TO, length } or { Stag, VA,
length }, depending on the addressing mode.
>  
> I think the non-ZBVA mode is really just a special case of the existing
one, but where the meaning of TO has changed from a small offset to some
other token, generated and managed only at the initiator. So, it seems
artificial to define it as distinct, and document it as possessing some new
properties. Isn’t it just a Target Offset, still?
> Tom.

I agree, so far we have not seen a protocol justification for the need to
add "Virtual Address" to the glossary as something distinct from Target
Offset
for the purpose of defining an IETF protocol.

I am sympathetic to the interoperability issues raised, but I don't think
those can justify something that has *no* justification in the protocol.

If someone could site a class of implementation where there is a real need
for this distinction in an iSER adapter, but as far as I can see the
adapters have
to be able to translate to TO *anyway* in order to use an RDMA Write or RDMA
Read.

Local Interface compatibility with IB can make a *lot* of sense, but why
does it have to make its way into the *wire* protocol?

---

From: Michael Ko <[email-masked]>
Date: Thu, 14 Jul 2011 18:50:12 -0700

Mallikarjun,

Nice to hear from you!

I have embedded my comments in your response.

Mike
  ----- Original Message ----- 
  From: Mallikarjun Chadalapaka 
  To: 'Michael Ko' ; Tom Talpey ; 'Caitlin Bestler' 
  Cc: [email-masked] 
  Sent: Thursday, July 14, 2011 6:26 PM
  Subject: RE: [storm] iSER draft

  I suggest we have to be explicit about the responsibilities of each protocol
  layer because the phrase "initiator side needs to translate the Tagged
  Offset" sounds vague to me.  

  [mk] That statement is only meant to convey what would be done by the initiator if it uses address for referencing the buffer rather than using buffer offset to reference the buffer, and the Steering Address is not sent to the target as currently defined in RFC 5046.  It does not appear in the spec.

  Also, it may be worth formalizing the notion of
  "if Virtual Address was used for this command" via a negotiated connection
  attribute. 

  [mk] The use of the Steering Address is up to the RCaP.  For iWARP, the Steering Address is set to zero.  By adding the explanation (as I indicated in my previous note) on how the Tagged Offset is computed from the Steering Address and the Buffer Offset, it is up to the RCaP to determine what they want to do.

  [mk] Both the Steering Address and the Tagged Offset will be required.  When the Steering Address is set to zero, we have the iWARP mode of behavior as defined in RFC 5046.

  [mk] I don't think a new connection attribute is needed.  The RCaP just needs to set the Steering Address to the appropriate value.  So for iWARP, set the Steering Address to zero.

  Mallikarjun

  Mike
  ----- Original Message ----- 
  From: Tom Talpey 
  To: Michael Ko ; Caitlin Bestler 
  Cc: Alexander Nezhinsky ; [email-masked] 
  Sent: Thursday, July 14, 2011 1:16 PM
  Subject: RE: [storm] iSER draft

  Changing the name doesn't address the core issues here. There are no
  processing rules defined in the draft to support the arithmetic you describe
  below, or even to use the Steering/Virtual Address.

  Mike
  ----- Original Message ----- 
  From: Caitlin Bestler 
  To: Tom Talpey 
  Cc: Alexander Nezhinsky ; Michael Ko ; [email-masked] 
  Sent: Wednesday, July 13, 2011 9:49 PM
  Subject: Re: [storm] iSER draft

  On Jul 13, 2011, at 8:12 AM, Tom Talpey wrote:

  > Yes, understood on the ZBVA/Infiniband issue. Regarding the VA term's
  > First, it seems to me that a "virtual address" has no real meaning to the
  the initiator, but it's not meaningful on the target in this way, it's only
  Control PDU, but where does it come back? The RDMA Read or Write as depicted
  in (xx) shows only a Tagged Offset. So, it's not clear what its protocol
  meaning is.
  > 
  > Third, I don't ever see a Tagged Buffer described by a fully qualified
  one, but where the meaning of TO has changed from a small offset to some
  other token, generated and managed only at the initiator. So, it seems
  artificial to define it as distinct, and document it as possessing some new
  properties. Isn't it just a Target Offset, still?

---

From: Tom Talpey <[email-masked]>
Date: Fri, 15 Jul 2011 18:16:38 +0000

This discussion is starting to fork, but I'll do my best to reply to the top of the thread.

I'm glad we're making progress, but I don't think we're on the same page yet. In particular, I disagree with your statement that "for iWARP, the Steering Address is set to zero." The choice of addressing mode is not made on a per-protocol basis, it is a local issue on the initiator. If ZBVA mode is used, then the addressing mode is zero-based. However local verbs support several other modes, and the "steering address" is not at all constrained to be an "address". Depending on several factors, it can be a virtual address, a physical address, or a completely arbitrary value.

To my reading, the new draft is proposing that an additional value be sent in the Control PDU header, to be added to the BufferOffset of the individual SCSI control PDU payload when the target performs RDMA Read or RDMA Write. To me, this value is not an address, and has no semantics beyond being an initiator-provided value needed to direct the expected RDMA, when it is performed by the target. Personally, I think it would be better defined as a "Base Offset", or "Tagged Base Offset", to which the BufferOffset is added to form the resulting TaggedOffset. There's no further requirement than to have the two values to add together.

I think that any proposed addition of a connection attribute for "iWARP" or "offset" to the normative discussion is incorrect, for the reasons above. While it's true that iWARP may support addressing modes that another RDMA protocol may not, it's neither necessary or desirable to require iSER or iSCSI to change behavior depending on the lower RDMA layer. iSER is appropriately a transport adaptation layer, and should be defined in such a way as to be agnostic to the needs of the lower layer.

So again, I would like to see:

-          Virtual Address renamed to agnostic term (perhaps) Base Offset

-          Clearly stated processing at the target to compute the necessary TaggedOffset

-          Avoidance of discussion of addressing mode requirements or local verb semantics.

From: Michael Ko [mailto:[email-masked]]
Sent: Thursday, July 14, 2011 9:50 PM
To: Mallikarjun Chadalapaka; Tom Talpey; 'Caitlin Bestler'
Cc: [email-masked]
Subject: Re: [storm] iSER draft

Mallikarjun,

Nice to hear from you!

Mike
----- Original Message -----
From: Mallikarjun Chadalapaka<mailto:[email-masked]>
To: 'Michael Ko'<mailto:[email-masked]> ; Tom Talpey<mailto:[email-masked]> ; 'Caitlin Bestler'<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Sent: Thursday, July 14, 2011 6:26 PM
Subject: RE: [storm] iSER draft

Mallikarjun

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of
Michael Ko
Sent: Thursday, July 14, 2011 1:53 PM
To: Tom Talpey; Caitlin Bestler
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER draft

Mike
----- Original Message -----
From: Tom Talpey
To: Michael Ko ; Caitlin Bestler
Cc: Alexander Nezhinsky ; [email-masked]<mailto:[email-masked]>
Sent: Thursday, July 14, 2011 1:16 PM
Subject: RE: [storm] iSER draft

From: Michael Ko [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Thursday, July 14, 2011 2:39 PM
To: Caitlin Bestler; Tom Talpey
Cc: Alexander Nezhinsky; [email-masked]<mailto:[email-masked]>
Subject: Re: [storm] iSER draft

Mike

---

From: Mallikarjun Chadalapaka <[email-masked]>
Date: Fri, 15 Jul 2011 11:21:16 -0700

Hi Mike,

A few responses to your comments, :)

>[mk] The use of the Steering Address is up to the RCaP.

Understood.  But iSER layer has differing protocol requirements (see my
bullets 2-b-c, 3-b-c), keyed to RCaP semantics.  So IMHO, it would be useful
to formally spec these differences and relate them to an iSER-level protocol
parameter.

>[mk] Both the Steering Address and the Tagged Offset will be required.

Sure, I agree for control-type PDUs.  That's the reason I said (emphasis
added)  " Steering Address ("Address") or Tagged Offset ("Offset") is
required by the RCaP in the *RDMA requests*."  RDMA Reads/Writes generated
due to iSCSI Data-type PDUs contain only one 64-bit reference - for iWARP,
it is an Offset when used with ZB-VA; for IB, it's a full Virtual Address
(although I am not an IB-expert, that's what I understand so far)

>[mk] I don't think a new connection attribute is needed.  

While adding clarification text in several places in the draft might address
the issue that Tom pointed out, I think a formal iSCSI or iSER state
variable (either through text key negotiation or iSER Hello negotiation)
gives a hook for you to hang those semantics off of in a formal way - both
to spec, and validate wire protocol behavior for compliance later on.
E.g., let's say the attribute name is "Reference Type" with 0 - for Address,
and 1 for Offset, you could say "When the operational Reference Type is 1
for the connection, initiator iSER layer MUST.....".  Finally, this is
probably better modeled as an iSER Hello parameter rather than iSCSI text
key, to minimize layer "rototilling".  For the RCaP that does not have a
Hello negotiation (e.g. IB), you could specify in the spec that Reference
Type=0 is the default iSER behavior.

My 0.02.

Mallikarjun

>Sent: Thursday, July 14, 2011 6:50 PM
>[mk] That statement is only meant to convey what would be done by the
initiator if it uses address for referencing the buffer rather than using
buffer offset to reference the buffer, and the Steering Address is not sent
to the target as currently defined in RFC 5046.  It does not appear in the
spec.
>[mk] The use of the Steering Address is up to the RCaP.  For iWARP, the
Steering Address is set to zero.  By adding the explanation (as I indicated
in my previous note) on how the Tagged Offset is computed from the Steering
Address and the Buffer Offset, it is up to the RCaP to determine what they
want to do.
>[mk] Both the Steering Address and the Tagged Offset will be required.
When the Steering Address is set to zero, we have the iWARP mode of behavior
as defined in RFC 5046.
>[mk] I don't think a new connection attribute is needed.  The RCaP just
needs to set the Steering Address to the appropriate value.  So for iWARP,
set the Steering Address to zero.