Subject: [lisp] IPv6 UDP Checksum Offload Hardware and Software

From: Greg Shepherd <[email-masked]>
Date: Wed, 12 Aug 2009 11:43:13 -0700

> It is _not_ RFC compliant to ignore non-zero UDP checksums in either IPv4 or IPv6.

Margret,

Which RFC are you referring to for IPv4?

>From RFC768 I find:

"Fields"

"If the computed checksum is zero, it is transmitted as all ones (the
equivalent in one's complement  arithmetic). An all zero transmitted
checksum value means that the transmitter  generated  no checksum (for
debugging or for higher level protocols that don't care)."

It almost reads as if Postel had LISP in mind when he wrote this. ;-)

Now we just need to fix IPv6 and we're golden.

Greg

> So, the behavior described in the LISP >document is _not_ RFC compliant for IPv4 or IPv6 at this point, nor will it be when you
> change the text to indicated that the ETR >_MAY_ ignore non- zero UDP checksums. The current RFCs require that receiving
> nodes _check_ non-zero UDP checksums. >This is just a fact, and I don't really understand how we could be arguing about it.
>
> It may be the case that there are good reasons for LISP to behave as it does. If so, we need to document those reasons very
> clearly and get IETF consensus that ignoring non-zero UDP checksums is a reasonable thing to do in this case.
>
> Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 12 Aug 2009 15:26:47 -0400

Hi Greg,

> Now we just need to fix IPv6 and we're golden.

Not quite...  It isn't the sending of zero UDP checksums in IPv4 that  
is non-compliant, it is what the LISP ETR is required to do when  
receiving them.

The current LISP specification requires ETRs to _all_ UDP checksums  
received in the outer header of encapsulated data packets, not just  
ones that are set to zero.  So, even if the ITR did calculate a  
checksum and insert a non-zero value into the UDP checksum field, the  
receiving ETR would not check it.

RFC 768 is silent on the issue of what level of UDP checksum support  
is required on IPv4 systems, but RFCs 1122 (requirements for internet  
hosts) and 1812 (requirements for internet routers) both require that  
IPv4 nodes that implement UDP include support for checking non-zero  
UDP checksums in received packets.  We decided earlier in this  
discussion that a LISP node is a router not a host, so RFC 1812 would  
apply. It says (among many other things):

" Although a particular application protocol may require that UDP
datagrams it receives must contain a UDP checksum, there is no
general requirement that received UDP datagrams contain UDP
checksums. Of course, if a UDP checksum is present in a received
  datagram, the checksum must be verified and the datagram discarded
if the checksum is incorrect."

So, to comply with RFC 1812, the LISP ETR would need to verify non- 
zero checksums and discard the datagram if the checksum incorrect.

There are two things that we are trying to do here:

- Avoid the performance overhead of calculating UDP checksums.  The  
current spec proposes we do this by having ITRs set the UDP checksums  
in the outer headers LISP encapsulated data packets to zero.  This is  
fine in IPv4, compliance-wise, but is forbidden by RFC 2460.  Marshall  
Eubanks has a draft out to allow this in IPv6, as well.
- Make LISP work across widely-deployed, buggy NAT boxes that  
overwrite zero UDP checksums with invalid values.  The proposal in the  
LISP spec is to accomplish this by having LISP ETRs ignore all UDP  
checksums, even non-zero ones.

As far as I can tell, we have three reasonable choices for what to do  
to bring LISP and the IETF standards into line:

(1) Decide that it is not that important for LISP to work across those  
buggy NAT boxes (or that other things are more important), AND  
convince the IETF that is okay to send zero UDP checksums in IPv6.

or

(2) Convince the IETF that it is okay to send zero UDP checksums in  
IPv6 AND that it is okay to ignore non-zero UDP checksums (in IPv4 and  
IPv6).

or

(3) Use a different LISP encapsulation (UDP-Lite or IP-in-IP) for both  
IPv4 and IPv6.

Unfortunately, choice three is unpalatable to many of us, because we  
want LISP to work with legacy LAG/ECMP systems that can only calculate  
their load-balancing has based on the IP addresses, the IP protocol/ 
next-header field and TCP or UDP ports.

So, we're basically stuck between a rock and a couple of hard places,  
and the WG needs to decide what to do...

Margaret

---

From: Greg Shepherd <[email-masked]>
Date: Wed, 12 Aug 2009 13:40:40 -0700

On Wed, Aug 12, 2009 at 12:26 PM, Margaret Wasserman<[email-masked]> wrote:
> Not quite...  It isn't the sending of zero UDP checksums in IPv4 that is
> non-compliant, it is what the LISP ETR is required to do when receiving
> them.
>
> The current LISP specification requires ETRs to _all_ UDP checksums received
> in the outer header of encapsulated data packets, not just ones that are set
> to zero.  So, even if the ITR did calculate a checksum and insert a non-zero
> value into the UDP checksum field, the receiving ETR would not check it.

An ITR won't do that. IF it did it is violating the LISP spec:

"UDP Checksum:  this field MUST be transmitted as 0 and ignored on
      receipt by the ETR.  Note, even when the UDP checksum is
      transmitted as 0 an intervening NAT device can recalculate the
      checksum and rewrite the UDP checksum field to non-zero.  For
      performance reasons, the ETR MUST ignore the checksum and MUST not
      do a checksum computation."

So let's limit our assumptions to the possible.

> RFC 768 is silent on the issue of what level of UDP checksum support is
> required on IPv4 systems, but RFCs 1122 (requirements for internet hosts)
> and 1812 (requirements for internet routers) both require that IPv4 nodes
> that implement UDP include support for checking non-zero UDP checksums in
> received packets.  We decided earlier in this discussion that a LISP node is
> a router not a host, so RFC 1812 would apply. It says (among many other
> things):
>
> So, to comply with RFC 1812, the LISP ETR would need to verify non-zero
> checksums and discard the datagram if the checksum incorrect.
>
> - Avoid the performance overhead of calculating UDP checksums.  The current
> spec proposes we do this by having ITRs set the UDP checksums in the outer
> headers LISP encapsulated data packets to zero.  This is fine in IPv4,
> compliance-wise, but is forbidden by RFC 2460.  Marshall Eubanks has a draft
> out to allow this in IPv6, as well.

Yes, I'm aware of Marshall's draft. And the motivation for his draft
was not LISP but AMT, so you can see the disallowing of zero checksums
of an outer header is not unique to LISP. Clearly RFC 2460 overlooked
the needs of the community.

> - Make LISP work across widely-deployed, buggy NAT boxes that overwrite zero
> UDP checksums with invalid values.

Well, since ITRs send packets to ETRs and ITRs are forbidden from
writing non-zero checksums any non-zero chechsum received has been
tampered with. We can either assume the entire packet is corrupt and
drop it or strip off the outer header and let the forwarding process
of the inner header determine the validity of the packet. The later
seems the better of the two choices.

> The proposal in the LISP spec is to
> accomplish this by having LISP ETRs ignore all UDP checksums, even non-zero
> ones.

I'm not sure if you have a problem with this specifically or just the
documentation and IETF process to provide for this. Which is it?

> As far as I can tell, we have three reasonable choices for what to do to
> bring LISP and the IETF standards into line:
>
> (1) Decide that it is not that important for LISP to work across those buggy
> NAT boxes (or that other things are more important), AND convince the IETF
> that is okay to send zero UDP checksums in IPv6.
>
> or
>
> (2) Convince the IETF that it is okay to send zero UDP checksums in IPv6 AND
> that it is okay to ignore non-zero UDP checksums (in IPv4 and IPv6).
>
> or
>
> (3) Use a different LISP encapsulation (UDP-Lite or IP-in-IP) for both IPv4
> and IPv6.
>
> Unfortunately, choice three is unpalatable to many of us, because we want
> LISP to work with legacy LAG/ECMP systems that can only calculate their
> load-balancing has based on the IP addresses, the IP protocol/next-header
> field and TCP or UDP ports.
>
> So, we're basically stuck between a rock and a couple of hard places, and
> the WG needs to decide what to do...

No rock. No hard place. We are the IETF. I think that is what we're
doing now, right? Number 2) above seems the best path to me and the
current path we are on. Am I missing something here?

Thanks,
Greg

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 12 Aug 2009 16:59:55 -0400

On Aug 12, 2009, at 4:40 PM, Greg Shepherd wrote:
>> received

It is certainly possible that a LISP ETR will receive an encapsulated  
LISP data packet with a non-zero UDP checksum.

The LISP spec documents at least one case where this could happen --  
the UDP checksum could be changed by a NAT box or otherwise corrupted  
in transit.

Also, there are a lot of widely-deployed systems that offload the  
checksum to hardware, and on those systems it may not be possible to  
turn off checksum calculation for outbound packets.  So, those systems  
will send encapsulated LISP packets with non-zero checksums.

Dino has already agreed to change the text for LISP -04 to say:

"An ITR MAY send a zero checksum, an ETR MUST accept a 0 checksum and  
MAY ignore the checksum completely."  (or something similar from Sam)

So, this implies that ITRs may also send non-zero checksums.

Personally, I would greatly prefer to retain the ability to enable UDP  
checksums in LISP, just in case we ever find out that there are  
significant operational problems caused by our choice to ignore  
decades of research and experience that indicate that it is a bad idea  
to do so.
>> current

It's quite a bit more complicated that than, if you consider our  
"community" to include all of the end-users of (non-LISP, non-AMT) UDP  
applications, whose applications may be disrupted by corrupted and  
misdirected LISP or AMT packets.
>> overwrite zero

The problems don't really come in if the packet makes it to the right  
destination with the destination UDP port intact.  They come in when  
that doesn't happen.
> documentation and IETF process to provide for this. Which is it?

I think that the choice to disable UDP checksum for LISP is  
technically unwise, perhaps even reckless.

I also think that it violates some RFCs, but that is a lesser  
consideration.  If the WG does decide that sending IPv6 zero checksums  
and ignoring non-zero checksums on receipt is the right thing to do,  
and if the WG can convince the rest of the IETF that they are right,  
then updating RFCs is a fairly trivial matter.  In that case, we'd  
need Marshall's draft (or something similar) and a draft to update RFC  
1812 to say it is okay to ignore non-zero checksums on receipt.

There are _reasons_, though, why the current RFCs were written the way  
they were.  There is research and experience that has shown that  
packets that get corrupted in transit and misdirected to the wrong  
host or the wrong application cause serious problems.  Those problems  
won't be constrained to LISP or even to nodes that implement LISP.

Margaret

---

From: Greg Shepherd <[email-masked]>
Date: Wed, 12 Aug 2009 14:16:18 -0700

On Wed, Aug 12, 2009 at 1:59 PM, Margaret Wasserman<[email-masked]> wrote:
> It is certainly possible that a LISP ETR will receive an encapsulated LISP
> data packet with a non-zero UDP checksum.

True, but that's not what you stated above. I'm both reading and
responding in kind.

> The LISP spec documents at least one case where this could happen -- the UDP
> checksum could be changed by a NAT box or otherwise corrupted in transit.

Which is different than the ETR setting a non-zero checksum as I was
addressing above.

> Also, there are a lot of widely-deployed systems that offload the checksum
> to hardware, and on those systems it may not be possible to turn off
> checksum calculation for outbound packets.  So, those systems will send
> encapsulated LISP packets with non-zero checksums.

Show me one such system that has a LISP implementation. Again, let's
stick with the possible here. We're developing the spec now for LISP
implementations to follow.

> "An ITR MAY send a zero checksum, an ETR MUST accept a 0 checksum and MAY
> ignore the checksum completely."  (or something similar from Sam)
>

Yes, a compromise to put this behind us and move on. :)

> checksums in LISP, just in case we ever find out that there are significant
> operational problems caused by our choice to ignore decades of research and
> experience that indicate that it is a bad idea to do so.

Can you please site such research, specifically for zero checksums in
UDP encapsulated headers where the inner header checksum is non-zero?
I clearly need to learn more.

> It's quite a bit more complicated that than, if you consider our "community"
> to include all of the end-users of (non-LISP, non-AMT) UDP applications,
> whose applications may be disrupted by corrupted and misdirected LISP or AMT
> packets.

Have you read draft-eubanks-chimento-6man-00? It says nothing of
generic UDP. It is intended to specifically address the UDP
encapsulated outer header where the inner header checksum is in fact
calculated. Please help me understand how this would affect any
application dependent on the inner packet.

> destination with the destination UDP port intact.  They come in when that
> doesn't happen.

And then they are dropped. End of problem.

> I think that the choice to disable UDP checksum for LISP is technically
> unwise, perhaps even reckless.

I believe you do. I just don't yet understand why. Sorry, but I'm trying.

Greg

> I also think that it violates some RFCs, but that is a lesser consideration.
>  If the WG does decide that sending IPv6 zero checksums and ignoring
> non-zero checksums on receipt is the right thing to do, and if the WG can
> convince the rest of the IETF that they are right, then updating RFCs is a
> fairly trivial matter.  In that case, we'd need Marshall's draft (or
> something similar) and a draft to update RFC 1812 to say it is okay to
> ignore non-zero checksums on receipt.
>
> There are _reasons_, though, why the current RFCs were written the way they
> were.  There is research and experience that has shown that packets that get
> corrupted in transit and misdirected to the wrong host or the wrong
> application cause serious problems.  Those problems won't be constrained to
> LISP or even to nodes that implement LISP.
>
> Margaret
>

---

From: Sam Hartman <[email-masked]>
Date: Wed, 12 Aug 2009 20:10:37 -0400

Greg, I appreciate that you're trying to join the discussion; it is
always great to get new people involved.  However you're asking people
to rehash a lot of ground and to repeat a bunch of things that have
been discussed in the last few days.

People, including Margaret, Dino and myself have spent a lot of time
reading and responding to LISP mail in the last couple of days.
Rehashing the arguments yet again doesn't really help.

I do have a suggestion for how you could come up to speed anh help us
all.  It looks like you work at Cisco, so you probably have reasonably
good access to the arguments about why it would be nice to send 0
checksums and ignore checksums on receipt.

I've often found that when jumping into the discussion it really helps
if I try and summarize the position I don't agree with.  By the time I
can get to a summary that the other side agrees accurately states
their position then I find I have a lot more understanding and am much
more likely to contribute constructively.

I think it would be great if you could help us all out and prepare
such a summary.

If you don't have time, I completely understand,but if you don't have
time, then it's probably best if you sit this discussion out and join
us with the next new issue rather than asking us to repeat what we've
been saying for the last week.

Sam Hartman
LISP co-chair

---

From: Greg Shepherd <[email-masked]>
Date: Wed, 12 Aug 2009 18:23:51 -0700

> Rehashing the arguments yet again doesn't really help.

I see you have, and I have read the archives. But I've been involved
in this discussion since before LISP was a working group as it first
came up while trying to take AMT to last call. So my interests cross
over a bit as well.

> more likely to contribute constructively.

I believe I was directly addressing open issues raised by others in
the thread. I asked questions. Is that still allowed?

> I think it would be great if you could help us all out and prepare

Sure. I don't agree with arguments about whether or not the outer
header checksum can/should be zero based on issues regarding general
UDP header checksum.

Thanks,
Greg

---

From: Shane Amante <[email-masked]>
Date: Wed, 12 Aug 2009 20:52:15 -0600

Margaret,

On Aug 12, 2009, at 13:26 MDT, Margaret Wasserman wrote:
[--snip--]
> As far as I can tell, we have three reasonable choices for what to  
> do to bring LISP and the IETF standards into line:
>
> (1) Decide that it is not that important for LISP to work across  
> those buggy NAT boxes (or that other things are more important), AND  
> IPv6 AND that it is okay to ignore non-zero UDP checksums (in IPv4  
> and IPv6).
>
> or
>
> (3) Use a different LISP encapsulation (UDP-Lite or IP-in-IP) for  
> both IPv4 and IPv6.
>
> want LISP to work with legacy LAG/ECMP systems that can only  
> calculate their load-balancing has based on the IP addresses, the IP  
> protocol/next-header field and TCP or UDP ports.

I've taken a step back from this discussion for a few days and had a  
chance to clear my mind and re-focus.  In choice #3, you suggest that  
LISP switch to a different encapsulations for IPv4 and IPv6, such as  
UDP-lite or IP-in-IP.  I'd like to focus on the latter (IP-in-IP) for  
a moment, specifically when IPv6 is used as the outer encapsulation.   
AFAICT, if LISP (or any other protocol) is doing IP-in-IPv6, then the  
outer IPv6 header is not protected by a header checksum[1], correct?   
Therefore, in the case of IP-in-IPv6, if the outer header is damaged  
the packet may end up: a) dropped; or, b) delivered to the wrong  
destination.  So, what is different from this case versus UDP-in-IPv6,  
(where UDP is using 0 checksums)?  Putting this another way, should IP- 
in-IPv6 be deprecated/forbidden, (if it's not already?), because the  
outer header isn't checksum protected?

-shane

[1] Perhaps I'm misinformed on this point and, if that's the case, I  
would certainly welcome a reference to the RFC describing and/or  
mandating that. I've briefly scanned RFC's 2460 & 2473 and didn't see  
it addressed there.

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 13 Aug 2009 00:38:09 -0400

Hi Shane,

On Aug 12, 2009, at 10:52 PM, Shane Amante wrote:
> ussion for a few days and had a chance to clear my mind and re- 
> focus.  In choice #3, you suggest that LISP switch to a different  
> encapsulations for IPv4 and IPv6, such as UDP-lite or IP-in-IP.  I'd  
> like to focus on the latter (IP-in-IP) for a moment, specifically  
> when IPv6 is used as the outer encapsulation.  AFAICT, if LISP (or  
> any other protocol) is doing IP-in-IPv6, then the outer IPv6 header  
> is not protected by a header checksum[1], correct?

Yes, that is correct.  Because there is no header checksum in IPv6,  
packets that are tunneled directly in IPv6 will not have any checksum  
for the outer IPv6 header fields.

It would, of course, be possible to add a checksum to the LISP header  
that includes the IPv6 pseudo-header checksum, if we are concerned  
about this lack of protection.  Unlike the UDP checksum, calculating a  
checksum of the IPv6 and LISP headers wouldn't require access to more  
than 128 bytes of the packet.

Of course, this option doesn't support the LAC/ECMP hash function, and  
that makes the IP-LISP-IPv6 approach less desirable than an IP-LISP- 
UDP-IPv6 approach, all other things being equal.

> destination.  So, what is different from this case versus UDP-in- 
> IPv6, (where UDP is using 0 checksums)?  Putting this another way,  
> should IP-in-IPv6 be deprecated/forbidden, (if it's not already?),  
> because the outer header isn't checksum protected?

Dropped packets aren't really a problem, since that's what will happen  
when a bad checksum is detected, anyway.  If we do the analysis of all  
the possible misdirections caused by corrupted IPv6, UDP or LISP  
headers and we find that the only thing that happens is that packets  
are dropped and/or decapsulated in the wrong place but sent on to the  
correct  destination, then I may not have any technical objection to  
allowing (MAY), or even recommending (SHOULD) that LISP ITRs use  zero  
checksums (for IPv4 and IPv6) in LISP data packets.  I think we still  
have some analysis to finish before we can be reasonably sure that  
this is safe to make this recommendation, though.

I've run through a number of scenarios so far, and the cases that  
still concern me include:

- What happens when a LISP packet is redirected to the wrong UDP  
destination port?   For example, what would happen if a LISP packet  
were received on the TFTP port in a LISP xTR during an active TFTP  
transfer?

- What happens when a LISP packet arrives at the wrong ETR?  I _think_  
it will just be discarded unless the ETR that received it can forward  
to the inner header destination EID, in which case it will  
serendipitously be delivered to the final destination, right?  I am  
also concerned about how errors would be detected and handled in this  
case.

- What happens when a LISP packet arrives at the right ETR but appears  
to come from the wrong destination?  We've discussed the possibility  
of an incorrect "gleaned" mapping, but I haven't fully explored the  
implications of that situation.  I'm also concerned about how errors  
would be handled in this case.

I think I've mentally covered all of the other cases and determined  
that the packets will just be dropped in those cases.

I'd describe myself as pretty soft on the issue of zero UDP checksums  
in IPv6.  After we get as many facts on the table as possible, I think  
we may decide that  this is the best available choice for LISP, and  
that it shouldn't pose any serious dangers for other nodes and  
protocols.  I think we have a bit more analysis to complete before we  
can be reasonably sure that this is a safe thing to do, though.  There  
have been a lot of cases where people have implemented protocols with  
zero UDP checksums and have come to regret that decision later.  I  
hope we won't be in that camp.

If we do decide that this is the best thing to do, I think that we  
_can_ get the IETF to agree to something similar to Marshall's draft,  
as long as we are very specific about when the use of zero IPv6  
checksums is applicable and when it is not.

I am much less comfortable with sanctioning the practice of ignoring  
inbound non-zero UDP checksums.  I understand the "buggy NAT" issue  
which is driving this choice, but I don't think that the ability to  
work behind those buggy NATs (that I _hope_ are a dying breed) is as  
important as leaving open the possibility that we can turn on UDP  
checksums in LISP later if we find that there _are_ problems in  
practice with using UDP zero checksums.  I would also like it to be  
possible to implement LISP on a fairly wide set of available hardware  
and software, and it doesn't seem like most systems have a way to turn  
off checking inbound UDP checksums.  Also, I'm concerned that there  
may be no good way to turn this off for _only_ the LISP protocol, and  
that we'd see boxes that don't check received UDP checksums for any of  
their protocols (TFTP, SNMP, etc...).

I also have concerns about the impacts of corruption in the LISP  
header itself, and I think we need more analysis there.  If we find  
that the LISP protocol is vulnerable to problems caused by LISP header  
corruption, we will need to do something to verify the integrity of  
the LISP header.  We haven't really started this analysis, though, so  
I won't raise that issue unless/until we see a problem.

Margaret

> mandating that. I've briefly scanned RFC's 2460 & 2473 and didn't  
> see it addressed there.