Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

From: Bill Jouris <[email-masked]>
Date: Mon, 3 Jun 2013 08:21:28 -0700 (PDT)

> P.S.: Before any attempt on this path, one should probably ask: are we 
> suppoed to have this ind of functionality at the Internet layer? Is this 
> worth the pain for most implementations? Is this going to work in the 
> general case, or is this going to be the subject of lots fo breakage? 
> (e.g., packet drops)

Fernando, the problem with using any other layer is that it would require implementation across ALL of the various protocols.  Which is far more disruptive than this implementation in the IP layer would be. 

Also, I'm not sure what kind of breakage you anticipate.  Is it that you think that there would be a problem processing the IP layer due to the additional option?  (And if so, why this option more than any of the existing ones?)  Or are you thinking that the option header itself would break when being processed?

Bill Jouris
Inside Products, Inc.
www.insidethestack.com
831-659-8360
925-855-9512 (direct)

---

From: Lorenzo Colitti <[email-masked]>
Date: Tue, 4 Jun 2013 10:47:24 +0900

On Tue, Jun 4, 2013 at 12:21 AM, Bill Jouris <[email-masked]
> wrote:

> Fernando, the problem with using any other layer is that it would require
> implementation across ALL of the various protocols.  Which is far more
> disruptive than this implementation in the IP layer would be.
>

But on the other hand, implementing it at the Internet layer pushes this
complexity to every host on the Internet, whereas this functionality is
obviously targeted at private network deployments (since it cannot work
anywhere else).

---

From: Nalini Elkins <[email-masked]>
Date: Mon, 3 Jun 2013 20:38:31 -0700 (PDT)

>>But on the other hand, implementing it at the Internet layer pushes this complexity to every host on the Internet, whereas this functionality is obviously targeted at private network deployments (since it cannot work anywhere else).

This is an optional header.  That means that a stack or operating system can be implemented so that the default is to NOT send this header.  End of complexity.

Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
 From: Lorenzo Colitti <[email-masked]>
To: Bill Jouris <[email-masked]> 
Cc: "[email-masked]" <[email-masked]> 
Sent: Monday, June 3, 2013 6:47 PM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

>Fernando, the problem with using any other layer is that it would require implementation across ALL of the various protocols.  Which is far more disruptive than this implementation in the IP layer w…

But on the other hand, implementing it at the Internet layer pushes this complexity to every host on the Internet, whereas this functionality is obviously targeted at private network deployments (since it cannot work anywhere else).
_______________________________________________
v6ops mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/v6ops

---

From: Lorenzo Colitti <[email-masked]>
Date: Tue, 4 Jun 2013 14:00:14 +0900

On Tue, Jun 4, 2013 at 12:38 PM, Nalini Elkins <
[email-masked]> wrote:

> This is an optional header.  That means that a stack or operating system
> can be implemented so that the default is to NOT send this header.  End of
> complexity.
>

Well, but complexity that is off by default is still complexity. Kernel
code has to be written to implement the header, testing must be done to
ensure it works, firewalls need to be modified to not drop it, and so on.
This complexity can only be avoided by not implementing the header.

If you implement the complexity above the Internet layer, then you don't
need to add complexity to the Internet layer and you can implement it at
the application layer only for those applications that need it. That way,
the applications implementing the complexity are also the ones getting the
benefit.

If you don't want to do it at the application layer, an alternative that
also doesn't involve adding complexity to general Internet hosts would be
an Internet-layer header that is not implemented in standard Internet hosts
and standard operating systems, but is only implemented by hosts in
networks that need this functionality. Is that approach acceptable, or do
you want this functionality to be part of every Internet-connected host?

---

From: Nalini Elkins <[email-masked]>
Date: Tue, 4 Jun 2013 06:59:57 -0700 (PDT)

>>If you don't want to do it at the application layer, an alternative that also doesn't involve adding complexity to general Internet hosts would be an Internet-layer header that is not implemented in standard Internet hosts and >>standard operating systems, but is only implemented by hosts in networks that need this functionality. Is that approach acceptable, or do you want this functionality to be part of every Internet-connected host?

How might this work?  Our folks use Windows, Unix, etc just as all others do.
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
 From: Lorenzo Colitti <[email-masked]>
To: Nalini Elkins <[email-masked]> 
Cc: Bill Jouris <[email-masked]>; "[email-masked]" <[email-masked]> 
Sent: Tuesday, June 4, 2013 12:00 AM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

>This is an optional header.  That means that a stack or operating system can be implemented so that the default is to NOT send this header.  End of complexity.

Well, but complexity that is off by default is still complexity. Kernel code has to be written to implement the header, testing must be done to ensure it works, firewalls need to be modified to not drop it, and so on. This complexity can only be avoided by not implementing the header.

If you implement the complexity above the Internet layer, then you don't need to add complexity to the Internet layer and you can implement it at the application layer only for those applications that need it. That way, the applications implementing the complexity are also the ones getting the benefit.

If you don't want to do it at the application layer, an alternative that also doesn't involve adding complexity to general Internet hosts would be an Internet-layer header that is not implemented in standard Internet hosts and standard operating systems, but is only implemented by hosts in networks that need this functionality. Is that approach acceptable, or do you want this functionality to be part of every Internet-connected host?

---

From: Lorenzo Colitti <[email-masked]>
Date: Tue, 4 Jun 2013 23:31:39 +0900

On Tue, Jun 4, 2013 at 10:59 PM, Nalini Elkins <
[email-masked]> wrote:

>  >>If you don't want to do it at the application layer, an alternative
> that also doesn't involve adding complexity to general Internet hosts would
> be an Internet-layer header that is not implemented in standard Internet
> hosts and >>standard operating systems, but is only implemented by hosts in
So then it's the case that you want every host running one of these
operating systems to support this functionality, just in case someone wants
to use it? The bar for that is quite high.

---

From: Nalini Elkins <[email-masked]>
Date: Tue, 4 Jun 2013 07:39:02 -0700 (PDT)

>> So then it's the case that you want every host running one of these operating systems to support this functionality, just in case someone wants to use it? The bar for that is quite high.

I don't know about the "just in case".  My feeling is that many corporations will want this.

Actually, I was just working on a problem this morning with a customer not 10 minutes ago where they could have used this feature.

Is the "bar" you are talking about getting the stack vendors to implement?  If so, I have already started discussions with some.
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

Sent: Tuesday, June 4, 2013 7:31 AM

>How might this work?  Our folks use Windows, Unix, etc just as all others do.
So then it's the case that you want every host running one of these operating systems to support this functionality, just in case someone wants to use it? The bar for that is quite high.

---

From: Bill Jouris <[email-masked]>
Date: Tue, 4 Jun 2013 08:49:46 -0700 (PDT)

> general case, or is this going to be the subject of lots fo breakage? 

Fernando,
 the problem with using any other layer is that it would require 
implementation across ALL of the various protocols.  Which is far more 
disruptive than this implementation in the IP layer would be. 

But 
on the other hand, implementing it at the Internet layer pushes this 
complexity to every host on the Internet, whereas this functionality is 
obviously targeted at private network deployments (since it cannot work 
anywhere else).

Why do you assume that this functionality is targetted at private network deployments?  

Certainly it will be very useful there.  But any company that does significant business on-line is going to be interested in being able to see WHY their customers are having problems or getting bad response when coming in over the Internet.  And that is also where they cannot rely on specialize software (e.g. agents) being installed along the route between host and user, or at the user's node.  Unless the information is available in the header somewhere, they are blind.

Bill Jouris
Inside Products, Inc.
www.insidethestack.com
831-659-8360
925-855-9512 (direct)

---

From: [email-masked]
Date: Tue, 04 Jun 2013 19:45:33 +0200 (CEST)

> Certainly it will be very useful there.  But any company that does significant business on-line is going to be interested in being able to see WHY their customers are having problems or getting bad…

Are you sure the *customers* are interested in this?

Speaking solely for myself, I feel that Google, Amazon, etc knows more
than enough about me already. I don't wish to share more information. I
may be old-fashioned here...

Steinar Haug, AS 2116

---

From: "Ackermann, Michael" <[email-masked]>
Date: Tue, 4 Jun 2013 18:07:59 +0000

I guess you could say only hosts for which availability, reliability and performance are issues?       In my world that is pretty much every host.

I would like to see it at least be an option on every host.       Some Vendors can choose to not implement it, which may make their customers less pleased.    Customers can choose to not turn it on or turn it on selectively, as appropriate.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lorenzo Colitti
Sent: Tuesday, June 04, 2013 10:32 AM
To: Nalini Elkins
Cc: [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

The information contained in this communication is highly confidential and is intended solely for the use of the individual(s) to whom this communication is directed. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information is prohibited. Please notify the sender, by electronic mail or telephone, of any unintended receipt and delete the original message without making any copies.
 
 Blue Cross Blue Shield of Michigan and Blue Care Network of Michigan are nonprofit corporations and independent licensees of the Blue Cross and Blue Shield Association.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 04 Jun 2013 19:20:49 +0100

On 04/06/2013 15:39, Nalini Elkins wrote:
> Actually, I was just working on a problem this morning with a customer not
> 10 minutes ago where they could have used this feature.

Nalini,

we understand that the concept is useful.

You've proposed two methods so far which don't seem to be runners:

- modifying the ipv6 header: ain't gonna happen.  The moment you change the
header, it's no longer ipv6.

- adding an extension header: which has been roundly trashed because
extension headers cause serious operational difficulties.

Can we park these two ideas and move on?  Everyone is arguing in circles
and the entire conversation has rat-holed once more.

Nick

---

From: Nalini Elkins <[email-masked]>
Date: Tue, 4 Jun 2013 11:32:27 -0700 (PDT)

>> adding an extension header: which has been roundly trashed because

Is this IETF consensus: IPv6 extension headers should no longer be used?

Maybe then, all the RFCs need to be updated.   Let's start with RFC2460 -- all extension headers are henceforth deprecated.   Who is going to take this on?

Is this what we are saying?  AH and ESP too?  Why not go backwards and take them out in IPv4 as well?

Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
 From: Nick Hilliard <[email-masked]>
To: Nalini Elkins <[email-masked]> 
Cc: Lorenzo Colitti <[email-masked]>; "[email-masked]" <[email-masked]> 
Sent: Tuesday, June 4, 2013 11:20 AM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

Nalini,

Nick

---

From: "Ackermann, Michael" <[email-masked]>
Date: Tue, 4 Jun 2013 18:54:36 +0000

I don't see this as providing more information about YOU, per say.    But rather more about the performance of your IP packets.    NO protected or personal information as we say in my field.  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, June 04, 2013 1:46 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

Speaking solely for myself, I feel that Google, Amazon, etc knows more than enough about me already. I don't wish to share more information. I may be old-fashioned here...

Steinar Haug, AS 2116

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 04 Jun 2013 22:49:03 +0100

> Is this IETF consensus: IPv6 extension headers should no longer be used?

The ietf hasn't deprecated them, but as you can see from the reaction in
this group, the operator community doesn't like them because they cause
serious operational difficulties.  The IETF won't deprecate them either,
because there are as many people who think they are wonderful as think they
are a disaster.  If this discussion opened up, consensus would not be
achieved either way.  There is no point going there.

> Maybe then, all the RFCs need to be updated.   Let's start with RFC2460 --
> all extension headers are henceforth deprecated.   Who is going to take
> this on?
> 

With regard to ipv6, they are a royal pain - AH and ESP included.

As an aside, for ipv4 I would dearly love to see AH deprecated in favour of
ESP (see draft-bhatia-ipsecme-avoiding-ah for why); alas there is a noisy
crowd in the ipsec working group which won't are fully intent on blocking
this sensible move.  I won't bore you with the details, but you can read
all about it on the ietf ipsec mailing list.

ESP/ipv4 is fine - it's easily handled by firewalls because you can run a
simple bitfield comparison on the incoming packet.  ESP is protocol 50 and
this ID is located in bits 72-79 in ipv4.  You can check for this really
fast in hardware and as the packet is encrypted anyway, there is no point
inspecting further inside the payload, which means that your forwarding
engine can handle this very quickly.

All of which point to the core problem of extension headers: they use TLVs.

This was one of these things that seemed like a good idea at the time, way
back in 1995.  People had realised that one of the annoying limitations of
ipv4 was that the header used a fixed-length construct and it was difficult
to extend the protocol.  So a TLV structure was devised to allow arbitrary
headers to be attached to an ipv6 packet.  This seemed sensible, because
most packet forwarding was done in software and it was relatively easy to
write code to inspect deep into packets - in software.  Also, for the most
part, people were more interested in forwarding packets than not forwarding
packets.

But as time moved on, more and more forwarding decisions got moved to ASICs
and NPUs, and these had more limitations, particularly with regard to
packet structures which were difficult to parse, especially on ASICs.  E.g.
lots of silicon today will perform a forwarding lookup based on only the
first N octets of the header, and if your extension header falls outside
that lookup window, it will not be parsed in the forwarding path.  Your
options are to drop regardless, to forward regardless or to punt to a slow
cpu.  I'm not sure which is the least broken of these options.

We don't have widespread silicon which can make forwarding decisions based
on arbitrary TLV structures, because that would mean inspecting ipv6
packets of pretty much arbitrary length.  I'm not saying it doesn't exist -
there are some chipsets and NPUs out there which will do this, but they are
not the norm.  Also, given that we're having trouble building hardware to
meet the demands of current forwarding requirements, I don't think we're
going to see tlv processing on 100ge / 400ge any time soon, although no
doubt it will happen at some stage.

Many firewall stacks have not bothered to implement proper stacked EH
processing because it's hard to do right.  And network operators don't
necessarily understand how they operate either, which means that you will
often end up with misconfigured firewalls which block EHs to an even
greater extent than firewall operators currently block e.g. ICMP today.

This leaves us in the awkward position that we've got this great protocol
which is too cool to actually work at the sorts of speed that we expect
from ipv4, and with the same feature levels.  Maybe it will work one day
when we get forwarding path feature parity with ipv4 at the speeds we need.
 Until then, they are a gross inconvenience which cause identifiable and
serious operational problems.

Nick

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 05 Jun 2013 10:17:29 +1200

> achieved either way.  There is no point going there.

Not there exactly, but see
https://datatracker.ietf.org/doc/draft-ietf-6man-ext-transmit/
because we can't leave things as messy as they are.
(If you want to discuss that draft, please do so on the 6man list.
It's intended to allow firewalls to do what they need to do, *and*
to allow standardised extensions to actually get deployed. That's
a narrow line to tread, as this thread has illustrated.)

   Brian

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 04 Jun 2013 23:44:32 +0100

> a narrow line to tread, as this thread has illustrated.)

there's also draft-ietf-6man-nd-extension-headers, which encourages safe
use of extension headers for ND stuff; and rfc6564, which attempts to limit
future damage.  The underlying theme of all these documents is that
extension headers cause trouble.

Nick

---

From: "Ackermann, Michael" <[email-masked]>
Date: Tue, 4 Jun 2013 23:34:35 +0000

What are the serious operational difficulties caused by extension headers?     

And are not extension headers an intrinsic component of IPV6?   

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Nick Hilliard
Sent: Tuesday, June 04, 2013 2:21 PM
To: Nalini Elkins
Cc: [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

Nalini,

- modifying the ipv6 header: ain't gonna happen.  The moment you change the header, it's no longer ipv6.

- adding an extension header: which has been roundly trashed because extension headers cause serious operational difficulties.

Can we park these two ideas and move on?  Everyone is arguing in circles and the entire conversation has rat-holed once more.

Nick

---

From: joel jaeggli <[email-masked]>
Date: Tue, 04 Jun 2013 16:41:08 -0700

On 6/4/13 2:49 PM, Nick Hilliard wrote:
> that lookup window, it will not be parsed in the forwarding path.
Other possible consequences of a longer header chain are  packet take a 
different path than a flow with a shorter one due to L4 header 
contribution hashing/ecmp decision making not being able to find it. 
Some ACLs either can't be applied (bad) because the L4 header falls 
outside of the parseable N octets of the packet or policy drops the 
packet because the L4 header can't be found (also bad).
>   Your

---

From: joel jaeggli <[email-masked]>
Date: Tue, 04 Jun 2013 16:55:28 -0700

> What are the serious operational difficulties caused by extension headers?
http://tools.ietf.org/html/draft-wkumari-long-headers-00
> And are not extension headers an intrinsic component of IPV6?
I'd say the fragmentation header is too...

http://tools.ietf.org/html/draft-taylor-v6ops-fragdrop-00

but we have issues.

---

From: Nalini Elkins <[email-masked]>
Date: Wed, 5 Jun 2013 07:40:15 -0700 (PDT)

Nick,

>> the operator community doesn't like them because they cause serious operational difficulties.  

We are operators also.  The people on my team run very large, complex private networks.   We are trying to address our long-term operational diagnostic difficulties.   I do really appreciate your thoughtful comments.   I hope that we can all work together to put our energies to fix the difficulties that we all have.  I will be commenting on the 6man list on the https://datatracker.ietf.org/doc/draft-ietf-6man-ext-transmit/ as well.  I believe that the issues you have brought up are some of the most acutely serious facing IPv6 implementations today.  On the other hand, the issues we are trying to address are long-term and chronic.

>> The IETF won't deprecate them either, because there are as many people who think they are wonderful as think they are a disaster. 

I would be interested in hearing from some of the people who think IPv6 extension headers are wonderful.  (I have a feeling that I am leaning in that direction myself!).    Are these people fools?   Do they run slow-as-molasses networks where every packet is being defaulted to being routed in software?  Do they not enable IPv6 extension headers on their networks?  What is the story?  I will reach out to some of the organizations that I know who have fairly widespread implementations of IPv6 on private networks.   

What do Joint Interoperability Test Command (JITC) and National Institute of Standards and Technology (NIST) in the US have to say about this?   They have been in the forefront of setting the guidelines for IPv6 implementation in the United States. Anybody know? Do they say "just kidding" about implementing IPv6 extension headers?  If you arbitrarily drop extension headers or route them very slowly, you are getting rid of some extremely basic functionality.

>> ESP/ipv4 is fine - it's easily handled by firewalls because you can run a simple bitfield comparison on the incoming packet.  ESP is protocol 50 and this ID is located in bits 72-79 in ipv4.  You can check for this really
>> fast in hardware and as the packet is encrypted anyway, there is no point inspecting further inside the payload, which means that your forwarding engine can handle this very quickly.

So IPv6 ESP not fine?  This is going to be a surprise to many.

As far as your comments on ASICs, a few questions:

1.  I am not getting why forwarding decisions are having to be made by looking at anything other than the IP addresses.  What am I missing here?  

2.  Are you talking about ASICs on large core switches, routers, firewalls, load balancers or all the above?

3.  It was my understanding the the ASIC size was in the range of 256 bytes to 1K.  This seems to leave plenty of room for the IPv6 main header & quite a few extensions.

>>We don't have widespread silicon which can make forwarding decisions based on arbitrary TLV structures, because that would mean inspecting ipv6 packets of pretty much arbitrary length.  

Well, it seems like maybe what is missing is the old IP header length field.  This time including extension headers as well as the main.

>> Also, given that we're having trouble building hardware to meet the demands of current forwarding requirements, I don't think we're going to see tlv processing on 100ge / 400ge any time soon, although no doubt it will happen at some >>stage

Realistically, we are talking about the extension header that we want implemented being available 5 - 6 years from now (or even more).  We are considering the time spent for the IETF approval process, then to get on the development priority list for the stack software vendors, then to get that particular release of the operating system actually out at customer sites, etc.  I hope the world will look a bit different at that point in this regard.

>> Many firewall stacks have not bothered to implement proper stacked EH processing because it's hard to do right.  

This sounds like another problem which needs to be addressed.   Why is it that there are no RFCs for firewalls?  At least that is my understanding.  This is a large hole that IMHO needs to be fixed.

>> And network operators don't necessarily understand how they operate either

Then they should learn.  In our country, they have a saying "ignorance of the law is no excuse".   If my daughter wants to drive a car, she can't just go out without learning how to do it.  How is it possible to run networks that way?  Sounds pretty irresponsible to me.

Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

Sent: Tuesday, June 4, 2013 2:49 PM

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 05 Jun 2013 16:33:38 +0100

On 05/06/2013 15:40, Nalini Elkins wrote:
> I would be interested in hearing from some of the people who think IPv6
> extension headers are wonderful.  (I have a feeling that I am leaning in
> that direction myself!).

The easiest way to figure this out would be to write a draft formally
deprecating EHs and make sure to cross-post to 6man, v6ops and [email-masked].

Let me know if you plan this so I can buy a huge batch of popcorn.  And a
bigger sofa, nomnomnom.

> Are these people fools?   Do they run
> slow-as-molasses networks where every packet is being defaulted to being
> routed in software?

They're not fools: they just don't run networks.  They design networking
protocols and leave implementation to someone else.  Do you see Ford car
designers driving formula one?  Probably not.

FWIW, it would make my day to permanently disable ipv4 at ietf meetings so
that everyone at the IETF would be forced to eat ipv6 dog food.  The DHCPv6
vs RA problem would come flying into focus if this happened, and I'd be
most pleased if the RA jihadis finally got their comeuppance.

> What do Joint Interoperability Test Command (JITC) and National
> Institute of Standards and Technology (NIST) in the US have to say about
> this?

No idea, and no idea why this would even be relevant to ietf standards any
more than the opinions of any other regional standards bodies.

> So IPv6 ESP not fine?  This is going to be a surprise to many.

It's just that there is plenty of hardware out there which won't handle it
very well, that's all.

> 1.  I am not getting why forwarding decisions are having to be made by
> looking at anything other than the IP addresses.  What am I missing
> here?

as Joel mentioned, ECMP on routers, or policy routing, or QoS.  Other
generic middleboxes will also make forwarding decisions based on arbitrary
packet data.

> 2.  Are you talking about ASICs on large core switches, routers,
> firewalls, load balancers or all the above?

pretty much any device which makes forwarding decisions on the internet.

> 3.  It was my understanding the the ASIC size was in the range of 256
> bytes to 1K.  This seems to leave plenty of room for the IPv6 main
> header & quite a few extensions.

it is very small on plenty of boxes, e.g. 64 bytes on c7600/rsp720.

> Realistically, we are talking about the extension header that we want
> implemented being available 5 - 6 years from now (or even more).  We are
> considering the time spent for the IETF approval process, then to get on
> the development priority list for the stack software vendors, then to
> get that particular release of the operating system actually out at
> customer sites, etc.  I hope the world will look a bit different at that
> point in this regard.

An alternative point of view is: extension headers are causing more
problems as time goes on, not fewer. Let's fix this problem first before we
define more of them and make our current / future problems worse.

Nick

---

From: Warren Kumari <[email-masked]>
Date: Wed, 5 Jun 2013 10:52:35 -0500

On Jun 4, 2013, at 4:49 PM, Nick Hilliard <[email-masked]> wrote:

> serious operational difficulties.  

I should mention that  we have had this almost exact set of conversations (not the headers bit, the "I want to squish some token somewhere in the packet" bit) a number of times before.

http://www.ietf.org/mail-archive/web/v6ops/current/msg15823.html
http://www.ietf.org/mail-archive/web/v6ops/current/msg15781.html
http://www.ietf.org/mail-archive/web/v6ops/current/msg15126.html
etc etc etc.

> The IETF won't deprecate them either,

--
"Working the ICANN process is like being nibbled to death by ducks,
it takes forever, it doesn't make sense, and in the end we're still dead in the water." 
    -- Tom Galvin, VeriSign's vice president for government relations.

---

From: Warren Kumari <[email-masked]>
Date: Wed, 5 Jun 2013 11:10:16 -0500

On Jun 5, 2013, at 10:33 AM, Nick Hilliard <[email-masked]> wrote:

> as Joel mentioned, ECMP on routers, or policy routing, or QoS.  

And filtering… (often the same logic as forwarding).
A large number of folk (including enterprises, content providers and ISPs (for DoS mitigation)) want to be able to filter traffic, either to only allow specific protocol / ports in, or to drop known attack traffic.
If there are extension headers your filtering logic may not be able to see the L4 info, and (obviously) an attacker can insert extensions to *cause* this. 

> pretty much any device which makes forwarding decisions on the internet.

Yup.

> it is very small on plenty of boxes, e.g. 64 bytes on c7600/rsp720.

and attackers can add extension headers to force the upper-layer past what the router can see. This makes the only safe assumption that the traffic is bad / attack traffic and drop it.

> define more of them and make our current / future problems worse.
+1
W

--
What our ancestors would really be thinking, if they were alive today, is: "Why is it so dark in here?"

    -- (Terry Pratchett, Pyramids)

---

From: Nalini Elkins <[email-masked]>
Date: Wed, 5 Jun 2013 09:50:14 -0700 (PDT)

>>define more of them and make our current / future problems worse.

Definitely we should fix the problem with extension headers.  Could not agree more.

BUT, I also think that we should proceed with our proposal for not a NEW extension header but an implementation of an existing header. - Destination Options.

I also asked a couple of organizations who have implemented IPv6 for a while if they disable IPv6 extension headers:

Here is the first reply:

"To answer your question, we don't disable or block IPv6 extension headers. I'm not sure if our routers process them (or some of them) in software or not, but  I'll try to find out. Either way, we haven't noticed an performance issue. Although I guess we won't know for sure of the impact until the volume of IPv6 traffic scales up significantly in our network - v6 is still only about 8% of our total traffic. The most common extension header observed in our network is the fragment header (usually because of fragmented DNS UDP response messages)."

BTW, I know that this organization uses DNSSec, so that is why the fragmentation of DNS.

2nd reply:

"It depends. (don’t you hate that kind of answer). This is what I have heard. On some of the older OS’s and equipment, they will drop the extension headers because they don’t know what to do with them. Then they started running in software because they didn’t have chipsets supporting IPv6. I’m not sure who is running IPv6 in ASIC now but I have heard that some company whose name I can’t remember is (the one that competes with CISCO on the big routers).  That being said, all of our traffic is outward facing systems that are not really inside our network. We’re not running v6 inside the network yet."

Now I think that some of the discussion and differing points of view that we are seeing on this set of discussions has to do with the difference between operators who run large private networks and operators who are on the Internet.  We can control our environment to a much higher degree than those on the Internet.  We also need diagnostics and response time data in a way that those on the Internet do not.  If I might use my analogy of animals in the savanna, we are giraffes and the others are zebras.  Our needs are similar but in some ways different.

The header we are suggesting is optional.  If people don't want to use it, then they don't have to.  We will use it in our networks & make sure that our hardware is configured to support it properly and that our vendors support it properly.

We are just trying to take care of our operational needs in running real networks every day. 

Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

---

From: Nalini Elkins <[email-masked]>
Date: Wed, 5 Jun 2013 09:51:46 -0700 (PDT)

>> and attackers can add extension headers to force the upper-layer past what the router can see. This makes the only safe assumption that the traffic is bad / attack traffic and drop it.

If on our controlled, secure private networks, we have attackers who are able to modify packets.  We have much bigger problems than packets with IPv6 extension being dropped.
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

---

From: Brian Haberman <[email-masked]>
Date: Wed, 05 Jun 2013 12:57:03 -0400

> If there are extension headers your filtering logic may not be able to see the L4 info, and (obviously) an attacker can insert extensions to *cause* this.

Yes, this concern has been raised in the past.  And my question then was 
: Can anyone show us a legitimate use of a long chain of extension headers?

I can't recall anyone providing such evidence, so my conclusion is that 
it is probably reasonable to drop packets that push the L4 header that 
far down the chain.  I would still welcome input on legitimate packets 
containing long header chains...

Regards,
Brian

---

From: Nalini Elkins <[email-masked]>
Date: Wed, 5 Jun 2013 11:58:06 -0700 (PDT)

>>containing long header chains...

It depends on what you are calling "long".  If long = 256 bytes or more, then OK.  If long = less than 64 bytes, I would say not OK as that precludes just about everything.
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
 From: Brian Haberman <[email-masked]>
To: [email-masked] 
Sent: Wednesday, June 5, 2013 9:57 AM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

Regards,
Brian

---

From: Brian Haberman <[email-masked]>
Date: Wed, 05 Jun 2013 15:09:09 -0400

> It depends on what you are calling "long".  If long = 256 bytes or more, then OK.  If long = less than 64 bytes, I would say not OK as that precludes just about everything.

The question was really directed at two sets of folks and looking for 
two types of input.  For the vendors, how far into a header chain can 
you reasonably parse?  For operators, how many extension headers have 
you seen in the wild?

Each has some interesting implications to this whole discussion.

Regards,
Brian

---

From: joel jaeggli <[email-masked]>
Date: Wed, 05 Jun 2013 12:43:21 -0700

On 6/5/13 12:09 PM, Brian Haberman wrote:
>> It depends on what you are calling "long".  If long = 256 bytes or 
>> more, then OK.  If long = less than 64 bytes, I would say not OK as 
>> that precludes just about everything.
>>
>
> you reasonably parse?
There are devices forwarding in the core of the internet that can only 
look out 53bytes (wonder how that number was picked)  there are other 
that can do 256. there are few if any asic based forwarding engines that 
can do longer.
>   For operators, how many extension headers have you seen in the wild?
>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 06 Jun 2013 08:12:24 +1200

On 06/06/2013 07:43, joel jaeggli wrote:

...
> look out 53bytes (wonder how that number was picked)  

Since I don't see a smiley, I need to say: ATM.

---

From: Nalini Elkins <[email-masked]>
Date: Wed, 5 Jun 2013 13:18:05 -0700 (PDT)

...

>> Since I don't see a smiley, I need to say: ATM.

Guys, correct me if I am wrong, but are the 53 byte ASICs for ATM switches?  If so, then are they even parsing IP layer stuff or just sticking to layer 2?
_______________________________________________
v6ops mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/v6ops

---

From: Warren Kumari <[email-masked]>
Date: Wed, 5 Jun 2013 15:24:00 -0500

On Jun 5, 2013, at 3:18 PM, Nalini Elkins <[email-masked]> wrote:

> Guys, correct me if I am wrong,

Sure.

> but are the 53 byte ASICs for ATM switches?

No.

>  If so, then are they even parsing IP layer stuff or just sticking to layer 2?

Layer3. As *an* example, Juniper Martini architecture boxes:
"The Martini architecture is applied most directly in the M40, M20, M5, and M10 routers, and is used in a scaled up form in theM160 and a scaled down form in the Calvin architecture routers, the M7i and M10i."

There are many many many deployed boxes like this (and operators who are currently dropping all traffic that doesn't have the next-header as TCP/UDP/ICMP).

W

-- 
Eagles soar but a weasel will never get sucked into a jet engine

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 06 Jun 2013 08:44:02 +1200

>> but are the 53 byte ASICs for ATM switches?

Oh? I bet you they are reusing existing integrated circuits designed to
match entire ATM cells and nothing more.

> There are many many many deployed boxes like this (and operators who are currently dropping all traffic that doesn't have the next-header as TCP/UDP/ICMP).

As long as they don't claim to support IPv6 (as standardised by RFC 2460)
there's no problem with that. However, I don't think the IETF should be
making excuses for them.

   Brian

---

From: Warren Kumari <[email-masked]>
Date: Wed, 5 Jun 2013 15:48:46 -0500

On Jun 5, 2013, at 3:44 PM, Brian E Carpenter <[email-masked]> wrote:

> As long as they don't claim to support IPv6 (as standardised byRFC 2460)

So, if the device support looking 128 bytes into the packet is it RFC 2460 "compliant"? 256 bytes? 512bytes? 

It sure would to be nice if *some* carrier class boxes were…

W

-- 
Militant Agnostic -- I don't know and you don't either...

---

From: Nalini Elkins <[email-masked]>
Date: Wed, 5 Jun 2013 13:54:00 -0700 (PDT)

Thanks guys!  Very illuminating.  

We will certainly be on the lookout for these kind of boxes on our networks as we proceed.

W

---

From: joel jaeggli <[email-masked]>
Date: Wed, 05 Jun 2013 15:13:27 -0700

> there's no problem with that. However, I don't think the IETF should be
I don't have to make excuses for them. just forward packets across them. 
Asic design from 10 years ago has some compromises... eventually they'll 
age out of the internet. That's not the only platform will similarly low 
limitations , as noted there are platforms with higher limits all of 
them that I'm aware of have limits however.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 06 Jun 2013 10:51:52 +1200

> So, if the device support looking 128 bytes into the packet is it RFC 2460 "compliant"? 256 bytes? 512bytes? 

Well, yes, there's a problem there. The IPv6 header is a linked list
of variable length. Hardware architectures that assume a consistent
signature in the first 2**N bytes won't work well on a linked list.

I don't know what to say since this has been the IPv6 architecture
since 1996. All we can do, which a couple of active drafts in 6man
are trying to do, is clarify and limit the scope of the issue.

> It sure would to be nice if *some* carrier class boxes were…

The design assumption was that line-speed boxes will not look
beyond the first 40 bytes. As soon as they do so, you're into
linked-list processing.

    Bian

---

From: joel jaeggli <[email-masked]>
Date: Wed, 05 Jun 2013 18:43:55 -0700

> are trying to do, is clarify and limit the scope of the issue.
It is useful as a cogent jumping-off point to be able to describe things 
as they are rather than as we think they should be. while I think the 
later is important, the former is as well.

If we're disappointed that Equipment Vendors (that is, some ietf 
participants) went off and built asics that allowed the internet to 
grow, businesses to be successful and incurred NRE associated with 
supporting protocols for which they were a decade or more away from 
recovering their investment on , then I don't know what to say except 
that I am not. The ipv6 forwarding architecture and requirements got to 
ride along with the ipv4 forwarding logic, if it hadn't or couldn't been 
that case for some reason it's entirely possible that it wouldn't exist. 
Their foresight and effort  mean I can run my service on ipv4 and ipv6 
today, with some caveats.
> beyond the first 40 bytes.
Line speed in 1994 was on a general purpose cpu. A lot of assumptions 
about how forwarding was to be done were simply wrong. As it turns out 
for example we can forward on variable length headers (ipv4) as long as 
they're short. hop-by-hop options fall into the domain of linked list 
processing and they pose a similar problem.
>   As soon as they do so, you're into
> linked-list processing.
>
>      Bian
>
>

---

From: [email-masked]
Date: Thu, 06 Jun 2013 07:51:58 +0200 (CEST)

> are trying to do, is clarify and limit the scope of the issue.

Isn't this a pretty fundamental problem/flaw? As a *user* of high
speed routers, I want to know that my boxes are going to forward a
packet within a certain small time - which means that the address
*lookup* time must be bounded. A potentially long chain of headers
which is *not* bounded doesn't give me confidence. So I will vote
with my wallet and purchase boxes from the equipment vendors that
bound the lookup time (and the number of bytes they look into the
headers).

Steinar Haug, AS 2116

---

From: Lorenzo Colitti <[email-masked]>
Date: Thu, 6 Jun 2013 15:16:34 +0900

> "compliant"? 256 bytes? 512bytes?

If you believe Nalini's argument above, to comply with RFC 2460, the router
must be able to look 65575 bytes into the packet. RFC 2460 places no limits
on the size of the extension header chain, the 16-bit payload length field
counts up 65535, and you have to add 40 bytes for the IPv6 header.

If you don't think that is realistic, then I agree with you.

---

From: Lorenzo Colitti <[email-masked]>
Date: Thu, 6 Jun 2013 15:20:09 +0900

On Thu, Jun 6, 2013 at 1:51 AM, Nalini Elkins <
[email-masked]> wrote:

> If on our controlled, secure private networks, we have attackers who are
> able to modify packets.  We have much bigger problems than packets with
> IPv6 extension being dropped.
>

Wait - a while ago I thought you said that you were targeting this option
for use on the general Internet and on general operating systems. Now
you're saying that it only needs to be work on private networks.

Which of the two is it?

---

From: Ole Troan <[email-masked]>
Date: Thu, 6 Jun 2013 11:22:25 +0200

> If you believe Nalini's argument above, to comply with RFC 2460, the router must be able to look 65575 bytes into the packet. RFC 2460 places no limits on the size of the extension header chain, the 16-bit payload length field counts up 65535, and you have to add 40 bytes for the IPv6 header.

a router as defined by RFC2460 does _not_ examine or process any extension headers (with one exception).

a router is RFC2460 compliant if it can parse the 40 byte IPv6 header.

I'm not sure I see the need for the IETF to get involved in anything 'deeper' than that.

cheers,
Ole

---

From: Lorenzo Colitti <[email-masked]>
Date: Thu, 6 Jun 2013 18:40:22 +0900

> a router is RFC2460 compliant if it can parse the 40 byte IPv6 header.

True.

> I'm not sure I see the need for the IETF to get involved in anything

That's a fair position, but it's also not very useful if we standardize
something that cannot be deployed in today's Internet.

---

From: Ole Troan <[email-masked]>
Date: Thu, 6 Jun 2013 11:52:46 +0200

Lorenzo,

> That's a fair position, but it's also not very useful if we standardize something that cannot be deployed in today's Internet.

true, but designing for the other extreme, for an Internet where there are middleboxes that inspect and wants to understand every bit in a packet is impossible.

my point was that we  (as in the IETF) shouldn't require any router to have to be able to parse deeper than 40 bytes.
if you need a router that can parse and twiddle every bit in a packet, go talk to your vendor (and I'll be happy to sell you one).

I'm fully with you with regards to taking the realities of the Internet into consideration when designing new protocols and extensions though.
hopefully we will be able to take a comprehensive look at that for this problem in 6man.

cheers,
Ole

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 06 Jun 2013 11:59:41 +0100

On 06/06/2013 10:52, Ole Troan wrote:
> true, but designing for the other extreme, for an Internet where there
> are middleboxes that inspect and wants to understand every bit in a
> packet is impossible.

we're not talking about that level of extremity - we're talking about basic
functionality of a router.  In my books this would require e.g. the ability
to handle packet filtering based on packet header characteristics using L3
port ACLs.  There's nothing interesting about a box which can only forward
packets and nothing else.

re:

> my point was that we  (as in the IETF) shouldn't require any router to
> have to be able to parse deeper than 40 bytes.

> I'm fully with you with regards to taking the realities of the Internet
> into consideration when designing new protocols and extensions though.

this is giving me an uncomfortable level of cognitive dissonance.

Nick

---

From: [email-masked]
Date: Thu, 06 Jun 2013 13:22:07 +0200 (CEST)

> true, but designing for the other extreme, for an Internet where there are middleboxes that inspect and wants to understand every bit in a packet is impossible.

There will be middleboxes that need to look at enough of the IPv6
headers to find for instance port numbers.

At my AS borders, I typically do some filtering which depends on more
of the protocol header than just the IPv4/IPv6 address. Examples could
be:

- Allow carefully policed UDP traffic in the classical Unix traceroute
range to reach my routers.
- Deny DNS queries to my recursive name servers while allowing the same
servers to be pinged.

If you simply ignore such boxes, or label them "firewalls which we don't
want to deal with", you are excluding a significant group of boxes which
in reality are high speed routers that happen to have stateless firewall
abilities.

Steinar Haug, AS 2116

---

From: Ole Troan <[email-masked]>
Date: Thu, 6 Jun 2013 13:41:21 +0200

Steinar,

> - Deny DNS queries to my recursive name servers while allowing the same

just to be clear I have no issue with this, were the hosts that you control
basically outsource filters they would have applied anyway out to the network.

> in reality are high speed routers that happen to have stateless firewall

what do you think should be the consequence for the work we do in the IETF?

I think the IETF should be on the side of the end to end Internet and follow
the principles outlined in RFC1958.

clearly there are some better design choices than others. e.g. RFC4821 vs RFC1981,
but I don't think the IETF should just give up the ida of the end to end Internet and stop designing a new L4 header ever again.

that said, I'm not in favour of making middleboxes' life harder just for the sake of it. from that perspective
extension headers are a bad idea, and I wouldn't be opposed to removing them altogether.

cheers,
Ole

---

From: Bill Jouris <[email-masked]>
Date: Thu, 6 Jun 2013 07:14:13 -0700 (PDT)

> From: Lorenzo Colitti <[email-masked]>>  

> be deployed in today's Internet.

But doesn't that amount to saying that *every* RFC must be backwards compatable?  Forever?  (When IPv6 was first specified, were there ANY parts of the then-current Internet which could deal with its headers?) 

I'm not saying we should just ignore current limitations.  But it doesn't seem reasonable to make them a complete bar to change either.

Bill Jouris
Inside Products, Inc.
www.insidethestack.com
831-659-8360
925-855-9512 (direct)

---

From: Nalini Elkins <[email-masked]>
Date: Thu, 6 Jun 2013 07:26:19 -0700 (PDT)

>>If you believe Nalini's argument above, to comply with RFC 2460, the router must be able to look 65575 bytes into the packet. RFC 2460 places no limits on the size of the extension header chain, the 16-bit payload length field >>counts up 65535, and you have to add 40 bytes for the IPv6 header.

>>If you don't think that is realistic, then I agree with you.

Guys, I am not saying 65,535 bytes.  But, I do think 256 bytes is reasonable.  Also, I do also think that a standard needs to be set and the IETF is the right body to set it.
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
 From: Lorenzo Colitti <[email-masked]>
To: Warren Kumari <[email-masked]> 
Cc: "[email-masked]" <[email-masked]> 
Sent: Wednesday, June 5, 2013 11:16 PM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 06 Jun 2013 15:28:38 +0100

On 06/06/2013 15:14, Bill Jouris wrote:
> (When IPv6 was first specified, were there ANY parts
> of the then-current Internet which could deal with its headers?)

yes and in fact much more so than now.  When it was specified first in 1995
or so, most packet forwarding was done without hardware acceleration.
These days, only tiny edge networks can use kit which doesn't use hardware
assisted forwarding.

> I'm not saying we should just ignore current limitations.  But it doesn't
> seem reasonable to make them a complete bar to change either.

given that the problem is arguably getting worse, not better, I think the
level of scepticism shown by v6ops regarding any further extension headers
is by far the most prudent approach to further EHs right now.  I understand
that you disagree.

Nick

---

From: Owen DeLong <[email-masked]>
Date: Thu, 6 Jun 2013 11:21:18 -0700

> packets and nothing else.

What is an L3 port ACL? 

If you are referring to TCP and UDP port numbers, then isn't that L4?

Owen

---

From: Joe Touch <[email-masked]>
Date: Thu, 06 Jun 2013 11:36:22 -0700

> packets and nothing else.

That box is called a router - see RFC 1812.

Maybe that makes it uninteresting to you, but I would argue then that 
you are interested in firewalls, not routers.

Joe

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 06 Jun 2013 19:39:24 +0100

> What is an L3 port ACL? 

uh, an ACL on a layer 3 port.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 06 Jun 2013 20:43:16 +0100

On 06/06/2013 19:36, Joe Touch wrote:
> Maybe that makes it uninteresting to you, but I would argue then that you
> are interested in firewalls, not routers.

on the L3 infrastructures I manage, all the routers have urpf and
infrastructure ACLs applied, and use lots of qos stuff to stop their
respective control planes from being trashed by unwanted traffic.  These
boxes were definitely routers when I installed them, and unless someone has
managed to sneak into a number of locked cages and done a surreptitious
forklift swapout of the whole lot for firewalls, I'm willing to bet that
they still are. :-)

Nick

---

From: Joe Touch <[email-masked]>
Date: Thu, 06 Jun 2013 13:16:59 -0700

They're sold as routers, but they have two distinct functions:

router functions
firewall functions

Routers don't *need* to do anything with L4 to be called a router. 
That's not how router vendors market them, agreed.

Joe

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 06 Jun 2013 21:36:42 +0100

On 06/06/2013 21:16, Joe Touch wrote:
> Routers don't *need* to do anything with L4 to be called a router. That's
> not how router vendors market them, agreed.

Joe, they are routers.  They need stateless packet filtering and various
other things which require packet header inspection to survive on the
internet, but that does not make them routers any the less.

We can pick nits about this all day if you want, but a firewall is
generally accepted to be a device which performs stateful/inspected flow
forwarding, and a router is generally accepted to be a device which does
stateless packet forwarding (albeit with features which allow stateless
filtering).  Now can we move on?

Nick

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 07 Jun 2013 08:43:06 +1200

On 06/06/2013 17:51, [email-masked] wrote:
> Isn't this a pretty fundamental problem/flaw? 

Yes. Designing a box that is supposed to look beyond the first
40 bytes that cannot handle the linked-list design of the IPv6
header is a fundamental flaw in the box design. I don't think
it's very hard to design silicon to flip through a linked list,
but it's not the same as silicon that matches a given bit
pattern in the first N bits.

> As a *user* of high
> *lookup* time must be bounded. 

It is bounded. What isn't bounded is the time to look up stuff
way beyond the destination address.

> A potentially long chain of headers

I do agree that a completely unbounded chain is worrisome
and that could be clarified in the specs.

If your parenthesis is relevant, it means they have the flaw of
not handling the linked list structure in hardware. It would be
good to ask them when they plan to do so, given that it was
defined in 1996.

Also when they plan to support RFC 6437 and 6438, which is
another part of this puzzle.

I realise it's no comfort for current operations, but the
vendors really do need to catch up with the specs.

    Brian

---

From: [email-masked]
Date: Thu, 06 Jun 2013 22:47:59 +0200 (CEST)

> That's not how router vendors market them, agreed.

That's a nice distinction in theory. How much does it matter in
practice? I use some rather basic, stateless, firewall functions for
my routers. A router which doesn't offer these functions is simply
not a relevant box in my network.

I think it would be useful for router vendors to have some idea of
how far into the IPv6 header they need to look, in order to be a
proper IPv6 router.

Steinar Haug, Nethelp consulting, [email-masked]

---

From: [email-masked]
Date: Thu, 06 Jun 2013 22:58:54 +0200 (CEST)

> way beyond the destination address.

And therein lies the problem. As pointed out by Nick Hilliard and
myself, we *need* those stateless firewall functions, which may
need to look up stuff way beyond the destination address. I don't
find a discussion of whether such functions *should* be needed in
a router or not all that relevant.

Steinar Haug, AS 2116

---

From: Joe Touch <[email-masked]>
Date: Thu, 06 Jun 2013 14:10:37 -0700

> how far into the IPv6 header they need to look, in order to be a

To be a *router*, they definitely need to process the HBH options header 
in its entirety if present.

To do things beyond that, they need to look far enough into the header 
chain to satisfy the vendor claiming that the device performs a 
non-router function, such as L4 filtering, etc.

I.e., if you want to be just a router, stop at HBH.

If being more than a router is what you want to do - and that's fine 
with me - then dig in and do the work to earn it.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 06 Jun 2013 14:17:00 -0700

> a router or not all that relevant.

Just as I don't find a discussion of how much the cost to implement such 
functions makes the life of router venders more complex.

If the kitchen is too hot...

Joe

---

From: Havard Eidnes <[email-masked]>
Date: Thu, 06 Jun 2013 23:36:57 +0200 (CEST)

> Just as I don't find a discussion of how much the cost to implement
> such functions makes the life of router venders more complex.
>

That's one vantage point.  Another would be one which talks about
ivory towers and losing touch with what is practically possible now.

Regards,

- Håvard

---

From: Joe Touch <[email-masked]>
Date: Thu, 06 Jun 2013 14:41:28 -0700

> ivory towers and losing touch with what is practically possible now.

If it's possible, then there's no problem.

What you're all suggesting here are ways to limit IPv6 to make it more 
convenient for router vendors to implement various features that were 
never part of the design (of either the Internet architecture nor of 
IPv6 in specific).

If you want to make sure that IP supports efficient router parsing of L4 
header information, IMO you can take that up when we design IPv7, but it 
was never a requirement of IPv6. Assuming it now is - as you are all 
discussing - impractical.

Joe

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 06 Jun 2013 22:04:59 -0400

In a formal sense Joe, you are correct.  There is no formal requirement 
in IPv6 for Layer 4 visibility to routers.

And one can argue that firewalling should be done in specialized boxes. 
  I tend to disagree, but that is not the point.

Even without firewalls, there is a current, real world, operational need 
for almost all routes to be able to find layer 4 information.  Operators 
depend upon effective ECMP (and at layer 2, LAG) operation.

While I would like things to work better (for example, by putting the 
needed information into the flow label), current deployments can not 
depend upon such an unspecified and undeployed host behavior.

As such, real world deployed routers depend upon being able to find L4 
information.  Some devices can chase all the header chains, so that 
requiring L4 be in the packet is sufficient.  Many routers have limits 
on what they can chase.

Claiming that the reasonable and effective limitations people have had 
to put in place are "wrong" and vendors should stop doing that, requires 
that there be an answer to the issue.

Yours,
Joel

PS: This seems largely orthogonal to Ms. Elkins original topic.

---

From: Nalini Elkins <[email-masked]>
Date: Thu, 6 Jun 2013 21:23:11 -0700 (PDT)

Speaking of Ms. Elkins original topic, some of us in the real world have used the old IPID field in IPv4 as a packet sequence number to do diagnostics more quickly even though it was not intended as such.   In the real world, you do what you have to to keep your network running and solve problems.  Theory goes out the window when real users are involved.

This is why we want the new implementation of the destination options header that we have defined.   

If I might, I seem to sense that some of the people on this list possibly may not really understand or may not have much sympathy for the kind of pressure we are under to solve problems quickly and to meet service level requirements for performance.  If you can think of the type of networks our team runs - the health care networks, the banking sector, the stock markets, you can see that we diagnosticians have a lot of pressure on us.

I understand that there are a number of issues with IPv6 extension headers, no real RFCs for middle boxes, and the ASIC issue.  And, tomorrow (or Saturday) when I can get away from actual work, I will respond on the very good discussion on the 6man list on draft-ietf-6man-ext-transmit-01.txt.   Definitely, these are very critical points.  And, if we can help in any way to help in creating standards to address these issues, we stand ready and willing to help in any way possible.  I feel sure that these problems will be solved in the future.

We would really appreciate support for what we are asking for which is a way for us to solve our chronic operational problems which we are afraid will get worse with IPv6.

Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
From: Joel M. Halpern <[email-masked]>
To: Joe Touch <[email-masked]> 
Cc: [email-masked] 
Sent: Thursday, June 6, 2013 7:04 PM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

In a formal sense Joe, you are correct.  There is no formal requirement in IPv6 for Layer 4 visibility to routers.

And one can argue that firewalling should be done in specialized boxes.  I tend to disagree, but that is not the point.

Even without firewalls, there is a current, real world, operational need for almost all routes to be able to find layer 4 information.  Operators depend upon effective ECMP (and at layer 2, LAG) operation.

While I would like things to work better (for example, by putting the needed information into the flow label), current deployments can not depend upon such an unspecified and undeployed host behavior.

As such, real world deployed routers depend upon being able to find L4 information.  Some devices can chase all the header chains, so that requiring L4 be in the packet is sufficient.  Many routers have limits on what they can chase.

Claiming that the reasonable and effective limitations people have had to put in place are "wrong" and vendors should stop doing that, requires that there be an answer to the issue.

Yours,
Joel

---

From: Lorenzo Colitti <[email-masked]>
Date: Fri, 7 Jun 2013 14:58:57 +0900

On Fri, Jun 7, 2013 at 1:23 PM, Nalini Elkins <
[email-masked]> wrote:

> If I might, I seem to sense that some of the people on this list possibly
> may not really understand or may not have much sympathy for the kind of
> pressure we are under to solve problems quickly and to meet service level
> requirements for performance.  If you can think of the type of networks our
> team runs - the health care networks, the banking sector, the stock
> markets, you can see that we diagnosticians have a lot of pressure on us.
>

Agreed. But all those networks are highly specialized, highly closed, and
highly integrated, and the solutions that are appropriate for them might
not be appropriate for the general Internet.

Also, if you want to solve this problem quickly, then... well, an extension
header is not a good idea, because it has to be implemented at the lowest
level possible (the host operating system). If you need a quick solution,
then what you want is the sequence number field in the GRE header.

---

From: Ray Hunter <[email-masked]>
Date: Fri, 07 Jun 2013 08:41:55 +0200

Joe Touch wrote:
> If you want to make sure that IP supports efficient router parsing of
> L4 header information, IMO you can take that up when we design IPv7,
> but it was never a requirement of IPv6. Assuming it now is - as you
> are all discussing - impractical.
>
> Joe
>
Why do you think it's impractical to make IPv6 more convenient for
middle-box and router vendors to parse without going back to the drawing
board?

What if there were a bunch of informational RFC's that said "if you want
your packet switched on the fast past in hardware make sure it ..... blah"

Where blah is something like:
has (a maximum of) n MPLS labels in the the stack
has (a maximum of) n QinQ tags
.....
the hop by hop header may or may not be present, but if present it
should always contain exactly these TLV hop-by-hop options in exactly
this transmission order.

Wouldn't that ensure that it's in everyones' best interests to adhere to
these informational RFCs, without having to completely redo standards
track RFCs? (Other packets confirming to standard track RFCs should
still be properly switched, but on a slightly slower, or very much
slower path)

At the moment, "blah" would seem to be "make sure your packet does not
contain any extension headers, or else it will likely be dropped" and
we've currently got the wrong optimization of flexibility and length
efficiency versus switching speed.

best regards,
RayH

---

From: Havard Eidnes <[email-masked]>
Date: Fri, 07 Jun 2013 09:06:24 +0200 (CEST)

> What if there were a bunch of informational RFC's that said "if
> you want your packet switched on the fast past in hardware make
> sure it ..... blah"

That way of implementing things has issues.  I"m explaining below.

> (Other packets confirming to standard track RFCs should still
> be properly switched, but on a slightly slower, or very much
> slower path)

If one isn't very careful when implementing such a scheme, it is
easy to create a DoS vector.  For some reason that's not very
popular in certain circles (imagine that!).

Regards,

- Håvard

---

From: Ray Hunter <[email-masked]>
Date: Fri, 07 Jun 2013 09:24:49 +0200

> Havard Eidnes <mailto:[email-masked]>
> 7 June 2013 09:06

So those operators with DoS concerns will continue to ignore the IETF
standards and drop soft-switched packets that don't conform to the
informational RFC for a strict-structured hop-by-hop header optimized
for hardware forwarding.

How's that worse than today, where people are apparently dropping all
packets containing any extension headers (because they're all soft
switched)?

The basic idea being that we can optimize/facilitate hardware parsing
further and further along the header chain by publishing additional
informational RFCs, and at some point in the future we'll get enough
extension headers supported together with hardware forwarding that they
become useful. I'd certainly like to be able to buy a firewall with ASIC
optimized forwarding of IPv6 (including extension headers).

The alternative would seem to be a stalemate of continual shouting from
the ivory tower that hardware vendors just need to learn to live with
arbitrary packet complexity, whilst operational people drop everything
containing an extension header so there's no market demand for the
hardware vendors to do anything.

regards,
RayH

---

From: [email-masked]
Date: Fri, 07 Jun 2013 09:50:33 +0200 (CEST)

> packets containing any extension headers (because they're all soft

Also, remember there is plenty of hardware out there that doesn't *do*
soft switching at all. If a packet cannot be forwarded in hardware, it
is dropped.

> optimized forwarding of IPv6 (including extension headers).

I definitely think this is worth pursuing, as long as the point of not
creating a DoS vector is kept in mind.

> hardware vendors to do anything.

Or operational people voting with their wallets, and buying the box 
that kan "only" look 256 bytes into the header, because it is N% less
expensive than the box that can look a full 64 kByte into the header.

Steinar Haug, AS 2116

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 7 Jun 2013 05:54:04 -0700 (PDT)

On Fri, Jun 7, 2013 at 1:23 PM, Nalini Elkins <[email-masked]> wrote:

>>Agreed. But all those networks are highly specialized, highly closed, and highly integrated, and the solutions that are appropriate for them might not be appropriate for the general Internet.

Well, the problem is that we all have to live together.   That is, I wish we had a set of rules for us and a set of rules for the Internet.  But, what has happened is that we have to live with the rules for the general Internet, and sometimes that doesn't work for us.   You are actually pointing out the fundamental problem.   RFCs defined for the general Internet are having to be used on Intranets!  So, as Ole puts it, maybe sometimes, we have to say, this is for 'consenting adults' only?  I don't know.  What do you think?  

>> Also, if you want to solve this problem quickly, then... well, an extension header is not a good idea, because it has to be implemented at the lowest level possible (the host operating >>system). If you need a quick solution, then what you want is the sequence number field in the GRE header.

Our networks don't always use GRE & I don't know that we want to change everything to encapsulate everything everywhere in GRE.

---

From: Joe Touch <[email-masked]>
Date: Fri, 7 Jun 2013 07:43:35 -0700

> This is why we want the new implementation of the destination options header that we have defined. 

The old IPv4 ID field never did support diagnostics correctly, because:

	- they were never intended for diagnostics at all

	- vendors decided it was too much work to generate unique IDs correctly:
		- endpoints that want to go faster than a few Mbps
		- NATs that want less state
		- cellphones that want even less state

So basically the device vendors decided they should make more money, at the expense of the diagnostics vendors.

A similar tension happened when IPv6 was being designed, and the compromise was to put everything a router needed in either the main header or the HBH header.

Putting more there - a hostID, packetID, or a copy of the transport ports - is the same tension between those who want to build "cheaper than chasing a chain" routers vs. those who want endpoints that run fast, can be implemented cheaply, or are secure (against DOS attacks, or just implement privacy).

So, correct me if I'm wrong, but isn't this discussion then about 

	what do you propose to put in the HBH header, 
	and is it safe and worth the cost?

But ignoring packets that use the chained headers that follow is a non-starter for IPv6.

Joe

---

From: Shane Amante <[email-masked]>
Date: Fri, 7 Jun 2013 08:46:50 -0600

Hi Ray, Joe, others,

Please see below.

> efficiency versus switching speed.

I would like to point out the following effort, initiated by Curtis Villamizar and to which I and others have contributed, in the MPLS WG to do what you're describing above specific to _MPLS_:
http://tools.ietf.org/html/draft-ietf-mpls-forwarding-00

From the Abstract of the above IETF draft:
---snip---
   This document provides guidelines for implementors regarding MPLS
   forwarding and a basis for evaluations of forwarding implementations.
   Guidelines cover many aspects of MPLS forwarding.  Topics are
   highlighted where implementors might potentially overlook practical
   requirements which are unstated or underemphasized or are optional
   for conformance to RFCs.
---snip---

Note first that the above document is _Informational_.  Second, the above document is intended to address MPLS _forwarding_.  To my knowledge, we have *not* stated anything specific to length/number of IPv6 extension headers, nor do I think we should.  However, we have stated that, for practical reasons, if a forwarding HW and/or ASIC vendor wants to be successful, then they SHOULD allow for inspection of IPv4 and IPv6 header information, including src_port + dst_port, to support fine-grained load-balancing of IP/MPLS traffic over LAG or ECMP paths in the core of a MPLS network.  IMHO, if anyone attempts to specify new IPv6 Extension Headers or other proposals that increase the length of IPv6 Extension Header chains that would *inhibit* extraction of Layer-4 information for the purposes of load-balancing or stateless FW filtering in routers, then operators (and, vendors they buy from) are not going to support it.

If you have any comments specific to this document, then please take them to the MPLS WG mailing list.

-shane

---

From: "Ackermann, Michael" <[email-masked]>
Date: Fri, 7 Jun 2013 16:36:40 +0000

Agree strongly that ignoring packets that have chained headers (which an intrinsic part of the protocol),  will be a HUGE detriment to IPV6 deployment in the real world.  

And regarding the comment about theory going out the window when it conflicts with reality.  Unfortunately TRUE DAT!   But I am hopeful  that we are not too late to get theory to somewhat match reality here and encourage (rather than discourage) IPV6 deployments.  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joe Touch
Sent: Friday, June 07, 2013 10:44 AM
To: Nalini Elkins
Cc: [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

Joe

---

From: "Ackermann, Michael" <[email-masked]>
Date: Fri, 7 Jun 2013 16:36:47 +0000

A big ECHO  from Blue Cross and the Health Industry in general,  on the comments below!!!

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Nalini Elkins
Sent: Friday, June 07, 2013 12:23 AM
To: Joel M. Halpern; Joe Touch
Cc: [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

Yours,
Joel

On 6/6/2013 5:41 PM, Joe Touch wrote:
>>>> find a discussion of whether such functions *should* be needed in a 
>>>> router or not all that relevant.
>>>

---

From: "Ackermann, Michael" <[email-masked]>
Date: Fri, 7 Jun 2013 16:36:47 +0000

Just wanted to tack on that most of our networks are not closed to the degree you might expect.   More and more of our connections  are client VPN's or Peer to Peer VPN's over the Internet.   So we are VERY interested in that path as well!

Secondly, we do NOT expect that ANYTHING in this space will happen quickly.   But if it is right and best, we are more than willing to put in the related time and effort to attain the strategic value our business demand.     Band aids usually fall off! :)

From: [email-masked] [mailto:[email-masked]] On Behalf Of Lorenzo Colitti
Sent: Friday, June 07, 2013 1:59 AM
To: Nalini Elkins
Cc: [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

On Fri, Jun 7, 2013 at 1:23 PM, Nalini Elkins <[email-masked]<mailto:[email-masked]>> wrote:
If I might, I seem to sense that some of the people on this list possibly may not really understand or may not have much sympathy for the kind of pressure we are under to solve problems quickly and to meet service level requirements for performance.  If you can think of the type of networks our team runs - the health care networks, the banking sector, the stock markets, you can see that we diagnosticians have a lot of pressure on us.

Also, if you want to solve this problem quickly, then... well, an extension header is not a good idea, because it has to be implemented at the lowest level possible (the host operating system). If you need a quick solution, then what you want is the sequence number field in the GRE header.

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 7 Jun 2013 10:07:26 -0700 (PDT)

>>    and is it safe and worth the cost?

Well, we are actually using the Destination Options header and not hop-by-hop.  But, same points.

>>But ignoring packets that use the chained headers that follow is a non-starter for IPv6.

YES!!!!!   Could not agree more!   So now academia and those of us with our hands completely dirty are in complete agreement.  There must be some kind of alignment of the stars or something!
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
 From: Joe Touch <[email-masked]>
To: Nalini Elkins <[email-masked]> 
Cc: Joel M. Halpern <[email-masked]>; "[email-masked]" <[email-masked]> 
Sent: Friday, June 7, 2013 7:43 AM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

Joe

---

From: Warren Kumari <[email-masked]>
Date: Fri, 7 Jun 2013 12:13:18 -0500

> And regarding the comment about theory going out the window when it conflicts with reality.  Unfortunately TRUE DAT!   But I am hopeful  that we are not too late to get theory to somewhat match rea…

If you are trying to encourage IPv6 deployment you need to consider the cost of deployment, what the incremental cost of looking further into the packet and what the value to the operators is.

Simply saying "you need to to be formally correct" (or "to meet my niche need") doesn't pay the bills.

W

--
For every complex problem, there is a solution that is simple, neat, and wrong.
                -- H. L. Mencken

---

From: "Ackermann, Michael" <[email-masked]>
Date: Fri, 7 Jun 2013 17:25:37 +0000

YES, I think we should consider costs of deployment.  We (customers) ultimately pay that price anyway.    I assume you are talking here about cost for router and firewall vendors to "tool up" to fully support IPV6.   As more of us actually start to utilize IPV6, the vendors will learn more and more about what that REALLY means and adapt to it.   I think the EH processing issue is a great example of that.   

If router and Firewall vendors need to make changes to accommodate  IPV6 and/or customer requirements, I am sensitive to their related efforts and costs, but it is what we must all do to survive and prosper.   

And while I am appreciative of the time, effort and cost this will take on one hand, I do not think that we should base planning for the architectures and protocols of the future, on hardware limitations of the past.   

We ALL will be expending cost, effort and time to accommodate IPV6 and any other fundamental changes we face.   It will be in very different areas for us, than for equipment vendors, but significant nonetheless.    My company isn't happy about it either, but it must be accepted.    

-----Original Message-----
From: Warren Kumari [mailto:[email-masked]] 
Sent: Friday, June 07, 2013 1:13 PM
To: Ackermann, Michael
Cc: Warren Kumari; Joe Touch; Nalini Elkins; [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

W

---

From: Warren Kumari <[email-masked]>
Date: Fri, 7 Jun 2013 12:56:02 -0500

> soft switching at all. If a packet cannot be forwarded in hardware, it

Yup. I think that plenty is actually "most deployed hardware with 10GB interfaces".

> expensive than the box that can look a full 64 kByte into the header.

Yup, and from talking to vendors, N is *large*. 

The big issue here isn't so much complexity of building ASICs that can parse the linked-list, it is the cost (bandwidth and memory) of dragging the headers up to the ASIC so that it can be inspected. 

This may take us down (yet another) rat-hole where folk express surprise at the cost of widgets, discussions on the actual cost vs discounted cost, If you can afford to pay X, you can afford X+10% (or 2*X), router-architecture, but I think it is worth folk having *some* understanding of the costs here.

Using public info:

Here is the example 'show chassis hardware models' for a Juniper T1600 from Juniper's site: http://www.juniper.net/techpubs/en_US/release-independent/junos/topics/task/operational/t4000-upgrade-verify-t640-t1600-components.html and using prices published here: http://www.technicacorp.com/federal/contracts/nasa_sewpiv/sewp_search?manufacturer=Juniper%20Networks,%20Inc.

We get: 

QTY     DESC    List    Total   Comment
3       T1600-FPC4-ES   "$300,000.00"   "$900,000.00"
1       T640-FPC2-E2    "$71,100.00"    "$71,100.00"
1       T640-FPC1-E2    "$47,400.00"    "$47,400.00"
1       T640-FPC3-E     "$118,500.00"   "$118,500.00"
1       T640-FPC3       $0.00   $0.00   Could not easily find public price.
5       SIB-I-T1600-S   "$50,000.00"    "$250,000.00"
3       PC-8GE-TYPE3-SFP-IQ2    "$120,000.00"   "$360,000.00"
2       PD-4XGE-XFP     "$250,000.00"   "$500,000.00"
2       PC-4OC48-SON-SFP        "$205,000.00"   "$410,000.00"
2       PC-1XGE-TYPE3-XFP-IQ2   "$141,000.00"   "$282,000.00"
2       PC-10GE-SFP     "$120,000.00"   "$240,000.00"
2       PB-1OC48-SON-SMSR       $0.00   $0.00   Could not easily find public price.
1       CB-T-S  "$9,000.00"     "$9,000.00"
1       SCG-T-S "$5,000.00"     "$5,000.00"
1       PD-4OC192-SON-XFP       "$660,000.00"   "$660,000.00"
1       PD-1OC768-SON-SR        "$620,000.00"   "$620,000.00"
1       PD-1CE-CFP-FPC4 "$650,000.00"   "$650,000.00"
1       PC-TUNNEL       "$120,000.00"   "$120,000.00"
1       PC-1OC192-SON-VSR       "$125,000.00"   "$125,000.00"
1       PC-1OC192-SON-SR2       $0.00   $0.00   Could not easily find public price.
1       PB-4OC12-SON-SMIR-REFURB        "$20,000.00"    "$20,000.00"    Refurb
1       PB-4GE-TYPE1-SFP-IQ2    "$32,000.00"    "$32,000.00"
1       PB-4GE-SFP      "$58,000.00"    "$58,000.00"
1       PB-1CHOC12SMIR-QPP      "$67,150.00"    "$67,150.00"

        Total           "$5,545,150.00"

This is for a large box, but not the most expensive-- it is also missing a control plane, the actual chassis / backplane, power, etc (I got bored).

Cisco pricing is similar - http://www.cisco.com/web/strategy/government/wsca/pricelist_archive.html Feel free to find 'show diag' for a CRS-1 (or -3) and do your own calculations…

In  the real world (what we are talking about here!), everyone gets a reasonable discount off list prices, but that discount is far from 100% :-P

What percentage more should I be willing to pay to be formally correct / RFC compliant? Or to support some niche use? How do I justify spending an additional X% to my bean-counters? What value can I point at? Seeing as things are working just fine without extensions headers at the moment, and the Internet Protocol Police have no enforcement power, it is very hard to justify any X, let alone the type of X that I've heard from vendors….

And no, additional bits in my packets for diagnostics is not the value here, I'm managing just fine without them, thanks though.

Yes, 'tis sad when the development of protocols / deployment of features is impacted by business[0] considerations -- welcome to the real world.

W

[0]: Or by physics ( like C / thermodynamics), or by technology (like power and cooling), or by politics, or by IPR, or by legacy considerations, or ...

--
"Go on, prove me wrong. Destroy the fabric of the universe. See if I care."  -- Terry Prachett

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Jun 2013 10:58:47 -0700

> hop-by-hop.  But, same points.

Well, HBH is the *correct* place to put things that are examined per-hop ;-)

> YES!!!!!   Could not agree more!   So now academia and those of us with
> our hands completely dirty are in complete agreement.  There must be
> some kind of alignment of the stars or something!

FWIW, ISI may be part of USC, but I've seen more academic positions in 
industry.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Jun 2013 11:14:55 -0700

Hi, all,

In the spirit of moving towards a solution, here's my understanding of 
the problems:

1. the need for load balancing

	- if packets had true E2E addresses, those addresses would be
	sufficient for load-balancing

	- NATs and tunnels interfere with address-based loadbal, so
	another solution needs to be found

	- that other solution typically uses L4 port info

	>> solution:

	- NATs should copy the L4 port pair to a HBH 'hostmux' ID
		NATs want the revenue of being an L4 device,
		and they need L4 info to accomplish this anyway,
		so they already do this work

	- tunnel ingresses should copy the inner L3 pair to
	the hostmux ID

2. the need for L4 port filtering or traffic differentiation

	>> solution:

	- parse the whole L4 header in the router;
	you want the revenue of calling yourself an L4 device,
	so you need to do the work to earn it ;-)

Joe

---

From: [email-masked]
Date: Fri, 07 Jun 2013 20:22:49 +0200 (CEST)

> YES!!!!!   Could not agree more!   So now academia and those of us with our hands completely dirty are in complete agreement.  There must be some kind of alignment of the stars or something!

Not sure how much further we're going to get in this discussion. In 
any case, I disagree at least to some degree with the top statement
here:

- I have a significant number of routers doing IPv6 forwarding in
hardware today. These routers may not necessarily be able to chase
long IPv6 header chains to find the L4 info.

- Given a choice between 1) finding the L4 info within that part of
the IPv6 header chain my routers *can* inspect (so that I can secure
my network) and 2) ignoring (dropping) a packet with a long IPv6
header chain - I'm going to drop the packet. Every time.

I'm not going to throw out my current equipment.

I may be able to put in stronger requirements on chasing long IPv6
header chains for later purchases - but I'm *not* going to request
the ability to inspect 64k bytes into the packet.

Steinar Haug, AS 2116

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Jun 2013 11:29:42 -0700

On 6/7/2013 11:22 AM, [email-masked] wrote:
>> YES!!!!! Could not agree more! So now academia and those of us
>> with our hands completely dirty are in complete agreement. There
>> must be some kind of alignment of the stars or something!
>

As long as those devices parse the main header and HBH options, that's 
fine. If what you *need* was for them to find L4 info in general, then 
you need to buy a better router, IMO.

> header chain - I'm going to drop the packet. Every time.

Which is your decision, but then if you were my ISP I'd find another ISP.

> the ability to inspect 64k bytes into the packet.

That is your *business* decision, but it doesn't magically make the use 
of chained headers incorrect or inappropriate, unless the market decides 
to keep using you as their ISP.

Joe

---

From: Ray Hunter <[email-masked]>
Date: Fri, 07 Jun 2013 21:10:26 +0200

Thanks for the reply. Inline.
> Warren Kumari <mailto:[email-masked]>
> 7 June 2013 19:56
OK. If we make a gross assumption that RFC 6437/6438 could "fix" the
core devices at 10Gbps and above, by replacing their requirements to
look at L4 headers with the flow label in the main IPv6 header [ECMP,
LAG, other load balancing use cases]

What about all the 1Gbps devices and below? The great unwashed at the edge?

There's plenty of devices in enterprise networks that also need access
to L4 headers apart from for LAG and ECMP use cases.
And they need to work at wire rates [albeit at lower wire
ratesgenerally, but not always]

> The big issue here isn't so much complexity of building ASICs that can parse the linked-list, it is the cost (bandwidth and memory) of dragging the headers up to the ASIC so that it can be inspecte…

OK. Interesting.

I understood that a significant problem was the inability to pre-fetch
the seemingly endless multiple possible branches in the extension header
+ options chain, so the parser could work efficiently in parallel. I'm
perfectly willing to be educated.
> This may take us down (yet another) rat-hole where folk express surprise at the cost of widgets, discussions on the actual cost vs discounted cost, If you can afford to pay X, you can afford X+10%…

I'm all for openness and am not at all surprised.

I want to buy multi-gigabit IPv6 site firewalls for $50K a piece. [which
is roughly what an ASIC accelerated IPv4 firewall costs today]

> What percentage more should I be willing to pay to be formally correct / RFC compliant? Or to support some niche use? How do I justify spending an additional X% to my bean-counters? What value can…

Are they working fine? IPv6 relies on PMTUD and fragmentation headers.
But AFAICS neither of these are a particularly good solution if
1) there are firewalls that block ICMPv6 so PMTUD doesn't detect the
drop [working on this with educating the security people]
2) there are devices on the path that drop all packets with extension
headers (including fragment extension headers as far as I read)
3) today's IPv6 firewalls can't handle today's IPv4 traffic volumes at
today's prices AFAICS. [correct me if I'm wrong]

= unhappy eyeballs and having to rewrite *all* applications to work with
1280 byte packets (enormous cost!)

Even if we just documented a common packet format in an informational
RFC that allowed everyone to reliably switch a hop by hop extension
header + a fragment extension header in hardware end to end, wouldn't
that be a huge step forward?

And then AH + ESP?

Doesn't that have real value?

I'm looking for a pragmatic compromise here between the long-term
academic position of "this is IPv6, deal with it" and the short-term ops
manager saying "we have no money this year".

> Yes, 'tis sad when the development of protocols / deployment of features is impacted by business[0] considerations -- welcome to the real world.

Never said it would be otherwise. DECnet V was mighty fine: I've never
seen it in operations. DECnet IV is still running in many networks.
> 0

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 08 Jun 2013 08:21:17 +1200

Joe,

>     sufficient for load-balancing

Absolutely not. Load balancing routinely looks at L4 headers.
See RFC 6438, draft-ietf-intarea-flow-label-balancing,
draft-ietf-opsawg-large-flow-load-balancing.

    Brian

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Jun 2013 13:43:29 -0700

Hi, Brian,

Both docs refer to cases I indicate in my later post:

> I don't quite understand why IP addresses are insufficient; the only
> way that would be true is if:
>
>     a) a host could send data at very high rates (possible, but
>     unlikely sustained)
>
>     b) a block of hosts share a single address and send data
>     out at an aggregate high rate
>
>     c) aggregated traffic is either NAT'd or tunneled

The first doc focuses on case (b). IMO, that's a failure of the 
configuration of the block of hosts.

The second doc indicates that there are several ways to differentiate 
flows, of which only the first - the 5-tuple - includes L4 information.

The other post explained how I suggested handling each of these three 
cases, FWIW.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Jun 2013 15:20:55 -0700

Hi, all,

I realized that part of my response below was based on a message I sent 
privately, but it needn't have been; here's the post:

-----

I'm trying to get at the core of this issue.

I don't quite understand why IP addresses are insufficient; the only way 
that would be true is if:

In the case of (c), the NAT/tunnel ingress should be part of the 
solution; it clearly is creating the problem, as I suggest below.

In case (a), the network won't benefit from *any* host identity 
solution; there's no point in solving that case.

In case (b), I would argue that the problem is sharing the address in 
the first place. That case should be solved by reconfiguration.

Is there some other case?

Or maybe, rather than "drop all packets that have extension headers", a 
better solution is to "load balance where the IP information is 
sufficient", and where it is not just drop traffic that exceeds the need 
to balance?

Joe

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 07 Jun 2013 18:53:05 -0400

One thing that was observed years ago was that for whatever reason, even 
in the absence of NAT, the source and dest IP address simply did not 
have enough entropy to give a good distribution of packets across ECMP 
groups.
Yours,
Joel

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Jun 2013 15:59:34 -0700

> have enough entropy to give a good distribution of packets across ECMP

Until we understand why this is true, it's not realistic to try to 
address it as a problem, IMO.

Joe

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 7 Jun 2013 16:16:48 -0700 (PDT)

Great info about the hardware prices!

>>And no, additional bits in my packets for diagnostics is not the value here, I'm managing just fine without them, thanks though.

This highlights the different constituencies.   Unfortunately, we both have to buy equipment from the same vendors who are following the same RFCs.  (Let's leave aside the argument that maybe they are not following ANY RFCs!).   You wouldn't like it if you were forced to live under rules that in many ways didn't apply to you.  Well, we don't either.
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
 From: Warren Kumari <[email-masked]>
To: [email-masked] 
Cc: [email-masked]; "[email-masked] WG" <[email-masked]> 
Sent: Friday, June 7, 2013 10:56 AM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

Using public info:

We get: 

        Total           "$5,545,150.00"

W

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 7 Jun 2013 16:18:44 -0700 (PDT)

>>Well, HBH is the *correct* place to put things that are examined per-hop ;-)

Maybe I am missing something but I am not seeing why the header we are proposing has to be looked at by ANY intermediate hop.  The source node sticks in the timestamp and packet sequence number & we are all set.  Why would that need to be in a HBH header?
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

Sent: Friday, June 7, 2013 10:58 AM

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Jun 2013 16:31:56 -0700

On 6/7/2013 4:18 PM, Nalini Elkins wrote:
> Maybe I am missing something but I am not seeing why the header we are
> proposing has to be looked at by ANY intermediate hop.  The source node
> sticks in the timestamp and packet sequence number & we are all set.
>   Why would that need to be in a HBH header?

It wounldn't unless you need it to be examined by a router. If it's only 
examined by the endpoints, it could be elsewhere.

The point of the HBH header is to place all of the info a router should 
*ever* need to look at (that isn't in the main IPv6 header) in one place 
that's easy to find and parse.

Joe

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 7 Jun 2013 16:59:50 -0700 (PDT)

>>It wounldn't unless you need it to be examined by a router. If it's only examined by the endpoints, it could be elsewhere.

>>The point of the HBH header is to place all of the info a router should *ever* need to look at (that isn't in the main IPv6 header) in one place that's easy to find and parse.

Sure.  That is why we put it in the Dest Ops header.  No need for a router to ever bother with our little guy!

Joe

---

From: Warren Kumari <[email-masked]>
Date: Fri, 7 Jun 2013 20:22:47 -0400

> Sure.  That is why we put it in the Dest Ops header.  No need for a router to ever bother with our little guy!

Sure, but the Dest Ops header shows up before the L4 information, and so pushed the L4 header further out.

Someone (it may have been Ron Bonica, hopefully he doesn't mind me mentioning it) suggested reordering the header chain so that it goes IP -> HBH -> L4 / Payload -:> Dest Options.
This is not a stupid idea….

Anyway, I decided that maybe having some data here would be useful, so I decided to see how many sites from Dan Wing's list (http://www.employees.org/~dwing/aaaa-stats/ipv6-aaaa.2013-06-07_0800.txt) I could:
A: make an IPv6 connection to and 
B: would accept packets where the first header was not the payload.

So, here are some stats. I connect (using scapy) and do an HTTP GET. If that works, I try again, with an empty Destination Options (the only thing I could figure out that would move the payload up and not influence routing / seem weird). My methodology may be broken, and the code (pasted at the bottom *certainly* is horrendous, but at least we have *some* data. And kvetching about methodology, data is (IMO) more interesting than where are currently are, which seems to not be progressing… 

[SNIP]
www.ssa.gov does not work if there is a Destination Option Extension Header
www.techtudo.com.br does not work if there is a Destination Option Extension Header
www.tradebit.com does NOT seem to work on IPv6
tutsplus.com does NOT seem to work on IPv6
www.turktelekom.com.tr does NOT seem to work on IPv6
www.ub.ac.id does not work if there is a Destination Option Extension Header
Processed 1100 sites, 6.173994 work with extension headers

www.unesp.br does NOT seem to work on IPv6
www.newsen.com does NOT seem to work on IPv6
Working V6 sites: 1069, Working v6 sites with EH: 66. Percent: 6.173994
------------------------------------------------------------
Working EH sites:
acesso.uol.com.br
www.adrenaline.uol.com.br
www.automobile.it
www.automobile.fr
band.uol.com.br
www.batepapo.uol.com.br
www.blogfolha.uol.com.br
blogdojuca.uol.com.br
www.blogosfera.uol.com.br
www.bol.uol.com.br
www.bpm.uol.com.br
www.busca.uol.com.br
caras.uol.com.br
www.celebridades.uol.com.br
www.clickjogos.uol.com.br
www.cms.gov
copadomundo.uol.com.br
www.clixboom.com
www.disponivel.uol.com.br
economia.uol.com.br
educacao.uol.com.br
www.empregocerto.uol.com.br
esporte.uol.com.br
www.folha.uol.com.br
gizmodo.uol.com.br
www.idgnow.uol.com.br
www.harvard.edu
www.jogosonlinegratis.uol.com.br
www.indosat.com
www.levelupgames.uol.com.br
www.ed.gov
www.mail.uol.com.br
www.medicare.gov
meusjogosdemeninas.uol.com.br
www.mobile.de
www.mobile.ro
www.mulher.uol.com.br
www.musica.uol.com.br
www.ne10.uol.com.br
www.noticias.uol.com.br
olhardigital.uol.com.br
www.omelete.uol.com.br
portalcorreio.uol.com.br
www.radio.uol.com.br
www.radardedescontos.com.br
www.ripe.net
sac.uol.com.br
www.shopping.uol.com.br
www.tecnologia.uol.com.br
www.televisao.uol.com.br
www.todaoferta.uol.com.br
www.tportal.hr
www.tpg.com.au
www.tudogostoso.uol.com.br
www.tvuol.uol.com.br
www.uol.com.br
www.uolhost.com.br
www.userapi.com
www.virgula.uol.com.br
www.vk.me
www.turkcell.com.tr
www.vk.com
www.xpg.uol.com.br
ziggi.uol.com.br
zipmail.uol.com.br
www.xl.co.id

The majority of these are uol.com.br hosts. If I count them as one "site", then 1.7% of sites can handle packets with a dest options header. 
This is more than I had expected, but keep in mind that this is a *single* Destination Option header (8 bytes (2 + PadN of 6))

I'll extend this sometime to see how the numbers change when there is more (16, 32, 48, etc) header.

Warning: Ugly tossed together code:

#!/usr/bin/
#
# $Revision::                                              $
# $Date::                                                  $
# $Author::                                                $
# $HeadURL::                                               $
# Copyright: Warren Kumari ([email-masked]) -- 2013
#

from scapy.all import *

def SupportsEH(name):
  try:
    a=sr1(IPv6(dst=name)/TCP(dport=80)/'GET / HTTP/1.0\r\n\r\n', timeout=2, verbose=0)
  except socket.gaierror:
    print 'Cannot resolve %s' % name
    return (False, False)
  if a:
    b=sr1(IPv6(dst=name)/IPv6ExtHdrDestOpt()/TCP(dport=80)/'GET / HTTP/1.0\r\n\r\n', timeout=2, verbose=0)
    if b:
      print '*** %s works with extension headers! ***' % name
      return (True, True)
    else:
      print '%s does not work if there is a Destination Option Extension Header' % name
      return (True, False)
  else:
    print '%s does NOT seem to work on IPv6' % name
    return (False, False)

def ReadSiteList(filename):
  sites=[]
  for line in open(filename).read().splitlines():
    try:
      site = line.split()[0]
    except IndexError:
      pass
    sites.append(site)
  return sites

def main():
  v6_sites=[]
  eh_sites=[]
  count = 0
  # From: http://www.employees.org/~dwing/aaaa-stats/ipv6-aaaa.2013-06-07_0800.txt
  sites = ReadSiteList('ipv6-aaaa.2013-06-07_0800.txt')
  for site in sites:
    count +=1
    (v6, eh) = SupportsEH(site)
    if v6:
      v6_sites.append(site)
    if eh:
       eh_sites.append(site)
    if not count % 20:
      print 'Processed %d sites, %f work with extension headers\n' % (count,
                                                                    float(len(eh_sites)) / len(v6_sites) * 100.0)

  print 'Working V6 sites: %d, Working v6 sites with EH: %d. Percent: %f' % (len(v6_sites), len (eh_sites), float(len(eh_sites)) / len(v6_sites) * 100.0)

  print '-'*60
  print 'Working EH sites:'
  print '\n'.join([site for site in eh_sites])

if __name__ == '__main__':
  main()

W

--
"I think perhaps the most important problem is that we are trying to understand the fundamental workings of the universe via a language devised for telling one another when the best fruit is." --Terry Prachett

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Jun 2013 17:34:03 -0700

On 6/7/2013 5:22 PM, Warren Kumari wrote:
> Someone (it may have been Ron Bonica, hopefully he doesn't mind me
> mentioning it) suggested reordering the header chain so that it goes IP
> -> HBH -> L4 / Payload -:> Dest Options.
>

Oh, don't be so sure ;-)

Where are the other headers? HBH and DestOpts are only two; you're 
forgetting fragmentation, encryption, etc.

If you move those two - in particular - after L4, then how do you even 
know that what you think is L4 control info (ports) is really there? vs. 
some other data, or an encrypted copy?

Joe

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 7 Jun 2013 18:16:52 -0700 (PDT)

Warren,

Very cool!

Now, I know one site has told me that they disable extension headers on the outside (facing the Internet) and enable them on the internal network.
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
 From: Warren Kumari <[email-masked]>
To: Nalini Elkins <[email-masked]> 
Cc: Warren Kumari <[email-masked]>; Joe Touch <[email-masked]>; "[email-masked]" <[email-masked]> 
Sent: Friday, June 7, 2013 5:22 PM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-ipv6-end-to-end-rt-needed

[SNIP]

#!/usr/bin/
#
# $Revision::                                              $
# $Date::                                                  $
# $Author::                                                $
# $HeadURL::                                               $

from scapy.all import *

def SupportsEH(name):
  try:

def main():
  v6_sites=[]
  eh_sites=[]
  count = 0

  print '-'*60

W

---

From: Ray Hunter <[email-masked]>
Date: Sat, 08 Jun 2013 07:39:32 +0200

> Sure, but the Dest Ops header shows up before the L4 information, and so pushed the L4 header further out.
Absolutely.

> So, here are some stats. I connect (using scapy) and do an HTTP GET. If that works, I try again, with an empty Destination Options (the only thing I could figure out that would move the payload up…
<snip>

Thanks for the data.

But your test only pushed the L4 header out by one Destination Option header (8 bytes (2 + PadN of 6))

That to me would suggest far more brokenness than simply that the L4
header had moved a small amount beyond the limit of the 'n' first bytes
of the packet (where 'n' is limited due to bandwidth limitations to the
ASIC), because 8 bytes is the absolute minimum a L4 header can be
displaced. Your test traffic would certainly not constitute an
abnormally long extension header chain IMHO. e.g. It might be that there
were problems with a broken parser not finding the L4 header at an exact
fixed location where it expected it, a filter that had been configured,
or some additional configuration to say drop all destination options.

Speaking entirely selfishly: accelerating fragment headers, and AH+ESP
through firewalls that examine L4 data would be the two use-cases I'd
want to see most supported.

Or is the IPv6 Internet "web only"?

regards,
RayH

---

From: joel jaeggli <[email-masked]>
Date: Sat, 08 Jun 2013 11:31:59 +0200

On 6/8/13 12:20 AM, Joe Touch wrote:
> Hi, all,
>
> I realized that part of my response below was based on a message I 
> sent privately, but it needn't have been; here's the post:
>
> -----
>
> Or maybe, rather than "drop all packets that have extension headers", 
> a better solution is to "load balance where the IP information is 
> sufficient", and where it is not just drop traffic that exceeds the 
> need to balance?
>
> Joe
>
This is the load balancer case yes... which is essentially a localized 
form of anycast, say for example 20 nexthop addresses for the same 
destination IP. each one of those a load balancer with a pool of hosts 
behind it. There is rather a lot of outgoing traffic with the same 
source ip on it.

---

From: Arturo Servin <[email-masked]>
Date: Sat, 08 Jun 2013 11:23:28 -0300

I agree we need to leave this debate about routers/firewalls and L3 and L4.

	But the worst thing that we can do now it is to leave the status quo
(Extension Headers as they exists today) without even questioning if we
need to change it.

	First, we need to ask ourselves if extension headers are worth at all
and if they really make IPv6 a better protocol. Then if so, we need to
try to find a compromise between performance/deployment (vendors) vs
protocol design (ietf). A compromise may be a fixed maximum size on EH
chains.

	If we found that EX are useless (or some of them) then we should
deprecate them.

my 20 cents.
as

---

From: Nalini Elkins <[email-masked]>
Date: Sat, 8 Jun 2013 08:53:36 -0700 (PDT)

>Or is the IPv6 Internet "web only"?

Very interesting point.

Let me stress that I have talked to real organizations on the IPv6 internet and some are blocking IPv6 extension headers on their Internet facing sides.   But not on the internal facing network.   Their concern is security.  So, what I think has happened here is NOT that Warren's code pushed the L4 header out but that the mere existence of a dest ops header was enough to cause the firewall to drop the packet.   

Just saying because I know one of the organizations that Warren tested against.  Now, I only asked 3 organizations and did not do some kind of systematic study.

Nalini

---

From: Nalini Elkins <[email-masked]>
Date: Sat, 8 Jun 2013 08:54:44 -0700 (PDT)

>>    I agree we need to leave this debate about routers/firewalls and L3 and L4.

  >>  But the worst thing that we can do now it is to leave the status quo
>>(Extension Headers as they exists today) without even questioning if we

  >>  First, we need to ask ourselves if extension headers are worth at all
>>protocol design (ietf). A compromise may be a fixed maximum size on EH

 >>  If we found that EX are useless (or some of them) then we should
>>deprecate them.
 
Arturo, very sensible statement.

Nalini

---

From: Warren Kumari <[email-masked]>
Date: Sat, 8 Jun 2013 12:28:38 -0400

> But your test only pushed the L4 header out by one Destination Option header (8 bytes (2 + PadN of 6))

Yup. I mentioned that in my mail

I am planning on retesting the sites that managed to pass *any* headers by adding more padding and seeing where various sites fall out.
Unfortunately scapy doesn't really like my adding the padding myself (or, at least I haven't yet figured out yet how). I'll try beat it into submission more, or just give up and hand-craft the packets myself. 

> or some additional configuration to say drop all destination options.

Yup, the *huge* majority of the "brokenness" was probably filters like the below - if you don't apply any filters / hashing / load-balancing the router is only looking at the IP header.

term discard-ext-header-packets {
               from {
                   next-header-except [ tcp udp icmpv6 ];
               }
               then {
                   count discarded-packets-with-headers;
                   discard;
	}
   }
}

But, as we say in: http://tools.ietf.org/html/draft-wkumari-long-headers-00

"In one widely deployed router architecture, if there are any
   extension headers between the IPv6 header and the upper-layer header,
   the forwarding logic never sees the upper-layer header, and so is not
   able to filter on it."

and 

"Network operators may filter traffic with extention header, because
   of the lack of fixed extention header size, the complexity of
   following header chains, and the inability of deployed routers to
   look sufficently deep into packets to see the upper-layer header."

But, as a [ web / content ] operator, why would you allow in any traffic that seems suspicious? 

We have an updated version of this draft sitting in an edit buffer (and a corresponding recommendation draft), which we intend to submit in a few days (as soon as we figure out who has the most recent revision, why they didn't check it into svn, and who exactly was supposed to post it…  :-P) 

> want to see most supported.

So, yes, there *are* extensions that are useful -- ESP, AH / Mobility spring to mind.

One of the updates to the draft suggests that devices be designed to be able to look N bytes into the packet to find the L4 info (where N is 64 / 128 / 256). This seems like a good compromise between extensibility and cost.   Most *sites* on the Internet will continue to only allow traffic in that they are expecting though.

As it is "hard" for machines / applications to know if they are emitting a packet that will traverse the big-I Internet they should assume that there will be a filter that will barf on EH headers and so not emit them.

"For this reason, appplications and IPv6 stacks should avoid the use
   of extention headers whenever possaible, and applications should be
   designed to deal with the posibility that packets containing
   extention headers may not be delivered."

What flavor of kink  you get up to in the privacy of your own network is (largely) your own decision, but keep in mind that:
1: hardware is designed for the majority / profitable cases.
2: adding complexity to the protocol has a cost to all (bugs per kloc, limited developer resources, edge cases and operational considerations, endless threads on v6ops@ ).

> Or is the IPv6 Internet "web only"?

Nope. No-one said that, but sometimes it does seem that way, doesn't it? :-). 

W

-- 
With Feudalism, it's your Count that votes.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 09 Jun 2013 07:59:18 +1200

> One of the updates to the draft suggests that devices be
> designed to be able to look N bytes into the packet to find
> the L4 info (where N is 64 / 128 / 256). 

If you mean "look" in the sense of draft-ietf-6man-ext-transmit,
it is very reasonable to say that in the real world, line speed
devices can only do so for a finite number of bytes, and we
could do some analysis to decide what is a reasonable number of
bytes to avoid problems most of the time. In fact, I would
personally see this as a good thing to add to the 6man draft,
which at the moment leaves it open-ended.

So, what is the longest reasonable header chain using any
reasonable combination of the extensions defined to date?

> This seems like a
> good compromise between extensibility and cost.   

Agreed.

> Most
> *sites* on the Internet will continue to only allow traffic
> in that they are expecting though.

That's exactly why all the rules need to be configurable, as
required by the 6man draft, so that sites can decide for themselves.

Regards
   Brian

---

From: Joe Touch <[email-masked]>
Date: Sat, 8 Jun 2013 14:52:46 -0700

> 	If we found that EX are useless (or some of them) then we should

How about talking about the server farms that cause the problem? Maybe they should be told to use one IP per host, or suffer from arbitrary packet reordering caused by random load balancing. Or they should incur the cost of labelling their flows, so load balancers can manage those flows.

Or about the load balancer, that wants to share paths without the cost of differential delay. They could fix that by creating a tunnel with a label and egress reordering. Or they could balance flows where they can, and either balance randomly or drop traffic that isn't flow-labeled when it exceeds a threshold.

Or about those who buy routers that can't process extension headers, who should realize they got an underpowered device that doesn't comply with the standards.

THere are a lot of 'practical' recommendations here - but, IMO, they ought to first and foremost put the pain where the problem is. The problem is sharing a single address without flow labeling, and devices that think it's OK to implement "most" of a standard.

Joe

---

From: Warren Kumari <[email-masked]>
Date: Sat, 8 Jun 2013 19:58:57 -0400

> Unfortunately scapy doesn't really like my adding the padding myself (or, at least I haven't yet figured out yet how). I'll try beat it into submission more, or just give up and hand-craft the pack…

And, while doing this (which I think I finally got to work right) I discovered that I had an embarrassingly  stupid bug in my initial code -- I was only checking to make sure that I got *some* response from the site, but not what the response was. This means that I was counting sites that returned e.g. an ICMPv6DestUnreach as though they worked.

The view of the world after fixing this (code below) is *very* different -- one site (www.sapo.pt) works with extension headers, but only sporadically - I'm guessing there is a LB and one node is missing filters / there is ECMP and one path is missing filters.

This is 0.09% of tested sites. This is more like what I would have expected -- I'm paranoid and so expect others to be as well, and to discard / reject packets that don't start with [TCP/UDP/ICMP]. 

It is also possible that there is something wrong with my methodology / scapy's packet crafting, but seeing as one site works I think the generated packets are ok...

wkumari@vimes:~/tmp$ time sudo python ./ipv6_eh_tester.py
[SNIP]
www.zbozi.cz does not work if there is a Destination Option Extension Header
www.zerolag.com does not work if there is a Destination Option Extension Header
### ziggi.uol.com.br returned an ICMPv6 Destination Unreachable ###
zipmail.uol.com.br does not work if there is a Destination Option Extension Header
www.zimbra.free.fr does not work if there is a Destination Option Extension Header
www.singtel.com does not work if there is a Destination Option Extension Header
www.zuhause.de does not work if there is a Destination Option Extension Header
www.zoover.nl does not work if there is a Destination Option Extension Header
www.zwame.pt does not work if there is a Destination Option Extension Header
www.zoover.de does not work if there is a Destination Option Extension Header
Cannot resolve www.skoob.com.br
### www.xl.co.id returned an ICMPv6 Destination Unreachable ###
www.slysoft.com does not work if there is a Destination Option Extension Header
www.socialsecurity.gov does not work if there is a Destination Option Extension Header
www.ssa.gov does not work if there is a Destination Option Extension Header
www.techtudo.com.br does not work if there is a Destination Option Extension Header
www.tradebit.com does not work if there is a Destination Option Extension Header
tutsplus.com does not work if there is a Destination Option Extension Header
www.turktelekom.com.tr does not work if there is a Destination Option Extension Header
www.ub.ac.id does not work if there is a Destination Option Extension Header
Progress: Processed 1100 sites, 0.092081 work with extension headers

www.unesp.br does not work if there is a Destination Option Extension Header
www.newsen.com does not work if there is a Destination Option Extension Header
Working V6 sites: 1088, Working v6 sites with EH: 1. Percent: 0.091912

www.sapo.pt

real	81m3.027s
user	2m4.080s
sys	0m13.525s

Here is the scapy show() for a working www.sapo.pt:

>>> b.show()
###[ IPv6 ]###
  version= 6L
  tc= 0L
  fl= 0L
  plen= 24
  nh= Destination Option Header
  hlim= 118
  src= 2001:8a0:2104:ff:213:13:146:140
  dst= 2606:700:e:551::100
###[ IPv6 Extension Header - Destination Options Header ]###
     nh= TCP
     len= 0
     autopad= On
     \options\
      |###[ PadN ]###
      |  otype= PadN [00: skip, 0: Don't change en-route]
      |  optlen= 4
      |  optdata= '\x00\x00\x00\x00'
###[ TCP ]###
        sport= www
        dport= ftp_data
        seq= 2802195972
        ack= 1
        dataofs= 6L
        reserved= 0L
        flags= SA
        window= 8192
        chksum= None
        urgptr= 0
        options= {}

So, it looks like 0.09% of sites on the big-I Internet will accept packets that look like:

>>> packet.show()
###[ IPv6 ]###
  version= 6
  tc= 0
  fl= 0
  plen= None
  nh= Destination Option Header
  hlim= 64
  src= 2606:700:e:551::100
  dst= <Net6 www.sapo.pt>
###[ IPv6 Extension Header - Destination Options Header ]###
     nh= TCP
     len= None
     autopad= On
     \options\
###[ TCP ]###
        sport= ftp_data
        dport= www
        seq= 0
        ack= 0
        dataofs= None
        reserved= 0
        flags= S
        window= 8192
        chksum= None
        urgptr= 0
        options= {}
###[ Raw ]###
           load= 'GET / HTTP/1.0\r\n\r\n'

Yes, I'm only testing web servers here, and those on the public Internet. Feel free to perform your own testing internally, etc.
W

Ugly code:
#!/usr/bin/python
#
# $Revision:: 88                                           $
# $Date:: 2013-06-08 18:26:06 -0400 (Sat, 08 Jun 2013)     $
# $Author:: wkumari                                        $
# $HeadURL:: svn+ssh://svn.kumari.net/data/svn/code/python#$
# Copyright: Warren Kumari ([email-masked]) -- 2013
#

from scapy.all import *

class PaddedDestOpt(Packet):
  name = "Padded IPv6 Destination Option Header"
  fields_desc = [ ByteEnumField("nh", 59, ipv6nh),
                  FieldLenField("len", None, length_of="padding", fmt="B",
                                adjust = lambda pkt,x: (x+2+7)/8-1),
                                  # Stuff in a PadN header
                  ByteField("otype", 0x01),
                  FieldLenField("optlen", None, length_of="padding", fmt="B"),
                  StrLenField("padding", "",
                               length_from = lambda pkt: pkt.optlen - 2)
                ]
  overload_fields = {IPv6: { "nh": 60 }}

def SupportsEH(name):
  try:
    response=sr1(IPv6(dst=name)/TCP(dport=80)/'GET / HTTP/1.0\r\n\r\n', timeout=2, verbose=0)
  except socket.gaierror:
    print 'Cannot resolve %s' % name
    return (False, False)
  if packet:
    response=sr1(IPv6(dst=name)/IPv6ExtHdrDestOpt()/TCP(dport=80)/'GET / HTTP/1.0\r\n\r\n', timeout=4, verbose=0)
    if response:
      if response.haslayer(ICMPv6DestUnreach):
        print '### %s returned an ICMPv6 Destination Unreachable ###' % name
        return (True, False)
      elif response.haslayer(ICMPv6TimeExceeded):
        print '!!! %s returned an ICMPv6 TTL Exceeded !!!' % name
        return (True, False)
      elif response.haslayer(TCP):
        print '*** %s works with extension headers! ***' % name
        return (True, True)
      else:
        print 'We got some other sort of error from %s. Packet:' % site
        response.show()
      return (True, False)
    else:
      print '%s does not work if there is a Destination Option Extension Header' % name
      return (True, False)
  else:
    print '%s does NOT seem to work on IPv6' % name
    return (False, False)

def main():
  v6_sites=[]
  eh_sites=[]
  count = 0
  # From: http://www.employees.org/~dwing/aaaa-stats/ipv6-aaaa.2013-06-07_0800.txt
  sites = ReadSiteList('ipv6-aaaa.2013-06-07_0800.txt')
  #sites = ReadSiteList('ipv6_eh_sites.txt')
  for site in sites:
    count +=1
    (v6, eh) = SupportsEH(site)
    if v6:
      v6_sites.append(site)
    if eh:
       eh_sites.append(site)

    if not count % 20:
      print 'Progress: Processed %d sites, %f work with extension headers\n' % (count,
                                                                   float(len(eh_sites)) / len(v6_sites) * 100.0)

  print '\n\n'
  print '-'*60

--
It is impossible to sharpen a pencil with a blunt axe.  It is equally vain
to try to do it with ten blunt axes instead.
    --  E.W Dijkstra, 1930-2002

---

From: Fernando Gont <[email-masked]>
Date: Sun, 09 Jun 2013 02:17:37 +0200

On 06/08/2013 02:22 AM, Warren Kumari wrote:

> mentioning it) suggested reordering the header chain so that it goes
> IP -> HBH -> L4 / Payload -:> Dest Options. This is not a stupid
> idea….

The problem is that if the destination system doesn't support the
corresponding transport protocol, it cannot process the Destination
Options (because transport protocol do not have a fixed TLV syntax).

-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Fernando Gont <[email-masked]>
Date: Sun, 09 Jun 2013 06:47:40 +0200

Hi, Warren,

On 06/09/2013 01:58 AM, Warren Kumari wrote:
> 
> And, while doing this (which I think I finally got to work right) I
> discovered that I had an embarrassingly  stupid bug in my initial
> code -- I was only checking to make sure that I got *some* response
> from the site, but not what the response was. This means that I was
> counting sites that returned e.g. an ICMPv6DestUnreach as though they
> worked.

Question: Do you actually send the HTTP GET?

If you're just probing with TCP SYN or ICMPv6 ech, you could use the
tcp6 and icmp6 tools at: <http://www.si6networks.com/tools/ipv6toolkit>,
which let you send fancy packets easily.

If you're really sending the HTTP GET, then, if interested, I can add
support to send arbitrary application-layer payloads to our tcp6 tool,
such that this (and other) test are feasible.

> The view of the world after fixing this (code below) is *very*
> different -- one site (www.sapo.pt) works with extension headers, but
> only sporadically - I'm guessing there is a LB and one node is
> missing filters / there is ECMP and one path is missing filters.

Only one?

> This is 0.09% of tested sites. This is more like what I would have
> expected -- I'm paranoid and so expect others to be as well, and to
> discard / reject packets that don't start with [TCP/UDP/ICMP].

What about fragmentation support?

Thanks!

Best regards,

---

From: Warren Kumari <[email-masked]>
Date: Mon, 10 Jun 2013 15:09:01 -0400

Yup.

I do the following:
1: Send 'GET / HTTP/1.0\r\n\r\n' to port 80.
  If the site responds, I classify it as "working" for IPv6 and continue.

2: Send 'GET / HTTP/1.0\r\n\r\n' to port 80, with an empty Destination Option (simply padding).
  If the site sends back an ICMPv6DestUnreach or a ICMPv6TimeExceeded I classify it as not working with extension headers and abort.
  If the site replies with a TCP payload I classify it as working with Extension Headers.
  Otherwise I assume it is some other error and print the packet -- this hasn't happened.

> such that this (and other) test are feasible.

Yes please.

>> missing filters / there is ECMP and one path is missing filters.

Only one site seems to support EH packets, and that only sporadically. I tried hitting it a few times (ten or so) and it only responded correctly twice.

>>> working=0
>>> for count in range (1,200):
...   b=sr1(IPv6(dst='www.sapo.pt')/IPv6ExtHdrDestOpt()/TCP(dport=80)/'GET / HTTP/1.0\r\n\r\n', timeout=2, verbose=0)
...   if b:
...     working +=1
>>> working
16

So, 16 in 200 connections got back a response. Maybe 12-13 machines behind a LB? I could fingerprint further, but….

> What about fragmentation support?

Didn't test, but many sites drop fragments:
  http://tools.ietf.org/html/draft-taylor-v6ops-fragdrop-01

> Thanks!

Nah, thank you!

W

---

From: Joe Touch <[email-masked]>
Date: Mon, 10 Jun 2013 12:18:01 -0700

Hi, Warren,

For sites where you got a response, can you let us know whether the ICMP 
destination unreachable was from an intermediate router or the final 
destination?

Based on the info below, it's clear that the option you tested fails, 
but not where or why. It could as easily be endpoint failure.

And if the error is at the endpoints, then any solution that presumes an 
endpoint fix could just as easily assume correcting this flaw...

Joe

---

From: Warren Kumari <[email-masked]>
Date: Mon, 10 Jun 2013 16:50:00 -0400

On Jun 10, 2013, at 3:18 PM, Joe Touch <[email-masked]> wrote:

> Hi, Warren,
> 
> For sites where you got a response, can you let us know whether the ICMP destination unreachable was from an intermediate router or the final destination?

They all came from intermediate routers - I removed the *.uol.com.br site, where were all of the form:
### www.xpg.uol.com.br (2804:49c:319:430::292) sent an ICMPv6 Destination Unreachable from 2001:12ff:aaaa::37:737:2 ###

### www.automobile.it (2001:67c:2d0:4200::33) sent an ICMPv6 Destination Unreachable from 2001:67c:2d0:114::2 ###
### www.automobile.fr (2001:67c:2d0:4200::30) sent an ICMPv6 Destination Unreachable from 2001:67c:2d0:114::2 ###
### www.cms.gov (2607:f028:0:400:146:123:140:205) sent an ICMPv6 Destination Unreachable from 2607:f028:0:80::4 ###
### www.clixboom.com (2001:1528:1:2100:250:56ff:fea1:2d17) sent an ICMPv6 Destination Unreachable from 2001:1528:1:2100:250:56ff:fea1:2d17 ###
### www.harvard.edu (2607:fb60:100:210::e6) sent an ICMPv6 Destination Unreachable from 2607:fb60:0:1:1b::2 ###
### www.indosat.com (2407:0:1000:2::2) sent an ICMPv6 Destination Unreachable from 2407::6 ###
!!! www.ed.gov (2604:b300:ad02:1::4a54:db99) returned an ICMPv6 TTL Exceeded from 2607:f758:2200::4 !!!
### www.medicare.gov (2607:f028:0:400:146:123:140:204) sent an ICMPv6 Destination Unreachable from 2607:f028:0:80::4 ###
### www.mobile.de (2001:67c:2d0:4200::25) sent an ICMPv6 Destination Unreachable from 2001:67c:2d0:110::2 ###
### www.mobile.ro (2001:67c:2d0:4200::24) sent an ICMPv6 Destination Unreachable from 2001:67c:2d0:114::2 ###
### www.radardedescontos.com.br (2804:49c:319:430::359) sent an ICMPv6 Destination Unreachable from 2001:12ff:aaaa::37:737:2 ###
### www.ripe.net (2001:67c:2e8:22::c100:68b) sent an ICMPv6 Destination Unreachable from 2001:7f8:1::a500:3333:2 ###
### www.tportal.hr (2a00:c30:fffd:2::2) sent an ICMPv6 Destination Unreachable from 2a00:c30:b000::113 ###
### www.uolhost.com.br (2804:49c:319:430::358) sent an ICMPv6 Destination Unreachable from 2001:12ff:aaaa::37:737:2 ###
### www.turkcell.com.tr (2a02:4e0:101:1101::14) sent an ICMPv6 Destination Unreachable from 2001:470:15:109::2 ###

> 
> Based on the info below, it's clear that the option you tested fails, but not where or why. It could as easily be endpoint failure.

It is being caused by router (and possibly, though unlikely LB / firewall) filters. Letting unknown traffic into your network is just asking for trouble, and so it is established practice to only allow known and expected traffic in through your edge filters. 

Some folk configure the filters to 'discard' the traffic (silent drop), some to 'reject' it (ICMP Dest Unreachable, Communication with destination administratively prohibited). Discard is often viewed as friendlier, as it stops you sending streams of ICMP replies at spoofed destinations.

> 
> And if the error is at the endpoints, then any solution that presumes an endpoint fix could just as easily assume correcting this flaw…

Assuming that the network operator wanted to accept the traffic….

W

---

From: Joe Touch <[email-masked]>
Date: Mon, 10 Jun 2013 13:59:43 -0700

On 6/10/2013 1:50 PM, Warren Kumari wrote:
> It is being caused by router (and possibly, though unlikely LB /
> firewall) filters. Letting unknown traffic into your network is just
> asking for trouble, and so it is established practice to only allow
> known and expected traffic in through your edge filters.

Sure, but existing defined end-host options are not "unknown traffic".

...
> Assuming that the network operator wanted to accept the traffic….

Seems like a "teachable moment". If network operators consider such 
options "unknown", it's time to inform them otherwise.

Joe

---

From: "cb.list6" <[email-masked]>
Date: Mon, 10 Jun 2013 15:18:33 -0700

On Jun 10, 2013 2:01 PM, "Joe Touch" <[email-masked]> wrote:
>>> For sites where you got a response, can you let us know whether the
ICMP destination unreachable was from an intermediate router or the final
destination?
>>
>>
>> They all came from intermediate routers - I removed the *.uol.com.brsite, where were all of the form:
>> ### www.xpg.uol.com.br (2804:49c:319:430::292) sent an ICMPv6
Destination Unreachable from 2001:12ff:aaaa::37:737:2 ###
>> ### www.cms.gov (2607:f028:0:400:146:123:140:205) sent an ICMPv6
Destination Unreachable from 2607:f028:0:80::4 ###
>> ### www.clixboom.com (2001:1528:1:2100:250:56ff:fea1:2d17) sent an
ICMPv6 Destination Unreachable from 2001:1528:1:2100:250:56ff:fea1:2d17 ###
>> ### www.medicare.gov (2607:f028:0:400:146:123:140:204) sent an ICMPv6
Destination Unreachable from 2607:f028:0:80::4 ###
>> ### www.radardedescontos.com.br (2804:49c:319:430::359) sent an ICMPv6
Destination Unreachable from 2001:12ff:aaaa::37:737:2 ###
>> ### www.ripe.net (2001:67c:2e8:22::c100:68b) sent an ICMPv6 Destination
Unreachable from 2001:7f8:1::a500:3333:2 ###
>> ### www.uolhost.com.br (2804:49c:319:430::358) sent an ICMPv6
Destination Unreachable from 2001:12ff:aaaa::37:737:2 ###
>> ### www.turkcell.com.tr (2a02:4e0:101:1101::14) sent an ICMPv6
Destination Unreachable from 2001:470:15:109::2 ###
>>> And if the error is at the endpoints, then any solution that presumes
an endpoint fix could just as easily assume correcting this flaw…
> Joe

I think Warren's data has shown that networks operators and gear
implementers are seizing the momemt to teach the ietf that these packets
types should be depricated to avoid future misunderstandings and to keep
the standars relevant.

The experiment with kinky EH is over.

Take the results and move on

CB

---

From: Joe Touch <[email-masked]>
Date: Mon, 10 Jun 2013 15:24:15 -0700

On 6/10/2013 3:18 PM, cb.list6 wrote:
...
> types should be depricated to avoid future misunderstandings and to keep

As with the rest of the Internet, what the operators will learn is that 
their lack of support for these extensions will force us to tunnel, 
which will kill their ability to see transport headers. Fine with me...

Joe

---

From: joel jaeggli <[email-masked]>
Date: Tue, 11 Jun 2013 00:49:06 +0200

On 6/11/13 12:24 AM, Joe Touch wrote:
> As with the rest of the Internet, what the operators will learn is 
> that their lack of support for these extensions will force us to 
> tunnel, which will kill their ability to see transport headers. Fine 
> with me...
Dumb network, smart endpoints oddly I'm not unreceptive to that. 
Remember I'm an end user though and I still need to see the transport 
header.

---

From: Joe Touch <[email-masked]>
Date: Mon, 10 Jun 2013 16:40:15 -0700

> Remember I'm an end user though and I still need to see the transport

If you want to see the transport header, convince the router guys not to 
drop well-formed IPv6 packets just because it's "hard".

Otherwise, I assure you the endpoints *will* tunnel - if the alternative 
is to have their packets dropped.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 10 Jun 2013 16:47:26 -0700

One other point, below...

> Dumb network, smart endpoints oddly I'm not unreceptive to that.

It's more than that. You think you're running a network, but I'll just 
start treating it like a link layer. Which means less opportunity for 
services for you (filtering, port-based forwarding, etc.), and pushes 
what you consider the "Internet" towards being irrelevant.

I'm OK with that, though, too.

Joe

---

From: Mark Andrews <[email-masked]>
Date: Tue, 11 Jun 2013 10:13:03 +1000

In message <CAD6AjGS14Ld_pKOkBv=x2eGbc_t8qPVW2srURy-=[email-masked]>
, "cb.list6" writes:
> --===============5004857423208636513==
> Content-Type: multipart/alternative; boundary=047d7bf0ca0246335a04ded42924
> 
> --047d7bf0ca0246335a04ded42924
> Content-Type: text/plain; charset=windows-1252
> Content-Transfer-Encoding: quoted-printable
> 
> >> They all came from intermediate routers - I removed the *.uol.com.brsite=
> , where were all of the form:
> an endpoint fix could just as easily assume correcting this flaw=85
> >>
> >>
> >> Assuming that the network operator wanted to accept the traffic=85.
> >
> >

No.  Equipment vendors take whatever shortcuts they think they can
get away with regardless of the impact they have because no one
calls them to account for it.

Other vendors then need to work around the breakages they cause.
As a DNS vendor I need to work around firewalls that think all
DNS/UDP packets are all <= 512 bytes of payload.  I need to work
around firewall that drop fragmented packets.  I need to work around
firewalls that drop ICMP packets.  I need to work around firewalls/servers
that drop EDNS packets.  I need to work around firewalls/servers
that drop unknown query types.  I need to work around firewalls/servers
that refuse to pass/answer what should be known query types.

True routers are quite able to pass fragmented UDP extend DNS packets.

Oh goolly gee, this packet is fragmented.  The poor defenceless
host can't be allowed to see this naughty packet it might hurt
itself.  This is despite the same boxes running exposed to the raw
internet fine elsewhere.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]