Subject: [RAM] what we should be working on

From: Jari Arkko <[email-masked]>
Date: Thu, 15 Mar 2007 15:24:49 +0200

I have spent some time thinking on what we should focus on
in this space. As you have surely observed, a wide variety of
things have been on the table during the discussion, both
in terms of the understanding the problem or the scope
of the solution. Solution ideas have ranged from small things
to what I would classify clean-slate work.

My first suggestion is that it makes sense to divide and
conquer. I think we have some low-hanging fruit that
deserve to picked early, and we some architectural
work that will take time to develop and even more to
deploy.

For instance, looking at some data that Geoff showed
me a while ago, it seems that highest BGP update activity is
concentrated to a small set of players. Is there something
that we can do in BGP to recognize these cases and deal
with them in a more efficient manner. I think this is one
of the topics that RTGAREA will talk about on Thursday.

Another update related issue is that I'm the final IESG
blessing away from getting NEMO WG rechartered to
produce an efficient scheme for the airline industry's
mobility problem. I think this is doable without incurring
either a lengthy tunneling delay or a huge BGP impact.

There are probably more issues like that in the routing
space, low hanging fruit that improves the general
performance of the routing system.

Hardware development engineers need to do things, too.
I have recently convinced myself that significant improvements
are possible here (moving to commodity components etc).
But this is not something that the IETF is really an expert on,
so I don't want to say more about that.

But even if we keep engineering the BGP system and
improving routers (perhaps even to the point of there
not being a problem with the current table growth in
the near future), I think it still makes sense to work on
architectural changes. Even if we can scale routers
to certain size of the Internet at a certain time, if we
had more architectural tools in our hands, perhaps we
could support a bigger Internet with them, or sooner. Or
with more multihoming, traffic engineering, or permanent
address space than otherwise possible. I can't get my personal
PI space, big companies can -- is that as it should be, or
something to be improved upon?

But what would be the properties of the solution we are
looking for be? So far, we have seen a lot of different ideas.
We are planning to spend most of the INTAREA meeting
next Thursday to talk about this topic (more than the
actual bits).

>From my perspective, these boundary conditions should
probably be the starting point so that we at least talk
about the same space:

- Useful for better scaling of the routing system, but also useful
  for other purposes.

  Details of this should be explored further. What is usel for the scaling
  of the routing system, exactly, to begin with? And should the solution
  be useful for traffic engineering? Mobility? Multihoming? Some new
  security benefits?

- Deployable. In particular, backwards compatible with current
  devices:

  o IP looks the same to devices that have not been updated.

  o Allow existing applications (and APIs) to work unchanged.
     Should think about referrals, too.

  o Not rehash old issues; assume IPv4, IPv6, NATs, etc. exist. Focus
     on what is useful now, given that we are where we are today.
     No redefinition IPv6.

- Goes somehow beyond the solutions we already have. For
  instance, a lot of people seem to believe we need a network-
  based approach in addition to (or instead of) a host based
  approach. Or maybe we need better host based mechanisms.
  In any case, it is important to realize that we have plenty of tunneling
  mechanisms, and some identifier-locator split or multi-level
  identifier designs.

- Seriously consider supporting IPv4. I know this may be hard with
  some of the solutions, so maybe its not achievable. But the current
  routing table growth problem is in the IPv4 side, and while we could
develop
  an IPv6-only solution, I fear that it would lead to having to develop
  some other solution for IPv4 later. And if there was no solution
  for IPv4, I'm not sure we would be have a big impact for real life
  devices or the use of the Internet, until quite far in the future
  (maybe 10 years).

- There must be deployment incentives that cause people who need
  to upgrade to actually upgrade. For instance, if the benefits and
  costs its easier to argue that something should be deployed
  than if they go to different parties.

Thoughts on these? Something to add, or take away?

Finally, I hope that the research community (RRG, HIPRG,
everyone else) will keep working on designs that go beyond
what is presented above, including clean-slate identifier-locator
split architectures, new routing paradigms such as compact
routing, etc.

Jari

_______________________________________________
RAM mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ram

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 15 Mar 2007 16:38:38 +0100

> work that will take time to develop and even more to

In addition, I think there is also some work that isn't exactly low  
hanging, but not architectural either. This is the fairly boring  
stuff that just needs to be done at some point so we can avoid future  
problems.

What I'm thinking of here is increasing the average size of IP  
packets, because the bad stuff, such as the number of FIB lookups  
required, typically scales at O(#packets) while the good stuff  
typically scales at O(bandwidth).

Currently, it's almost inconceivable that an IP packet crossing the  
commercial internet is larger than 1500 bytes. The reason is that  
ethernet rules the world these days, and although today's 1 and 10  
Gbps ethernet has very little in common with the original 10 Mbps  
version, the faster versions did inherit the 1500 byte maximum packet  
size because there is no fragmentation at the ethernet layer so  
without a common maximum you can't interconnect the different speeds.

So what we need to do is make it possible for IP systems with  
different maximum packet sizes to live on the same subnet and we  
probably have to do some work on PMTUD as well to increase its  
reliability. Boring, annoying and little short term gain. But if we  
can do this now, we'll be in a much better position to take advantage  
of advancements in transmission technology.

On the low hanging fruit side: a new BGP metric that helps with  
traffic engineering could be very helpful in reducing the practice of  
announcing lots of small address blocks rather than a single large one.

> of the topics that RTGAREA will talk about on Thursday.

Hm, didn't we have flap dampening for that?

> performance of the routing system.

My big question that is still waiting for a good answer: do we really  
want to continue running some form of BGP4 for years to come? We can  
fix all the incidental problems but the architectural ones won't go  
away. (Meaning problems with the BGP architecture, not with the R/A  
architecture at large.)

> Even if we can scale routers

It works the other way around: anyone who can satisfy the baroque RIR  
policy requirements can get address space and an AS number, and  
subsequently inject one or more prefixes into the global routing  
table and then we all have to buy bigger routers to accommodate the  
growth. Those RIR policies are basically the only barrier to entry,  
and they're not designed to function as such, so they're not doing a  
good job of it. Increasing router capacity won't provide any benefits  
to the community (as opposed to us who are having these discussions)  
because the community is already behaving as though router capacity  
is limitless.

> looking for be?

0. It has to actually work
1. Cost and benefits are aligned
2. Deployable

---

From: Tony Li <[email-masked]>
Date: Thu, 15 Mar 2007 09:32:23 -0700

> either a lengthy tunneling delay or a huge BGP impact.

Why is this necessary?  A solution seems straightforward:

Establish a VPN that carries per-plane prefixes.  Have the VPN touch  
the Internet at numerous large interconnects and advertise an  
aggregated prefix.  Traffic to the plane now travels to the nearest  
interconnect and is then tunneled to the actual uplink.

BGP is protected as only fixed aggregates are injected.  Stretch is  
not bad, as packets would have most likely flowed to the major  
interconnect anyway.

Regards,
Tony

---

From: Jari Arkko <[email-masked]>
Date: Thu, 15 Mar 2007 19:01:48 +0200

Tony,

> A solution seems straightforward:
>

Right. I was thinking along these lines, actually. There are
some details to think about of course, and maybe even
some specification work. In any case, provide some guidance
to the airline community on how they can deal with their
problem.

Jari

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 15 Mar 2007 14:08:14 -0400

> not bad, as packets would have most likely flowed to the major  

This basic approach seems reasonable at first, and I believe that ICAO
(the standards body for aeronautics) has considered it to some extent.
One of the things to consider though is that VPN gateway redundancy
mechanisms may be non-standard (at best) at the moment, and this is
important to the stake-holders.  For a few other reasons, those of us
here who have been considering the problem think that NEMO offers the
strongest solution.

One point to clarify is that the "airline industry" (corporate) is
really only one facet of aeronautical communications.  Networking for
air-traffic control (international governmental) is also important and
coordinating a VPN for this purpose may be practically much more
difficult than it is for an airline's needs.

We've been working on a requirements draft for NEMO route optimization
in aeronautics that explains some of this more, and can share in a
couple weeks time.

-- 
Wesley M. Eddy
Verizon Federal Network Systems

---

From: Tony Li <[email-masked]>
Date: Thu, 15 Mar 2007 11:22:42 -0700

> in aeronautics that explains some of this more, and can share in a

Roger that, looking forward to it.

> important to the stake-holders.

Please be sure that your draft expounds on this a bit more, as the  
normal business VPN requirements would seem to subsume this at first  
glance.

> Networking for

Interesting.  What would ATC have to do with NEMO?  Given the highly  
critical nature of ATC, I would hope that it would be strongly  
secured with respect to both passenger traffic and the rest of the  
Internet.

Tony

---

From: Andrew McGregor <[email-masked]>
Date: Fri, 16 Mar 2007 12:23:39 +1300

On 16/03/2007, at 7:22 AM, Tony Li wrote:

> Interesting.  What would ATC have to do with NEMO?  Given the  
> highly critical nature of ATC, I would hope that it would be  
> strongly secured with respect to both passenger traffic and the  
> rest of the Internet.
>
> Tony

The interesting thing about ATC traffic is that during cruise the  
critical requirements are that the messages be authenticated and  
reasonably timely, but it isn't all that critical that they actually  
be delivered since the whole system is designed to safely deal with  
the occasional loss of communication (usually at the expense of a  
whole lot of fuel if the outage is prolonged).

One of the issues that the airspace system is facing is congestion on  
voice frequencies, especially over the oceans because there are not  
that many frequencies to use, and I'm sure an IP based system using  
textual communication would greatly reduce that problem.

You could do ATC over Jabber using certificates for message  
authentication, and that would certainly be an improvement over what  
is there now, provided you can get the IP connectivity.

Andrew

---

From: Jari Arkko <[email-masked]>
Date: Sat, 17 Mar 2007 18:02:05 +0200

Iljitsch,

> hanging, but not architectural either. This is the fairly boring stuff
> that just needs to be done at some point so we can avoid future problems.

Yes.

> typically scales at O(bandwidth).

I am all for increasing the average size of packets. But I'll note that
its probably not a panacea. Router cost, for instance, is a function
of many factors, including FIB looks, filtering, fancy prioritizing
schemes as well as line speed. And Path MTU increase is one of those
nasty things where the weakest link counts, so getting that deployed
requires updates in several places. But yes, I think it would still
be useful.

> announcing lots of small address blocks rather than a single large one.

Yes.

> Hm, didn't we have flap dampening for that?

We did -- what I have been told is that looking at the
data, a large part of the update activity comes from
a small number of sources that do this either intentionally
or unintentionally. The question is, can we do something
smarter about the high-activity sources than we currently
do? I do not pretend to be an expert on what we could
do, but I understood that there might be some ideas.
Improve over the flap dampening design, perhaps, or
make it more suitable for our current situation.
Ross, John, Geoff should know more about this.

> away. (Meaning problems with the BGP architecture, not with the R/A

That is a good question, too. Not a low hanging fruit! Maybe
another long-term architecture topic for the routing area
to consider? Do other people feel that this work is something
should be pursued?

> 1. Cost and benefits are aligned

Nice summary. Those are the key issues.

Jari

---

From: John G. Scudder <[email-masked]>
Date: Sun, 18 Mar 2007 11:30:20 -0400

>> announcing lots of small address blocks rather than a single large  

The historical record indicates that this is easier said than done.   
The most recent attempt at "helping with traffic engineering" is  
draft-ietf-as-pathlimit.  I wouldn't say it "reduces the practice of   
announcing lots of small address blocks rather than a single large  
one", though it does potentially restrict their scope.

If you have something more powerful in mind, you should propose it,  
especially if it's straightforward enough to qualify as "low hanging  
fruit".

--John

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 18 Mar 2007 21:49:08 +0100

> The historical record indicates that this is easier said than  
> done.  The most recent attempt at "helping with traffic  
> engineering" is draft-ietf-as-pathlimit.  I wouldn't say it  
> "reduces the practice of  announcing lots of small address blocks  
> rather than a single large one", though it does potentially  
> restrict their scope.

> especially if it's straightforward enough to qualify as "low  
> hanging fruit".

Last year, I decided it would be fun to try and design a (somewhat)  
link state inter-domain routing protocol. It was indeed fun, so the  
time was well spent. (See http://www.muada.com/drafts/draft-van- 
beijnum-idle-00.txt for those who are curious.)

One of the new aspects of that protocol was a real inter-AS metric.  
(Note that the BGP MED is sometimes called "inter-AS metric" but  
that's not what I'm talking about.)

This metric would (in BGP) be a path attribute that holds an integer  
value that is increased at every hop. The length of the AS path,  
which is increased by one at each hop unless the operator takes  
measures to do something else, so in most cases, its value is one,  
two or three in the majority of cases. This is simply not granular  
enough. So the new inter-AS metric is increased by a value of at  
least 1 and maximum 127. The default would be 11. (The actual values  
depend on the choice for the number of bits in the iASm and the  
maximum number of hops that must be supported, we'll assume 15 and  
256 for this example.)

    A -- B -- C
   /     |     \
me      |      observer
   \     |     /
    D -- E -- F

This means that after three intermediate hops, the AS path would hold  
four ASes at the observer. If the observer decides to send me traffic  
over C, B and A, and I don't like that, I can prepend the AS path  
over A but so for the observer, the path over C-B-A is now five hops  
and F-E-D four. But as a side effect, for B, the path over E and D  
has the same length as the prepended one over A, so I can very easily  
end up disrupting traffic flow severely.

With the new iASm on the other hand, I can just send out a metric of  
12 to A rather than the default 11. So for the observer, the metric  
of the path over C-B-A is 45 while D-E-F is 44. That's enough to make  
her send traffic over C-B-A if all else is equal. But for B, it's now  
23 over A and 33 over E-D so nothing has changed. On the other hand,  
if I WANT to change the way B sends its traffic, I can increase my  
metric towards A to 22.

So an inter-AS metric allows for more granular traffic engineering,  
which should make more specific announcements unnecessary in a  
certain fraction of all cases.

Although this isn't entirely trivial and it will take some time to  
take effect, it doesn't disrupt current practice if implemented in  
BGP in a backward compatible way, so I think it's low hanging as IDR  
fruit goes.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 19 Mar 2007 16:12:05 +0100

> requires updates in several places. But yes, I think it would still

Yes, there are many reasons why this is and has been a low-return  
investment. However, we're rapidly approaching the situation where,  
for a significant number of paths, at every hop, the hardware  
supports packet sizes larger than 1500 bytes, but we have no  
reasonable way of using this capability where it exists. But if we  
can make things such that the largest packets possible are used, over  
time, more and more packets will be larger than 1500 bytes so the  
growth in the total amount of per-packet work will start to trail the  
growth in bandwidth. Over time, the savings in processing power /  
energy use will become more and more significant.

I guess I have a draft to write.  :-)

---

From: Pekka Savola <[email-masked]>
Date: Mon, 19 Mar 2007 17:21:55 +0200 (EET)

On Mon, 19 Mar 2007, Iljitsch van Beijnum wrote:
> reasonable way of using this capability where it exists.
...

IMHO, the bigger problem is that if we end up with any kind of 
solution that employs encapsulation, we end up in a potentially nasty 
mess of PMTUD issues unless we're able to raise the MTU on links 
between the encapsulators and decapsulators.

The pain is felt worst in Ethernet-based (all of them? :-) exchange 
points due to the lowest common denominator MTU.  (Some IXs have tried 
to create separate, optional "big MTU VLANs" but I don't think those 
have been very popular).  On direct peerings from the technology and 
O&M perspective, usually raising MTU is possible.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: "Puneet Agarwal" <[email-masked]>
Date: Mon, 19 Mar 2007 08:23:35 -0700

Hi Iljitsch,

Lookup bandwidth is dominated by small packets (all the TCP acks that
are flowing). Increasing the MTU to be > 1500 will not solve the lookup
bandwidth problem. Almost 50% of all packets are 64 bytes (on Ethernet)
and a little smaller over PPP.

Hence increased MTU (greater than 1500) will not solve the lookup bw
problem.

Thanks.

-Puneet

-----Original Message-----
From: Iljitsch van Beijnum [mailto:[email-masked]] 
Sent: Monday, March 19, 2007 8:12 AM
To: Jari Arkko
Cc: [email-masked]
Subject: Re: [RAM] what we should be working on

> I am all for increasing the average size of packets. But I'll note 
> that its probably not a panacea. Router cost, for instance, is a 
> function of many factors, including FIB looks, filtering, fancy 
> prioritizing schemes as well as line speed. And Path MTU increase is 
> one of those nasty things where the weakest link counts, so getting 
> that deployed requires updates in several places. But yes, I think it 
> would still be useful.

Yes, there are many reasons why this is and has been a low-return
investment. However, we're rapidly approaching the situation where, for
a significant number of paths, at every hop, the hardware supports
packet sizes larger than 1500 bytes, but we have no reasonable way of
using this capability where it exists. But if we can make things such
that the largest packets possible are used, over time, more and more
packets will be larger than 1500 bytes so the growth in the total amount
of per-packet work will start to trail the growth in bandwidth. Over
time, the savings in processing power / energy use will become more and
more significant.

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 19 Mar 2007 08:46:34 -0700

> I guess I have a draft to write.  :-)

And the problem with *my* draft is... ??

http://ipvlx.net

Fred
[email-masked]

---

From: Jari Arkko <[email-masked]>
Date: Mon, 19 Mar 2007 16:51:54 +0100

Iljitsch,
>> I guess I have a draft to write.  :-)

Do you actually need to write something? Seems like we
have the technology (unless want to modify TCP ACKs) to
do what we need... it might be more useful to go and
configure a few exchange points to bigger MTU...

Jari

---

From: Jari Arkko <[email-masked]>
Date: Mon, 19 Mar 2007 16:51:54 +0100

Iljitsch,

Jari

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 19 Mar 2007 16:54:31 +0100

> Hence increased MTU (greater than 1500) will not solve the lookup bw

Not sure what you mean by "solve". General purpose computers almost  
always use less power when idle than when doing work. If this is true  
for routers doing FIB lookups (not a given, but I don't see how it  
would be possible to authoritatively say it can NEVER be true in the  
future) then having fewer packets for a given amount of bandwidth is  
helpful.

Obviously there are applications that generate small packets, so a  
larger MTU won't make those packets larger. But for TCP it does help.  
If you have to send 100 packets with 1500 byte packets but only 20  
with large packets, this means that the number of TCP ACKs will go  
down from approximately 50 to approximately 10 and the total from 150  
to 30.

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 19 Mar 2007 08:57:55 -0700

> configure a few exchange points to bigger MTU...

Are you meaning the work that came out of the PMTUD wg?
That is good for discovering the optimum MTU over the
entire path, but it doesn't help to discover the optimum
MTU from *within the tunnel*. The tunnel is just an
ordinary link, and may be only a single hop on the path
btw the TCP endpoints, but the tunnel ingress needs to
discover and enforce an optimum MTU from within if the
TCPs are ever going to take advantage of the larger MTUs.

Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 19 Mar 2007 17:22:55 +0100

> And the problem with *my* draft is... ??

That (from a quick scan) it doesn't actually make it possible to have  
systems with different MTUs on the same subnet?

Have a look at the discussion under "Re: jumbo frame of GbE and IPv6  
-- A proposal" on http://www1.ietf.org/mail-archive/web/ipv6/current/ 
mail74.html to get an idea of what I'm thinking of.

Iljitsch

	<[email-masked]>
From: Iljitsch van Beijnum <[email-masked]>
Subject: Re: [RAM] what we should be working on
Date: Mon, 19 Mar 2007 17:22:55 +0100
To: "Templin, Fred L" <[email-masked]>
Cc: [email-masked]

> And the problem with *my* draft is... ??

That (from a quick scan) it doesn't actually make it possible to have  
systems with different MTUs on the same subnet?

Have a look at the discussion under "Re: jumbo frame of GbE and IPv6  
-- A proposal" on http://www1.ietf.org/mail-archive/web/ipv6/current/ 
mail74.html to get an idea of what I'm thinking of.

Iljitsch

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 19 Mar 2007 09:41:20 -0700

> systems with different MTUs on the same subnet?

Huh? I don't see any problem at all. The tunnel is configured
over an underlying link and can discover an MTU up to the MTU
that is configured on the underlying link, and that can be
independent and different from the MTUs configured by any other
hosts connected to the same underlying link. Or, maybe I don't
understand what you mean by "subnet"?
 
> mail74.html to get an idea of what I'm thinking of.

This discussion has been going on for a long time, hasn't
it? :^} Maybe sit down with the draft and let's compare
notes. IMHO, the mechanism is all there, but could benefit
from some fine-tuning. But, I don't see the limitation you
are referring to.

Thanks - Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 19 Mar 2007 18:09:43 +0100

On 19-mrt-2007, at 17:41, Templin, Fred L wrote:

>> That (from a quick scan) it doesn't actually make it possible
>> to have systems with different MTUs on the same subnet?

> understand what you mean by "subnet"?

For simplicity, let's assume this to be a logical ethernet segment.  
(I mean, who uses anything other that ethernet nowadays anyways?)  
Suppose:

host A (8192)                                     host C (9216)
              \                                   /
               switch 1 (2048) --- switch 2 (9000)
              /                                   \
host B (9000)                                     host D (16384)

The IEEE solution here is extremely simple: use 1500 bytes. The  
operational solution is to look at the diagram and either:

1. set up a 2048 byte MTU on all hosts manually
2. determine that going from 1500 to 2048 isn't worth the trouble and  
do nothing
3. replace switch 1 and configure a 8192 byte MTU manually on all hosts

This is just too hard, especially if most traffic is towards other  
destinations so you're probably limited to a 1500 byte path MTU anyway.

But if we can automate all of this on the IP level, hosts and routers  
will automatically adopt the lowest common MTU, so over time, more  
and more places will be able to hande 1500+ packets without headaches  
for operators.

> it? :^}

I don't think it has been going all this time... Didn't follow  
through back then.

> Maybe sit down with the draft and let's compare

Sure, let me know when, I'll read your draft in more detail beforehand.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 19 Mar 2007 18:09:43 +0100

---

From: [email-masked]
Date: Mon, 19 Mar 2007 13:42:14 -0400

Doubling the size of the IP packet is certainly not a help with respect to VOIP or video transmission.
Heiner

-----Ursprüngliche Mitteilung----- 
Von: [email-masked]
An: [email-masked]
Cc: [email-masked]
Verschickt: Mo., 19. Mrz. 2007, 8:12
Thema: Re: [RAM] what we should be working on

 
> I am all for increasing the average size of packets. But I'll note > that 
 
Yes, there are many reasons why this is and has been a low-return investment. However, we're rapidly approaching the situation where, for a significant number of paths, at every hop, the hardware supports packet sizes larger than 1500 bytes, but we have no reasonable way of using this capability where it exists. But if we can make things such that the largest packets possible are used, over time, more and more packets will be larger than 1500 bytes so the growth in the total amount of per-packet work will start to trail the growth in bandwidth. Over time, the savings in processing power / energy use will become more and more significant. 

_______________________________________________ 
RAM mailing list 
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ram 
________________________________________________________________________
Kostenlos: AOL eMail
2 GB Speicherplatz sowie erstklassiger Spam- und eMail Virenschutz.
Sichern Sie sich Ihre persönliche eMail Adresse noch heute!
_______________________________________________
RAM mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ram

---

From: Marshall Eubanks <[email-masked]>
Date: Mon, 19 Mar 2007 13:50:24 -0400

> respect to VOIP or video transmission.

This is a MAY not a MUST, so it wouldn't hurt. But at any case, while  
VOIP would generally not gain much
from larger packet MTU sizes, video certainly would, at least for the  
higher bandwidths. At 1 Mbps and
30 fps, for example, video would certainly benefit from jumbo frames.

Regards
Marshall

> can make things such that the largest packets possible are used,  
> over time, more and more packets will be larger than 1500 bytes so  
> the growth in the total amount of per-packet work will start to  
> trail the growth in bandwidth. Over time, the savings in processing  
> power / energy use will become more and more significant.
>

---

From: "Puneet Agarwal" <[email-masked]>
Date: Mon, 19 Mar 2007 11:40:31 -0700

Hi Iljitsch,

Software lookup based routers are definitely helped with lower
packets/sec. However, most (all?) of the core routers deployed implement
hardware based lookups (for the required lookup rates on even a
reasonably density 1G solution). Moreover, the benchmarks for these
devices are not based on average packet rates but based on peak packet
rates (ie for 64 bytes). Larger MTU helps the average pkt rate but not
the peak pkt rate.

Hence larger MTU (while helpful under some constrained scenarios) would
not really help the stringent lookup bandwidth requirements for
core/edge routers with large FIBs.

Thanks.

-Puneet

-----Original Message-----
From: Iljitsch van Beijnum [mailto:[email-masked]] 
Sent: Monday, March 19, 2007 8:55 AM
To: Puneet Agarwal
Cc: Jari Arkko; [email-masked]
Subject: Re: [RAM] what we should be working on

On 19-mrt-2007, at 16:23, Puneet Agarwal wrote:

> are flowing). Increasing the MTU to be > 1500 will not solve the 
> lookup bandwidth problem. Almost 50% of all packets are 64 bytes (on
> Ethernet)

Not sure what you mean by "solve". General purpose computers almost
always use less power when idle than when doing work. If this is true
for routers doing FIB lookups (not a given, but I don't see how it would
be possible to authoritatively say it can NEVER be true in the
future) then having fewer packets for a given amount of bandwidth is
helpful.

Obviously there are applications that generate small packets, so a
larger MTU won't make those packets larger. But for TCP it does help.  
If you have to send 100 packets with 1500 byte packets but only 20 with
large packets, this means that the number of TCP ACKs will go down from
approximately 50 to approximately 10 and the total from 150 to 30.

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 19 Mar 2007 13:58:09 -0700

Puneet,

The reasons that larger packets make sense for Packetization
Layer Path MTU Discovery (draft-ietf-pmtud-method) are the
same reasons that larger packets make sense for path MTU
assurance over tunnels, i.e., the question of whether large
packets make sense or not is not specific to tunnels. But,
tunnels also have the added responsibility of presenting
an assured minimum MTU to upper layers without causing
unnecessary fragmentation in the network.

Fred

> From: Puneet Agarwal [mailto:[email-masked]] 
> Sent: Monday, March 19, 2007 11:41 AM
> To: Iljitsch van Beijnum
> Cc: [email-masked]
> Subject: RE: [RAM] what we should be working on
> 
> Hi Iljitsch,
>

---

From: "Puneet Agarwal" <[email-masked]>
Date: Mon, 19 Mar 2007 17:51:51 -0700

Hi Fred,

I have no issue with your argument that to support overlay models (ip
tunnels, mpls etc), you want the MTU to be a little larger (say extra
200 bytes) than end node MTU. Fragmentation in the network never did
anyone any good.

I was merely objecting to the implication that were being drawn that
larger mtu will somehow reduce the lookup bandwidth requirements for
core/edge routers (which it doesn't).

The good news is that service providers have almost always insisted that
their network equipment support at least FDDI/T3 MTUs of ~4500 bytes on
all links. Hence I do not think that supporting 1500+200=1700 byte MTU
should be much of a challenge for the SPs (though I do not want to speak
for the SPs).

Thanks.

-Puneet 

-----Original Message-----
From: Templin, Fred L [mailto:[email-masked]] 
Sent: Monday, March 19, 2007 1:58 PM
To: Puneet Agarwal; Iljitsch van Beijnum
Cc: [email-masked]
Subject: RE: [RAM] what we should be working on

Puneet,

The reasons that larger packets make sense for Packetization Layer Path
MTU Discovery (draft-ietf-pmtud-method) are the same reasons that larger
packets make sense for path MTU assurance over tunnels, i.e., the
question of whether large packets make sense or not is not specific to
tunnels. But, tunnels also have the added responsibility of presenting
an assured minimum MTU to upper layers without causing unnecessary
fragmentation in the network.

Fred

> packets/sec. However, most (all?) of the core routers deployed 
> implement hardware based lookups (for the required lookup rates on 
> even a reasonably density 1G solution). Moreover, the benchmarks for 
> these devices are not based on average packet rates but based on peak 
> packet rates (ie for 64 bytes). Larger MTU helps the average pkt rate 
> but not the peak pkt rate.
>

---

From: Tony Li <[email-masked]>
Date: Tue, 20 Mar 2007 06:52:23 +0100

On Mar 19, 2007, at 6:50 PM, Marshall Eubanks wrote:

> This is a MAY not a MUST, so it wouldn't hurt. But at any case,  
> while VOIP would generally not gain much
> from larger packet MTU sizes, video certainly would, at least for  
> the higher bandwidths. At 1 Mbps and

Raising the average packet size would increase jitter.  This may well  
be irrelevant at GigE speeds, but at lower speeds could become  
significant.  Someone should work the figures...

Tony

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 20 Mar 2007 08:26:45 +0100

On 20-mrt-2007, at 6:52, Tony Li wrote:

> Raising the average packet size would increase jitter.  This may  
> well be irrelevant at GigE speeds, but at lower speeds could become  

How much jitter is acceptable?

It would also be helpful to know the amount of buffering that happens  
in routers and switches, but that seems to be all over the map.

---

From: Tony Li <[email-masked]>
Date: Tue, 20 Mar 2007 08:31:30 +0100

On Mar 20, 2007, at 8:26 AM, Iljitsch van Beijnum wrote:

>> well be irrelevant at GigE speeds, but at lower speeds could  
>> become significant.  Someone should work the figures...
>

This is also all over the map and is really the domain of our VOIP  
and video brethren.  I'll let them speak for themselves.

>
> It would also be helpful to know the amount of buffering that  
> happens in routers and switches, but that seems to be all over the  
> map.

Correct.  It very much depends on make and model.  There is no hard  
and fast standard.

Tony

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 20 Mar 2007 02:01:41 -0700

Puneet,
 
> 200 bytes) than end node MTU.

No; I didn't say that at all. I want that the tunnel not fragment
but also that it can scale its MTU up to larger sizes as appropriate
to the path MTU from within the tunnel; 1500 bytes-plus-change is
too limiting.

> Fragmentation in the network never did anyone any good.

Right, and one of the main goals of the proposal is to avoid
in-the-network fragmentation.
 
> core/edge routers (which it doesn't).

I didn't mean to imply that (I believe you may have been the one
who first asserted this?), because I think that is only one facet
of the more complex question as to whether larger packet sizes in
the Internet are desireable. And again, this is a question for the
larger issue of raising the MTU in the Internet; it is not specific
to tunnel MTU assurance.

> should be much of a challenge for the SPs (though I do not 

Already spoke to the 1500-plus-change assertion above; that's
too limiting and it is setting the bar too low for what we want
to accomplish.

Fred

---

From: "Chris L. Morrow" <[email-masked]>
Date: Tue, 20 Mar 2007 10:17:37 +0000 (GMT)

> reasonable way of using this capability where it exists. But if we

So... just thinking out loud here, the majority of end-system media is
ethernet, that's where the packet size problem exists isn't it? can't the
upstream of the majority of these systems (cable or dsl) support >1500 bytes
so the real issue is:
1) get ethernet mtu 'fixed'
2) providers will/should increase MTU on tail circuits

Does 1000bt (like in many/most 'new' computers 10/100/1000 copper) do
jumbo-frames already? So is this a problem?

---

From: "Chris L. Morrow" <[email-masked]>
Date: Tue, 20 Mar 2007 10:27:28 +0000 (GMT)

> Hence increased MTU (greater than 1500) will not solve the lookup bw

Note that I'm not convinced there IS a problem with MTU size, but could it
be that increasing MTU will decrease ack flows? For say a 10k total data
xfer today you'd send ~7 full packets if your MTU were 5k you'd send 2, so
5 less acks.

What's interesting is that over a short 10min sample of data (admittedly a
small sample) packet average size seems to be 489 bytes.

---

From: Tony Li <[email-masked]>
Date: Tue, 20 Mar 2007 11:31:46 +0100

> jumbo-frames already? So is this a problem?

AFAIK, jumbo frames are *not* standardized and have been repeatedly  
rejected by the IEEE.  The IETF is less than willing to step on the  
IEEE's turf and so has historically declined to make its own  
standards in this area.

Tony

---

From: <[email-masked]>
Date: Tue, 20 Mar 2007 10:29:55 -0000

________________________________

From: Chris L. Morrow [mailto:[email-masked]]
Sent: Tue 20/03/2007 10:27
To: Puneet Agarwal
Cc: [email-masked]
Subject: RE: [RAM] what we should be working on

[alb] from what i remember, you get a lot of very short  (100byte ish) packets (acks, dns queries etc) and a lot of 1500 byte packets (ie if you plotted number of packets agaisnt packet size, you get 2 peaks, each containing about half the packets)

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 20 Mar 2007 11:55:10 +0100

> IEEE's turf and so has historically declined to make its own  

Jumboframe capability of some sort is very common, but certainly not  
universal in NICs that support gigabit ethernet, with or without 10  
and 100 Mbps ethernet capability. There is no common jumboframe size.  
I've seen stuff in the 1500 - 2048 byte range and 9000, 9216, 16384  
and 64000 bytes. Note that jumboframes must invariably be enabled  
manually.

Not being privvy to IEEE internals, I assume the problem the IEEE has  
with changing the 802.3 maximum packet size is twofold:

1. Existing 802.3 standards mandate 1500 bytes and there is no  
support for fragmentation so if a newer standard supports larger  
packets, it won't be possible to internetwork equipment adhering to  
the old and new standards because the old stuff can't handle the 1500 
+ packets. (The graybeards among us may at this point fondly think  
back to bridging between ethernet and FDDI.)

2. 802.3 has a length field where Ethernet II has a type field. The  
type and length values don't clash for ~1500 bytes but they do for  
larger packets.

Argument 1. disappears when we fix this at the IP layer because  
unlike ethernet, IP can fragment. Argument 2. is irrelevant for IP  
because we use Ethernet II and not 802.3 encapsulation.

---

From: Tony Li <[email-masked]>
Date: Tue, 20 Mar 2007 12:05:59 +0100

On Mar 20, 2007, at 11:55 AM, Iljitsch van Beijnum wrote:

> unlike ethernet, IP can fragment.

I refer you to www.ecse.rpi.edu/Homepages/shivkuma/teaching/sp2001/ 
readings/ccr-9501-mogulf1.pdf.

> Argument 2. is irrelevant for IP because we use Ethernet II and not  
> 802.3 encapsulation.

Except for an annoying IGP called IS-IS.

Tony

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 20 Mar 2007 12:20:00 +0100

> readings/ccr-9501-mogulf1.pdf.

"On a high-speed LAN, throughput can increase almost linearly with  
packet size over a wide range of sizes. Therefore, we prefer to make  
our packets as large as possible."

:-)

The point is that IP and its higher layer protocols can adjust the  
size of the packets sent so IP can adapt to varying MTUs. In some  
cases this requires fragmentation but in general, TCP or the  
application protocol on top of UDP makes sure the packets are of an  
optimum size, which is obviously much better. Ethernet does not have  
these capabilities.

>> Argument 2. is irrelevant for IP because we use Ethernet II and  
>> not 802.3 encapsulation.

> Except for an annoying IGP called IS-IS.

Obviously if we can make IP use varying MTUs towards different hosts  
on the same subnet we should only use jumboframes for IP, not for non- 
IP protocols.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 20 Mar 2007 05:57:36 -0700

Ethernet jumbo frames may not be standardized, but they are real
and they do appear in widely-deployed products. As a layer 3
mechanism, our job is not to standardize jumbo frames or legislate
how they should be deployed; our job is only to take best advantage
of the resources the network is offering us.

Here are some links on raising the Internet MTU and ethernet
jumbo frames:

http://www.psc.edu/~mathis/MTU/
http://sd.wareonearth.com/~phil/jumbo.html
http://darkwing.uoregon.edu/~joe/jumbo-clean-gear.html

Fred

> Sent: Tuesday, March 20, 2007 3:55 AM
> To: Tony Li
> Cc: [email-masked]

---

From: "John G. Scudder" <[email-masked]>
Date: Tue, 20 Mar 2007 22:30:22 +0100

On Mar 20, 2007, at 11:17 AM, Chris L. Morrow wrote:
> jumbo-frames already?

I can't speak to the standardization status, but lots of consumer  
1000bT gear supports jumbos.  For example, all current Apple  
computers support jumbos, as do the dirt-cheap GE switches in my  
home.  Not my DSL router, though (or I'd be awfully surprised if it  
did, anyway).

--John

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 30 Mar 2007 12:42:31 +0200

(in catch up mode)

On 2007-03-20 08:31, Tony Li wrote:
> This is also all over the map and is really the domain of our VOIP and 
> video brethren.  I'll let them speak for themselves.

I'm not one of them, but I believe the answer is that on links
where this is an issue you will run a couple of diffserv classes,
and in the class used for real time traffic the packet sizes will
be uniform and an EF queuing discipline will manage jitter. I would
be shocked if we needed to do this on underloaded GigE access links
or on provider backbones. There, an MTU that readily accommodates
encapsulation overhead seems appropriate.

     Brian
> Correct.  It very much depends on make and model.  There is no hard and 
> fast standard.
> 
> Tony
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 30 Mar 2007 14:23:07 +0200

> encapsulation overhead seems appropriate.

After thinking about this for a bit, my conclusion is that there are  
two issues: the up/downlink between an end-user and a service  
provider, and the stuff in the middle.

The up/downlink is typically (but not always) the bottleneck  
bandwidth-wise and most likely to see congestion. On the sending  
side, it should be easy enough to use a queuing mechanism that gives  
priority to real-time traffic (with or without diffserv). On the  
receiving side this is more difficult, because the queuing is under  
the control of the ISP there. But the receiving side has the option  
to stick to standard packet sizes if and when needed so I don't think  
this is a show stopper. So the problem we have to consider here is  
just head of line blocking by one extra large packet. This is 1.2 ms  
for 1500 bytes at 10 Mbps, so even at that speed, a moderate increase  
is possible and at 100 Mbps or more, this isn't an issue at all.

On intra- and inter-ISP links we can probably not assume smart  
queuing. First of all, AFAIK, most ISPs simply don't use it today,  
and secondly, because this requires relatively deep packet inspection  
or trusting diffserv code points set by customers and/or other ISPs.  
Although most ISP links are fast (especially the jumbo frame capable  
ones), the problem is that there can be a lot of them, so the jitter  
gets to build up along the path.

I'm thinking something like 1 - 2 ms of extra jitter per hop would  
likely be acceptable. At Gbps speeds that's 120 to 250 kilobytes that  
can be buffered, or 13 to 26 packets for 9000 byte packets. However,  
average packet sizes are a lot smaller. The figure I've seen most is  
500 bytes. If we assume (for simplicity) that's an average of 70% 64- 
byte packets and 30% 1500-byte packets, then a similar mix with 9000- 
byte packets would result in an average packet size of 2750 bytes,  
which should be a safe upper limit. This allows for 45 - 90 packet  
buffers.

By the way, it only takes 25% of the 1500-byte packets = 7% of all  
packets to become 9000 bytes to double the average packet size to  
1000 bytes.

---

From: Marshall Eubanks <[email-masked]>
Date: Fri, 30 Mar 2007 09:40:00 -0400

Hello;

> I'm not one of them, but I believe the answer is that on links

I am one of them (video, not VOIP)

> where this is an issue you will run a couple of diffserv classes,

Yes,

> be uniform and an EF queuing discipline will manage jitter. I would

Uniform packet sizes ? Not in modern-day video, at least, not if you  
have any control traffic or
any audio traffic or a MPEG GOP type encoding
or are using variable bit rate (all of which are used in most  
practice). For example, looking at one of my H.264 1 Mbps streams,  
fairly few of the packets are near the MTU (set at 1450 bytes).

For broadcasts, buffers are huge (seconds), so the issue is with  
video conferencing and other real time video. There, there are  
typically two causes of issues

- the dynamics of human interactions mean that you like to keep  
latency down below a few 100 milliseconds, end to end. This means  
that jitter needs to be kept to less than, say, 100 msec at most.

- The arrival of certain packets may be taken as signals, for  
example, the first packet of a new video frame may signal the decoder  
to flush the old video frame. (This may even be required, depending  
on memory available, etc.)

- Generally, there is a timer, and if a packet is expected but not  
received by a certain time, it is assumed lost.

Jitter that causes certain packets to be out of order or too late can  
thus cause effective packet loss in videoconferencing. This is more  
likely than it causing human factor problems directly.

The direction of all Internet video is strongly towards increasing  
the bit rate and the frame rate, which makes raising the MTU at least  
a little attractive.

Suppose you are sending 10 Mbps HD 60 fps for telepresence. (I will  
ignore overheads to keep the math simple.) This would be (at least)  
834 pps for a 1500 byte MTU, and 140 pps for a 9000 byte MTU. For a 1  
Mbps 30 fps SD stream, there would be (again, at a minimum) 84 pps at  
1500 bytes  and 14 pps for 9000 bytes. The latter is worrisome, as 14  
pps < 30 fps, but the encoder packetizer would almost certainly  
respond by simply not making use of it (i.e., in the 1 Mbps 30 fps  
case, the actual MTU used would effectively be no more than ~ 4200  
bytes, and the average packet size would thus be even less). You  
would occasionally see a longer packet (say, at the beginning of a  
new GOP), but they would be rare.

So in practice I think that what this means is that increasing the  
MTU limit will quickly reach a point of diminishing return, at least  
for real time systems with current video systems. Going from 9000  
bytes to 90,000 bytes would be pretty irrelevant for  
videoconferencing, for example.

If we ever started doing a lot of uncompressed HD at multiple Gbps,  
then these considerations would of course change.

> encapsulation overhead seems appropriate.

My advise is to set the MTU to be efficient, and let the codecs worry  
about their packet sizes.

Regards
Marshall

>>> happens in routers and switches, but that seems to be all over  
>>> the map.
>> Correct.  It very much depends on make and model.  There is no  
>> hard and fast standard.
>> Tony