Subject: Re: [lisp] LISP Over Lossy Links

From: [email-masked] (Noel Chiappa)
Date: Wed, 16 Sep 2009 11:14:24 -0400 (EDT)

> From: Margaret Wasserman <[email-masked]>

    > Do you think we should suggest that LISP routers queue packets with
    > no map cache hit, and send them when the map response is received? 

Holding packets like that is a real no-no in a router. (Routers do not, for
instance, generally even hold packets waiting for an ARP reply, and that's a
lot shorter RTT.)

If we discover that there's a problem, we would probably want to look at
other ways of solving it (e.g. sending packets through the resolution
hierarchy) rather than holding packets.

But let's see how things actually work in practise, before we add a 'fix'?
In theory, ARP and DNS could cause problems too, but in practise they work
fine. (Heck, in theory old Ethernet - the non-hub kind - doesn't work well
under heavy loads... but in practise it worked just fine.) It's one thing
to pro-actively fix something that could lead to a real problem, something
else to fix something that _might_ be a performance issue - before we know
if there _is_ a performance issue.

	Noel

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 16 Sep 2009 11:41:17 -0400

> instance, generally even hold packets waiting for an ARP reply, and  

Are you sure that routers don't queue packets waiting for an ARP  
reply?  It's my understanding that routers _do_ queue packets when  
they send an ARP request,, and that those packets are transmitted when  
a matching ARP response is received.  IP stacks that I've worked on  
have had small packet queues associated with each outstanding ARP  
request for this purpose.

It is also suggested by RFC 1812 (Router Requirements), which says:

"The link layer MUST NOT report a Destination Unreachable error to IP  
solely because there is no ARP cache
entry for a destination; it SHOULD queue up to a small number of  
datagrams breifly while performing the ARP
request/reply sequence, and reply that the destination is unreachable  
to one of the queued datagrams only
when this proves fruitless."

Margaret

---

From: Sam Hartman <[email-masked]>
Date: Wed, 16 Sep 2009 11:52:14 -0400

>>>>> "Margaret" == Margaret Wasserman <[email-masked]> writes:

    Margaret> Are you sure that routers don't queue packets waiting
    Margaret> for an ARP reply?  It's my understanding that routers
    Margaret> _do_ queue packets when they send an ARP request,, and
    Margaret> that those packets are transmitted when a matching ARP
    Margaret> response is received.  IP stacks that I've worked on
    Margaret> have had small packet queues associated with each
    Margaret> outstanding ARP request for this purpose.

be sure to scope answers to either higher-end routers or low-end
routers unless you're sure they apply globally.

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 16 Sep 2009 15:51:53 -0400

I sent the information below in response to a private e-mail, and then  
I realized that the rest of the group might be interested in my  
(admittedly informal and unscientific) findings...

There is an important difference between LISP and ARP that may make  
the trade-offs quite different for LISP than they are for ARP.

With ARP, a given router will only have L + P = N ARP entries, where L  
is the number of locally attached hosts, and P is the number of other  
router interfaces attached to the same link.  However, a LISP node  
will have a much larger number of possible map cache entries -- one  
for every EID prefix on the Internet.

Let's take a real-life example...

Sandstorm has a fairly simple leaf network with a single Internet  
connection serving a network with ~30 attached hosts.  We have one  
main router, and an e-mail/web server on that communicates directly  
with the outside.

I put a forensics tool just outside of our router/NAT/firewall, where  
the LISP xTR function would go if we were using it for address  
independence and multihoming.  In a typical hour of workday traffic  
(captured from 10am-11am this morning), our forensics tool recorded  
~18000 "connections" with ~2050 nodes outside of our site.  This  
includes connections initiated from within our company, including DNS  
"connections", as well inbound connection attempts (most of which did  
not succeed).  In that hour, hosts behind our firewall initiated 3006  
DNS requests with 43 different servers, 3728 HTTP connections with 493  
different servers, 96 SSL connections with 35 different servers, 45  
NTP connections with 13 different servers, and a smattering of other  
connections.  All told, we communicated _outbound_ with about 650  
different systems.  During the same period about 30 different systems  
established inbound connections with us.

In that hour, our main router ARP'ed 5 different IP addresses on its  
outside link -- all of them routers or servers attached to its local  
interfaces.  It probably also arp'ed for most, if not all, of our 30  
internal nodes (I didn't see those messages, because I wasn't watching  
the internal network).  So, our router maintained an ARP cache of < 50  
entries.

On our (pretty typical small business) network in a typical hour, our  
(theoretical) LISP ITR would need to do > 7000 map cache lookups  
(assuming it performed map cache lookups only for outbound traffic).   
I looked at the external IP addresses in use, and only a very small  
number of them were grouped in a way that suggested they might be from  
the same network, so it is likely that, in a typical working hour, our  
LISP router would need to send map requests and establish map cache  
entries for > 600 EID prefixes.

The huge majority of these 600 entries represent the start of a  
connection (either with a DNS request or a TCP <SYN>).  In many cases  
(where, for example, URLs with hostnames are used to refer to external  
HTTP images), the establishment of a single connection might require  
the creation of two different map cache entries, one for the DNS  
server and one for the TCP connection.  This doubles the delay before  
an image can even start loading.

I am sympathetic to arguments that we should take a "wait and see"  
attitude about whether fixes are needed in this area, but I do  
consider this to be a major area of concern for LISP.  So, when we  
define the actual experiments that we intend to run (which I believe  
we have to do, in order to get our experimental RFCs published), I  
think we should pay special attention to covering the performance  
impact of LISP on connection establishment.

Margaret

P.S.  BTW, it appears from the traffic I captured that our small  
enterprise router/NAT/firewall _does_ queue at least one outbound  
packet when it sends an ARP request, and that it sends the packet as  
soon as the ARP response is received (just judging from the temporal  
proximity of the two events).