Subject: [lisp] LISP Over Lossy Links

From: Margaret Wasserman <[email-masked]>
Date: Tue, 15 Sep 2009 13:37:13 -0400

Hi All,

I have a new question for the group...

I have some concerns about how LISP will perform when TCP is used over  
LISP when there is a lossy link somewhere in the mapping network and/ 
or the link from the mapping network to the remote ETR.

I've been mapping out packet flows on my whiteboard, and I have a  
scenario that looks like this...

Assume no existing LISP connections/mapping between site A (served by  
xTR A) and site B (served by xTR B).

A host (Host A with EID A) behind ITR A initiates a TCP connection to  
a host (Host B with EID B) behind ETR B.  The <SYN> packet gets to ITR  
A, and ITR A sends a map request for EID B to its map resolver.  ITR A  
also drops the original <SYN> packet.  What happens if the map request  
gets lost before it gets to ETR B?

A very short time passes, and Host A retransmits the <SYN>.

The retransmission gets to ITR A, but the LISP spec indicates that map  
requests for the same EID should be rate limited to no more than one  
per second.  So, the retransmission is dropped.  This continues until  
a second has passed.

After 1 second, Host A retransmits the <SYN> again, and it is dropped  
by ITR A because there is no mapping.  At that point, ITR A sends  
another map request for EID B.  If this request gets a response, the  
Nth retransmission will finally be sent after 1 second + ~1 round trip  
of repeated retransmission.

By this point, Host A has determined that there is significant  
congestion between Host A and Host B, when actually only one packet  
was lost.  Host A will have a large retransmission timeout for Host B,  
and its slow-start mechanism will have significantly backed off due to  
(apparent) congestion.  So, it will take a while for Host A to begin  
sending to EID B again at full speed.

To avoid this scenario, I think we either need to institute some sort  
of retransmission for map requests that don't get a reply, or consider  
a more sophisticated rate limiting mechanisms than just one per second.

Margaret

P.S.  I am also concerned about rate limiting the sending of map  
requests per-EID...  Are there any cases where you would want to check  
the liveness of multiple locators using a map request probe?  Wouldn't  
this effectively limit you to checking the liveness of one locator per  
second?

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 15 Sep 2009 13:40:03 -0400

On Sep 15, 2009, at 1:37 PM, Margaret Wasserman wrote:
> was lost.

Oh, and to make this worse, the packet that was lost (the map request)  
wasn't (necessarily) lost on a link that will be traversed by the TCP  
traffic...

Margaret

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 15 Sep 2009 17:58:06 -0700

Let me show you what happens from this trace. I indented the trace  
data and my comments follow. This is an ITR that is ssh'ing to an ETR  
when the map-cache for the ETR does not exist.

Note EIDs are in 240.0.0.0/8 space and locators are in 1.0.0.0/8.

> dr22(config-if)# ssh 240.23.0.23
> 18:55:09.290262 netstack: [3620] (default) Send packet on  
> Ethernet2/2 (prty 7): s=1.22.23.22, d=1.11.22.11, nh=1.11.22.11,  
> proto=17 (udp), sport=65375, dport=4341, udp_len=120, chksum=0, tos/ 
> dscp=0xc0/0x30,

This is the encapsulated Map-Request. Yes, the SYN is dropped.

> 18:55:09.292337 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp),  
> sport=4342, dport=65375, udp_len=60, chksum=c0ce, tos/dscp=0xc0/0x30,

This is the returning Map-Reply. Notice 2ms RTT because boxes are in  
the same rack.

> 18:55:12.310387 netstack: [3620] (default) Send packet on  
> Ethernet2/6 (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23,  
> proto=17 (udp), sport=61567, dport=4341, udp_len=76, chksum=0, tos/ 
> dscp=0x0/0x0,

This is the retransmitted SYN LISP encapsulated packet. Note the  
retransmission timer is 3 seconds.

> 18:55:12.311510 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp),  
> sport=61567, dport=4341, udp_len=72, chksum=0, tos/dscp=0x0/0x0,  
> ip_len=92, id=f02c, ttl=64
> 18:55:12.311585 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp),  
> sport=22, dport=33390, seq=ce934c58, flags: SYN ACK, tos/dscp=0x0/0x0,

The first line is the encapsulated SYN/ACK and then when decapsulated,  
you see the SYN/ACK packet.

> 18:55:12.311843 netstack: [3620] (default) Send packet on  
> Ethernet2/6 (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23,  
> proto=17 (udp), sport=61567, dport=4341, udp_len=68, chksum=0, tos/ 
> dscp=0x0/0x0,

This is the ACK going back LISP encapsulated.

What follows is data going back and forth:

> 18:55:12.363370 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp),  
> sport=61567, dport=4341, udp_len=88, chksum=0, tos/dscp=0x0/0x0,  
> ip_len=108, id=f02d, ttl=64
> 18:55:12.363455 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp),  
> sport=22, dport=33390, seq=ce934c59, flags: ACK PUSH, tos/ 
> dscp=0x0/0x0,
> 18:55:12.365107 netstack: [3620] (default) Send packet on  
> Ethernet2/6 (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23,  
> proto=17 (udp), sport=61567, dport=4341, udp_len=88, chksum=0, tos/ 
> dscp=0x0/0x0,
> 18:55:12.370773 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp),  
> sport=61567, dport=4341, udp_len=592, chksum=0, tos/dscp=0x0/0x0,  
> ip_len=612, id=f02e, ttl=64
> 18:55:12.370856 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp),  
> sport=22, dport=33390, seq=ce934c6d, flags: ACK, tos/dscp=0x0/0x0,  
> ip_len=576, id=0e8e, ttl=64
> 18:55:12.371121 netstack: [3620] (default) Send packet on  
> Ethernet2/6 (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23,  
> proto=17 (udp), sport=61567, dport=4341, udp_len=592, chksum=0, tos/ 
> dscp=0x0/0x0,
> 18:55:12.371380 netstack: [3620] (default) Send packet on  
> Ethernet2/6 (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23,  
> proto=17 (udp), sport=61567, dport=4341, udp_len=328, chksum=0, tos/ 
> dscp=0x0/0x0,
> 18:55:12.371484 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp),  
> sport=61567, dport=4341, udp_len=280, chksum=0, tos/dscp=0x0/0x0,  
> ip_len=300, id=f02f, ttl=64
> 18:55:12.371552 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp),  
> sport=22, dport=33390, seq=ce934e79, flags: ACK PUSH, tos/ 
> dscp=0x0/0x0,
> 18:55:12.446838 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp),  
> sport=61567, dport=4341, udp_len=68, chksum=0, tos/dscp=0x0/0x0,  
> ip_len=88, id=f030, ttl=64
> 18:55:12.446919 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp),  
> sport=22, dport=33390, seq=ce934f4d, flags: ACK, tos/dscp=0x0/0x0,  
> ip_len=52, id=0e90, ttl=64
> 18:55:12.447168 netstack: [3620] (default) Send packet on  
> Ethernet2/6 (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23,  
> proto=17 (udp), sport=61567, dport=4341, udp_len=92, chksum=0, tos/ 
> dscp=0x0/0x0,
> 18:55:12.452147 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp),  
> sport=61567, dport=4341, udp_len=220, chksum=0, tos/dscp=0x0/0x0,  
> ip_len=240, id=f031, ttl=64
> 18:55:12.452217 netstack: [3620] (default) Rcvd packet on  
> Ethernet2/6 (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp),  
> sport=22, dport=33390, seq=ce934f4d, flags: ACK PUSH, tos/ 
> dscp=0x0/0x0,
> 18:55:12.456347 netstack: [3620] (default) Send packet on  
> Ethernet2/6 (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23,  
> proto=17 (udp), sport=61567, dport=4341, udp_len=212, chksum=0, tos/ 
> dscp=0x0/0x0,
> 18:55:12.467629 netstack2008 Mar 30 18:55:12.640315 netstack: [3620]  
> (default) Send packet on Ethernet2/6 (prty 7): s=1.22.23.22,  
> d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), sport=615The  
> authenticity of host '240.23.0.23 (240.23.0.23)' can't be established.

And here you see visual data:

> RSA key fingerprint is 24:cb:8b:1f:8b:d7:a1:3d:a3:59:db:de: 
> 9b:b6:f7:f1.
> Are you sure you want to continue connecting (yes/no)?

This only happens for the first connection (if no other packet was  
sent before) between Site A and Site B.

The point here is that TCP RTT estimates are not affected since early  
retransmission timers are larger than the Map-Request rate-limiting.

I know we don't show a Map-Request loss case here but the next Map- 
Request would have been sent 3 seconds later.

If a packet today would get dropped the same thing would happen. The  
TCP connection would take 3 seconds to come up. But *it's not for  
every connection* in the LISP case.

Dino

On Sep 15, 2009, at 10:37 AM, Margaret Wasserman wrote:

> I have some concerns about how LISP will perform when TCP is used  
> over LISP when there is a lossy link somewhere in the mapping  
> network and/or the link from the mapping network to the remote ETR.
>
> Assume no existing LISP connections/mapping between site A (served  
> by xTR A) and site B (served by xTR B).
>
> A host (Host A with EID A) behind ITR A initiates a TCP connection  
> to a host (Host B with EID B) behind ETR B.  The <SYN> packet gets  
> to ITR A, and ITR A sends a map request for EID B to its map  
> resolver.  ITR A also drops the original <SYN> packet.  What happens  
> if the map request gets lost before it gets to ETR B?
>
> The retransmission gets to ITR A, but the LISP spec indicates that  
> map requests for the same EID should be rate limited to no more than  
> one per second.  So, the retransmission is dropped.  This continues  
> until a second has passed.
>
> After 1 second, Host A retransmits the <SYN> again, and it is  
> dropped by ITR A because there is no mapping.  At that point, ITR A  
> sends another map request for EID B.  If this request gets a  
> response, the Nth retransmission will finally be sent after 1 second  
> + ~1 round trip of repeated retransmission.
>
> was lost.  Host A will have a large retransmission timeout for Host  
> B, and its slow-start mechanism will have significantly backed off  
> due to (apparent) congestion.  So, it will take a while for Host A  
> to begin sending to EID B again at full speed.
>
> To avoid this scenario, I think we either need to institute some  
> sort of retransmission for map requests that don't get a reply, or  
> consider a more sophisticated rate limiting mechanisms than just one  
> per second.
>
> Margaret
>
> requests per-EID...  Are there any cases where you would want to  
> check the liveness of multiple locators using a map request probe?   
> Wouldn't this effectively limit you to checking the liveness of one  
> locator per second?
>
>
>
> _______________________________________________
> lisp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lisp

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 15 Sep 2009 22:49:11 -0400

On Sep 15, 2009, at 8:58 PM, Dino Farinacci wrote:

> data and my comments follow. This is an ITR that is ssh'ing to an  
> ETR when the map-cache for the ETR does not exist.
>

This isn't the same case as I was describing.  The map request  
obviously wasn't dropped, as the reply came back.

> retransmission timer is 3 seconds.

That seems like a long initial TCP retransmission timer.  I see the  
initial retransmission happening in under a second on my system.
> The point here is that TCP RTT estimates are not affected since  
> early retransmission timers are larger than the Map-Request rate- 
> limiting.

The initial RTT estimates on your system seem to be considerably  
longer than on mine.  Is that something you needed to set in order to  
make TCP work properly with LISP?
> every connection* in the LISP case.

I understand that.

Margaret

---

From: Loránd Jakab <[email-masked]>
Date: Wed, 16 Sep 2009 14:36:59 +0200

Hi Dino,

> sent before) between Site A and Site B.

Indeed, but take for example typical web pages today: they have dozens 
of external references on different domains, with different IP 
addresses. The first time I load any web page, load time will increase 
with 3 seconds for each external IP it references, in addition to DNS 
lookups and download times.

Of course this is not really an issue if the ITR is a border router of a 
busy domain, most popular destinations will be cached. But if we 
consider the other deployment scenarios discussed recently on list, it 
may become an issue. Waiting 30+ seconds for each new web page to load 
is really bad user experience.

Regards,
Loránd Jakab

---

From: Damien Saucez <[email-masked]>
Date: Wed, 16 Sep 2009 14:45:23 +0200

Dino Farinacci wrote:
>> 18:55:09.290262 netstack: [3620] (default) Send packet on Ethernet2/2 
>> (prty 7): s=1.22.23.22, d=1.11.22.11, nh=1.11.22.11, proto=17 (udp), 
>> sport=65375, dport=4341, udp_len=120, chksum=0, tos/dscp=0xc0/0x30,
>
>> 18:55:09.292337 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp), sport=4342, 
>> dport=65375, udp_len=60, chksum=c0ce, tos/dscp=0xc0/0x30,
>
>> 18:55:12.310387 netstack: [3620] (default) Send packet on Ethernet2/6 
>> (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), 
>> sport=61567, dport=4341, udp_len=76, chksum=0, tos/dscp=0x0/0x0,
>
>> 18:55:12.311510 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp), sport=61567, 
>> dport=4341, udp_len=72, chksum=0, tos/dscp=0x0/0x0, ip_len=92, 
>> id=f02c, ttl=64
>> 18:55:12.311585 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp), sport=22, 
>> dport=33390, seq=ce934c58, flags: SYN ACK, tos/dscp=0x0/0x0,
>
>> 18:55:12.311843 netstack: [3620] (default) Send packet on Ethernet2/6 
>> (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), 
>> 18:55:12.363370 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp), sport=61567, 
>> dport=4341, udp_len=88, chksum=0, tos/dscp=0x0/0x0, ip_len=108, 
>> id=f02d, ttl=64
>> 18:55:12.363455 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp), sport=22, 
>> dport=33390, seq=ce934c59, flags: ACK PUSH, tos/dscp=0x0/0x0,
>> 18:55:12.365107 netstack: [3620] (default) Send packet on Ethernet2/6 
>> (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), 
>> 18:55:12.370773 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp), sport=61567, 
>> dport=4341, udp_len=592, chksum=0, tos/dscp=0x0/0x0, ip_len=612, 
>> id=f02e, ttl=64
>> 18:55:12.370856 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp), sport=22, 
>> dport=33390, seq=ce934c6d, flags: ACK, tos/dscp=0x0/0x0, ip_len=576, 
>> id=0e8e, ttl=64
>> 18:55:12.371121 netstack: [3620] (default) Send packet on Ethernet2/6 
>> (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), 
>> 18:55:12.371380 netstack: [3620] (default) Send packet on Ethernet2/6 
>> (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), 
>> sport=61567, dport=4341, udp_len=328, chksum=0, tos/dscp=0x0/0x0,
>> 18:55:12.371484 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp), sport=61567, 
>> dport=4341, udp_len=280, chksum=0, tos/dscp=0x0/0x0, ip_len=300, 
>> id=f02f, ttl=64
>> 18:55:12.371552 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp), sport=22, 
>> dport=33390, seq=ce934e79, flags: ACK PUSH, tos/dscp=0x0/0x0,
>> 18:55:12.446838 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp), sport=61567, 
>> dport=4341, udp_len=68, chksum=0, tos/dscp=0x0/0x0, ip_len=88, 
>> id=f030, ttl=64
>> 18:55:12.446919 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp), sport=22, 
>> dport=33390, seq=ce934f4d, flags: ACK, tos/dscp=0x0/0x0, ip_len=52, 
>> id=0e90, ttl=64
>> 18:55:12.447168 netstack: [3620] (default) Send packet on Ethernet2/6 
>> (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), 
>> sport=61567, dport=4341, udp_len=92, chksum=0, tos/dscp=0x0/0x0,
>> 18:55:12.452147 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=1.22.23.23, d=1.22.23.22, proto=17 (udp), sport=61567, 
>> dport=4341, udp_len=220, chksum=0, tos/dscp=0x0/0x0, ip_len=240, 
>> id=f031, ttl=64
>> 18:55:12.452217 netstack: [3620] (default) Rcvd packet on Ethernet2/6 
>> (prty 7): s=240.23.0.23, d=240.22.0.22, proto=6 (tcp), sport=22, 
>> dport=33390, seq=ce934f4d, flags: ACK PUSH, tos/dscp=0x0/0x0,
>> 18:55:12.456347 netstack: [3620] (default) Send packet on Ethernet2/6 
>> (prty 7): s=1.22.23.22, d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), 
>> sport=61567, dport=4341, udp_len=212, chksum=0, tos/dscp=0x0/0x0,
>> RSA key fingerprint is 24:cb:8b:1f:8b:d7:a1:3d:a3:59:db:de:9b:b6:f7:f1.
> I know we don't show a Map-Request loss case here but the next 
> Map-Request would have been sent 3 seconds later.
>
3 seconds seems to be the best case.

Source                                                Destination
---------SYN---------> Dropped
-------------Request -->...
<-----------Reply---------...

---------SYN -------------------------------------------->
                      Dropped <----------- SYN + ACK-----
                        ... <----------------- Request ------------
                        ... ---------------------- Reply ------------>

----- SYN ------------------------------------------------>
<---------------------------------------------------SYN + ACK -----

The minimum time here is 6 seconds. If the reply needs more than 3 
seconds to come, it jump easily to 18 seconds. And a 3+second mapping 
time could be observed on lossy links.

Damien Saucez

>> over LISP when there is a lossy link somewhere in the mapping network 
>> and/or the link from the mapping network to the remote ETR.
>>
>> a host (Host B with EID B) behind ETR B.  The <SYN> packet gets to 
>> ITR A, and ITR A sends a map request for EID B to its map resolver.  
>> ITR A also drops the original <SYN> packet.  What happens if the map 
>> request gets lost before it gets to ETR B?
>>
>> Nth retransmission will finally be sent after 1 second + ~1 round 
>> trip of repeated retransmission.
>>
>> due to (apparent) congestion.  So, it will take a while for Host A to 
>> begin sending to EID B again at full speed.
>>
>> of retransmission for map requests that don't get a reply, or

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 16 Sep 2009 09:10:10 -0400

> lookups and download times.

I absolutely agree. Some major sites (such as news aggregation sites)  
have literally hundreds of pieces of
other people's content to download. As of right now

Google News - http://news.google.com/news?ned= -  37 files (mostly non- 
Google images)
Washington Post http://www.washingtonpost.com/ - 133 files (who knows,  
but at least some are external)

Maybe LISP would change the way that people construct web sites, but I  
wouldn't count on it.

> is really bad user experience.

It seems to me that this would make LISP caching into a good business  
model for a CDN, but it would not find
favor otherwise.

Regards
Marshall

---

From: Loránd Jakab <[email-masked]>
Date: Wed, 16 Sep 2009 16:09:37 +0200

Hi Margaret,

> initial retransmission happening in under a second on my system.

RFC1122 recommends (with SHOULD) a 3 second initial RTO. This is the 
default on most current operating systems. There is a presentation from 
Stockholm by a Googler, advocating for a 1 second initial RTO here:

http://www.ietf.org/proceedings/75/slides/tcpm-1.pdf

According to they survey only Mac OS X has a 1 second initial RTO, the 
rest use 3 (or 3.38 in the case of Solaris).

I have checked, and it is possible to change this value in Windows by 
modifying the registry. I couldn't find the appropriate sysctl on Linux 
and right now I have no access to a FreeBSD machine, but I suppose it is 
also possible to do it.

> make TCP work properly with LISP?

Actually, LISP would benefit from a _lower_ initial RTO, just like 
Google is advocating for.

Loránd Jakab

---

From: Eliot Lear <[email-masked]>
Date: Wed, 16 Sep 2009 16:32:28 +0200

Marshall, Loránd,

Indeed there are worst case scenarios that could make things seem
"bad".  However, these are likely to not be all that common.  There is a
whole lot of shared infrastructure out there that means that you'll
already have the lookup cached from one web site to another.  Also,
there are ample methods to mitigate such scenarios.
Here are just a few possibilities:

    * Carry the first packet over the ALT backbone.  There are
      challenges with handling bearer traffic.
    * Have a slightly more robust mapping timer that doesn't simply
      discard expired entries, but holds them in reserve.
    * Combine the above with an LRU very large secondary cache.  This
      would give you performance approximating today, assuming the
      mapping entries don't actually change.
    * Pre-cache popular sites.

I'm sure there other possibilities.  These are just a few.

Eliot

On 9/16/09 3:10 PM, Marshall Eubanks wrote:
> Google News - http://news.google.com/news?ned= -  37 files (mostly
> non-Google images)

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 16 Sep 2009 10:58:43 -0400

Hi Loránd,

> rest use 3 (or 3.38 in the case of Solaris).

Thanks for the information.  I am using a Mac, which (in my testing,  
ftp'ing to a non-existing host) retransmitted in just over 900ms the  
first time, followed by subsequent 1 second retransmissions until the  
application layer timed out.

So, on the Mac, I'd normally see a 1 second delay in establishing  
every TCP connection (initial packet dropped for map request, first  
retransmission acknowedged).

However, on most systems we'd see a 3 second delay, even when no  
packets are lost at all.

Do you think we should suggest that LISP routers queue packets with no  
map cache hit, and send them when the map response is received?  At  
least then we'd only introduce a single RTT of delay for session  
establishment, instead of 3 seconds per connection.

Dino indicated that this delay would only happen for the first packet  
of a connection where there is no map cache entry, but won't there be  
a packet drop and retransmission for existing connections whenever the  
LISP mapping TTL expires?  Or are we expecting implementations to  
renew LISP mappings more frequently than that?

Margaret

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 16 Sep 2009 11:02:48 -0400

I had always assumed that if there is a cache entry, and it is being 
used, that the ITR would refresh the cache entry before it expires.  (I 
see no reason to standardize exactly when.)

Yours,
Joel

Margaret Wasserman wrote:
> So, on the Mac, I'd normally see a 1 second delay in establishing every 
> TCP connection (initial packet dropped for map request, first 
> However, on most systems we'd see a 3 second delay, even when no packets 
> are lost at all.
> 
> Dino indicated that this delay would only happen for the first packet of 
> a connection where there is no map cache entry, but won't there be a 
> packet drop and retransmission for existing connections whenever the 
> LISP mapping TTL expires?  Or are we expecting implementations to renew 
> LISP mappings more frequently than that?
> 
> Margaret
>

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 16 Sep 2009 11:03:41 -0400

Hi Elliot,

On Sep 16, 2009, at 10:32 AM, Eliot Lear wrote:
> "bad".  However, these are likely to not be all that common.  There  
> is a whole lot of shared infrastructure out there that means that  
> you'll already have the lookup cached from one web site to another.

It's actually pretty common that a lot of people get up (or get into  
work) around the same time and start loading websites that haven't  
been loaded for hours.  In our network, at least, you see a lot of DNS  
traffic at that time, as the DNS caches have timed out over night.

> Also, there are ample methods to mitigate such scenarios.
> 	• Carry the first packet over the ALT backbone.  There are  

Wouldn't this cause the same sorts of problems that folks are trying  
to avoid by making sure that all of the packets from a given flow are  
load-balanced across the same path?
>
> 	• Have a slightly more robust mapping timer that doesn't simply  

I think that this is a better, and simpler, choice than above.

> 	• Combine the above with an LRU very large secondary cache.  This  
> would give you performance approximating today, assuming the mapping  
> entries don't actually change.

It doesn't matter how large the cache is if the map cache entries time  
out when the network is quiet.

> 	• Pre-cache popular sites.

I _suppose_ this could work, although which sites are popular seems to  
change rather rapidly.

Margaret

---

From: Eliot Lear <[email-masked]>
Date: Wed, 16 Sep 2009 17:12:22 +0200

Hey Margaret,
> traffic at that time, as the DNS caches have timed out over night.

That's right, and so one area of research I think would be useful would
be to look at caching strategy.  I suspect there is a whole lot of
existing work from which one could borrow in the area of file systems,
by the way.

> load-balanced across the same path?

It really depends on how many packets ACTUALLY traverse the ALT.  But it
would seem safe to say that the path behavior would shift as the map
reply is answered.

> out when the network is quiet.

Right -- again, I would suggest this as an area for research, but what I
would try is literally storing old entries on disk and using them for
data until you have a better route.  This will fall back to what is
currently discussed today, but it would make the worst case all the more
rare.

> I _suppose_ this could work, although which sites are popular seems to

Right- you could even envision pre-caching services, based on observed
popularity in the core.

Eliot

---

From: Damien Saucez <[email-masked]>
Date: Wed, 16 Sep 2009 17:24:42 +0200

Margaret,

> establishment, instead of 3 seconds per connection.
Loránd has presented that approach in Stockholm. Whatever the speed of 
the link, you can do buffering, when the buffer is full, just drop new 
packets in miss and thus have the same behavior for these packets than 
without buffering.

Regards,

Damien Saucez

---

From: Florin Coras <[email-masked]>
Date: Wed, 16 Sep 2009 18:43:03 +0300

Hi Dino,

Dino Farinacci wrote:
> Let me show you what happens from this trace. I indented the trace data
> and my comments follow. This is an ITR that is ssh'ing to an ETR when
> the map-cache for the ETR does not exist.
> 
> This is the returning Map-Reply. Notice 2ms RTT because boxes are in the
> same rack.
> 
>> d=1.22.23.23, nh=1.22.23.23, proto=17 (udp), sport=615The authenticity
>> of host '240.23.0.23 (240.23.0.23)' can't be established.
> 
> This only happens for the first connection (if no other packet was sent
> before) between Site A and Site B.
> 
> If a packet today would get dropped the same thing would happen. The TCP
> connection would take 3 seconds to come up. But *it's not for every
> connection* in the LISP case.
> 
> Dino
> 

>From what I see in your example, if the Map-Reply takes more than 3s due
to the delay brought by the Mapping System or is lost,  the ITR would
send another Map-Request because the SYN timer (3 seconds) has expired
on the host that initiated the communication.
Such delay in the mapping system isn't impossible in a global mapping
system and we showed in Stockholm that the ALT might have, occasionally,
delays larger than 3s. Moreover, having LISP closer to the edge will
further increase the delay.
Therefore the ITR should retransmit Map-Requests using an independent
timer instead of depending on receiving the next TCP syn. I guess this
can be implemented easily because there is state related to the nonce
already kept on the ITR.

Thanks,
Florin

---

From: John Zwiebel <[email-masked]>
Date: Wed, 16 Sep 2009 10:25:36 -1000

On Sep 16, 2009, at 4:58 AM, Margaret Wasserman wrote:

>
>
> Do you think we should suggest that LISP routers queue packets with  
> no map cache hit, and send them when the map response is received?   
> At least then we'd only introduce a single RTT of delay for session  

Absolutely not.

We went around and around on this for multicast with
app developers insisting that the first packet not be dropped.
A hole lot of extra protocol was developed to "fix" this problem.
None of that worked, a lot of time was wasted.  And in the
real world no one misses it.

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 16 Sep 2009 18:21:50 -0400

>> establishment, instead of 3 seconds per connection.

+1

Marshall

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 16 Sep 2009 15:55:28 -0700

> send another Map-Request because the SYN timer (3 seconds) has expired

The 3 second delay is TCP not the mapping system. And if any other  
host at the site sent to the same destination sort during that 3  
second period, then a Map-Request would be generated.

> can be implemented easily because there is state related to the nonce

The delay of the mapping system is based on the paths the Map-Request  
takes. The path can be pretty much on path into the core.

In our test network, my site has an EID-prefix out of RIPE's  
allocation. My site is in San Jose. I register to the Map-Server in  
Amsterdam. So when I send a Map-Request from another site in San Jose  
to my site, the Map-Request travels to Amsterdam, then gets  
encapsulated to the ETR in San Jose. Then the Map-Reply goes to the  
source site in San Jose. We have measured times of 200-300ms.

Dino

---

From: Jesper Skriver <[email-masked]>
Date: Thu, 17 Sep 2009 15:58:13 +0300

+1

In real life it doesn't make any difference.

For example, when a typical Cisco IOS router needs to send a packet to  
a next-hop for which it does not have an ARP entry, it will send out  
an ARP request after having dropped the user packet, that it could not  
forward due to an ARP entry missing.

/Jesper

> real world no one misses it.

/Jesper