Subject: Re: [lisp] LISP NAT Traversal [Was: Virtual meeting]

From: Albert López <[email-masked]>
Date: Wed, 1 Apr 2020 11:28:48 +0200

On 1/4/20 2:20, Dino Farinacci wrote:
>> Thanks Dino for your reply. Answers inline
>
> No prob.
>
>>
>> On 31/3/20 1:27, Dino Farinacci wrote:
>>>> I would also like to add the NAT Traversal draft discussion into 
>>>> the agenda.
>>>>
>>> I agree it should on the agenda and an over-due work item we need to 
>>> get working.
>>>
>>>
>>>> We have experience implementing the NAT traversal functionality in 
>>>> our LISP open source implementation (OOR) which have support for 
>>>> Android and IOS devices. We believe that NAT traversal is a 
>>>> critical point for LISP-MN and we would like to share our 
>>>> experience on that. .
>>>>
>>>> Our implementation is based on the draft-ermagan-lisp-nat-traversal 
>>>> and during its study some questions have emerged that could be 
>>>> interesting to discus:
>>>>
>>> I have an implementation of NAT-traversal in the lispers.net 
>>> <http://lispers.net>
>>>  implementation which is a subset and simpler version of the 
>>> draft-ermagan-lisp-nat-traversal. I think that NAT-traversal is so 
>>> important as a core functionality for LISP because many people use 
>>> it today just to traverse NATs.
>>>
>>> I would suggest we make this a working group document. It is in the 
>>> charter and is long over-due to move from individual submission.
>>>
>>>
>>>> • What happens when a device handovers between NAT and not NAT 
>>>> interfaces? The xTR has to notify the remote ITRs, however an xTR 
>>>> behind NAT usually only has a default map cache entry with the RTR 
>>>> as a default gateway. Therefore, the xTR cannot notify directly 
>>>> ITRs of the change of the mapping.
>>>>
>>> If the remote ITR subscribes for 0.0.0.0/0 then any more specific 
>>> mappings which are registered and Map-Notified by the Map-Server to 
>>> the subscribers. That is one way the problem can be solved.
>> So here, you are suggesting to use the publish/subscribe draft isn’t 
>> it? Yes this could be a good solution. Is there any study 
>> of scalability regarding this option?
>
> Yes, that is what I’m suggesting.
>
>>> Another way is now that the xTR is not behind a NAT, it can SMR 
>>> currently RLOCs it is talking to. Which also may be an RTR that the 
>>> remote ITRs can be using (when they are behind NATs). That causes 
>>> the non-NAT ITR and the remote RTR to do mapping lookup to get the 
>>> global/non-translated RLOC of this xTR.
>> When the xTR is only using the RTR (I believe is the usual case), xTR 
>> doesn’t know who are its remote ITRs to send and SMR so it depends on 
>> how the RTR react to SMR.
>
> When the remote ETR isn’t behind NAT, the RTR can reach it directly 
> but so can the ITR.
You say " *That causes the non-NAT ITR *and the remote RTR *to do 
mapping lookup* to get the global/non-translated RLOC" but who sends 
them an SMR to update the mapping? ETR can not do it because he doesn't 
have their entries in the cache.
>
>>>> • What happens in the previous case if the not NAT interface is 
>>>> IPv6? IPv6 is a special case. We want to maintain the connectivity 
>>>> with established connections (keep using the same RTR, which has 
>>>> the map-cache). For that to work, the RTR needs to have an IPv6 to 
>>>> receive an SMR from the xTR after the handover (from the new xTR’s 
>>>> IPv6 RLOC). Therefore RTRs announced by the Map Server should have 
>>>> both IPv4 and IPv6. As we don't know which RTR IPv6 addresses map 
>>>> to which IPv4 addresses, all RTRs should be notified.
>>>>
>>> It can be a local decision if an ITR uses an RTR for IPv6 traffic or 
>>> just encapsulate directly to the ETRs that register the EID. Most 
>>> customers are going to want the shortest path so I have seen the 
>>> latter be deployed.
>>>
>>> I would suggest you only use RTRs for IPv6 when you are either doing 
>>> (1) signal-free IPv6 multicast or (2) have a traffic-engineering 
>>> requirement. Both in general, specify that IPv4 and IPv6 can and 
>>> should run ships-in-the-night.
>>>
>> What I was referring with this question is , If we are behind NAT 
>> using an RTR as a default gateway and then we handover to an IPv6 
>> connection (at that point we are not considering the 
>> publish/subscribe solution), If we don't want to lose established 
>> connections, xTR has to notify the RTR (SMR) of the change of the 
>> mapping. To notify it, 
>
> Yes. But now I’m not clear if you the “IPv6 connection” is referring 
> to an IPv6 EID or RLOC. Please clarify.
I am referring to IPv6 RLOCs
>
>> the RTR should have an IPv6 RLOC that should be included in the list 
>> of RTRs of the Info Reply message.  Depending on what the RTR does 
>> with the SMR,  remote ITRs could move stablished connections to 
>> direct path or continue using the RTR.
>
> If IPv6 is an RLOC, it is assumed anyone can reach it and a site xTR 
> need not encap packets to an RTR. RTRs for NAT-traversal are used only 
> for IPv4 RLOCs, regardless if the EID is IPv4 or IPv6.
If we want to maintain established connections, when we handover from 
NAT to IPv6 RLOC, we need to notify the change of mapping. If we use SMR 
procedure, our only entry is the RTR, so if it doesn't have an IPv6 
RLOC, we can not SMR it and the xTR will loose the already established 
connections.
>
>>>> • Which should be the procedure of an RTR in front of receiving an 
>>>> SMR from the xTR that just handover from NAT to not NAT? Probably 
>>>> the answer is that the xTR sends an SMR to the RTR which then has 
>>>> two options 1) the
>>>>
>>>> RTR can do an SMR to the appropriates ITRs based on the map-cache 
>>>> entries (but then how to know which of the map-cache entries were 
>>>> used by the xTR) or 2) it can act as a PxTR for the xTR that 
>>>> performed the handover. In that case, until when? Until the 
>>>> expiration of the TTL? Furthermore, what happens when an xTR roams 
>>>> again before the previous TTL expires.
>>>>
>>> This is no different than any other type of RLOC change. And we have 
>>> many mechanisms documented for this. They are:
>>> TTL-expiry, SMRs, pubsub Map-Notify from mappingn system.
>>>
>> TTL-expiry is not realistic for mobility scenarios where handovers 
>> can be frequent and you don’t know when will happen
>
> Well you will find with experience that link-layer assocaition 
> estalishment takes seconds. So having the map-cache TTL around 3-5 
> seconds is reasonable. Follow the numbered flows.
>
>> The behaviour of an RTR receiving an SMR is what I would like to 
>> clarify with this question
>>
>>
>> pubsub is a good option but at this point I would like to clarify 
>> what happens when an RTR receive an SMR.
>
> Note, that SMRs will cause a flurry of messages depending on how 
> aggressive the ETR wants the ITRs sending to it to be updated. And 
> requires queuing demands on the ETR for reliable delivery. Hence why 
> data-driven SMRs were introduced in implementations. But that only 
> happens for EID-mobility (EID is not co-located with RLOC) and not in 
> LISP-MN (where EID and RLOC are co-located).

 From my point of view a TTL of 3-5 seconds is also quite aggressive 
from the perspective of the Map Server that receive the queries 
(probably will have several MN registered to it). May be, a good point 
would be to evaluate all the pros and cons of each mechanism and 
recommend to use one for each case. In any case, I believe that we need 
to specify what is the behaviour of an RTR when receiving an SMR.

>
>>>> • Which is the mechanism to change the RTR used by and xTR?
>>>>
>>> Note the xTR gets periodic Info-Replies back from the map-server. 
>>> The list of RTRs can change. So when the list of RTRs is changed, it 
>>> is configured in the map-server and the next Info-Request that the 
>>> NAT-assisted xTR sends is returned with the new list. The 
>>> lispers.net <http://lispers.net> implementation updates RTRs when 
>>> the list changes and RLOC-probes each one to make sure they have 
>>> reachability before using.
>> I believe this answer is replying  next question. With this question, 
>> my intention was to make constance that the draft doesn’t specify 
>> (may be it is not needed to go into this detail level) a mechanism, 
>> from the xTR point of view, to use a different RTR from the list of 
>> the info reply (for example, due to a better latency time)
>
> We should fix that in the spec then. Thanks for finding it.
>
>>>> • Is there any procedure from MS point of view to notify associated 
>>>> xTRs to change the RTR they are using?
>>>>
>>> Yes, in the Info-Reply messages the map-sever send, that is in 
>>> draft-ermagan-lisp-nat-traversal.
>>>
>>>
>>>> • If a NATed device wants to manage their map request and map 
>>>> replies, which ITR RLOCS should use to fill the Map Request?
>>>>
>>> It needs to be reachable RLOCs. And in the lispers.net 
>>> <http://lispers.net> implementation, an xTR behind a NAT registers 
>>> it’s translated RLOC and the RTR RLOCs. That is so the remote RTR 
>>> can use the translated RLOC (and the translated port it learns from 
>>> Info-Requests) and remote ITRs to use the RTR’s RLOC (since that RTR 
>>> is the only node that can get packets through the NAT).
>> I don’t have the full details of the lispers implementation of 
>> NAT-Traversal solution. Is any document describing it?.
>
> I don’t have something written up but I do have slides that show 
> protocol flow. See at the end of this email. It can all be explained 
> in one slide. ;-)
The Info Request message contain authentication section, is the key 
shared between MS,xTR and RTRs?
>
>> My concern here is regarding section 5.1.2 of the draft (Map-Request 
>> and Map-Reply handling). When we sent an Encap Map-Request to get a 
>> mapping, we add a list of iTR addresses to receive the Map-Reply. We 
>> can add here the global address of the xTR but if the NAT has not 
>> been previously punched, we will not receive never a Map-Reply, at 
>> least for symetrics NATs.
>
> You MUST put public-RLOCs in the mapping and the only node that can do 
> that is the ETR. And it discovers its public-RLOC but receiving a 
> Info-Reply from the MS that echos the outer source address and source 
> UDP port from the Info-Request message. The message that the NAT 
> translated the source address and source port.
May be I am wrong but the scenario of open the NAT with the Info-Request 
Info-Reply messages is only valid for a scenario with a single MS where 
all EIDs are registered with proxy mode. If the reply of the Map-Request 
comes form other MS or remote ETR, then we don't have a hold in the NAT 
and the Map-Reply  message will not be received.
>
>>
>>> Another topic but related to NAT-traversal.
>>>
>>> Note there is a problem I came across during deployment of an IoT 
>>> use-case. This is a description of the problem with an example. Say 
>>> I am xTR1 and learn from the Map-Server RTRs A, B, and C. and I want 
>>> to encapsulate packets for an EID behind xTR2. The encapsulation 
>>> algorithms indicate that xTR1 use a 5-tuple hash to load-split 
>>> packets across A, B, and C. And xTR1 knows that A, B, and C are 
>>> reachable via RLOC-probing. However, xTR1 has no idea if any of A, 
>>> B, or C, can reach xTR2. So xTR1 can choose an RTR that would drop 
>>> packets if it couldn’t reach xTR2.
>>>
>>> I solved this in the
>>> lispers.net <http://lispers.net>
>>>  implementation to have the ETRs only register the RLOC addresses of 
>>> RTRs if they are currently reachable via RLOC-probing. This is how 
>>> it works using the example from above:
>>>
>>> (1) xTR2 learns of RTRs A, B, and C from the map-server.
>>> (2) It uses those RTRs as the RLOCs for the map-cache entry 0.0.0.0/0.
>>> (3) It RLOC-probes A, B, and C.
>>> (4) It registers its translated RLOC and RTRs A, B, and C in 4 
>>> RLOC-records of the Map-Register.
>>> (5) That means any remote ITR (xTR1) would encap to either A, B, or C.
>>> (6) If xTR2 lost reachability to A, then what is registered is only 
>>> B and C (and the translated RLOC).
>>> (7) Then xTR1 only has a choice of B and C to encapsulate to for the 
>>> EID behind xTR2.
>>>
>>> What limits the solution is asymmetry. Meaning if the path from xTR2 
>>> to RTR A is down but the path from RTR A to xTR2 is up, xTR1 would 
>>> not use RTR A. So bidirectional connectivity is required (for the 
>>> pair of RLOCs that are part of the RLOC-probe).
>>>
>> In the scenario you are proposing, both xTR are behind NAT using the 
>> same MS or at least the same RTRs isn’t it?
>
> Well as a LISP-MN roams, it uses the same Map-Server no matter where 
> it attaches to the network. Thereby using the same RTR-list for what 
> the Map-Server is configured with. The Map-Server with policy COULD 
> return a different RTR-list based on the LISP-MN RLOC address or GPS 
> coordinates it registers with.
>
>> What would happen if the xTR 2 is not behind NAT  or it is behind NAT 
>> but using different RTRs?
>
> I don’t have it implemented for a multiple sets of RTR-lists.
>
> NAT-traversal, doing it dynamically and seamless to the user and 
> network administrator is one of the toughest problems I had to solve 
> in my career. It is so messy and has too many combinations that need 
> to work.
>
I agree
> Also note, there is a requirement for an xTR to have two interfaces, 
> each connected to different NATs. And load-splitting is required 
> across the two intefaces (inbound)

I think that the draft already contemplate this scenario, at least when 
both interfaces use the same RTR despite using different NATs. If they 
are using different RTRs, both RTR's RLOCs will be included in the 
mapping, then is the ITR the responsible to split the traffic between 
RLOCs. Does it cover your scenario?

Albert

> Dino
>> Regards,
>> Albert L.

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 1 Apr 2020 14:23:38 -0700

Pruning some texts. Answers to your questions inline.

> On Apr 1, 2020, at 2:28 AM, Albert López <[email-masked]> wrote:
> 
>> When the remote ETR isn’t behind NAT, the RTR can reach it directly but so can the ITR. 
> You say " That causes the non-NAT ITR and the remote RTR to do mapping lookup to get the global/non-translated RLOC" but who sends them an SMR to update the mapping? ETR can not do it because he doesn't have their entries in the cache. 

The 3 mechanisms I mentioned in previous email. SMRs (both control or data-driven), Map-Notify with pubsub from Map-Server, or map-cache TTL management.

>>>> What I was referring with this question is , If we are behind NAT using an RTR as a default gateway and then we handover to an IPv6 connection (at that point we are not considering the publish/subscribe solution), If we don't want to lose established connections, xTR has to notify the RTR (SMR) of the change of the mapping. To notify it, 
>> 
>> Yes. But now I’m not clear if you the “IPv6 connection” is referring to an IPv6 EID or RLOC. Please clarify.

Then you don’t need NAT-traversal.

>> 
>>> the RTR should have an IPv6 RLOC that should be included in the list of RTRs of the Info Reply message.  Depending on what the RTR does with the SMR,  remote ITRs could move stablished connections to direct path or continue using the RTR.
>> 
>> If IPv6 is an RLOC, it is assumed anyone can reach it and a site xTR need not encap packets to an RTR. RTRs for NAT-traversal are used only for IPv4 RLOCs, regardless if the EID is IPv4 or IPv6.
> If we want to maintain established connections, when we handover from NAT to IPv6 RLOC, we need to notify the change of mapping. If we use SMR procedure, our only entry is the RTR, so if it doesn't have an IPv6 RLOC, we can not SMR it and the xTR will loose the already established connections. 

This is true. That is why you use an RTR for NAT-traversal only for IPv4 RLOCs.

>> 
>> Note, that SMRs will cause a flurry of messages depending on how aggressive the ETR wants the ITRs sending to it to be updated. And requires queuing demands on the ETR for reliable delivery. Hence why data-driven SMRs were introduced in implementations. But that only happens for EID-mobility (EID is not co-located with RLOC) and not in LISP-MN (where EID and RLOC are co-located).
> From my point of view a TTL of 3-5 seconds is also quite aggressive from the perspective of the Map Server that receive the queries (probably will have several MN registered to it). May be, a good point would be to evaluate all the pros and cons of each mechanism and recommend to use one for each case. In any case, I believe that we need to specify what is the behaviour of an RTR when receiving an SMR.

It depends where your mobility domain is. And how fast the requirements are to change mappings. We have 3 mechanisms and they all have their disadvantages.

>>>>> 
>>>> It needs to be reachable RLOCs. And in the lispers.net implementation, an xTR behind a NAT registers it’s translated RLOC and the RTR RLOCs. That is so the remote RTR can use the translated RLOC (and the translated port it learns from Info-Requests) and remote ITRs to use the RTR’s RLOC (since that RTR is the only node that can get packets through the NAT).
>>> I don’t have the full details of the lispers implementation of NAT-Traversal solution. Is any document describing it?.
>> 
>> I don’t have something written up but I do have slides that show protocol flow. See at the end of this email. It can all be explained in one slide. ;-)
> The Info Request message contain authentication section, is the key shared between MS,xTR and RTRs?

I don’t use authentication for Info-Request and Info-Replies. Otherwise, you need a security association with the RTR that can change quickly. You don’t want to provision keys between xTRs and RTRs but since xTR to MS is more of a provision based admission control to the control-plane system, it is acceptable and necessary there.

>> 
>>> My concern here is regarding section 5.1.2 of the draft (Map-Request and Map-Reply handling). When we sent an Encap Map-Request to get a mapping, we add a list of iTR addresses to receive the Map-Reply. We can add here the global address of the xTR but if the NAT has not been previously punched, we will not receive never a Map-Reply, at least for symetrics NATs.
>> 
>> You MUST put public-RLOCs in the mapping and the only node that can do that is the ETR. And it discovers its public-RLOC but receiving a Info-Reply from the MS that echos the outer source address and source UDP port from the Info-Request message. The message that the NAT translated the source address and source port.
> May be I am wrong but the scenario of open the NAT with the Info-Request Info-Reply messages is only valid for a scenario with a single MS where all EIDs are registered with proxy mode. If the reply of the Map-Request comes form other MS or remote ETR, then we don't have a hold in the NAT and the Map-Reply  message will not be received.

No, not true. My implementation allows an ETR to send Info-Requests to 2 map-servers and assumes/requires that the RTR-list configured on each map-server is the same.

Yes, and proxy mode is required because in my implementation, the map-server returns ETR public-RLOCs to requesting RTRs and RTR RLOCs to requesting ITRs. Plus the map-server can’t send packets to destination port 4342 through the NAT. But this part is solvable. Just not documented or implemented.

>> Also note, there is a requirement for an xTR to have two interfaces, each connected to different NATs. And load-splitting is required across the two intefaces (inbound)
> I think that the draft already contemplate this scenario, at least when both interfaces use the same RTR despite using different NATs. If they are using different RTRs, both RTR's RLOCs will be included in the mapping, then is the ITR the responsible to split the traffic between RLOCs. Does it cover your scenario?

Yes, and it also supports two xTRs behind the same NAT to know the fact so they can encapsulate each other via private-RLOCs. I call this short-cut NAT-traversal in my implementation.

Cheers,
Dino