Subject: [Ice] NAT UDP Hole punching - question about a particular configuration

From: Dima Kogan <[email-masked]>
Date: Fri, 3 Aug 2018 22:10:35 -0700

While experimenting with ICE NAT traversal, I came across a class
of network configurations that currently does not result in a
direct P2P connection (but rather falls back to TURN), yet I
believe it could be in principle. I wanted to ask whether the
limitation for this kind of configurations is ''known to the ICE
community'', and how do you view the trade-off of supporting it.

Specifically, the configuration is where both peers are behind a
NAT which implements
* endpoint-independent mappings,
* endpoint-dependent filtering, and
* does not drop all unsolicited incoming packets, but rather generated
  ICMP unreachable replies.

An example for such a configuration is Linux Iptables with a MASQUERADE
rule on the FORWARD chain, yet without a DROP rule on the INPUT chain.

In this configuration the following scenario can happen:

1. Both peers learn their external IP address and port from a
STUN server.  Assume that Peer1 binds local
address/port (X1,x1), which gets mapped by NAT1 to (Y1,y1),
and for Peer2 binds (X2,x2), which gets mapped by NAT2 to (Y2,y2).

2. Peer 1 sends an outgoing UDP packet to (Y2,y2) (as part of the
hole punching), and this packet reaches NAT2 *before* Peer 2
sends an outgoing packet to (Y1,y1). In this case, since NAT2
implements endpoint-dependent filtering, NAT2 does not forward
the packet to Peer2 (this is expected). However, if NAT2 does not
drop the incoming packet, but rather generates an ICMP packet in
reply, the flow
(Y1,y1)<-->(Y2,y2) gets allocated (I believe this is called a
Null Binding in Netfilter terminology).

3. Subsequently, when Peer2 sends its UDP packet (as part of the
hole punching) from (X2,x2) to (Y1,y1), NAT2 modifies the
external port of this packet to y3, since y2 is already allocated
due to the previous unsolicited packet. When this packet reaches
NAT1, it cannot traverse it since the source port y3, is
different than the port for which the hole in NAT1 has been
punched.  Note that had the unsolicited packet not be previously
received, the external port would not be modified, since the NAT
implements endpoint-independent mappings.

This prevents direct P2P connectivity.

I believe one semi-standard solution to this approach is to begin
the hole punching by sending packets with short TTL numbers,
which would guarantee that a hole in each peer's NAT is punched
before any incoming packets arrive. This would prevent the above
scenario of unsolicited packets creating erroneous bindings in
the NAT that prevent succesful hole punching.
Obviously there are a lot of details that need to be fleshed out (e.g.,
which TTL to use, double NAT etc.) but hopefully some additional
cases could result in a P2P connection.

---

From: Justin Uberti <[email-masked]>
Date: Sun, 5 Aug 2018 18:12:17 -0700

> Null Binding in Netfilter terminology).

OK. Since Y2:y2 has already been allocated as the endpoint-independent
mapping for X2:x2 (by virtue of sending a packet from X2:x2 to a STUN
server), this is simply an application of endpoint-dependent filtering
(rejecting the packet from Y1:y1).

> due to the previous unsolicited packet.
>

This is surprising. If the mapping is truly endpoint-independent, it
shouldn't matter if X2:x2 is sending to Y1:y1, it should use the existing
(endpoint-independent) Y2:y2 mapping.

How prevalent is this situation?

> When this packet reaches
> _______________________________________________
> Ice mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ice
>

---

From: Dima Kogan <[email-masked]>
Date: Sun, 5 Aug 2018 19:48:10 -0700

> How prevalent is this situation?

I think the reason is that Linux IPTables/Netfilter implements
endpoing-independent mapping
that conforms with the following descriptions from RFC7857:

If destination addresses and ports are different for outgoing
         connections started by local clients, a NAT MAY assign the same
         external port as the source ports for the connections.  The
         port overlapping mechanism manages mappings between external
         packets and internal packets by looking at and storing their
         5-tuple (protocol, source address, source port, destination
         address, and destination port).

---

From: Dima Kogan <[email-masked]>
Date: Sun, 5 Aug 2018 19:55:29 -0700

Sorry, hit send prematurely.

As a result, when the packet from (Y1,y1) reaches NAT2, even though (Y2,y2)
is allocated for (X2,x2),
it can still be shared at this point since the remote addresses differ
(STUN for the first flow, (Y1,y1) for the second one).
However, when (X2,x2) subsequently sends a packet to (Y1,y1), it no longer
can be shared as there's a collision
on the entire tuple.

It's hard for me to know how prevalent this is, but to me the fact that is
(a) described in the RFC, and (b) is sort of the default
behavior on Linux, suggest this might be not too rare.

---

From: Nils Ohlmeier <[email-masked]>
Date: Sun, 5 Aug 2018 21:09:37 -0700

> On Aug 3, 2018, at 22:10, Dima Kogan <[email-masked]> wrote:
> 

Let me start with a clarifying question: are the two ICE endpoints in this scenario are physically separated boxes from the Linux Iptables machine?
Or are the ICE endpoints running on the same machine?

I’m asking, because I haven’t see Linux generating ICMP responses on the wire. But if you have for example a DROP rule in the local Iptables rules the local socket will behave as if an ICMP error was received. Which can be really annoying when you are trying to test something by setting up local DROP rules. Because it’s no longer behaving like a DROP rule but like a REJECT rule. But I’m only aware of this when everything runs locally on the same machine.

> cases could result in a P2P connection.

My biggest concern would be that sending with increasing TTL’s will slow things down. Especially if the two endpoints are lots of hops apart, e.g. opposite sides of the world.

Further more I would be concerned about packet loss of the TTL packets. The only way I can think of right now to ensure that your TTL trick worked would be to wait for ICMP responses. But that obviously introduces another round trip waiting for ICMP. Plus the risk of the ICMP error not getting back to the ICE endpoint at all.

Best regards
  Nils Ohlmeier

---

From: Justin Uberti <[email-masked]>
Date: Sun, 5 Aug 2018 22:04:48 -0700

The section in that RFC refers to port preservation for local (internal)
clients, and doesn't say anything about packets from external addresses.

Generally, I think that the packet inbound from Y1:y1 to Y2:y2 will simply
be discarded by the endpoint-specific filtering behavior. This packet
shouldn't cause any change to the existing mapping or cause one to be
created (or else this technique could be used from the outside to exhaust
ports on the NAT).

On Sun, Aug 5, 2018 at 7:55 PM Dima Kogan <[email-masked]> wrote:

> As a result, when the packet from (Y1,y1) reaches NAT2, even though
> (Y2,y2) is allocated for (X2,x2),

---

From: Dima Kogan <[email-masked]>
Date: Sun, 5 Aug 2018 22:14:03 -0700

On Sun, Aug 5, 2018 at 9:09 PM Nils Ohlmeier <[email-masked]> wrote:

> Let me start with a clarifying question: are the two ICE endpoints in this
> scenario are physically separated boxes from the Linux Iptables machine?
> I’m asking, because I haven’t see Linux generating ICMP responses on the
> wire. But if you have for example a DROP rule in the local Iptables rules
> the local socket will behave as if an ICMP error was received. Which can be
> really annoying when you are trying to test something by setting up local
> DROP rules. Because it’s no longer behaving like a DROP rule but like a
> REJECT rule. But I’m only aware of this when everything runs locally on the
> same machine.
>

As far as the the scenario I'm describing, I've reproduced it both when the
NAT is on a physically separate box, and when setting up a test environment
on a single machine (using ip namespaces).
To reproduce this on a single machine, I've been using the script below
(slightly adapted from a version someone posted on a discussion on the
netfilter list: https://www.spinics.net/lists/netfilter/msg58226.html).

Just to understand your question, are you concerned with hole punching
failing in the local scenario even if the local NAT is setup to DROP? If
ICMP errors cause local sockets to behave badly, it's possible to
artificially drop incoming ICMP packets on the "hosts" (again thanks to
Adel Belhouane for pointing this out on a past netfilter thread):
iptables -A INPUT -p icmp -j DROP

> My biggest concern would be that sending with increasing TTL’s will slow
> things down. Especially if the two endpoints are lots of hops apart, e.g.
> opposite sides of the world.
>

I agree that incrementing all the way from 1 until connectivity is
established (especially if waiting for responses) could be slow.
Since in most cases, the local NAT (which is the actual target of these
packets) is at distance 1-3 probably, I wonder whether some heuristic could
be a good compromise (say even something stupid as only incrementing the
TTL from 1 to 3 and then going straight to 255).

I'm not an ICE expert, so I don't know all the design considerations that
would go into a change like this, but I could imaging this only being used
if both peers think they are behind an endpoint-independent NAT (which
should theoretically allow P2P) yet the direct connection method fails. The
alternative being TURN, maybe it's worth for them to try.

>
> Further more I would be concerned about packet loss of the TTL packets.
> The only way I can think of right now to ensure that your TTL trick worked
> would be to wait for ICMP responses. But that obviously introduces another
> round trip waiting for ICMP. Plus the risk of the ICMP error not getting
> back to the ICE endpoint at all.
>
> Best regards
>   Nils Ohlmeier
>
>

(run everything as root)

#############################################################
#!/bin/sh

for i in s1 r1 in r2 s2; do ip netns del "$i" 2>/dev/null || : ; done

ip netns add s1
ip netns add r1
ip netns add in
ip netns add r2
ip netns add s2

ip -n s1 link add eth0 type veth peer name lan0 netns r1
ip -n r1 link add wan0 type veth peer name left0 netns in
ip -n in link add right0 type veth peer name wan0 netns r2
ip -n r2 link add lan0 type veth peer name eth0 netns s2

# Uncomment the next line to fix hole punching
#ip netns exec r1 iptables -P INPUT DROP
ip netns exec r1 iptables -P FORWARD DROP
ip netns exec r1 iptables -A FORWARD -m conntrack --ctstate
RELATED,ESTABLISHED -j ACCEPT
ip netns exec r1 iptables -A FORWARD -i lan0 -j ACCEPT
ip netns exec r1 iptables -t nat -A POSTROUTING -o wan0 -j MASQUERADE

# Uncomment the next line to fix hole punching
#ip netns exec r2 iptables -P INPUT DROP
ip netns exec r2 iptables -P FORWARD DROP
ip netns exec r2 iptables -A FORWARD -m conntrack --ctstate
RELATED,ESTABLISHED -j ACCEPT
ip netns exec r2 iptables -A FORWARD -i lan0 -j ACCEPT
ip netns exec r2 iptables -t nat -A POSTROUTING -o wan0 -j MASQUERADE

ip -n s1 address add 192.0.2.2/24 dev eth0
ip -n r1 address add 192.0.2.1/24 dev lan0
ip -n r1 address add 198.51.100.11/24 dev wan0
ip -n in address add 198.51.100.1/24 dev left0
ip -n in address add 203.0.113.1/24 dev right0
ip -n r2 address add 203.0.113.12/24 dev wan0
ip -n r2 address add 192.0.2.1/24 dev lan0
ip -n s2 address add 192.0.2.2/24 dev eth0

ip -n in route add unreachable 192.0.2.0/24 # enforce routers' NAT

for i in s1 r1 in r2 s2; do ip -n "$i" link set lo up; done
ip -n s1 link set eth0 up
ip -n s1 route add default via 192.0.2.1
ip -n r1 link set lan0 up
ip -n r1 link set wan0 up
ip -n r1 route add default via 198.51.100.1
ip -n in link set left0 up
ip -n in link set right0 up
ip -n r2 link set wan0 up
ip -n r2 route add default via 203.0.113.1
ip -n r2 link set lan0 up
ip -n s2 link set eth0 up
ip -n s2 route add default via 192.0.2.1

#########################################

Then run in multiple terminals:

# terminal 1: to setup inspection of the NAT on r1
ip netns exec r1 conntrack -E

# terminal 2: to setup a "STUN server" on "the internet"
ip netns exec in nc -u -l 2222

# terminal 3: Peer 1
ip netns exec s1 nc 198.51.100.1 2222 -u -p 1111

# enter some text on terminal 3 and on terminal 2. You should see the data
going both ways.
# you should see the following entries in terminal 1
# [NEW] udp      17 30 src=192.0.2.2 dst=198.51.100.1 sport=1111
dport=2222  [UNREPLIED] src=198.51.100.1 dst=198.51.100.11 sport=2222
dport=1111

# [UPDATE] udp      17 30 src=192.0.2.2 dst=198.51.100.1 sport=1111
dport=2222 src=198.51.100.1 dst=198.51.100.11 sport=2222 dport=1111
# [UPDATE] udp      17 180 src=192.0.2.2 dst=198.51.100.1 sport=1111
dport=2222 src=198.51.100.1 dst=198.51.100.11 sport=2222 dport=1111
[ASSURED]

# terminal 4: start punching a hole from Peer 2 to Peer 1
ip netns exec s2 nc -u -p 2222 198.51.100.11 1111
# enter some text. Nothing should appear on Peer 1(terminal 3)
# The following entry should appear in terminal 1
# [NEW] udp      17 30 src=203.0.113.12 dst=198.51.100.11 sport=2222
dport=1111 [UNREPLIED] src=198.51.100.11 dst=203.0.113.12 sport=1111
dport=2222

# back to terminal 3 (you can kill the netcat running there): now try
punching the hole from Peer 1 to Peer 2
ip netns exec s1 nc 203.0.113.12 2222 -u -p 1111
# You should see nothing reaching peer 2 and the problematic entry
appearing in terminal 1
# [NEW] udp      17 30 src=192.0.2.2 dst=203.0.113.12 sport=1111 dport=2222
[UNREPLIED] src=203.0.113.12 dst=198.51.100.11 sport=2222 dport=1024

Note the last port number that gets allocated (1024) which is different
from the source port 1111.

---

From: Dima Kogan <[email-masked]>
Date: Sun, 5 Aug 2018 22:43:58 -0700

> created (or else this technique could be used from the outside to exhaust

Thank you for taking the time to answer my questions.

I'm not 100% sure but maybe the creation of the mapping is triggered by the
outgoing ICMP error.
I've first asked this question on the netfilter forum, and a few people
have confirmed this behavior (
https://www.spinics.net/lists/netfilter/msg58234.html). (I've also attached
a script reproducing it in my other message).

I agree that it looks like the Linux behavior here could cause other
trouble as well (like the port exhaustion), and perhaps this is why the
good way to configure a NAT is to add the DROP rule on the incoming packets
(which would solve all trouble).
I guess I'm curios what's the right way to look at this scenario in which a
DROP rule is not present, and this behavior appears:
A) This is a corner case which, similarly to probably dozen other weird NAT
behaviors, not worth the effort for ICE.
B) Iptables should be changed to behave differently even without the DROP
rule.
C) It's worth working around it in ICE

I don't know who would have such data, but are there stats about how often
peers that are classified by STUN as being endpoint-independent still fail
to create a P2P connection?

>> However, when (X2,x2) subsequently sends a packet to (Y1,y1), it no
>> longer can be shared as there's a collision
>> on the entire tuple.
>>
>> It's hard for me to know how prevalent this is, but to me the fact that
>> is (a) described in the RFC, and (b) is sort of the default

---

From: Justin Uberti <[email-masked]>
Date: Sun, 5 Aug 2018 23:04:03 -0700

On Sun, Aug 5, 2018 at 10:44 PM Dima Kogan <[email-masked]> wrote:

>> Generally, I think that the packet inbound from Y1:y1 to Y2:y2 will
>> simply be discarded by the endpoint-specific filtering behavior. This
>> packet shouldn't cause any change to the existing mapping or cause one to
>> be created (or else this technique could be used from the outside to
>> exhaust ports on the NAT).
>>
>
> I'm not 100% sure but maybe the creation of the mapping is triggered by
> the outgoing ICMP error.
> https://www.spinics.net/lists/netfilter/msg58234.html). (I've also
> attached a script reproducing it in my other message).
>

Interesting. So, once the remote STUN message has been received by the NAT,
and the ICMP error sent, future STUN messages sent from X2:x2 to the STUN
server will yield the port Y2:y3?

> I guess I'm curios what's the right way to look at this scenario in which
> a DROP rule is not present, and this behavior appears:
> A) This is a corner case which, similarly to probably dozen other weird
> NAT behaviors, not worth the effort for ICE.

In the absence of data indicating this is a widespread issue, I'm inclined
to file this under a).

> to create a P2P connection?

This data would indeed be hard to come by, since classification is fairly
flaky (as noted in the specs), and ICE success depends on the interaction
between the two NATs, so attribution of errors would be challenging. That
said, you might be able to use the inconsistent STUN address mentioned
above to test for this specific behavior.

---

From: Dima Kogan <[email-masked]>
Date: Mon, 6 Aug 2018 00:28:31 -0700

On Sun, Aug 5, 2018 at 11:04 PM Justin Uberti <[email-masked]> wrote:

> Interesting. So, once the remote STUN message has been received by the
> NAT, and the ICMP error sent, future STUN messages sent from X2:x2 to the
> STUN server will yield the port Y2:y3?
>

Yes, that's a good way to confirm this. Here's a trace of my STUN execution:

>>>> STUN test 1 to 217.10.68.152:3478 gets the port 50000 preserved

[NEW] udp      17 30 src=100.64.0.2 dst=217.10.68.152 sport=50000
dport=3478 [UNREPLIED] src=217.10.68.152 dst=172.31.6.83 sport=3478
dport=50000 mark=29512

[UPDATE] udp      17 29 src=100.64.0.2 dst=217.10.68.152 sport=50000
dport=3478 src=217.10.68.152 dst=172.31.6.83 sport=3478 dport=50000
mark=29512

>>>> STUN test 2 begins (same destination, but STUN request now has
change_address =1, change_port=1)

[UPDATE] udp      17 180 src=100.64.0.2 dst=217.10.68.152 sport=50000
dport=3478 src=217.10.68.152 dst=172.31.6.83 sport=3478 dport=50000
[ASSURED] mark=29512

>>>> the reply comes back from a different address, get filtered, but
creates the overloaded mapping on port 50000

[NEW] udp      17 30 src=217.116.122.137 dst=172.31.6.83 sport=3479
dport=50000 [UNREPLIED] src=172.31.6.83 dst=217.116.122.137 sport=50000
dport=3479

>>>> STUN test 3 sends to the second address, and gets a new port (1024)
due to the collision
[NEW] udp      17 30 src=100.64.0.2 dst=217.116.122.137 sport=50000
dport=3479 [UNREPLIED] src=217.116.122.137 dst=172.31.6.83 sport=3479
dport=1024 mark=29512

>> I don't know who would have such data, but are there stats about how
>> often peers that are classified by STUN as being endpoint-independent still
>> fail to create a P2P connection?
>>
>

Yes, I guess that a reasonable heuristic is if the local port gets
preserved when talking to the first STUN server, and then gets modified
when talking to the second server (though to handle the case when the first
port is modified (but would have remained consistent for outgoing
connections), one should probably use 3 remote STUN addresses).

>>>>>> OK. Since Y2:y2 has already been allocated as the
>>>>>> endpoint-independent mapping for X2:x2 (by virtue of sending a packet from
>>>>>> X2:x2 to a STUN server), this is simply an application of
>>>>>> endpoint-dependent filtering (rejecting the packet from Y1:y1).
>>>>>>
>>>>>>