Subject: [Taps] User-Space Networking in iOS 11

From: Tommy Pauly <[email-masked]>
Date: Thu, 08 Jun 2017 12:50:14 -0700

Hello,

I wanted to point the TAPS group to some of the work that we announced this week at WWDC that relates to the Post-Sockets API effort. You can see a video of the session here (relevant section at ~13:50), along with the slides:

https://developer.apple.com/videos/play/wwdc2017/707 <https://developer.apple.com/videos/play/wwdc2017/707>

In the current betas of iOS 11, we have introduced “User-Space Networking” beneath our networking APIs. The transport and IP protocols are now being co-located with the security and application protocols in the process, meaning that we are no longer using sockets within the implementation of these APIs. This shift allows us to reduce the context switches between protocol layers, and could potentially open opportunities for the kind of stack flexibility and customization that the TAPS group is looking at. We’re excited to be making some first steps into a truly “Post-Sockets” world!

Thanks,
Tommy

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 8 Jun 2017 22:13:31 -0400

On 8. Jun 2017, at 15:50, Tommy Pauly <[email-masked]> wrote:
> https://developer.apple.com/videos/play/wwdc2017/707
> 
Hi Tommy,

so this means I can run my own TCP stack and my own IPv6 stack in my application, if I understood that correctly.
How is the port number space managed between the different applications? I'm wondering how the demultiplexing is
done?
Can I also use the API for a transport protocol supporting the port number concept, but not being UDP or TCP,
like SCTP or UDP-Lite or DCCP or whatever might come up in the future?

Best regards
Michael
> 
> Thanks,
> Tommy
> _______________________________________________
> Taps mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/taps

---

From: Tommy Pauly <[email-masked]>
Date: Fri, 09 Jun 2017 09:16:29 -0700

HI Michael,

> On Jun 8, 2017, at 7:13 PM, Michael Tuexen <[email-masked]> wrote:
> 

The ability to run your own stack is not currently exposed; the user-space stack for TCP, etc, is part of the system library. However, this architecture does open up the possibility to extend or modify this in the future. If people have specific ways that they think this would be most useful to extend and modify, please let me know.

The kernel is still responsible for managing the port namespace between applications, and demultiplexing incoming packets to the correct processes. Since that is part of the kernel, the port demultiplexing still needs to understand where to look for port numbers in known protocols. It seems like for the use-case you’re mentioning for new protocols, it could be useful to allow that logic to be extended, or ‘learn’ new protocols. This isn’t something we’re doing now, but definitely an interesting idea!

Thanks,
Tommy

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 9 Jun 2017 19:04:32 +0200 (CEST)

On Fri, 9 Jun 2017, Tommy Pauly wrote:

> The kernel is still responsible for managing the port namespace between 
> applications, and demultiplexing incoming packets to the correct 
> processes. Since that is part of the kernel, the port demultiplexing 
> still needs to understand where to look for port numbers in known 
> protocols. It seems like for the use-case you’re mentioning for new 
> protocols, it could be useful to allow that logic to be extended, or 
> ‘learn’ new protocols. This isn’t something we’re doing now, but 
> definitely an interesting idea!

Also sounds like a use-case for the work on "device should be handed 
multiple addresses". If it were, you could multiple in a way that each 
process that needed to have its own port space or run special protocols, 
could get unique usage of a newly allocated IPv6 address.

If this is something that would be useful, it would be good if this 
use-case was brought to 6man and v6ops. It would probably be more credible 
if you brought it than if I did.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 9 Jun 2017 19:55:32 -0400

> On 9. Jun 2017, at 12:16, Tommy Pauly <[email-masked]> wrote:
> 
> HI Michael,
> 
Leaving the port multiplexing / demultiplexing at the kernel makes a lot of sense. Are you doing something
like:
* open a TCP socket.
* bind it.
* set a socket option that you want to send/recv raw TCP packets
That way you could pass the packets up and down to a userland stack and could
coexist with a kernel stack. (Assuming that the kernel/userland boundary still uses sockets).
However, that doesn't fit the figure shown a WWDC, where you had the IP stack and the TCP
as part of the userland stack. So does the application the route selection, interface
selection?

Is there any documentation available for the interface between kernel land and userland?
Is the same architecture available on Mac OS X?

Best regards
Michael

---

From: Tommy Pauly <[email-masked]>
Date: Sat, 10 Jun 2017 10:04:43 -0700

> On Jun 9, 2017, at 4:55 PM, Michael Tuexen <[email-masked]> wrote:
> 

The figure shown on the slides shows the TCP/IP stack in the process-space since the actual formation of the TCP and IP headers is done there, along with all of the TCP state machine logic and IP fragment reassembly, etc. The kernel/user boundary is not using sockets at all for these connections, but instead is using a new mechanism for memory-mapping to the kernel. The application can influence the route and interface selection with preferences (no-cellular, for example), which are combined with the kernel policies and routing tables, as in the Policy Context in the Post-Sockets draft. This provides a binding for ports and interface, which is used for subsequent traffic flows and is validated by the kernel on egress. Effectively, this means that the IP part of the stack in user space can be thinner, since the routing information is more fixed and doesn’t require more routing lookups as is typically done in a kernel implementation.

> Is there any documentation available for the interface between kernel land and userland?

No, not yet at this time. This is the initial roll-out of the architecture, so it’s still early days.

> Is the same architecture available on Mac OS X?

We do not currently expose the same functionality on macOS. As mentioned in the talk, this is in part due to the fact that it is not compatible with network kernel extensions, which are common on macOS, but will be deprecated at some point in the future.

Thanks,
Tommy

>>>> [email-masked] <mailto:[email-masked]>
>>>> https://www.ietf.org/mailman/listinfo/taps <https://www.ietf.org/mailman/listinfo/taps>
>>> 
>>> [email-masked] <mailto:[email-masked]>
>>> https://www.ietf.org/mailman/listinfo/taps <https://www.ietf.org/mailman/listinfo/taps>

---

From: Tommy Pauly <[email-masked]>
Date: Sat, 10 Jun 2017 10:20:46 -0700

> On Jun 9, 2017, at 10:04 AM, Mikael Abrahamsson <[email-masked]> wrote:
> 
> Also sounds like a use-case for the work on "device should be handed multiple addresses". If it were, you could multiple in a way that each process that needed to have its own port space or run special protocols, could get unique usage of a newly allocated IPv6 address.
> 
> If this is something that would be useful, it would be good if this use-case was brought to 6man and v6ops. It would probably be more credible if you brought it than if I did.

Hi Mikael,

A per-process IPv6 address is not something we’re doing at this time, but it’s certainly a natural extension of an in-process networking stack. If the device is being allocated multiple IPv6 address, and ideally a /64, as recommended by RFC 7934, then the host system could assign an address to each process, or at least each process that has beyond-ordinary networking needs.

This could also have interesting implications for a device that’s dealing with multiple Provisioning Domains (PvDs). Processes could be grouped to share the address/port namespaces of a given PvD; for example, enterprise applications could be using a common address and set of ports, while personal applications could be sharing another set.

The main open question I would have would be around how an application would express that it should be using a separate address-space, or how that would be indicated as some sort of TAPS parameters. Any ideas are welcome =)

Thanks,
Tommy

---

From: Michael Tuexen <[email-masked]>
Date: Sun, 11 Jun 2017 10:33:21 +0200

> On 10. Jun 2017, at 19:04, Tommy Pauly <[email-masked]> wrote:
> 
> 
> 
Ahh, I see. Similar to the multistack approach described in
https://www.ietf.org/proceedings/95/slides/slides-95-tsvarea-2.pdf
> We do not currently expose the same functionality on macOS. As mentioned in the talk, this is in part due to the fact that it is not compatible with network kernel extensions, which are common on m…
I see. Thanks a lot for your explanations. They really help to understand what Apple is providing!

Best regards
Michael