Subject: Re: [v6ops] Please review draft-donley-behave-deterministic-cgn

From: Dave Thaler <[email-masked]>
Date: Wed, 4 Jan 2012 22:27:04 +0000

Yes operational commentary would be very welcome.

Also FYI, the author of the draft has declared IPR on it:
https://datatracker.ietf.org/ipr/1617/

-Dave

-----Original Message-----
From: Fred Baker [mailto:[email-masked]] 
Sent: Tuesday, October 11, 2011 6:52 AM
To: v6ops v6ops WG
Cc: Behave Chairs; [email-masked]
Subject: Please review draft-donley-behave-deterministic-cgn

Operators subject to law enforcement subpoenas and using Carrier Grade NAT are having difficulties with the syslog rate for per-connection logging. Chris Donley has proposed a simplification; the provider allocates a deterministic set of source ports to his subscribers, and only needs to log exceptions. Research in the area suggests that usage of source ports is pareto distributed; a typical user has a requirement on the order of 4 port numbers in simultaneous use (median), but port scans and other large volume uses drive the average quite a bit higher. 

They haven't asked for it, but I suspect the behave chairs would appreciate operational commentary on the draft.

http://tools.ietf.org/html/draft-donley-behave-deterministic-cgn
  "Deterministic Address Mapping to Reduce Logging in Carrier Grade NATs",
  Chris Donley, Chris Grundemann, Vikas Sarawat, Karthik Sundaresan,
  26-Sep-11

---

From: Fred Baker <[email-masked]>
Date: Wed, 4 Jan 2012 14:31:08 -0800

> https://datatracker.ietf.org/ipr/1617/

I wish them luck when they try to enforce it.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 04 Jan 2012 17:32:16 -0500

When I asked on operator about the port distribution, their answer was 
somewhat different.
What they said was that when the users are idle, they don't use many 
ports.  But when the user is busy, they use a lot of ports.
This makes repeated, largeish, block allocation sensible, but makes 
pre-allocation of a small number, and then individual allocation 
thereafter, a bad strategy, as it will basically log as much as current.

I do not have access to any research results to disambiguate these two 
reports.

Yours,
Joel

On 1/4/2012 5:27 PM, Dave Thaler wrote:
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Cameron Byrne <[email-masked]>
Date: Wed, 4 Jan 2012 14:56:28 -0800

> Yes operational commentary would be very welcome.

OK then.

As someone who has operated LSN/CGN before such terms were used, i
feel qualified to say that we do not need this function.  Others may
find it useful, but not my network.

Regarding port usage, it can be north of 1,000 ports simultaneously.
Depends on the user, it's very bursty.  The trend is more and more
ports.

Regarding logging, my CGN does not have this problem and does not need
this solution.

Regarding is it deploy-able?  I don't think so. "You cannot get there
from here".  If you are following your ARIN policies and are 80% to
100% utilizing your allocated blocks, how do you get to this model of
putting your users behind these bounded static allocations?  It seems
like there is a high barrier to entry here, that is not achievable for
the target audience of IPv4 constrained networks.

And if you want geographic redundancy from N locations, you must cut
your address allocation by 1/N?

This is yet another IPv4 life support effort that i would like to see go away.

CB

---

From: Fred Baker <[email-masked]>
Date: Wed, 4 Jan 2012 15:18:20 -0800

On Jan 4, 2012, at 2:32 PM, Joel M. Halpern wrote:

> When I asked on operator about the port distribution, their answer was somewhat different.
> What they said was that when the users are idle, they don't use many ports.  But when the user is busy, they use a lot of ports.
> This makes repeated, largeish, block allocation sensible, but makes pre-allocation of a small number, and then individual allocation thereafter, a bad strategy, as it will basically log as much as current.
> 
> I do not have access to any research results to disambiguate these two reports.

Chris should step in here. 

https://www.ietf.org/proceedings/82/slides/behave-5.pdf, quoting
http://www.wand.net.nz/~salcock/someisp/flow_counting/result_page.html
also referenced by RFC 6269

In the wand.net.nz HTML, look specifically for "mean number of peak sessions for active subscribers only for each 30 minute period"; the HTML doesn't have tags that would allow me to simply get you there.

I suspect that Chris/WAND and your service provider might be defining "user" and "a lot" differently. The Waikato numbers are very likely typical of a residential network behind a NAT - in a home, even if several computers are in use, it's unlikely that everyone in the home is web surfing during the same minute. Common browsers usually limit the number of TCPs they open in parallel to something like 2 or 4, and those sessions usually stay open single digit numbers of seconds; this says that in a typical residential network, they see 2-4 computers simultaneously doing that, for a peak of 4-8 TCP sessions simultaneously active. That would be very surprising in a corporate network, but makes sense for residential network.

Is the service provider you asked talking about corporate subscribers?

---

From: Joel jaeggli <[email-masked]>
Date: Wed, 04 Jan 2012 16:15:30 -0800

On 1/4/12 15:18 , Fred Baker wrote:
>> When I asked on operator about the port distribution, their answer
>> was somewhat different. What they said was that when the users are
>> idle, they don't use many ports.  But when the user is busy, they
>> use a lot of ports. This makes repeated, largeish, block allocation
>> sensible, but makes pre-allocation of a small number, and then
>> individual allocation thereafter, a bad strategy, as it will
>> basically log as much as current.
>> 
> In the wand.net.nz HTML, look specifically for "mean number of peak
> sessions for active subscribers only for each 30 minute period"; the
> HTML doesn't have tags that would allow me to simply get you there.
> 
> I suspect that Chris/WAND and your service provider might be defining
> "user" and "a lot" differently. The Waikato numbers are very likely
> typical of a residential network behind a NAT - in a home, even if
> several computers are in use, it's unlikely that everyone in the home
> is web surfing during the same minute. Common browsers usually limit
> the number of TCPs they open in parallel to something like 2 or 4,

per page... pardon my the lack of regex foo.

as mostly idle mac says the following.

Joels-MacBook-Pro:~ jjaeggli$ netstat -ln |egrep '(80|443)' | grep EST|
wc -l
      74

20 minutes later it says:

Joels-MacBook-Pro:~ jjaeggli$ netstat -ln |egrep '(80|443)' | grep EST|
wc -l
      58

your facebook page, and gmail and  ap foo are going to hold those
connections open forever if they can.

> and those sessions usually stay open single digit numbers of seconds;
> this says that in a typical residential network, they see 2-4
> computers simultaneously doing that, for a peak of 4-8 TCP sessions
> simultaneously active. That would be very surprising in a corporate
> network, but makes sense for residential network.
> 
>> Yours, Joel
>>> -----Original Message----- From: Fred Baker
>>> [mailto:[email-masked]] Sent: Tuesday, October 11, 2011 6:52 AM 
>>> To: v6ops v6ops WG Cc: Behave Chairs;
>>> [email-masked] Subject:
>>> Please review draft-donley-behave-deterministic-cgn
>>> 
>>> Operators subject to law enforcement subpoenas and using Carrier
>>> Grade NAT are having difficulties with the syslog rate for
>>> per-connection logging. Chris Donley has proposed a
>>> simplification; the provider allocates a deterministic set of
>>> source ports to his subscribers, and only needs to log
>>> exceptions. Research in the area suggests that usage of source
>>> ports is pareto distributed; a typical user has a requirement on
>>> the order of 4 port numbers in simultaneous use (median), but
>>> port scans and other large volume uses drive the average quite a
>>> bit higher.
>>> 
>>> They haven't asked for it, but I suspect the behave chairs would
>>> appreciate operational commentary on the draft.
>>> 
>>> "Deterministic Address Mapping to Reduce Logging in Carrier Grade
>>> NATs", Chris Donley, Chris Grundemann, Vikas Sarawat, Karthik
>>> Sundaresan, 26-Sep-11
>>> 
>>> 
>>> 
>>> _______________________________________________ v6ops mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
>>> 
>> _______________________________________________ v6ops mailing list 
>> [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
> 
> _______________________________________________ v6ops mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Fred Baker <[email-masked]>
Date: Wed, 4 Jan 2012 16:38:13 -0800

> connections open forever if they can.

OK. Let's compromise at 128. That's bigger than your or WAND's numbers (BTW, I don't leave a FB or GMail page open; neither do the other big users in my life. We close applications we're not using, and gmail comes to my mail tool using POP).

Define "big". We can theoretically have 512=2^(16-7) subscribers share a single address.

Issues, yes. 512 subscribers per address... a whole lot more than we have now.

And for anything else, there's MasterCard. Oops, wrong commercial...

---

From: Cameron Byrne <[email-masked]>
Date: Wed, 4 Jan 2012 18:43:10 -0800

On Jan 4, 2012 4:38 PM, "Fred Baker" <[email-masked]> wrote:
> OK. Let's compromise at 128. That's bigger than your or WAND's numbers
(BTW, I don't leave a FB or GMail page open; neither do the other big users
in my life. We close applications we're not using, and gmail comes to my
mail tool using POP).
> Issues, yes. 512 subscribers per address... a whole lot more than we have
now.

Who is we? I have 30+ million subscribers and 100k ipv4.

Cb

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 05 Jan 2012 03:08:33 -0500

The operator was a residential operator, reporting onnot for attribution 
on observed residential patterns.  (I had asked because I was looking at 
some ideas loosely related to the ones Chris is proposing.)

Apparently, it has to do with the number of connections that various 
common applications have taken to opening, within the browser context. 
Not being an applications guy, I took his word on that front.
I will look at the wand data.

Yours,
Joel

---

From: Joel jaeggli <[email-masked]>
Date: Thu, 05 Jan 2012 00:14:16 -0800

> Issues, yes. 512 subscribers per address... a whole lot more than we have now.

I agree that's more realistic. whether it's enough to make the model
work I guess is something I'm not convinced of...

I've got enough experience with non-overloaded nats in non-residential
circumstances that I know they can suck up more ip addresses than I have
available to me at the time.

---

From: Ray Hunter <[email-masked]>
Date: Thu, 05 Jan 2012 14:31:48 +0100

IMVH operational experience of processing firewall/ NAT/ netflow logs, 
any solution that relies on a certain fixed statistical distribution of 
mean port usage per user for it's long term success is likely to fail, 
simply because there is no standard behavior for how many ports an 
individual machine or user or household or application should or may 
open simultaneously. Applications change. Traffic patterns change. Port 
usage varies greatly per machine over time, and as Cameron says, can be 
bursty.

A scheme more like those used to avoid excessive disk fragmentation 
might be more appropriate for bursty allocations that extend far above 
the long-term mean. This could be based on "allocation groups" of blocks 
of outside ports in a range, which are then dynamically associated on 
the fly with individual inside port mappings. Only a single record 
containing the outside IP address and outside port range plus the the 
inside IP address/UID (not the individual inside ports) would have to be 
logged in order to be able trace back from an outside session to a 
unique identifiable source covering many inside sessions. Use of 
allocation groups could also allow a certain amount of parallelism in 
the allocation process of establishing NAT mappings. And it might avoid 
intellectual property issues, due to prior art.

I can't see any definition in the draft of appropriate behavior when the 
Maximum ports per user (M) parameter is exceeded. Should the CGN drop 
user traffic if it cannot be logged? Where's the standard that says an 
individual IPv4 address cannot source more than 5040 ports 
simultaneously (over the period of the CGN NAT association timeout)? I 
know many CGN manufacturers  allow you to create different user 
profiles, but good luck in retrospectively enforcing such restrictions 
in a fair use agreement.

There should also be a note in Additional Logging Considerations that 
the CGN operator should check whether this form of CGN logging complies 
with their local telecommunication data retention law, as the exact 
requirements for the individual entries in a data traffic log may vary 
per jurisdiction. Some would seem to require logging the detail of every 
single session, incuding source and destination address, source and 
destination port, and protocol.

regards,
RayH

Cameron Byrne wrote:
> On Wed, Jan 4, 2012 at 2:27 PM, Dave Thaler<[email-masked]>  wrote:
>

---

From: "George, Wes" <[email-masked]>
Date: Thu, 5 Jan 2012 09:16:01 -0500

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Cameron Byrne
> Sent: Wednesday, January 04, 2012 5:56 PM
> Subject: Re: [v6ops] Please review draft-donley-behave-deterministic-
> cgn
>
[WEG] yes, others do need it. We don't have the experience of operating it yet, but our networks are somewhat different. I'm glad you don't. I wish we didn't.

> Depends on the user, it's very bursty.  The trend is more and more
[WEG] Given that your application is typically one user doing periodic mobile web access (NAT44), one could extrapolate that the average residential environment where each customer has multiple such hosts behind a common device (NAT444) and longer periods of user activity due to the device format and lack of battery life and data cap concerns, the numbers on simultaneous port use will be higher, perhaps linearly based on the number of simultaneous users behind the CPE device.

> Regarding logging, my CGN does not have this problem and does not need
[WEG] That's because IIRC, mobile has other places to log the relationship between user/device and IP, so addressing abuse/LEA requests can be handled somewhat differently than it is in a non-mobile network. In a former life, I remember having variants of this discussion, and it seemed that for the most part, LEAs haven't figured out that they might care about mobile data based on the IP address (and port) - the LEA compliance folks in mobility looked at me funny when I asked about it, because all they'd ever gotten was requests associated with a phone number.

We've done the math (based on some assumptions). I'm not at liberty to share specifics, but I can say that worst case if we log every session is *very* bad - hundreds of terabytes of logging per year. SAN vendors would love us. Anything that reduces the logging by orders of magnitude is a big win.
I do need to caveat my last statement. I make it assuming that CGN is a foregone conclusion, so it probably should be deployable, because I do not believe in the notion of punishing operators by making it difficult in an attempt to prevent people from deploying it. I'd prefer to never have to use this, but if I do, it's useful. I view a "win" in this category as being told, "I'm going to flog you" instead of "I'm going to flog you while forcing you to watch the Twilight movies"  -- it simply sucks less than the alternative.

Thanks
Wes George

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Cameron Byrne <[email-masked]>
Date: Thu, 5 Jan 2012 07:33:22 -0800

On Jan 5, 2012 6:16 AM, "George, Wes" <[email-masked]> wrote:
> [WEG] yes, others do need it. We don't have the experience of operating
it yet, but our networks are somewhat different. I'm glad you don't. I wish
we didn't.

Ack. I said *my* network for just that reason.

> [WEG] Given that your application is typically one user doing periodic
mobile web access (NAT44), one could extrapolate that the average
residential environment where each customer has multiple such hosts behind
a common device (NAT444) and longer periods of user activity due to the
device format and lack of battery life and data cap concerns, the numbers
on simultaneous port use will be higher, perhaps linearly based on the
number of simultaneous users behind the CPE device.

It is always interesting to hear outsiders views of my network. But don't
feel burdened to scope and caveat my feedback about how a draft applies to
my network or mobile networks.  I know it is strange and usual for a mobile
operator to participate in the IETF, but it is no longer strange and usual
for mobile devices to communicate on the internet.

That said, I disagree with your statement above and say the trend is that
most devices I ship have mobile hot spot functionality , which is nat444.
Thanks Android.

Also, the trend is that most phones are connected to many clouds
constantly: Gmail, updates, Facebook, twitter, email (@home, @work, @school
...) ...everything is "always on":and "push".  Oh. And that "group on" app
wants to push you a coupon because you walked into Starbucks.  And BBC has
breaking news alerts. ... and that all goes via long lived Nat cloud
attachment.

> [WEG] That's because IIRC, mobile has other places to log the
relationship between user/device and IP, so addressing abuse/LEA requests
can be handled somewhat differently than it is in a non-mobile network. In
a former life, I remember having variants of this discussion, and it seemed
that for the most part, LEAs haven't figured out that they might care about
mobile data based on the IP address (and port) - the LEA compliance folks
in mobility looked at me funny when I asked about it, because all they'd
ever gotten was requests associated with a phone number.
>
> We've done the math (based on some assumptions). I'm not at liberty to
share specifics, but I can say that worst case if we log every session is
*very* bad - hundreds of terabytes of logging per year. SAN vendors would
love us. Anything that reduces the logging by orders of magnitude is a big
win.
> I do need to caveat my last statement. I make it assuming that CGN is a
foregone conclusion, so it probably should be deployable, because I do not
believe in the notion of punishing operators by making it difficult in an
attempt to prevent people from deploying it. I'd prefer to never have to
use this, but if I do, it's useful. I view a "win" in this category as
being told, "I'm going to flog you" instead of "I'm going to flog you while
forcing you to watch the Twilight movies"  -- it simply sucks less than the
alternative.

That's fine for *you*.  I am just making it clear that this is not at all
appealing in my network. It is not deployable in networks that look my
mobile network, and we already do a lot of cgn.

Cb
> Thanks
> Wes George
>
> This E-mail and any of its attachments may contain Time Warner Cable
proprietary information, which is privileged, confidential, or subject to
copyright belonging to Time Warner Cable. This E-mail is intended solely
for the use of the individual or entity to which it is addressed. If you
are not the intended recipient of this E-mail, you are hereby notified that
any dissemination, distribution, copying, or action taken in relation to
the contents of and attachments to this E-mail is strictly prohibited and
may be unlawful. If you have received this E-mail in error, please notify
the sender immediately and permanently delete the original and any copy of
this E-mail and any printout.
 On Jan 5, 2012 6:16 AM, "George, Wes" <[email-masked]> wrote:

> [WEG] yes, others do need it. We don't have the experience of operating it
> yet, but our networks are somewhat different. I'm glad you don't. I wish we
> didn't.
>
> >
> [WEG] Given that your application is typically one user doing periodic
> mobile web access (NAT44), one could extrapolate that the average
> residential environment where each customer has multiple such hosts behind
> a common device (NAT444) and longer periods of user activity due to the
> device format and lack of battery life and data cap concerns, the numbers
> on simultaneous port use will be higher, perhaps linearly based on the
> number of simultaneous users behind the CPE device.
>
> >
> [WEG] That's because IIRC, mobile has other places to log the relationship
> between user/device and IP, so addressing abuse/LEA requests can be handled
> somewhat differently than it is in a non-mobile network. In a former life,
> I remember having variants of this discussion, and it seemed that for the
> most part, LEAs haven't figured out that they might care about mobile data
> based on the IP address (and port) - the LEA compliance folks in mobility
> looked at me funny when I asked about it, because all they'd ever gotten
> was requests associated with a phone number.
>
> We've done the math (based on some assumptions). I'm not at liberty to
> share specifics, but I can say that worst case if we log every session is
> *very* bad - hundreds of terabytes of logging per year. SAN vendors would
> love us. Anything that reduces the logging by orders of magnitude is a big
> win.
> I do need to caveat my last statement. I make it assuming that CGN is a
> foregone conclusion, so it probably should be deployable, because I do not
> believe in the notion of punishing operators by making it difficult in an
> attempt to prevent people from deploying it. I'd prefer to never have to
> use this, but if I do, it's useful. I view a "win" in this category as
> being told, "I'm going to flog you" instead of "I'm going to flog you while
> forcing you to watch the Twilight movies"  -- it simply sucks less than the
> alternative.
>
> Thanks
> Wes George
>
> This E-mail and any of its attachments may contain Time Warner Cable
> proprietary information, which is privileged, confidential, or subject to
> copyright belonging to Time Warner Cable. This E-mail is intended solely
> for the use of the individual or entity to which it is addressed. If you
> are not the intended recipient of this E-mail, you are hereby notified that
> any dissemination, distribution, copying, or action taken in relation to
> the contents of and attachments to this E-mail is strictly prohibited and
> may be unlawful. If you have received this E-mail in error, please notify
> the sender immediately and permanently delete the original and any copy of
> this E-mail and any printout.
>

---

From: Cameron Byrne <[email-masked]>
Date: Thu, 5 Jan 2012 08:33:55 -0800

On Thu, Jan 5, 2012 at 7:33 AM, Cameron Byrne <[email-masked]> wrote:
>> residential environment where each customer has multiple such hosts behind a
>> common device (NAT444) and longer periods of user activity due to the device
>> format and lack of battery life and data cap concerns, the numbers on
>> simultaneous port use will be higher, perhaps linearly based on the number
>> of simultaneous users behind the CPE device.
>>
>
> Also, the trend is that most phones are connected to many clouds constantly:
> Gmail, updates, Facebook, twitter, email (@home, @work, @school ...)
> ...everything is "always on":and "push".  Oh. And that "group on" app wants
> to push you a coupon because you walked into Starbucks.  And BBC has

Alas, sometimes my words are not enough, and other times i just cant
stop talking so you are going to get some more of it.

Here is *one* data point on why more and more apps are "push" / always-on

http://urbanairship.com/blog/2011/11/04/customer-success-story-swirl-uses-push-to-boost-mobile-orders-by-20/

And, they even describe how they handle the scale on their side in the
cloud, they can get about 500K "sessions" per EC2 instance.  Their
cost per session is cheaper than mine on the middlebox...

http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-for-c500k/

Great.

So, what does that mean for a mobile operator? Not "periodic mobile
web access" as you say.  It means there are a lot of long lived and
sustained CGN sessions.  Or, an alternative solution, dare i say the
forbidden word on an IPv4 life support thread? ... ipv6.

Let me also clear up something, most of mobile operator traffic is
volume is video.

http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white_paper_c11-520862.html

"Mobile video traffic will exceed 50 percent for the first time in
2011. Mobile video traffic was 49.8 percent of total mobile data
traffic at the end of 2010, and will account for 52.8 percent of
traffic by the end of 2011."

Once again, "not periodic mobile web"

cb

>> somewhat differently than it is in a non-mobile network. In a former life, I
>> remember having variants of this discussion, and it seemed that for the most
>> part, LEAs haven't figured out that they might care about mobile data based
>> on the IP address (and port) - the LEA compliance folks in mobility looked
>> at me funny when I asked about it, because all they'd ever gotten was
>> requests associated with a phone number.
>>
>> attempt to prevent people from deploying it. I'd prefer to never have to use
>> this, but if I do, it's useful. I view a "win" in this category as being
>> told, "I'm going to flog you" instead of "I'm going to flog you while
>> copyright belonging to Time Warner Cable. This E-mail is intended solely for
>> the use of the individual or entity to which it is addressed. If you are not
>> the intended recipient of this E-mail, you are hereby notified that any
>> dissemination, distribution, copying, or action taken in relation to the
>> contents of and attachments to this E-mail is strictly prohibited and may be
>> unlawful. If you have received this E-mail in error, please notify the
>> sender immediately and permanently delete the original and any copy of this
>> E-mail and any printout.
>
>> residential environment where each customer has multiple such hosts behind a
>> common device (NAT444) and longer periods of user activity due to the device
>> format and lack of battery life and data cap concerns, the numbers on
>> simultaneous port use will be higher, perhaps linearly based on the number
>> of simultaneous users behind the CPE device.
>>
>> >
>> somewhat differently than it is in a non-mobile network. In a former life, I
>> remember having variants of this discussion, and it seemed that for the most
>> part, LEAs haven't figured out that they might care about mobile data based
>> on the IP address (and port) - the LEA compliance folks in mobility looked
>> at me funny when I asked about it, because all they'd ever gotten was
>> requests associated with a phone number.
>>
>> attempt to prevent people from deploying it. I'd prefer to never have to use
>> this, but if I do, it's useful. I view a "win" in this category as being
>> told, "I'm going to flog you" instead of "I'm going to flog you while
>> copyright belonging to Time Warner Cable. This E-mail is intended solely for
>> the use of the individual or entity to which it is addressed. If you are not
>> the intended recipient of this E-mail, you are hereby notified that any
>> dissemination, distribution, copying, or action taken in relation to the
>> contents of and attachments to this E-mail is strictly prohibited and may be
>> unlawful. If you have received this E-mail in error, please notify the
>> sender immediately and permanently delete the original and any copy of this
>> E-mail and any printout.

---

From: "George, Wes" <[email-masked]>
Date: Thu, 5 Jan 2012 11:51:05 -0500

From: Cameron Byrne [mailto:[email-masked]]
Sent: Thursday, January 05, 2012 10:33 AM
To: George, Wes
Cc: Behave Chairs; Dave Thaler; [email-masked]; v6ops v6ops WG
Subject: RE: [v6ops] Please review draft-donley-behave-deterministic-cgn

It is always interesting to hear outsiders views of my network. But don't feel burdened to scope and caveat my feedback about how a draft applies to my network or mobile networks.  I know it is strange and usual for a mobile operator to participate in the IETF, but it is no longer strange and usual for mobile devices to communicate on the internet.

[WEG] It's not necessary to have a chip on your shoulder, Cameron. :-)  I wasn't caveatting or trying to minimize your feedback. I was trying to contribute to the overall discussion regarding a reasonable assumption about port numbers based on my own experience with a reasonably data-centric mobile network, and what I know about other broadband networks. Tl;dr version of my comment was "it could be even worse than this"

That said, I disagree with your statement above and say the trend is that most devices I ship have mobile hot spot functionality , which is nat444. Thanks Android.

[WEG] no argument here. However, simply because most have it doesn't mean that they all use it, nor does it mean that their usage profile becomes exactly like a residential broadband customer, especially on the extreme end of the usage (bandwidth or number of devices behind the NAT) spectrum. That's the distinction I was making. Mobile devices still have limitations which drive tradeoffs in user behavior - most notably, batteries and data caps. And yes, there are exceptions that prove me wrong. Don't read too much into it :-)

Also, the trend is that most phones are connected to many clouds constantly: Gmail, updates, Facebook, twitter, email (@home, @work, @school ...) ...everything is "always on":and "push".  Oh. And that "group on" app wants to push you a coupon because you walked into Starbucks.  And BBC has breaking news alerts. ... and that all goes via long lived Nat cloud attachment.

[WEG] Yeah, Android (and iOS, et al) is a chatty beast, and that lives on the mobile network... until it gets offloaded to a WiFi link, in which case it's one of multiple devices doing the same thing behind the first round CPE NAT. Again, I was simply trying to reinforce the point that your experiences show a baseline, but when you consider a home where all 3-5 residents have at least one (if not 2 or 3) devices doing exactly that, the problem is magnified, which is why the problem is maybe different for me. Was not trying to imply that I know what goes on in your network better than you do. Apologies if that's how it came across.

Thanks

Wes

________________________________
This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Chris Donley <[email-masked]>
Date: Tue, 10 Jan 2012 11:13:55 -0700

This comes down to compression density.  You're right that most
subscribers use few ports, but a few subs do use a very large number (see
the WAND data).  If you pack a lot of subscribers (let's say 1,000) behind
a single IP address, each gets only a handful of ports, and this proposal
almost turns into bulk port logging since many subscribers will be
requesting additional allocations (we're not suggesting per-connection
logging on the overflow).  However, if you pack fewer subs behind a single
public IP (let's say 60), each sub gets ~1000 ports, and 99% or so will
never request another block. Then, using bulk port logging for the
remainder makes for a very small quantity of log messages.  In the example
I presented in Taipei, we can get 100,000x-1,000,000x reduction in log
messages, depending on the size of the port blocks offered for subsequent
allocation.

You're correct that there is a tradeoff between compression density
(per-connection logging wins) and minimizing logging volume (Juniper's
SD-NAT wins). On that continuum, we believe our proposal provides a
balance that keeps log sizes small, but allows a reasonable compression
density while allowing some flexibility to accommodate the 1% of users who
consume significant numbers of ports.

Chris

On 1/4/12 4:18 PM, "Fred Baker" <[email-masked]> wrote:

>sessions for active subscribers only for each 30 minute period"; the HTML
>doesn't have tags that would allow me to simply get you there.
>
>several computers are in use, it's unlikely that everyone in the home is
>web surfing during the same minute. Common browsers usually limit the
>number of TCPs they open in parallel to something like 2 or 4, and those
>sessions usually stay open single digit numbers of seconds; this says
>that in a typical residential network, they see 2-4 computers
>simultaneously doing that, for a peak of 4-8 TCP sessions simultaneously
>active. That would be very surprising in a corporate network, but makes
>sense for residential network.
>
>>> Operators subject to law enforcement subpoenas and using Carrier Grade
>>>NAT are having difficulties with the syslog rate for per-connection
>>>logging. Chris Donley has proposed a simplification; the provider
>>>allocates a deterministic set of source ports to his subscribers, and
>>>only needs to log exceptions. Research in the area suggests that usage
>>>of source ports is pareto distributed; a typical user has a requirement
>>>on the order of 4 port numbers in simultaneous use (median), but port
>>>scans and other large volume uses drive the average quite a bit higher.
>>> 
>>>NATs",

---

From: Chris Grundemann <[email-masked]>
Date: Wed, 11 Jan 2012 10:02:27 -0700

>forbidden word on an IPv4 life support thread? ... ipv6.

Yes, as mobile Internet usage becomes more and more like fixed Internet
usage, mobile operators are going to have to look more and more to
solutions that look more like those of the fixed operators (lower IPv4 CGN
compression ratios and native IPv6 to name a couple).

>Once again, "not periodic mobile web"

Yes, mobile data usage is growing, but the delta is still massive between
what people do online at home and on the go. Specific to video, Nielson
reports that the average user watches 27 minutes a month of video online
at home and only 7 minutes a month on mobile.

http://techcrunch.com/2012/01/08/how-people-watch-tv-online/

So mobile's still about 4x shy (on average) in time and don't forget that
many apps are "optimized" for mobile viewing, giving the user less ports,
lower bit-rates, etcŠ

Chris,
~Chris

---

From: Chris Grundemann <[email-masked]>
Date: Wed, 11 Jan 2012 10:45:41 -0700

>thereafter, a bad strategy, as it will basically log as much as current.

We (the authors) actually agree that very small port allocations are a bad
idea with this methodology. Deterministic CGN should be used with low
compression ratios for the greatest gains over dynamic block allocations.

Remember that with just a 2:1 compression ratio (2 subscribers to each 1
public IPv4 address) you double the number of customers possible with your
existing IP allocation(s). Plus, each of those customers has access to
over 30,000 ports. However, you probably don't want to subject ALL of your
customers to CGN, so higher compression ratios are likely needed. If you
go all the way up to 64:1, you can put roughly 16,000 subscribers behind a
single /24 and still dedicate about 1,000 ports to each of them (with zero
need for logging). Then, you add in the ability to overflow into a shared,
block-allocation pool for extreme bursts (because, again, we agree that
some users are port-hogs and are often bursty in their usage). We feel
that this approach truly is the best strategy, as long as you are able to
keep the compression ratio's low (low enough that the static/deterministic
allocation covers the majority of your customer's peak port utilization)
in your network.

Cheers,
~Chris

---

From: Cameron Byrne <[email-masked]>
Date: Wed, 11 Jan 2012 12:06:20 -0800

> compression ratios and native IPv6 to name a couple).

I don't think suggesting incrementally lower IPv4 efficiency is a
helpful solution to the IPv4 exhaust problem.

---

From: Chris Grundemann <[email-masked]>
Date: Wed, 11 Jan 2012 13:21:20 -0700

>helpful solution to the IPv4 exhaust problem.

I wasn't suggesting lower IPv4 efficiency as a solution to the IPv4
exhaust problem. I was stating the fact that if mobile operators want
their data services to look/feel/behave more like fixed ISP services
(DSL/cable/etc), then they will have to operate their networks
accordingly. I do agree that this creates a problem for mobile operators
who choose not to deploy IPv6. Since we're veering off the topic of this
thread, I'll leave it at that (I'm happy to continue the discussion
off-line if you'd like).

Cheers,
~Chris

>>>http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/
>>>ns
>>>827/white_paper_c11-520862.html
>>>