Subject: [v6ops] draft minutes ietf 81, 3 meetings...

From: Joel Jaeggli <[email-masked]>
Date: Tue, 2 Aug 2011 10:24:11 -0700

There were at least 2 note-takers plus 2 people scribing on jabber at various times. I'm am deeply appreciative of the contribution to making these notes useful.

IETF 81 
V6OPS WG Minutes

-------------------------------

--Tuesday July 26 0900-1130 EDT
Approximately 200 participants

Fred Baker - Introduction
     The meeting is divided into two sections 
     	 1. World ipv6 day
	 2. Old Business

* Presentation - World IPV6 Day Measurement results - Keranen
	     http://tools.ietf.org/agenda/81/slides/v6ops-0.pdf
	     http://tools.ietf.org/html/draft-keranen-ipv6day-measurements

* Presentations - Microsoft Measurement report - Chris Palmer
	      http://tools.ietf.org/agenda/81/slides/v6ops-1.pptx

the sad graph, basically half a percent of th euser base came in via
v6, 7% of those folks were windows xp which is surprising becuase you
have to turn it on. as far as method, 91% native 8% 6to4 and 1%
teredo.

CDN's delivered, really at the last minute but they were there.

It's hard to say given the usage that we've validated this at scale.

Ralph D - Clarification Question ipv6 geolocation is it
unavailable or is it incorrect.

Chris - We use third parties and they're lacking the data.

? - What would it take from miscrosofts perspective to roll a patch to
enable windows XP

Chris P - That's something we've been generically thinking about but
there's only a 1000 days of Windows XP left.

Lorenzo C - one of the things you achieved with this is the removal of
fear. How do you lock in those gains? if you do nothing the fear will
come back.

John B - We see duplicate DUID when windows devices do DHCPv6, which
creates and operational complication.

Chris P - Stacks changes are coming in Sept.

Eric Kline - Will IE get ahppy eyeballs or fast fallback so it can
join the other browsers? 

Chris P - So things like happy eyeballs are useful but fuzy towards
the no. We're inclined to make changes to 3484.

Dan W - Hey your v6 is broken is is not generic to the whole internet,
it may work in your office or home, so a solution has to have scope.

* Presentations - Checkpoint's World IPv6 Day Experience - Bob hinden
	      http://tools.ietf.org/agenda/81/slides/v6ops-3.pdf

Mark T - In our experience there are a lot of things we could just
leave up afterwards, yet there is a lot of legacy stuff that has to be
validated.

Bob H - By doing this deployment exercise we're now having
conversations about internal deployment now that the IT team feels a
lot more comfortable and turns out to have not been that hard.

* Presentation - World IPv6 Day, What did we learn (Ripe measurement report) - Bert Wijnen 
	     http://tools.ietf.org/agenda/81/slides/v6ops-4.pdf

Bert W - Ripe had about 40 measurement points from which to observe
world ipv6 day. 

Looking at DNS queries, 2 days before, negative cache TTL dropped for
several participants.

The internet is is a collection of internets connected to each
other. Some of our vantage points revealed partitioning which was
confirmed with further discussion.

when monitoring performance ipv4 vs ipv6 for source destination pairs,
v4 was a little better, but very close however.

Alain D - Did you monitor the propigation of /48 prefixes , there are
rumors of edge providers still filtering.

Joel J - We advertise /48  prefixes and these days, we see them
basically everywhere we can see with routeviews.

Hue Deng - use of nat 64 for services can you detect that in your
data?

Joel J - Question everyone uses a load-balancer at scale, what are you
trying to find?

Alain D - With nat64 in front of the a server we have lost the identity
of the user.  

* Presentation - IPv6 day operators report (hurricane electric) - Martin Levy
	     http://tools.ietf.org/agenda/81/slides/v6ops-19.pdf

Only port 80/443 traffic volume changed on IPv6 day. We credit much of
this to youtube.

Teredo tunnels require that the tunnel-end point return an icmp6 ping.

About 11% of ASes are IPv6 enabled, if you count adjacencies, by the
time that you get to 7 adjacencies about 50% of those players are ipv6
enabled.

Geoff H - Your commnetary about Teredo isn't gelling for me, at the
client side the ICMP is in UDP, the modified stun protocol in teredo
guesses wrong about 37% of the time with the NAT binding. 

* Presentation - Comcast expereince with ipv6 deployment - John Browsowki
	     http://tools.ietf.org/agenda/81/slides/v6ops-20.pdf

Traffic increased notably during world ipv6 day.

An additional trial is being deployed (DS-Lite)

Native dual stack validated, preparing for production deployment.

We took Jason Fesler's IPv6 testing code and did some 30,000 tests
with end users, found some users that were potentially impacted.

Hue D - How is your backoffice deployment supporting IPv6? 

John B - We've been working on it for about 6 years. The edge of the
broadband network is where the effort is.

Lee H - I thank you John because we started three years ago which
means things have gone much faster. 

Chris P - Do you have visibility into  household customers with
gateways that don't support V6? 

John B - That vast majority, when we enable it in the broadband
network we can tall people hey guys you need to upgrade. 

Chris P - my concern is that maybe if we're lucky we can reach 1 in 4
households.

Lorenzo C - As a customer V6 is great and it works as well as ipv4. If
you went all out how long would it take you to bring V6 to 1% of your
customers? 

John B - 1% is doable pretty easy thequestion is more like the 90% number.

Fred Baker - Status report World IPv6 day call to arms
     http://tools.ietf.org/html/draft-chown-v6ops-call-to-arms

It doesn't have a lot of value as an RFC so Tim will let it lapse but
the info will go into some other drafts.
     
* Presentation - Happyeyeballs implementation report - Dan Wing
	     http://tools.ietf.org/agenda/81/slides/v6ops-5.pptx
	     http://tools.ietf.org/html/draft-ietf-v6ops-happy-eyeballs

After discussion on the list, instead of describing the algorithm we
decided to describe requirements. There are two versions stateless,
e.g. chrome or firefox vs stateful Algorithms.

Chris P - I think the direction is fantastic, requirements are good.

* Presentation - IPv6 DNS Whitelisting - Jason Livingood
	     http://tools.ietf.org/agenda/81/slides/v6ops-6.pptx
	     http://tools.ietf.org/html/draft-ietf-v6ops-v6-aaaa-whitelisting-implications

Lots of feedback from the IESG, generating version 3-6.

We talked to implmentors, and it needs another rev, to make it
acceptable to a swap of the community. 

Wes G -  When this approach was first tried manual was the only
approach. 

Lorenzo C - When we started this it was because we had no
alternative. We removed ourselves from the discussion because we
thought it was a critique of us. It's possible that we can produce a
document that we can accept.

Focus on intent and consequences in this draft. 

Fred B - Is there something (a taxonomy of GTM/GSLB) that we
should take to DNSOP?

Fred Baker - Response to Appeal on 6to4 to historic
     http://tools.ietf.org/agenda/81/slides/v6ops-24.pptx
     http://tools.ietf.org/html/draft-ietf-v6ops-6to4-to-historic

Keith M - I'm convinced that you did due dilligence with respect to wg
consensus. 

Fred B - It is worth noting that the draft did have some impact (with
vendors).

New business

* Presentation - RFC for IPv6 IP identification field (header) - Mike Ackerman
	     http://tools.ietf.org/agenda/81/slides/v6ops-7.ppt
	     http://tools.ietf.org/html/draft-elkins-6man-ipv6-diagnostic-header

Fred B - Question for the group, do you use the IPID in ipv4? (no
respondants other than presenters)

Weg G - We use it all the time? I've never heard of it...

Francis D - it's potentially a covert channel, also why not use the
fregmentation header?

Bob H - We diefintently want to hear if this is useful. I don't know
who puts this header in the packet?

Nalini ? - The host clearly

Andrei Y - typical use case for this header would be to see if a
middle box is modifying the packet. 

Fred B - What I'm out of this discussion is that the application isn't
a bad one, it could be simulated in a fregment.

Fred Baker - Meeting complete, go to Lunch

-------------------------------
 
-- Thursday July 28 1520-1720 EDT
Approximately 200 participants

* Presentation - Stateless 4Via6 - Wojec Dec
	     http://tools.ietf.org/agenda/81/slides/v6ops-8.ppt
	     http://tools.ietf.org/html/draft-dec-stateless-4v6

4v6 index provides enough information to compute an ipv4 address and
port index.

Example scenario - A CPE, inside it has a port contrained nat44
implmentation along with the 4v6 function. inside the cpe we nat the
source ip to a public v4 address and port rnage provided by the v6
address encapsulated and sent to the 4v6 gateway where it is sent on
the internet.

Alternative modes of operation, mapped, vs translation. Translation
has signficantly less impact in the 3gpp case.

Oel T - Wanted to ste for the minutes that there isn't any practical
difference between encapsulation ad double translation.

Wojek D - We didn't get to have the discussion in softwires we belive
that V6OPS is the best place to capture the operational issues.

Hue D - would like less impact

? Shima - Going to be a work item for softwires.

Mark T - The vast majority of both operations are the same, the
question is do we compress the headers out or not. I support the
minimal effort which is encapsulation.

Alain D - This will be done at the interim metting in sept.

Jari A - we should put them in the same place (softwire) rather than
split them all over the place.

Sing Y (cernet) - Sometimes we deploy single translation sometimes
double, we haven't found many corner case where one works and the
other doesn't.

Fred B - Would suggest that people here read the draft, and comment to
softwire where the work will be done.

Dave T - the difference between tunneling and translation is very
small and it would be an odd division to put those in two places.

* Presentation - advanced requirements for cpe routers - Ole Troan
	     http://tools.ietf.org/agenda/81/slides/v6ops-9.pdf
	     http://tools.ietf.org/html/draft-ietf-v6ops-ipv6-cpe-router-bis

Request from homenet to move the lan side of this document to the
homenet wg. thus the document will be split in wan requirements vs
lan side which will go there.

Options
	give up on the document
	move to softwires (the ds-lite part) 
	resubmitt 6204 with ds-lite and 6rd requirements
	continue.

Barbara S - I like 6204 option or option 4

Ole T - include requirement for PCP?

Alain D - PCP has been through two WG last calls

Mark T - 6rd and DS-lite and PCP and making them come up and behave
well together, that work is already here.

Barbara S - Inclusion of PCP will start the question of do we also
include UPnP

Lorenzo C - what 6204 and homenet are doing is a relatively
partitionable problem.

Fred B - Question for Barbara, does this start here or in the BBF
(broadband forum)

Barbara S - Start them whereever someone is referencing PCP

Hemant S - Lan side requirements are where we have real urgency from
operators.

Fred B - unless homenet feels badly them my expression would be to go
for the thing with the nearest completion.

Erik K - we need to have the cpe vendors be able to build this early
next year.

Jari A - we should publish what we have, if there's falky stuff we
should pull it.

Lorenzo C - In a closer reading of some of the lan side stuff dhcp
will definitly hurt homenet work.

* Presentation - IPv6 RA-Guard  Evasion - Aurturo Servin
	     http://tools.ietf.org/html/draft-gont-v6ops-ra-guard-evasion
	     http://tools.ietf.org/agenda/81/slides/v6ops-11.pdf

Fred B - so you made this presentation in 6man, what was the upshot?

Aurturo - Was asking for acceptance becaue it's updates 6105, in v6ops
case it's documentation of the issue and the operational solutions.

Fernando G - Ra-guard is/was an infomational document produced in
v6ops not 6man.

Ron B - We would be stepping on the charter of 6Man

Joel J - (outside of the problem statement)

Fred B - We'll talk with Bob and Brian but it seems like the work can
be done in 6man.

* Presentation - IPv6 Practices on China Mobile  IP Network - Chen Gun 
	     http://tools.ietf.org/html/draft-chen-v6ops-ipv6-bearer-network-trials
	     http://tools.ietf.org/agenda/81/slides/v6ops-16.ppt

Chris P - empathize strongly, I think it's great but I have a
question are you drving load from your user's against this?

Chen G - we have a plan for that, it's a lab exercise.

Hue D - we have a national project that will deploy later this year.

* Presentation - NAT64 CPE Operation Usages and Requirements - Chen Gun 
	     http://tools.ietf.org/agenda/81/slides/v6ops-25.ppt
	     http://tools.ietf.org/html/draft-chen-v6ops-nat64-cpe

Fred B - If the other draft is covering the same material at least
they could be merged.

* Presentation - A Discard prefix for IPv6 - Dave Freedman 
	     http://tools.ietf.org/html/draft-hilliard-v6ops-ipv6-discard-prefix
	     http://tools.ietf.org/agenda/81/slides/v6ops-12.pdf

Dave Freeman - Where should the prefix be from?
     documentation range?
     cute hex?
     more than one address?
     Global Unicast?

Hard-Wired discard prefix is a standards track item.

Nick H - There are concievable situations where you might want to leak
this prefix e.g. to downstreams for DOS analysis etc (Warren K,
Lorenzo C dispute this)

Rob ? - We don't have a BCP for implementation

Joel J - as one of the coauthor's on the ipv4 (RTBH BCP) the ipv4 one
would be virtually identical.

Fred - hearing a fair amount of support (hum) in favor no opposed 
short prefix (e.g. /64) vs /128 
      wg prefernce for /64
This is an arguement for  BCP,
Lets take this to a working group last call and move it ahead.

* Presentation - DHCPv6 Prefix delegation as IPv6 Migration Tool in
	     mobile netoworks - behcet Sarikaya
	     http://tools.ietf.org/agenda/81/slides/v6ops-13.pptx
	     http://tools.ietf.org/html/draft-sarikaya-v6ops-prefix-delegation

Fred B - WG do you want to see this as a WG item?
(show of hands shows no strong support each way)

Nor support (for or against) individal submission

Fred Baker - meeting over

-------------------------------

Friday July 29 1300-1500 EDT 
Approximately 80 participants

* Presentation - IPv6 Address accountability Considerations - Tim
  	       Chown 
  	       http://tools.ietf.org/agenda/81/slides/v6ops-29.pdf
  	       http://tools.ietf.org/html/draft-chown-v6ops-address-accountability

Tim C - At the moment people in ipv4 have a reasonable task of
providing address accountability in v6, things like privacy addresses
make that harder.

Dave T - What do you mean by accountability

Tim C - either a hardward port or 802.1x giving you a binding between
port and ip address correlation between ip4 arp and ipv6 nd tables and
polling rapidly enough that you know your data is relevant,
alternatively record all nd traffic on the link.

Fred B - Zigbee has Pana 

Unkown (Back mic) - Use Savi

Tim C - Currently documents say logging with Savi should only log
potential spoofing events, Logging all mapping data would be required.

Fred B - as a Savi author switches in general have places where they
log all sorts of things.

James W - Shouldn't this rightfully be taken up in NOG organizations,
One person's accountability is another Persons Pen Register.

Wes G - The privacy concerns are something that should be included in
the document.

* Presentation - Operational Neighbor Discovery problem and
  	       enchancements - Joel Jaeggli
	       http://tools.ietf.org/html/draft-gashinsky-v6nd-enhance
	       http://tools.ietf.org/agenda/81/slides/v6ops-21.pdf

Joel - 6Man owns the solutions, V6ops feedback on whether there's a
problem or not is helpful.

Ole T - What's the relationship of this work to 6 lowpan nd 
Joel J - Don't know haven't explored that.

Lorenzo C - Propose splitting the draft into operational guidance vs
protocol changes.

Fred B - Are address scans a problem? 

If this draft is split would the working group take the operational
and implementation details (consensus, no opposition)

* Presentation - Implementing AplusP in the provider's IPv6 only
  	       network - X Deng
  	       http://tools.ietf.org/html/draft-deng-v6ops-aplusp-experiment-results
	       http://tools.ietf.org/agenda/81/slides/v6ops-14.ppt

X Deng - implemented both port-range and scatter port a+p 

Fred B - So this is the Demo you showed in the terminal room?

X Deng - Yeah
What Breaks? UPNP clients. 

Marshall E / Dan W - Is there a security implication to the mask of
ports? If I'm running a Kaminsky Style attack I can guess your next port.

Stuart C - Lot of focus on making UPNP 1.0 work, it never will,
you have to throw it out and start again hence IGD 2.0

* Presentation - Rapid Transition of IPv4 contents to be IPv6
  	       accessible - Q Sunq
	       http://tools.ietf.org/html/draft-sunq-v6ops-contents-transition
	       http://tools.ietf.org/agenda/81/slides/v6ops-27.pptx

Wes G - In the case where you turn on content providers, they
need to do that on their own edge. Has this show us a practical
application worth replicating or is this a proprietary one-off worth
documenting. 

Dan W - if the consensus in the room is that nat64 is inappropiate
in front of an ip4-only server because of a loss of the IPv6 source
address, then v6ops would benifit from documenting that as a not
recomended deplyoment.

Lorenzo C / Jason W - There's a disparity where content is ahead
of access. Access provders are currently working on that.

Fred Baker - Meeting adjourned

---

From: Carlos Pignataro <[email-masked]>
Date: Sun, 14 Aug 2011 13:03:16 -0400

Hi,

I believe there was one further point raised in regards to
draft-elkins-6man-ipv6-diagnostic-header (though I don't recall if it
was at the mike by Andrew Y., in a corridor, or only in my head): the
Heisenbug principle -- if this is used in debugging (and diagnostics),
the issue being studied or attempted to reproduce alters its behavior or
goes away when modifying the packet (like compiling with different -o).
I think this is one of the most important arguments against this
proposal. The use of fragment header is interesting, a new header could
show much more pronounced deviation of observed behaviour than without it.

For completeness, I did ask some services escalation engineers about how
they use the ip-id in IPv4 for troubleshooting, and basically got two
main uses for very corner case situations in: i. identify middleboxes
and their characteristics (by check IP ID increment patterns, find if
packets generated by the same box), and ii. track set of packets along
different points in the network (e.g., back-to-back packets and see
rate, drops, out of order). But it's built into the v4 header.

Thanks,

-- Carlos.

---

From: [email-masked]
Date: Sun, 14 Aug 2011 13:31:16 -0700 (PDT)

I am working on contacting a number of our large corporate customers to weigh in on this topic.  I think that the biggest issue here is that when a large commercial network is having problems, anything that can help speed diagnostics is worth doing. Outages and poor performance can cost a great deal of money.  

Situations such as comparing packets at various points on the network to determine packet loss is a necessary part of network problem determination.  I am working to get the technicians from the corporations themselves to get onto this email list and tell their opinions themselves.

I wish that the IPID was in the main IPv6 header.  That would be the best place for it.  That is not possible, so what we are looking at is doing our best under the circumstances.  One problem with having a Fragmentation Header generated is that it skews statistics on fragmentation.  Many corporate networks keep track of how much fragmentation is on their networks and if we generate a spurious indication, then this invalidates the real fragmentation.

Nalini

--- On Sun, 8/14/11, Carlos Pignataro <[email-masked]> wrote:

> From: Carlos Pignataro <[email-masked]>
> Subject: [v6ops] draft-elkins-6man-ipv6-diagnostic-header (Was: draft minutes ietf 81, 3 meetings...)
> To: [email-masked]
> Date: Sunday, August 14, 2011, 10:03 AM
> Hi,
> 
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Keith Moore <[email-masked]>
Date: Sun, 14 Aug 2011 18:04:34 -0400

> goes away when modifying the packet (like compiling with different -o).

Yes, this can happen.  But that doesn't mean the mechanism is useless.   It just means that the potential error introduced by the mechanism needs to be taken into account when making measurements based on it.

(Just as Heisenberg's principle doesn't prevent us from measuring either present position or future momentum of a particle.  It just says that there's a limit to the precision with which both can be measured simultaneously.)

I basically think that it's a potentially good idea.  The real trick is understanding how to implement it in hosts so that it can be turned on when appropriate without affecting (much) the applications generating the affected traffic.

Keith

---

From: Joel Jaeggli <[email-masked]>
Date: Sun, 14 Aug 2011 23:21:57 -0700

On Aug 14, 2011, at 1:31 PM, [email-masked] wrote:

> I wish that the IPID was in the main IPv6 header.  That would be the best place for it.  That is not possible, so what we are looking at is doing our best under the circumstances.  One problem with having a Fragmentation Header generated is that it skews statistics on fragmentation.

I presume that if one is parsing that far into the optional headers, that you can distinguish between a fragment header that has an offset of zero and a more fragments flag of 0 which would indicate the existence of a fragmentation header but that no other fragments exist and the other possible combinations.

>  Many corporate networks keep track of how much fragmentation is on their networks and if we generate a spurious indication, then this invalidates the real fragmentation.

should be straight forward to distinguish between the two use-cases.

---

From: Carlos Pignataro <[email-masked]>
Date: Mon, 15 Aug 2011 05:45:20 -0400

On 8/14/2011 6:04 PM, Keith Moore wrote:
>> draft-elkins-6man-ipv6-diagnostic-header (though I don't recall if
>> it was at the mike by Andrew Y., in a corridor, or only in my
>> head): the Heisenbug principle -- if this is used in debugging (and
>> diagnostics), the issue being studied or attempted to reproduce
>> alters its behavior or goes away when modifying the packet (like
>> compiling with different -o).
> 
> Yes, this can happen.  But that doesn't mean the mechanism is
> useless.   It just means that the potential error introduced by the
> mechanism needs to be taken into account when making measurements
> based on it.

I agree; I did not imply that the mechanism was "useless" -- just that
it has (much?) more limited applicability than the v4 counterpart,
because additionally, it is external instrumentation (if I have a packet
trace capture of a problem happening, that does not help, I need to
reproduce it again modifying the packet). However see below.

> 
> (Just as Heisenberg's principle doesn't prevent us from measuring
> either present position or future momentum of a particle.  It just
> says that there's a limit to the precision with which both can be
> measured simultaneously.)
> 

Similarly, since in the first sentence of the Abstract this is compared
with:

	The IPv4 main header contained a 16-bit IPID for fragmentation
	and reassembly.  This field was commonly used by network
	diagnosticians for tracking packets on multi-tier networks. In

just pointing out that this is with less precision.

Not to say it's useless but completing the minutes.

> I basically think that it's a potentially good idea.  The real trick
> is understanding how to implement it in hosts so that it can be
> turned on when appropriate without affecting (much) the applications
> generating the affected traffic.

Yes, that would be a requirement for host implementation. There is,
however, an assumption being made which is that the mechanism to debug
and troubleshoot needs to be carried in the packet -- that is ideal, but
if that does not exist I'd actually compare this proposal (or just using
fragment headers instead of a new header) to an offline trick
(implemented in a network protocol analyzer instead of in a host).

Thanks,

-- Carlos.

---

From: Carlos Pignataro <[email-masked]>
Date: Mon, 15 Aug 2011 05:54:32 -0400

On 8/15/2011 2:21 AM, Joel Jaeggli wrote:
>> I am working on contacting a number of our large corporate
>> customers to weigh in on this topic.  I think that the biggest
>> issue here is that when a large commercial network is having
>> problems, anything that can help speed diagnostics is worth doing.
>> Outages and poor performance can cost a great deal of money.
>> 
>> Situations such as comparing packets at various points on the
>> network to determine packet loss is a necessary part of network
>> problem determination.

I agree with this. To compare packets, we could potentially have the
network protocol analyzer (e.g., Wireshark) compute a hash on invariant
fields of the packet (i.e., exclude Hop Limit, etc), and present that as
packet metadata that is not actually carried on the wire. This would
work for packet traces not being instrumented. Would that help?

>> I am working to get the technicians from
>> the corporations themselves to get onto this email list and tell
>> their opinions themselves.
>> 
>> I wish that the IPID was in the main IPv6 header.  That would be
>> the best place for it.  That is not possible, so what we are
>> looking at is doing our best under the circumstances.  One problem
>> with having a Fragmentation Header generated is that it skews
>> statistics on fragmentation.
> 
> I presume that if one is parsing that far into the optional headers,
> that you can distinguish between a fragment header that has an offset
> of zero and a more fragments flag of 0 which would indicate the
> existence of a fragmentation header but that no other fragments exist
> and the other possible combinations.

This would also be less disruptive as it's a known header instead of a
new header (potentially for existing nodes.)

Thanks,

-- Carlos.

> 
>> Many corporate networks keep track of how much fragmentation is on
>> their networks and if we generate a spurious indication, then this
>> invalidates the real fragmentation.
> 
>>> From: Carlos Pignataro <[email-masked]> Subject: [v6ops]
>>> draft-elkins-6man-ipv6-diagnostic-header (Was: draft minutes ietf
>>> 81, 3 meetings...) To: [email-masked] Date: Sunday, August 14,
>>> 2011, 10:03 AM Hi,
>>> 
>>> draft-elkins-6man-ipv6-diagnostic-header (though I don't recall
>>> if it was at the mike by Andrew Y., in a corridor, or only in my 
>>> head): the Heisenbug principle -- if this is used in debugging
>>> (and diagnostics), the issue being studied or attempted to
>>> reproduce alters its behavior or goes away when modifying the
>>> packet (like compiling with different -o). I think this is one of
>>> the most important arguments against this proposal. The use of
>>> fragment header is interesting, a new header could show much more
>>> pronounced deviation of observed behaviour than without it.
>>> 
>>> For completeness, I did ask some services escalation engineers
>>> about how they use the ip-id in IPv4 for troubleshooting, and 
>>> basically got two main uses for very corner case situations in:
>>> i. identify middleboxes and their characteristics (by check IP ID
>>> increment patterns, find if packets generated by the same box),
>>> and ii. track set of packets along different points in the
>>> network (e.g., back-to-back packets and see rate, drops, out of
>>> order). But it's built into the v4 header.
>>> 
>>> Thanks,
>>> 
>>> -- Carlos.
>>> 
Fred B - Question for the group, do you use the IPID
>>> in ipv4? (no
>>> _______________________________________________ v6ops mailing
>>> list [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
>>> 
>> _______________________________________________ v6ops mailing list 
>> [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
>> 
> 
> _______________________________________________ v6ops mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: [email-masked]
Date: Mon, 15 Aug 2011 05:50:30 -0700 (PDT)

--- On Sun, 8/14/11, Joel Jaeggli <[email-masked]> wrote:

> From: Joel Jaeggli <[email-masked]>
> Subject: Re: [v6ops] draft-elkins-6man-ipv6-diagnostic-header (Was: draft minutes ietf 81, 3 meetings...)
> To: [email-masked]
> Cc: [email-masked]
> Date: Sunday, August 14, 2011, 11:21 PM
> 
> > I am working on contacting a number of our large
> corporate customers to weigh in on this topic.  I think
> that the biggest issue here is that when a large commercial
> network is having problems, anything that can help speed
> diagnostics is worth doing. Outages and poor performance can
> cost a great deal of money.  
> > 
> > Situations such as comparing packets at various points
> on the network to determine packet loss is a necessary part
> of network problem determination.  I am working to get
> the technicians from the corporations themselves to get onto
> this email list and tell their opinions themselves.
> > 
> > I wish that the IPID was in the main IPv6
> header.  That would be the best place for it. 
> That is not possible, so what we are looking at is doing our
> best under the circumstances.  One problem with having
> a Fragmentation Header generated is that it skews statistics
> on fragmentation.
> 
> I presume that if one is parsing that far into the optional
> headers, that you can distinguish between a fragment header
> that has an offset of zero and a more fragments flag of 0
> which would indicate the existence of a fragmentation header
> but that no other fragments exist and the other possible
> combinations.
> 
> >  Many corporate networks keep track of how much
> fragmentation is on their networks and if we generate a
> spurious indication, then this invalidates the real
> fragmentation.
> 

Straight forward if you are using a packet analyzer.  But, many people use SNMP to keep track of counters.  With SNMP, the fragmentation counter would just be incremented with no knowledge of if it was real or not.

> >> Subject: [v6ops]
> draft-elkins-6man-ipv6-diagnostic-header (Was: draft minutes
> ietf 81, 3 meetings...)
> >> To: [email-masked]
> >> draft-elkins-6man-ipv6-diagnostic-header (though I
> don't
> >> recall if it
> >> was at the mike by Andrew Y., in a corridor, or
> only in my
> >> head): the
> >> Heisenbug principle -- if this is used in
> debugging (and
> >> diagnostics),
> >> the issue being studied or attempted to reproduce
> alters
> >> its behavior or
> >> goes away when modifying the packet (like
> compiling with
> >> different -o).
> >> I think this is one of the most important
> arguments against
> >> this
> >> proposal. The use of fragment header is
> interesting, a new
> >> header could
> >> show much more pronounced deviation of observed
> behaviour
> >> than without it.
> >> 
> >> For completeness, I did ask some services
> escalation
> >> engineers about how
> >> they use the ip-id in IPv4 for troubleshooting,
> and
> >> basically got two
> >> main uses for very corner case situations in: i.
> identify
> >> middleboxes
> >> and their characteristics (by check IP ID
> increment
> >> patterns, find if
> >> packets generated by the same box), and ii. track
> set of
> >> packets along
> >> different points in the network (e.g.,
> back-to-back packets
> >> and see
> >> rate, drops, out of order). But it's built into
> the v4
> >> header.
> >> 
> >> Thanks,
> >> 
> >> -- Carlos.
> >> 
> >>> * Presentation - RFC for IPv6 IP
> identification field
> >> (header) - Mike Ackerman
> >>> Weg G - We use it all the time? I've never
> heard of
> >> it...
> >>> 
> >>> Francis D - it's potentially a covert channel,
> also
> >> why not use the
> >>> fregmentation header?
> >>> 
> >>> Bob H - We diefintently want to hear if this
> is
> >> useful. I don't know
> >>> Andrei Y - typical use case for this header
> would be
> >> to see if a
> >>> Fred B - What I'm out of this discussion is
> that the
> >> application isn't

---

From: [email-masked]
Date: Mon, 15 Aug 2011 06:13:59 -0700 (PDT)

--- On Mon, 8/15/11, Carlos Pignataro <[email-masked]> wrote:

> Date: Monday, August 15, 2011, 2:54 AM
> 
> 
> corporate

Well, that is a really good idea that we had explored also.  Now, we ourselves make an intelligent packet analyzer and work with the WireShark folks.  If I look at this selfishly for having an extra selling point for my products, then I can say, 'Yes, we allow you to do this!  Buy my product!'  But, I was trying to be a good citizen and allow all to do this without our products.  Now, if our proposal is not adopted, then we will do the checksum or hash on the packets in our products as I think it is needed for diagnostics. 

This idea also helps technicians who have taken a trace at one point using multiple facilities - that is, WireShark and IBM mainframe CTrace.  Then, it is useful to have the IPID shown in sequence.

But, the idea of the hash or checksum leaves aside the case where you have only one trace at one point along the path.  Being able to view the sequence of packet transmission is useful even then.  

Also, I really feel that down the road here, the 32-bit IPID is going to run into scalability issues.  We are proposing a 64-bit IPID.   The IPID is also not standardized.  Some implementations use the same IPID for all connections, others use a different IPID for each connection. Some increment by 1, some by two.

Our other future thoughts, and agenda, if you will, is that as networks and traffic grow, more and more diagnostics and troubleshooting will have do be done by computer systems rather than humans.  So, we are looking at what metrics need to be saved for that.  I feel that the IPID is one such.  There are others such as TCP Window Scaling - but that belongs on another email list, so I did not want to bring that up.

> header.  That would be

Now, actually, I am setting up a meeting with the IBM TCP/IP architects, as they are the host implementation that we work most closely with, to ask them what exactly would their stack do if they got a fragmentation header with 0 in the fragmentation header.  I wonder if that would cause any problems.

And, yes, the fragmentation header is known.  But, you know, we feel that diagnostics is very important and as we all grow in our sophistication and hopefully ability to do network diagnostics and recovery automatically, we may have other uses for a 'Diagnostic Header'.  We see the IPID as potentially the first use of this.
 
> fragmentation is on
> escalation engineers
> >>>> Francis D - it's potentially a covert
> channel, also
> >>> why not use the
> >>>> fregmentation header?
> >>>> 
> >>>> Bob H - We diefintently want to hear if
> this is
> >>> useful. I don't know
> >>>> Andrei Y - typical use case for this
> header would be
> >>> to see if a
> >>>> Fred B - What I'm out of this discussion
> is that the
> >>> application isn't

---

From: Shane Amante <[email-masked]>
Date: Mon, 15 Aug 2011 08:41:55 -0600

Nalini, All,

> Our other future thoughts, and agenda, if you will, is that as networks and traffic grow, more and more diagnostics and troubleshooting will have do be done by computer systems rather than humans.…

Have you taken a look at flow-spec for IPv4:
http://tools.ietf.org/html/rfc5575
... and, flow-spec for IPv6:
http://tools.ietf.org/html/draft-ietf-idr-flow-spec-v6-00
(Both the RFC and I-D are within the IDR WG).

In summary, using flow-spec one can define an ACL that then get distributed across all routers within an ASN, via BGP, that can then be used to count matching packets, log packet header information, etc.  It's up to the operator to decide what ACL to be applied and what 'action' (logging, counting, etc.) to be taken for packets that match a given 'rule'.  

From where I'm sitting, this seems like it might already solve the problem you're having, without having to invent a wholly new IPv6 Destination Header Option.  So, I would like to know if you have taken a look at flow-spec and, if so, why it was ruled out?

-shane

---

From: [email-masked]
Date: Mon, 15 Aug 2011 07:57:25 -0700 (PDT)

Shane,

> In summary, using flow-spec one can define an ACL that then
> get distributed across all routers within an ASN, via BGP,
> that can then be used to count matching packets, log packet
> header information, etc.  It's up to the operator to
> decide what ACL to be applied and what 'action' (logging,
> counting, etc.) to be taken for packets that match a given
> 'rule'.  
> 
> From where I'm sitting, this seems like it might already
> solve the problem you're having, without having to invent a
> wholly new IPv6 Destination Header Option.  So, I would
> like to know if you have taken a look at flow-spec and, if
> so, why it was ruled out?

Certainly an interesting set of RFCs but a few questions I would have:

1.  What if BGP is not being used?

2.  What about matching packets at non-router devices (that is, hosts)

3.  I believe what we need in diagnostics is the entire packet not just a logging that such a packet existed.  For example, the problem we are looking for may be in the body of the protocol data.  Ex. a problem with FTP.  So, we would need the entire packet as captured by a packet analyzer rather than a log that says 'a packet from dest x to source y was sent'.  I do not imagine that the log is going to keep the entire packet.  Am I missing something?

---

From: Warren Kumari <[email-masked]>
Date: Mon, 15 Aug 2011 11:09:38 -0400

On Aug 14, 2011, at 4:31 PM, [email-masked] wrote:

> 
> I am working on contacting a number of our large corporate customers to weigh in on this topic.  I think that the biggest issue here is that when a large commercial network is having problems, anything that can help speed diagnostics is worth doing.

No. Anything that can help speed diagnostics, and that doesn't cause collateral damage... or cost too much... or annoy the users... or destabilize the control plane.... or involve mucking with physical layer... or opens interesting security vectors... or....

> Outages and poor performance can cost a great deal of money.  

Yes.

> Situations such as comparing packets at various points on the network to determine packet loss is a necessary part of network problem determination.

No -- again you are making blanket assertions with nothing to back them up...

Comparing packets at various points on the network to determine packet loss is NOT a "necessary part of network problem determination."

It's not even a necessary part of what I'm assuming you meant -- finding packet loss.

There are a huge number of way to do this, including such obvious things as looking at SNMP / CLI packet counters (number of packets in should be roughly number of packets out (roughly is to account for mgmt / routing protocol overhead)), physical layer counters, netflow / sflow, flowspec, traceroute, BFD / LACP, fabric sanity checks, etc.

If for some reason you still feel the need to capture and compare packets at various points on the network, whats wrong with simply calculating a hash over relevant fields in the header / payload? 

>  I am working to get the technicians from the corporations themselves to get onto this email list and tell their opinions themselves.
> 

Ok, I've been doing networking for 15+ years, and have worked on networks from tiny ISPs to enterprise networks to startups to consulting to large ISPs to things that don't quite fit in other buckets. The only times (that I can remember!) that I have viewed IPID as an interesting diagnostic during  is to see if lots of packets have identical IDs, and so are likely from a poorly written DoS tool. This seems to line up with the views of other operators in the V6OPS meeting in Quebec.

I'm not saying the idea might not be useful in some circumstances, but rather that you are overselling the technique...

---

From: Warren Kumari <[email-masked]>
Date: Mon, 15 Aug 2011 11:13:10 -0400

> 2.  What about matching packets at non-router devices (that is, hosts)

If you are at the host you can simply match whatever tuple you are interested in and log the entire packet, and then do whatever analysis after the fact.

> 
> 3.  I believe what we need in diagnostics is the entire packet not just a logging that such a packet existed.  For example, the problem we are looking for may be in the body of the protocol data.  Ex. a problem with FTP.  
> So, we would need the entire packet as captured by a packet analyzer rather than a log that says 'a packet from dest x to source y was sent'.  I do not imagine that the log is going to keep the entire packet.  Am I missing something?

So:
A: Capture at the end host
or
B: Capture on the network if you really must (although having taps on the physical layer causes most security minded operators to get the heebie-jeebies) and match upon tuples...

W

---

From: [email-masked]
Date: Mon, 15 Aug 2011 08:32:01 -0700 (PDT)

Warren,

> Ok, I've been doing networking for 15+ years, and have
> worked on networks from tiny ISPs to enterprise networks to
> startups to consulting to large ISPs to things that don't
> quite fit in other buckets. The only times (that I can
> remember!) that I have viewed IPID as an interesting
> diagnostic during  is to see if lots of packets have
> identical IDs, and so are likely from a poorly written DoS
> tool. This seems to line up with the views of other
> operators in the V6OPS meeting in Quebec.
>

The people we work with have also done networking and diagnostics for 20 - 30 years and have a different way of doing diagnostics than you do. I have talked to a number of organizations and they are all surprised that the IPID is not in the main IPv6 header.  We need to provide as many helpful ways to do diagnostics as possible.

> I'm not saying the idea might not be useful in some
> circumstances, but rather that you are overselling the
> technique...
> 

Well, I do think that this has gotten a bit out of control in that I am not saying that the IPID is the be-all and end-all of diagnostics.  I am just saying that it is one interesting and helpful piece of information.  

Having said that, I do think that better metrics need to be kept for automated network diagnostics and this kind of header is the start of one way to have the metrics that we need.  I think that exactly what metrics we need is still evolving.

Nalini

---

From: Shane Amante <[email-masked]>
Date: Mon, 15 Aug 2011 11:02:11 -0600

Nalini,

> 1.  What if BGP is not being used?

Turn it on?

There are a plethora of books, training materials/sessions, know-how, best practices, etc. documented and presented over the years, including those given at various *NOG's ... if folks want/need pointers on where to start, I (and others) would be happy to provide them.

> 2.  What about matching packets at non-router devices (that is, hosts)

Use tcpdump and/or wireshark?

> 3.  I believe what we need in diagnostics is the entire packet not just a logging that such a packet existed.  For example, the problem we are looking for may be in the body of the protocol data.…

Take a look at RFC 5575.  It supports the idea of redirecting packets that match a flow-spec 'rule' to an IPVPN VRF.  I believe the reason this was done was to ensure (a copy of) the packet would be sent to a centralized collector(s) that could perform detailed packet analysis on the contents/payload of recv'd packets that are being sampled.

In short, I think this is a solved problem and there isn't a need for a IPv6 Diagnostic Header Destination Option.

-shane

---

From: Fred Baker <[email-masked]>
Date: Mon, 15 Aug 2011 10:15:15 -0700

>> 1.  What if BGP is not being used?

I don't think her question relates to how to turn it on. I think her question relates to the fact that it is not universally needed. BGP is used in inter-corporate networks including ISP-ISP and ISP-customer; it is less commonly used within enterprise and especially data center networks. So when it is not in use, ...

---

From: Shane Amante <[email-masked]>
Date: Mon, 15 Aug 2011 14:38:17 -0600

> I don't think her question relates to how to turn it on. I think her question relates to the fact that it is not universally needed. BGP is used in inter-corporate networks including ISP-ISP and IS…

But, in the case of flow-spec, we're talking about it being used as a "policy distribution mechanism" and not as a routing protocol.  In fact, I think this is amplified by the fact that flow-spec uses an entirely separate AFI/SAFI for flow-spec routes (1/133).  So, in this case it's entirely possible to (ab)use BGP as a information distribution mechanism, without concerns that BGP must also used for routing.  In this case BGP is just used for automating the deployment of filters through an ASN.  And, best of all, it's already been in shipping code + HW for a while now ...

Look, if BGP is too onerous, then deploy/use sFlow/NetFlow/IPFIX.  The only concern with that may be that depending on the scale/size of platform it may only be able to do statistical sampling, (not 1:1 sampling), if you need to use the central RP.  If that's a concern, take it up with your favorite vendor, they have the ability to either sell you HW (today) that will do 1:1, or close to it, sampling.

I still maintain there are existing tools out there, today, that _are_ already being used to trace these types of issues.  Please let's use them and/or improve upon them.

-shane

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 15 Aug 2011 22:50:08 +0200

+1

Thx,
R.

>> I don't think her question relates to how to turn it on. I think
>> her question relates to the fact that it is not universally needed.
>> BGP is used in inter-corporate networks including ISP-ISP and
>> ISP-customer; it is less commonly used within enterprise and
>> especially data center networks. So when it is not in use, ...
>
>
> But, in the case of flow-spec, we're talking about it being used as a
> "policy distribution mechanism" and not as a routing protocol.  In
> fact, I think this is amplified by the fact that flow-spec uses an
> entirely separate AFI/SAFI for flow-spec routes (1/133).  So, in this
> case it's entirely possible to (ab)use BGP as a information
> distribution mechanism, without concerns that BGP must also used for
> routing.  In this case BGP is just used for automating the deployment
> of filters through an ASN.  And, best of all, it's already been in
> shipping code + HW for a while now ...
>
> Look, if BGP is too onerous, then deploy/use sFlow/NetFlow/IPFIX.
> The only concern with that may be that depending on the scale/size of
> platform it may only be able to do statistical sampling, (not 1:1
> sampling), if you need to use the central RP.  If that's a concern,
> take it up with your favorite vendor, they have the ability to either
> sell you HW (today) that will do 1:1, or close to it, sampling.
>
> I still maintain there are existing tools out there, today, that
> _are_ already being used to trace these types of issues.  Please
> let's use them and/or improve upon them.
>
> -shane _______________________________________________ v6ops mailing

---

From: Fred Baker <[email-masked]>
Date: Mon, 15 Aug 2011 13:57:04 -0700

On Aug 15, 2011, at 1:38 PM, Shane Amante wrote:

> But, in the case of flow-spec, we're talking about it being used as a "policy distribution mechanism" and not as a routing protocol.

We could discuss "it slices, it dices, and it juliennes too." In my experience, the implied exponential complexity is often of questionable value. If she was using the routing protocol, having it also do something else might make sense. If she is using a different routing protocol or doesn't need one, turning on a routing protocol in order to turn off its routing functionality but... 

my head hurts.

> Look, if BGP is too onerous, then deploy/use sFlow/NetFlow/IPFIX.

Now, there's a sensible solution. At least it targets the intended functionality.

IPFIX, however, reports on sessions (in the target case, relatively large sets of packets) after they happen. What she's looking for is packet information while they're happening.

> The only concern with that may be that depending on the scale/size of platform it may only be able to do statistical sampling, (not 1:1 sampling), if you need to use the central RP.

... packet ... while they're happening ...

> If that's a concern, take it up with your favorite vendor, they have the ability to either sell you HW (today) that will do 1:1, or close to it, sampling.

No doubt.

> I still maintain there are existing tools out there, today, that _are_ already being used to trace these types of issues.  Please let's use them and/or improve upon them.

Well, and funny thing, she's building at least in part on wireshark.

---

From: Shane Amante <[email-masked]>
Date: Mon, 15 Aug 2011 18:37:46 -0600

Fred,

> We could discuss "it slices, it dices, and it juliennes too." In my experience, the implied exponential complexity is often of questionable value. If she was using the routing protocol, having it a…

So, does mine.  :-)  In any case, I think the key point to bear in mind, there's only a one-time set-up of BGP flow-spec across all devices in the network.  After that's done, administrators can, on an ad-hoc basis, easily inject a BGP flow-spec NLRI that gets automatically propagated (and withdrawn) across all those devices.  

I would argue that flow-spec is a superior tool in that regard, because it's automating the deployment of the filters concurrently across a [potentially large] set of devices.  Any other method would either have to rely on humans to manually deploy filters along a path of devices or they would have to spend time creating a tool that would automate that process, and keep it up-to-date.

> IPFIX, however, reports on sessions (in the target case, relatively large sets of packets) after they happen. What she's looking for is packet information while they're happening.

Depending on platform, this number could comfortably be 1:100 sampling.  That may be too large (or, too small) depending on one's network, HW and other variables at play.  

> ... packet ... while they're happening ...

Well, if IPFIX isn't sufficient, then how about RSPAN?  Other vendors typically refer to this as port or traffic mirroring.  Some allow you to apply ACL's on the routers/switches to allow you to filter the copied traffic before it is sent toward the monitoring station, in order that you're monitoring device is only recv'ing packets/frames of interest.
http://www.cisco.com/en/US/docs/switches/lan/catalyst3550/software/release/12.1_13_ea1/configuration/guide/swspan.html
---snip---
You can apply an output access control list (ACL) to RSPAN traffic to selectively filter or monitor specific packets. Specify these ACLs on the RSPAN VLAN in the RSPAN source switches
---snip---

> Well, and funny thing, she's building at least in part on wireshark.

I think I've stated my piece on this and will now quietly retire from this thread.

Thanks,

-shane