Subject: Re: [its] RoLL and Draft Charter proposal

From: Alexandru Petrescu <[email-masked]>
Date: Mon, 11 Mar 2013 23:08:50 +0100

Le 18/02/2013 12:43, Abdussalam Baryun a écrit :
> On 2/16/13, Alexandru Petrescu <[email-masked]> wrote:
>> RoLL stands for Routing over Low-Power and Lossy Links.
>
> ROLL is Routing over LLN, its lossy network not only lossy links, I
> think ITS is runing for lossy networks,

I am not at much ease with RoLL terminology, sorry.

But let me say that when a network loses packets, maybe TCP could be
used.  In that sense, TCP would compare to RPL?

>> But with V2V one would be looking more at high-power links, because
>> the higher the power the higher the speed of vehicles (e.g. in
>> France at 100mW WiFi a V2V entertaining videoconference allows for
>> two convoyed vehicles speed of around 90km/h, because the 100mW
>> corresponds to approximately 50meter distance, which is the
>> security distance in France for speed of 90km/h, no more).  This
>> would mean it's impossible to use RoLL on a highway where the
>> recommended speed is much above 90km/h.
>
> ROLL is not for high power, but don't forget that Vehicles are not
> all large as cars or trucks, they may be small transporters machines
>  in buildings/homes.

Ah!  I didnt think that use case...

When I say high-power emitters I mean more in terms of distance between
vehicles, not necessarily the size of the vehicle (although the two may
be related).

So... two small golf-cars may communicate using short-range links like
WiFi, if the golf court is small, ie the distance between the vehicles
is small.

Two small golf-cars would not be able to communicate using WiFi if the
diestance between them is too long.

RPL is mostly for short-range links, like WiFi and even shorter like
802.15.4.

>> RoLL is designed for links in the short-range family, like
>> 802.15.4. Whereas for V2V one would look at longer-range.
>
> Why we make the scope narrow to longer-range, I think ITS needs both
> long and short range to become intelligent otherwise it will not
> become intelligent,

:-) maybe we should rename ITS into DTS - dumb transportation systems
(like in 'smart end - dumb network').

But I agree with you both long-range and short-range links should be
considered.

Just that in the case of V2V protocols the long-range links are more
important because only them can make V2V possible.  Today nobody uses
long-range no-infrastructure links between vehicles in a V2V manner.

If we are able to use a long-range technology between two vehicles then
there is chance of novelty.  To make that work we need the highest-power
no-infrastructure technology available.

>> Inside a vehicle, RoLL would run ok on lossy links.  But we also
>> consider in a vehicle to use more reliable links, like Ethernet
>> cable, OBD-II interfaces, CAN.  Currently AUTOSAR SDO seems
>> interested in IP in vehicles, but I am not sure whether they
>> mention RoLL?
>
> Yes I like to have cables and/or wireless inside vehicles, depends
> on the transport resource and reason actually.

I agree.

In a vehicle, there exist already a number of links standardized, like
'CAN'.  I think it is little reasonable to imagine that the links
designed for office/building are to be reused in vehicles.  And on these
links RPL is made to work.  In a vehicle it's different...

Rather than reusing 802.15.4 in a vehicle, I'd start with considering
CAN and OBD-II.  But that's just an oppinion.

>> Now of course, one could say that the RPL protocol could run on
>> anything, not necessarily low-power links and lossy links.  At
>> which point I could reply that there are other routing protocols
>> which could run on anything as well.
>
> The bueaty of RPL is that it used for buildings and homes, so it can
> be used in our intelligence for our ITS.

Not sure about that :-)

Buildings and homes have many things in common, which are not in cars:
issued from the fact that walls dont move, dont change their attachment
points.

Assign a set of IP addresses to devices fixed to walls and there they
stay so for years.  In a vehicle on the other hand...

Turn on the wrong button in a building and then check the other button,
but in a vehicle dont press break if you mean to accelerate.

> Most transports in the world are between buildings and homes :-)

Yett the vehicle is different from both (from a communicaiton standpoint).

Alex
>
>>
>> But what RoLL doesn't have and what we'd propose to do here is a
>> mechanism to exchange routes, and to do address autoconfiguration
>> for vehicles.
>>
>
> IMO, we only need to use the RoLL techniques which help in our scope
> and ignore its techniques that is not related. We may take some other
> techniques as well or create a new mechanism depending which is
> better way on the discussion table.
>
>> Depending on these aspects, we need to identify how to add RoLL in
>>  the ITS picture.
>
> Yes, I agree totally,
>
>>
>> What do you think?
>>
>
> I think we are on the right track to best transports :)
>
>

---

From: Alexandru Petrescu <[email-masked]>
Date: Mon, 11 Mar 2013 23:42:14 +0100

Participants to the ITS effort at IETF,

Following suggestions I have updated the Draft Charter Proposal for ITS.
  It reads below.  Please comment. (I hope I got your comments right).

It has additions about RoLL, PMIP, smartcities, 2 new milestones.

Yours,

Alex

--------------------------------------------------------------------

	      Intelligent Transportation Systems at IETF
			Draft Charter Proposal
			 February 15th, 2013
		 http://ietf.org/mailman/listinfo/its

Intelligent Transportation Systems (its)
----------------------------------------
Current Status: Informal bar BoF

Chairs:
   TBD
   TBD

Assigned Area Director:
   TBD ()

Mailing list
   Address: [email-masked]
   To Subscribe: https://www.ietf.org/mailman/listinfo/its
   Archive:
   http://www.ietf.org/mail-archive/web/its/current/maillist.html

Draft Charter Proposal:

Context
-------

Intelligent Transportation Systems (ITS) are composed of
interconnected systems of machines; vehicles, mobile devices and fixed
embedded devices (actuator and sensors).  These machines communicate
with fixed access points which are are deployed along terrestrial
roads (e.g. Road-Side Units), shores along water paths; alternatively,
mobile or fixed access points may be deployed above ground; they all
offer wireless access to equipment deployed in mobile vehicles (e.g.
On-Board Units).  The entire system is connected to the Internet
through one or several IP paths.  In this system, disjoint and
heterogeneous radio systems are used (e.g. a vehicle uses two
different radios).

Due to the mobile nature of the system, several problems exist with
respect to IP addressing and route establishment such that Internet
communications can be realized (a typical IP network is using mostly
IP addresses topologically valid at a single point, and relatively
stable routing system).  In addition, several new problems specific to
vehicular communications at application layer are exposed when the
communication protocol is IP.

Several Standards Development Organizations outside IETF work towards
developping protocols for vehicular communications.  In some cases IP
protocols are used for transport, in other cases IP protocols are
modified for the purpose.  The examples of SDOs and respective groups
and documents are: ETSI ITS, ISO TC, IEEE 802.11p, AUTOSAR, IPSO.

A number of IETF protocols are being considered in the context of
Intelligent Transportation Systems - most notably IPv6, Mobile IPv6,
ND, ULA, DHCPv6, RPL, PMIPv6.  Others, such as Global HAHA, are not
excluded.  At IETF several Working Groups such as 6MAN, V6OPS, DHC,
MIF, MANET, DMM, RoLL, NETEXT, LISP produce work which is highly
pertinent to the ITS context.

New efforts at IETF are also relevant: 6TSCH effort (for communication
inside a vehicle) and I2RS WG.

Ongoing work
------------
Overall, in this group, the ongoing work items are to identify the
following:
(0) Scenarios and Requirements for IP in Intelligent Transportation Systems.
(1) vehicular-specific problems to be solved with IETF protocols,
(2) interactions with SDOs,
(3) existing IETF protocols which may solve the problems.
(4) EtherTypes for transporting IP over a MAC specific to vehicular
     communications.
(5) country-level frequencies for long-range vehicular communications
     with an IP-friendly MAC layer.

Potential new work items
------------------------
The following potential work items exist:

Scenarios and requirements for vehicular communications will be
developped which introduce at IETF the terms V2V, V2I, V2R and
intra-vehicular paradigms.

One particular use case should be identified as supporting basis for
further developments.

One particular use case could be under the form of an instrumented
ambulance.  The ambulance is connected to the Internet and offers
wireless and wired access to a large number of individual equipments
deployed in-vehicle.  There is a need to identify which existing
protocols may be reused to realize the connection between the
ambulance and the hospital.

The smart-city use case is also important for networking country- and
world-wide transports.  The use case should lead to development of
requirements and services for transports within cities, or countries,
making them smart.  The ITS technologies may use techniques from MANET
and from RoLL adapted to this smart-city use case.  One may need to
consider ETSI IPv6 over GeoNetworking, and also add OLSRv2 as used in
community networks.  A hybrid routing protocol could be used.

IP-over-foo, where foo is one particular link-layer which is
identified as very pertinent for vehicular communications.  One
particular example is IEEE 802.11p, but is not the only one;
clarifications whether IP-over-80211p is actually possible, due to
lack of country-level regulation.  IP is not 'safety' but rather
'entertainment'.  Other link-layers which are pertinent are
802.11abgn, and 802.11ai, and 802.11ak.  The preferred ones will be
those allowing long-range communications between vehicles, at
high-power levels.

Addressing mechanisms for vehicular communications: methods of
converting a Vehicular Identification Number, or geographical
coordinates, into an IPv6 address, and vice-versa.  Scoping of IPv6
addresses: addresses within a vehicle, between vehicles, to
infrastructure.  Distributing addresses between vehicles, using
dynamic mechanisms, such as DHCP or ND, which respect constraints of
applications.

Route exchanges between vehicles, and between vehicles and road-side
units.  This includes the identification of a need to realize the
exchanges, the constraints related to loop-free graphs and networks
which scale to large sizes, their interaction with the imposed
physical topology.

It is possible to adapt MANET and RoLL protocols to use in ITS.  A
distinction could be made between protocols used within a vehicle, and
protocols used between two vehicles.

Fast Network Transport Protocol is a protocol designed at ISO for
vehicular communications.  Feedback is solicited from IETF about this
protocol.

GeoNetworking is a protocol designed at ETSI.  Feedback is solicited
from IETF about this protocol.

Coordinate work with ISO SDO, about developping a V2V protocol based
on route exchanges using Router Advertisements.

Milestones:
   Mar 2013 - Meet in Orlando
   Jul 2013 - Meet in Berlin
   Dec 2013 - Present drafts "Scenarios and Requirements for IP in
              Intelligent Transportation Systems"
   Dec 2013 - Present status from ISO about V2V

---

From: "Ivancic, William D. (GRC-RHN0)" <[email-masked]>
Date: Mon, 11 Mar 2013 18:26:40 -0500

Propose

From: "In this system, disjoint and
heterogeneous radio systems are used (e.g. a vehicle uses two
different radios)."

To:  In this system, disjoint and
heterogeneous radio systems are used (e.g. a vehicle uses two OR MORE
different radios SIMULTANEOUSLY)."

Question:

Do you want simultaneous use of multiple radios or not?

Will Ivancic

> From: Alexandru Petrescu <[email-masked]>
> Date: Mon, 11 Mar 2013 17:42:14 -0500
> To: "[email-masked]" <[email-masked]>
> Subject: [its] Updated  Draft Charter proposal
> 
>  In this system, disjoint and

---

From: Abdussalam Baryun <[email-masked]>
Date: Tue, 12 Mar 2013 11:38:21 +0100

IMHO, the charter is well representing ITS, I hope that this opens
more discussions at Orlando to see other opinions.

thanking you
AB

On 3/11/13, Alexandru Petrescu <[email-masked]> wrote:
> _______________________________________________
> its mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/its
>

---

From: Abdussalam Baryun <[email-masked]>
Date: Tue, 12 Mar 2013 11:40:42 +0100

I think I understand from that as example of possible multiple radio use,

AB

---

From: <[email-masked]>
Date: Tue, 12 Mar 2013 10:50:57 +0000

Dear Alex, Dear all,

Thanks for setting up the text for the ITS charter!

Sorry for being silent the last months, but I was somehow overloaded!

I see that some issues/topics that have been presented during the previous IETF ITS meetings (including some that I had presented) are included in the charter and that is good!

I could include more issues/topics, but I think that it is good to first start with a small number of issues/topics!

Please note that I will not be attending the IETF meeting in Orlando, but I would like to emphasize that I am supporting this activity!

Best regards,
Georgios

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Abdussalam Baryun
> Sent: dinsdag 12 maart 2013 11:38
> To: Alexandru Petrescu
> Cc: [email-masked]
> Subject: Re: [its] Updated Draft Charter proposal
> 
> IMHO, the charter is well representing ITS, I hope that this opens more
> discussions at Orlando to see other opinions.
> 
> thanking you
> AB
>

---

From: Alexandru Petrescu <[email-masked]>
Date: Tue, 12 Mar 2013 23:02:10 +0100

Will,

Thanks for the comment.

I have inserted something about this in the Draft Charter Proposal, but
let me discuss.

The simultaneous use of multiple egress interfaces is a technique
allowing to expand the available bandwidth for a vehicle connection.

A known car network problem is the discrepancy between the outside
bandwidth and the inside bandwith.  The egress interface is typically a
cellular link, which is originally intended for unique personal use;
that is bandwidth 50Mbit/s and latency 50ms (e.g. LTE).  But within same
vehicle, one uses a Ethernet or WiFi links, which may use 300Mbit/s or
even 1Gbit/s.  If all the devices within the vehicle talk to the Servers
at the same time, it becomes obvious that the LTE link is a bottleneck.

The problem is specific to vehicular networks, and different from office
networks.  In office networks the upstream link is typically much higher
bandwidth than the link for office PCs.  E.g. 10GByte vs 1GByte
Ethernet.  In vehicular networks, the cellular link is the bottleneck
because it is much less bandwidth than the in-car link.

To solve this, there exist approaches where one proposes to use several
such egress external links on the vehicle, at the same time.  This would
add up the bandwidth; e.g. use 10 USB LTE dongles to achieve an
aggregated bandwidth of 500Mbit/s towards the infrastructure.  This
would enlarge this bottleneck.

At IETF, there is this draft-ietf-mip4-multiple-tunnel-support-05.txt
"Flow Binding Support for Mobile IPv4" that I co-author there.  I dont
think it advances ok, but should advance there.

However, that draft is not really corresponding to my intention about
simultaneous use of multiple interfaces.  It's more an evolution of what
'flow binding updates' means to Mobile IPv6.

I am not aware of any other draft at IETF doing simultaneous use of
multiple interfaces for bandwidth augmentation.

If you read until here, then please reply to see whether there is
interest in developping such solution.  This would mean to add it in the
Charter Proposal, and commit to some work items.  We are already very
short on space left about new things to do.

Yours,

Alex

Le 12/03/2013 00:26, Ivancic, William D. (GRC-RHN0) a écrit :
> Propose
>
> From: "In this system, disjoint and heterogeneous radio systems are
> used (e.g. a vehicle uses two different radios)."
>
> To:  In this system, disjoint and heterogeneous radio systems are
> used (e.g. a vehicle uses two OR MORE different radios
> SIMULTANEOUSLY)."
>
> Question:
>
>> From: Alexandru Petrescu <[email-masked]> Date: Mon,
>> 11 Mar 2013 17:42:14 -0500 To: "[email-masked]" <[email-masked]>
>> In this system, disjoint and heterogeneous radio systems are used
>> (e.g. a vehicle uses two different radios).
>
>
>

---

From: Rex Buddenberg <[email-masked]>
Date: Tue, 12 Mar 2013 14:22:13 -0800

Alex,

Your numbers are essentially correct -- radio links are about 10-4
smaller sized than wired ones.  

But a more powerful argument for multiple interfaces is availability.
In addition to radio being four orders of magnitude less capacious than
wired, radio has about 4 orders of magnitude higher error rates.  There
are myriad reasons that a radio link of whatever flavor will go away.
For short term or longer term duration.  

So the obvious way to counter this is to have >1 radio link.  The
arithmetic to support is standard high Ao engineering stuff.

> multiple interfaces for bandwidth augmentation.

Not sure what you're trying to say with this quote: every router has
multiple interfaces and they are often used in parallel for bandwidth
augmentation.  This is an inherent part of IP ... RFC 791.  Getting them
to behave properly has sometimes been a problem, but it's generally
outside of the standardization realm.  

Current reality: US Navy has a router on each ship.  There are usually
three LANs on the ship side (segregation for security reasons, VPN).
But there are multiple interfaces on the radio side -- one for each
satcom channel, for example.  

Will's question is easily answered: yes.

---

From: Alexandru Petrescu <[email-masked]>
Date: Wed, 13 Mar 2013 16:49:19 +0100

Ok,

So we talk radios as egress interfaces.  I.e. have several egress
interfaces on a Mobile Router (gateway in the vehicle).  When one of the
radio fails, the MR uses another one, such that the Internet is always
available to vehicle.

This sounds reasonable.

It is for me a sort of architectural requirement, to be strongly kept in
mind, for designing any new protocol.

Le 12/03/2013 23:22, Rex Buddenberg a écrit :
> But a more powerful argument for multiple interfaces is
> availability. In addition to radio being four orders of magnitude
> less capacious than wired, radio has about 4 orders of magnitude
> higher error rates. There are myriad reasons that a radio link of
> whatever flavor will go away. For short term or longer term
> duration.
>
>> I am not aware of any other draft at IETF doing simultaneous use
>> of multiple interfaces for bandwidth augmentation.
>>
>
> augmentation.  This is an inherent part of IP ... RFC 791. Getting
> them to behave properly has sometimes been a problem, but it's
> generally outside of the standardization realm.

I wonder what the others think about this?  (I do have some idea, but
not sure I as contributor would like to do anything about this
particular topic right now here, because other things here in ITS are
more mature.)

> Current reality: US Navy has a router on each ship.  There are
> usually three LANs on the ship side (segregation for security
> reasons, VPN). But there are multiple interfaces on the radio side --
> one for each satcom channel, for example.

Sure... this not surprising and is needed so!

However, what is the mechanism used by US Navy to make all these egress
interfaces look as one single interface in the router on the ship?  What
is the scheduling intelligence to transparently divide a traffic flow
evenly among all these interfaces?  (without trolling, I think there
isnt any; I think they put one application on one radio, another
application on another radio, and so on)

> Will's question is easily answered: yes.

Yes, I agree.

And from that answer, what we need right now is to know how to put this
in a Charter - is it by making it into a Problem Statement such that
scenarios and solutions for this particular multi-interface scenario be
developped?  Or just have it there as a design principle?  Or just to
reuse some existing protocol?

Knowing that there are already two things that are maturing as we speak
and that should get more attention in the Charter proposal: V2V and
addressing.

And that we should focus the Charter proposal.

And that for any thing that we put in the Charter proposal we shoudl
have commitment of effort to write drafts.

Alex

>> I have inserted something about this in the Draft Charter Proposal,
>> but let me discuss.
>>
>> bandwidth and the inside bandwith.  The egress interface is
>> typically a cellular link, which is originally intended for unique
>>  personal use; that is bandwidth 50Mbit/s and latency 50ms (e.g.
>> LTE).  But within same vehicle, one uses a Ethernet or WiFi links,
>>  which may use 300Mbit/s or even 1Gbit/s.  If all the devices
>> within the vehicle talk to the Servers at the same time, it becomes
>> obvious that the LTE link is a bottleneck.
>>
>> The problem is specific to vehicular networks, and different from
>> office networks.  In office networks the upstream link is typically
>> much higher bandwidth than the link for office PCs.  E.g. 10GByte
>> vs 1GByte Ethernet.  In vehicular networks, the cellular link is
>> the bottleneck because it is much less bandwidth than the in-car
>> link.
>>
>> To solve this, there exist approaches where one proposes to use
>> several such egress external links on the vehicle, at the same
>> time.  This would add up the bandwidth; e.g. use 10 USB LTE dongles
>> to achieve an aggregated bandwidth of 500Mbit/s towards the
>> infrastructure.  This would enlarge this bottleneck.
>>
>> At IETF, there is this
>> draft-ietf-mip4-multiple-tunnel-support-05.txt "Flow Binding
>> Support for Mobile IPv4" that I co-author there.  I dont think it
>> advances ok, but should advance there.
>>
>> However, that draft is not really corresponding to my intention
>> about simultaneous use of multiple interfaces.  It's more an
>> evolution of what 'flow binding updates' means to Mobile IPv6.
>>
>> I am not aware of any other draft at IETF doing simultaneous use
>> of multiple interfaces for bandwidth augmentation.
>>
>> interest in developping such solution.  This would mean to add it
>> in the Charter Proposal, and commit to some work items.  We are
>> already very short on space left about new things to do.
>>
>> Yours,
>>
>> Alex
>>
>>
>>> From: "In this system, disjoint and heterogeneous radio systems
>>> are used (e.g. a vehicle uses two different radios)."
>>>
>>> To:  In this system, disjoint and heterogeneous radio systems
>>> are used (e.g. a vehicle uses two OR MORE different radios
>>> SIMULTANEOUSLY)."
>>>
>>> Question:
>>>
>>>> From: Alexandru Petrescu <[email-masked]> Date:
>>>> Mon, 11 Mar 2013 17:42:14 -0500 To: "[email-masked]"
>>>> <[email-masked]> Subject: [its] Updated  Draft Charter proposal
>>>>
>>>> In this system, disjoint and heterogeneous radio systems are
>>>> used (e.g. a vehicle uses two different radios).
>>>
>>>
>>>
>>
>>
>> _______________________________________________ its mailing list
>> [email-masked] https://www.ietf.org/mailman/listinfo/its
>
>
>
>

---

From: "Stan Ratliff (sratliff)" <[email-masked]>
Date: Wed, 13 Mar 2013 16:15:23 +0000

Alex,

This describes almost exactly the deployments I've worked with. It is yet another reason I say that starting a new working group for the ITS effort is over-kill. Please don't get me wrong - this is interesting work, and should be addressed. However, it needs to be done (IMO) within the auspices of the MANET working group. Possibly ROLL, but I'm thinking MANET is a better fit. 

Regards,
Stan

---

From: "Ivancic, William D. (GRC-RHN0)" <[email-masked]>
Date: Wed, 13 Mar 2013 12:40:09 -0500

This is more of what I was getting at.  I was pretty active in mext, monami6
and a few other mobile-ip related groups working mobile networking.  Some of
the first code I used working the Cisco allowed you to use multiple egress
interfaces, but only one at a time.  If the preferred link when down, the
next best (as pre configured) link would be used.  If I had 5 egress
interfaces with 5 different radio systems, I could use any of these, but
only one at a time.  I might have wired, wifi, 4G, UHF with priority in that
order. Note, they all may be provided by different service providers.

My question was do you want to be able to simultaneously use multiple link?
If so, the problems becomes very interesting.  It also may be one wants
specific traffic to use specific links.

Buy the way, I am currently working on spacesuit communications. Spacesuits
are mini-spacecraft with full life support and such.  They really are an
intelligent vehicle, but probably less intelligent than a car.  There is a
current concept for the next generation to have two radios, a low-rate
highly reliable radio and a wifi-type radio with large bandwidth.  Weather
or no they would both be used simultaneously is an open question.
Obviously, high-rate, high-resolution video would not be permitted over the
UHF low-rate link.

Multi-homed from IETF 67 in 2006. Slides 5 and 11

"Multi-Homed Mobile Networks," 67th IETF, nemo and monami working groups,
November 9-10, 2006 San Diego, California ( Powerpoint ) file size 679,424
bytes 

http://roland.grc.nasa.gov/~ivancic/papers_presentations/2006/IETF67nemo-mon
ami6-final.ppt

Will

---

From: Rex Buddenberg <[email-masked]>
Date: Wed, 13 Mar 2013 10:06:34 -0800

Will,

I think we're thinking alike.

> order. Note, they all may be provided by different service providers.

Good use case.

The basic design of IP (and especially TCP) accommodates.  TCP is
designed to deal with packets that arrive out of order, which is likely
if you have diverse routes.  
> My question was do you want to be able to simultaneously use multiple link?

Yes, as we both said before.

> If so, the problems becomes very interesting.

There's been a whole area of research on 'best route'.  Early routing
information protocols defined that as 'least number of hops' and were
immune to explanations that the least-hop route might be the lowest
capacity one.  And a lot of load-balancing algorithms were pretty
elementary.

>   It also may be one wants

Smells like a linkage to DSCP to me.  

We've tried linking priority to users and linking priority to port
numbers... both too easy to be gamed.  And the prioritization mechanism
becomes counterproductive if it's too cute.
> intelligent vehicle, but probably less intelligent than a car.

Depends on how you define 'intelligent'.  Interesting use case.  I'd
guess that a space suit (and the body inside) is somewhat better
instrumented than a car.

>   There is a

This maps onto automobiles just fine.  The instrumented ambulance use
case works too: use the high capacity link when it's available
(suburbia) but use anything that works (rural).

---

From: "JP Vasseur (jvasseur)" <[email-masked]>
Date: Fri, 15 Mar 2013 18:03:27 +0000

Hi,

This is clearly a working intersecting two WG: MANET for sure, and ROLL for routing, since RPL has been used for car to car communication. 

Cheers.

JP.

On Mar 13, 2013, at 9:19 PM, Richard Roy wrote:

> Definitely plan for multiple simultaneous radio links.  Whether or not a
> particular ITS station (ITS terminology for a (possibly mobile) network
> node) can support such simultaneous transmission will be assessed by ITS
> station management, and if for some reason the radio modules interfere with
> each other on Tx, simultaneous Tx from both will be prohibited.  However,
> this is not something that concerns the network layer protocols directly and
> for the purposes of this effort, I suggest you just assume simultaneous Tx.
> ITS stations assume multiple flows (to the same or different app processes)
> and ITS station management manages the communication resources in realtime.
> The issue is getting the back end networking protocols to set up and support
> these flows.
> 
> RR
> 
>> From: Stan Ratliff (sratliff) [mailto:[email-masked]]
>> Sent: Wednesday, March 13, 2013 9:15 AM
>> To: Alexandru Petrescu; Joe Macker; JP Vasseur (jvasseur)
>> Cc: Rex Buddenberg; [email-masked]; Ivancic, William D. (GRC-RHN0)
>> Subject: Re: [its] simultaneous use o fmultiple radios or not
>> 
>> Alex,
>> 
>> This describes almost exactly the deployments I've worked with. It is yet
>> another reason I say that starting a new working group for the ITS effort
>> is over-kill. Please don't get me wrong - this is interesting work, and
>> should be addressed. However, it needs to be done (IMO) within the
>> auspices of the MANET working group. Possibly ROLL, but I'm thinking MANET
>> is a better fit.
>> 
>> Regards,
>> Stan
>>

---

From: Michelle Wetterwald <[email-masked]>
Date: Fri, 15 Mar 2013 21:22:23 +0100

Dear all,

You may also be interested by the work done in the MIF (Multiple 
Interfaces) WG , which addresses the mechanisms required to support the 
attachment to multiple provisioning domains, cf RFCs 6418 and 6419. It 
falls directly in this topic from networking point of view.

Regards,

Michelle

---

From: Kevin Lahey <[email-masked]>
Date: Wed, 13 Mar 2013 17:21:12 -0700

On Wed, 13 Mar 2013 10:06:34 -0800

> if you have diverse routes.  

Careful -- TCP is actually very sensitive to reordering.  Fast 
retransmission will be triggered after a small number of duplicate
ACKs, which are triggered when a packet arrives out of order (or
is lost).  Some stacks can recognize repeated reorderings and 
relax this rule, at the cost of lower performance.

The multipath schemes with which I'm familiar try hard to steer 
all packets in a flow to the same link.

Cheers,

Kevin
[email-masked]

---

From: Alexandru Petrescu <[email-masked]>
Date: Sun, 17 Mar 2013 02:44:11 +0100

Hello,

There is indeed re-chartering discussion in the MIF Working Group, 
according to these intarea slides of Ted Lemon (new Area Director):

http://www.ietf.org/proceedings/86/slides/slides-86-intarea-3.pdf

It says 'Multiple Provisioning Domains'.

Alex

Le 15/03/2013 21:22, Michelle Wetterwald a écrit :
>> This is clearly a working intersecting two WG: MANET for sure, and
>> ROLL for routing, since RPL has been used for car to car communication.
>>
>> Cheers.
>>
>> JP.
>>

---

From: Alexandru Petrescu <[email-masked]>
Date: Sun, 17 Mar 2013 02:52:04 +0100

Le 14/03/2013 01:21, Kevin Lahey a écrit :
> On Wed, 13 Mar 2013 10:06:34 -0800 Rex Buddenberg
> <[email-masked]> wrote:
>
>> designed to deal with packets that arrive out of order, which is
>> likely if you have diverse routes.
>
> ACKs, which are triggered when a packet arrives out of order (or is
> lost).

I noticed a similar behaviour of TCP.

> Some stacks can recognize repeated reorderings and relax this rule,
> at the cost of lower performance.

And, just to say that it is possible to use such multi-interface
technique with other things than TCP - e.g. a UDP video stream upstream.
And the non-TCP software on the Correspondent Node would be in charge
of reordering.

Alex

>
> The multipath schemes with which I'm familiar try hard to steer all
> packets in a flow to the same link.
>
> Cheers,
>
> Kevin [email-masked]

---

From: Wesley Eddy <[email-masked]>
Date: Sun, 17 Mar 2013 21:07:40 -0400

> all packets in a flow to the same link.

I agree that for normal TCP, multipath routing that causes reordering
is a really bad idea.

Recently, Multipath TCP, (MPTCP) was developed which handles the use
of multiple links in the transport layer, and avoids the issues that
normal TCP would have.

MPTCP can utilize multiple L3 paths simultaneously:
http://tools.ietf.org/html/rfc6824

This requires the end-host to have multiple addresses visible to it,
which could be a design constraint on the way the mobile network is
setup.

-- 
Wes Eddy
MTI Systems

---

From: Rex Buddenberg <[email-masked]>
Date: Sun, 17 Mar 2013 19:07:08 -0800

below

> I agree that for normal TCP, multipath routing that causes reordering

I think you have the problem backwards.  Rather than fit the plumbing to
the transport protocol, we need to have a transport protocol that
handles the reality of the media.

Radio is hard.  It always will be hard (at least harder than wireline).
 - four orders of magnitude less capacity (mbits vice gbits)
 - four orders of magnitued more noise (= bit errors, = damaged and
discarded packets).
 - a number of mission critical applications lining up (my favorite use
case is 'instrumented ambulance' where high availability is a driving
requirement (probably the dominant one).

> of multiple links in the transport layer, and avoids the issues that

I suspect that rather than thrash TCP, we have a situation where a lot
of data will be multiCAST in nature.  So we might be better off
thrashing NORM instead.

---

From: Wesley Eddy <[email-masked]>
Date: Mon, 18 Mar 2013 00:40:35 -0400

> handles the reality of the media.

Well, I don't think the group is chartering to include new transport
work, so doing things that break badly with existing transports seems
like a poor idea.  All packets within a flow need to either go over
the same link, or some kind of tunnel protocol for fixing reordering
in the transport stream is necessary.

> of data will be multiCAST in nature.  So we might be better off

If there are actually applications using NORM for ITS ... that doesn't
seem to be the common assumption.

-- 
Wes Eddy
MTI Systems

---

From: "Velt, R. (Ronald) in 't" <[email-masked]>
Date: Mon, 18 Mar 2013 10:29:21 +0000

Hi,

>Wesley Eddy
>Sent: maandag 18 maart 2013 2:08
>To: Kevin Lahey
>Cc: [email-masked]
>Subject: Re: [its] simultaneous use of multiple radios or not
>
>> lost).  Some stacks can recognize repeated reorderings and relax this
>> rule, at the cost of lower performance.
>>
>I agree that for normal TCP, multipath routing that causes reordering is a really
>bad idea.
>
>Recently, Multipath TCP, (MPTCP) was developed which handles the use of
>multiple links in the transport layer, and avoids the issues that normal TCP
>would have.
>
>This requires the end-host to have multiple addresses visible to it, which could
>be a design constraint on the way the mobile network is setup.

Right. Can I ask what model is being assumed here? Is this entity with multiple radio interfaces supposed to be a router or a multi-homed host? I tend to think it is the former, with hosts residing on some in-vehicle (ethernet or CAN-bus based) link attached to it. If that's the case, one would have to perform some special tricks to make the hosts see multiple source addresses. 

Ronald in 't Velt
This e-mail and its contents are subject to the DISCLAIMER at http://www.tno.nl/emaildisclaimer

---

From: Alexandru Petrescu <[email-masked]>
Date: Mon, 18 Mar 2013 16:17:51 +0100

Le 18/03/2013 05:40, Wesley Eddy a écrit :
>>>>> The basic design of IP (and especially TCP) accommodates.
>>>>> TCP is designed to deal with packets that arrive out of
>>>>> order, which is likely if you have diverse routes.
>>>>
>>>> retransmission will be triggered after a small number of
>>>> duplicate ACKs, which are triggered when a packet arrives out
>>>> of order (or is lost).  Some stacks can recognize repeated
>>>> reorderings and relax this rule, at the cost of lower
>>>> performance.
>>>>
>>>> The multipath schemes with which I'm familiar try hard to
>>>> steer all packets in a flow to the same link.
>>>>
>>>
>>>
>>> I agree that for normal TCP, multipath routing that causes
>>> reordering is a really bad idea.
>>
>> I think you have the problem backwards.  Rather than fit the
>> plumbing to the transport protocol, we need to have a transport
>> protocol that handles the reality of the media.
>
>
> work,

Right - it's not, unless we want it to.  If enough interest is expressed
that could be done as well.

There was a proposal from ISO participant to consider an ISO standard
under development - FNTP.  I have had it in the Charter proposal under
the form of reviewing FNTP for ISO.  But then I was told that this may
not be a specific Working Item in a WG - to which I agree.  So I removed it.

But let me write another email about this specific topic - should we
review documents for other SDOs and have that as specific WG items?

Alex

  so doing things that break badly with existing transports seems
>>> Recently, Multipath TCP, (MPTCP) was developed which handles the
>>> use of multiple links in the transport layer, and avoids the
>>> issues that normal TCP would have.
>>
>> I suspect that rather than thrash TCP, we have a situation where a
>> lot of data will be multiCAST in nature.  So we might be better
>> off thrashing NORM instead.
>
>
> If there are actually applications using NORM for ITS ... that
> doesn't seem to be the common assumption.
>
>

---

From: Alexandru Petrescu <[email-masked]>
Date: Mon, 18 Mar 2013 16:50:08 +0100

> ROLL for routing, since RPL has been used for car to car

JP,

I am happy to learn that RPL has been used for car to car communications.

RPL is based on ND messages and there are other protocols based on ND 
messages which have been used in car to car communications.

Intersection is clearly there in these ND messages.

Alex

>> Definitely plan for multiple simultaneous radio links.  Whether or
>> not a particular ITS station (ITS terminology for a (possibly
>> mobile) network node) can support such simultaneous transmission
>> will be assessed by ITS station management, and if for some reason
>> the radio modules interfere with each other on Tx, simultaneous Tx
>> from both will be prohibited.  However, this is not something that
>> concerns the network layer protocols directly and for the purposes
>> of this effort, I suggest you just assume simultaneous Tx. ITS
>> stations assume multiple flows (to the same or different app
>> processes) and ITS station management manages the communication
>> resources in realtime. The issue is getting the back end networking
>> protocols to set up and support these flows.
>>
>> RR
>>
>>> -----Original Message----- From: Stan Ratliff (sratliff)
>>> [mailto:[email-masked]] Sent: Wednesday, March 13, 2013 9:15
>>> AM To: Alexandru Petrescu; Joe Macker; JP Vasseur (jvasseur) Cc:
>>> Rex Buddenberg; [email-masked]; Ivancic, William D. (GRC-RHN0)
>>> This describes almost exactly the deployments I've worked with.
>>> It is yet another reason I say that starting a new working group
>>> for the ITS effort is over-kill. Please don't get me wrong - this
>>> is interesting work, and should be addressed. However, it needs
>>> to be done (IMO) within the auspices of the MANET working group.
>>> Possibly ROLL, but I'm thinking MANET is a better fit.
>>>
>>> Regards, Stan
>>>
>>>> So we talk radios as egress interfaces.  I.e. have several
>>>> egress interfaces on a Mobile Router (gateway in the vehicle).
>>>> When one of the radio fails, the MR uses another one, such that
>>>> the Internet is always available to vehicle.
>>>>
>>>> This sounds reasonable.
>>>>
>>>> It is for me a sort of architectural requirement, to be
>>>> strongly kept in mind, for designing any new protocol.
>>>>
>>>>> Your numbers are essentially correct -- radio links are about
>>>>> 10-4 smaller sized than wired ones.
>>>>>
>>>>> availability. In addition to radio being four orders of
>>>>> magnitude less capacious than wired, radio has about 4 orders
>>>>> of magnitude higher error rates. There are myriad reasons
>>>>> that a radio link of whatever flavor will go away. For short
>>>>> term or longer term duration.
>>>>>
>>>>> So the obvious way to counter this is to have >1 radio link.
>>>>> The arithmetic to support is standard high Ao engineering
>>>>> stuff.
>>>>>
>>>>>
>>>>>> I am not aware of any other draft at IETF doing
>>>>>> simultaneous use of multiple interfaces for bandwidth
>>>>>> augmentation.
>>>>>>
>>>>>
>>>>> Not sure what you're trying to say with this quote: every
>>>>> router has multiple interfaces and they are often used in
>>>>> parallel for bandwidth augmentation.  This is an inherent
>>>>> part of IP ... RFC 791. Getting them to behave properly has
>>>>> sometimes been a problem, but it's generally outside of the
>>>>> standardization realm.
>>>>
>>>> I wonder what the others think about this?  (I do have some
>>>> idea, but not sure I as contributor would like to do anything
>>>> about this particular topic right now here, because other
>>>> things here in ITS are more mature.)
>>>>
>>>>> Current reality: US Navy has a router on each ship.  There
>>>>> are usually three LANs on the ship side (segregation for
>>>>> security reasons, VPN). But there are multiple interfaces on
>>>>> the radio side -- one for each satcom channel, for example.
>>>>
>>>> However, what is the mechanism used by US Navy to make all
>>>> these egress interfaces look as one single interface in the
>>>> router on the ship?  What is the scheduling intelligence to
>>>> transparently divide a traffic flow evenly among all these
>>>> interfaces?  (without trolling, I think there isnt any; I think
>>>> they put one application on one radio, another application on
>>>> another radio, and so on)
>>>>
>>>> And from that answer, what we need right now is to know how to
>>>> put this in a Charter - is it by making it into a Problem
>>>> Statement such that scenarios and solutions for this particular
>>>> multi-interface scenario be developped?  Or just have it there
>>>> as a design principle?  Or just to reuse some existing
>>>> protocol?
>>>>
>>>> Knowing that there are already two things that are maturing as
>>>> we speak and that should get more attention in the Charter
>>>> proposal: V2V and addressing.
>>>>
>>>> And that for any thing that we put in the Charter proposal we
>>>> shoudl have commitment of effort to write drafts.
>>>>
>>>> Alex
>>>>
>>>>>
>>>>>
>>>>>
>>>>>> I have inserted something about this in the Draft Charter
>>>>>> Proposal, but let me discuss.
>>>>>>
>>>>>> The simultaneous use of multiple egress interfaces is a
>>>>>> technique allowing to expand the available bandwidth for a
>>>>>> vehicle connection.
>>>>>>
>>>>>> A known car network problem is the discrepancy between the
>>>>>> outside bandwidth and the inside bandwith.  The egress
>>>>>> interface is typically a cellular link, which is originally
>>>>>> intended for unique personal use; that is bandwidth
>>>>>> 50Mbit/s and latency 50ms (e.g. LTE).  But within same
>>>>>> vehicle, one uses a Ethernet or WiFi links, which may use
>>>>>> 300Mbit/s or even 1Gbit/s.  If all the devices within the
>>>>>> vehicle talk to the Servers at the same time, it becomes
>>>>>> The problem is specific to vehicular networks, and
>>>>>> different from office networks.  In office networks the
>>>>>> upstream link is typically much higher bandwidth than the
>>>>>> link for office PCs.  E.g. 10GByte vs 1GByte Ethernet.  In
>>>>>> vehicular networks, the cellular link is the bottleneck
>>>>>> To solve this, there exist approaches where one proposes to
>>>>>> use several such egress external links on the vehicle, at
>>>>>> the same time.  This would add up the bandwidth; e.g. use
>>>>>> 10 USB LTE dongles to achieve an aggregated bandwidth of
>>>>>> 500Mbit/s towards the infrastructure.  This would enlarge
>>>>>> this bottleneck.
>>>>>>
>>>>>> At IETF, there is this
>>>>>> draft-ietf-mip4-multiple-tunnel-support-05.txt "Flow
>>>>>> Binding Support for Mobile IPv4" that I co-author there.  I
>>>>>> dont think it advances ok, but should advance there.
>>>>>>
>>>>>> However, that draft is not really corresponding to my
>>>>>> intention about simultaneous use of multiple interfaces.
>>>>>> It's more an evolution of what 'flow binding updates' means
>>>>>> to Mobile IPv6.
>>>>>>
>>>>>> I am not aware of any other draft at IETF doing
>>>>>> simultaneous use of multiple interfaces for bandwidth
>>>>>> augmentation.
>>>>>>
>>>>>> If you read until here, then please reply to see whether
>>>>>> there is interest in developping such solution.  This would
>>>>>> mean to add it in the Charter Proposal, and commit to some
>>>>>> work items.  We are already very short on space left about
>>>>>> new things to do.
>>>>>>
>>>>>> Yours,
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>>
>>>>>>> From: "In this system, disjoint and heterogeneous radio
>>>>>>> systems are used (e.g. a vehicle uses two different
>>>>>>> radios)."
>>>>>>>
>>>>>>> To:  In this system, disjoint and heterogeneous radio
>>>>>>> systems are used (e.g. a vehicle uses two OR MORE
>>>>>>>> Date: Mon, 11 Mar 2013 17:42:14 -0500 To:
>>>>>>>> "[email-masked]" <[email-masked]> Subject: [its] Updated
>>>>>>>> Draft Charter proposal
>>>>>>>>
>>>>>>>> In this system, disjoint and heterogeneous radio
>>>>>>>> systems are used (e.g. a vehicle uses two different
>>>>>>>> radios).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________ its mailing
>>>>>> list [email-masked]
>>>> _______________________________________________ its mailing
>>>> list [email-masked] https://www.ietf.org/mailman/listinfo/its
>>>
>>
>>
>
>
>

---

From: Rex Buddenberg <[email-masked]>
Date: Mon, 18 Mar 2013 09:30:39 -0800

Gee, I didn't think I'd have to defend this one.  Warning: this is a
longer message, trying to roll up several questions you folks are
asking.

I contributed the 'instrumented ambulance' example to Alex's set of use
cases, so let me expand it a bit.  

Ronald, you're right -- there should be a LAN (actually a good case for
about three LANs) in an ambulance.  End systems such as casualty vital
signs monitors, radionav receivers, voice handsets for both the EMT and
the driver, etc, should be attached to the LAN.  Systems engineering
sanity: keep them modularly separated from the radio issues.  

Next up the food chain is a router.  Nothing unusual about it, except
that it has more ports than the el cheapo one you have in your
residence;-).  

Of course we need a radio network to get from the mobile platform to
anywhere else and there are several possibilities.  Since the ambulance
operation can be considered 'mission critical', high availability is a
driving factor.  And since radio is much more iffy than wireline, and
the ambulance can be expected to travel in and out of various footprints
in its travails ... you want more than one link to the wired internet.  
 - at least in the US, the most likely single link will be a cellphone
one -- the emergency services and Congress are off in the direction of
LTE over 700Mhz spectrum.  
 - in rural areas, wider coverage is necessary so at least some
ambulances may have satellite comms links.
 - at least some in [its] started with vehicle-vehicle daisy chains.  
 - it's plausible, in urban and suburban areas, to exploit the WiFi
access point density.  If we find one house in each city block with a
public-accessible AP, we'd probably have pretty solid coverage.  Close
cousin to the v2v above except it's handoffs instead.  

Ronald, does this answer your question?  There's an economic premium
with keeping the equipment in the back of the ambulance identical to
that in the ER/ICU. And never do with radio what you can do with wire
(e.g. LAN in your question).  Multi-homed hosts make no scale sense to
me.

The ambulance use case illustrates some related requirements.  Since
[its] does not seem to have defined its scope yet, you can't pooh-pooh
these as out of scope ... at least not yet;-)  
 - security.  End-to-end authenticity and confidentiality are required.
This is an application design issue (layer 6/7 and operating system).
In the ambulance case, at least within the US, we have the law (HIPPA)
on the same side of the issue.  
 - manageability.  This is also an app design issue; need SNMP agents in
end systems.  (the reason here, to be fair, has nothing to do with radio
but with the high availability needs).  
 - multicast.  A lot of emergency services data is multicast in nature
(and the bandwidth-limited radio links are, after all, shared media,
giving us multicast payoff exactly where we need it).  Emergency
services (and military) in general have much higher proportion of data
that is multicast in nature than an all-civilian setting.  But
all-civilian settings do need to be multicast-receive friendly (weather
warnings, for example). 

Radio characteristics.  We can't re-engineer the media -- God did that,
and as near as I can tell from Genesis, on the First Day.  It's four
orders of magnitude less capacious than wireline and its four orders of
magnitude noisier than wireline.  So we have to accommodate.  Thus my
previous comments about transport protocols:
 - consider the case of the ambulance transporting a casualty from a
rural area (suggesting satcom channel) to a suburban area where the
hospital is located (suggesting terrestrial radio-WAN of some sort).
Propagation time for the 4G cellular link will be in 10s of msec;
propagation time for the satcom link will be in 100s of msec.  Is it
reasonable to expect that packets will never arrive out of order?  Or
that we can jigger the underlying infrastructure so that doesn't
happen ... without infringing on a bunch of other desiderata?
 - IEEE 802.16 (at least) has a layer 2 NAK-retransmit algorithm.  This
would also cause packets to arrive in scrambled order ... over a single
link.
 I picked these observations because they will cause out-of-order
arrival, whether or not you have multiple-radio-hop daisy chains (which
I would expect to exacerbate).  The robustness exemplified by TCP has
been soft-pedaled as the internet migrated to lots more fiber optic, but
it's all back with radio.  

Topology.  Vehicle-to-vehicle daisy chains appear to present the most
volatile topology situations.  So I suspect that if we get that part
right (and [MANET] has that issue within its scope) then we probably
have a super-solution to the rest (rapid handoffs to fixed internet, for
example).  (Org comment: if topology is the only issue in [its] scope,
then some of the critics here are right -- [its] has nothing to do.)

On Mon, 2013-03-18 at 10:29 +0000, Velt, R. (Ronald) in 't wrote:
> Right. Can I ask what model is being assumed here? Is this entity with
> multiple radio interfaces supposed to be a router or a multi-homed
> host? I tend to think it is the former, with hosts residing on some
> in-vehicle (ethernet or CAN-bus based) link attached to it. If that's
> the case, one would have to perform some special tricks to make the
> hosts see multiple source addresses. 
> 
> Ronald in 't Velt

---

From: "Stan Ratliff (sratliff)" <[email-masked]>
Date: Mon, 18 Mar 2013 18:46:32 +0000

Rex, 

Excellent discussion. I agree with you 100%. 

Regards,
Stan

---

From: Alexandru Petrescu <[email-masked]>
Date: Tue, 19 Mar 2013 18:06:22 +0100

I read the email.

Yes, the ambulance scenario comes from you first - the instrumented
part, and from some EU project participants (the part of
ambulance-makes-way-thru-traffic).

I agree with many points described which are excellent network design
guidance.

If I can modify the Charter proposal accordingly - I will be more than
happy, I just need to know how.

I am trying to figure out whether the use of multiple interfaces is just
guidance, or whether there is some specific Item to work on for multiple
radios.  Or that the concept of multiple radios and wirelines is already
there deeply entrenched in the essence of IP addressing and routing.

Guidance is some text in the Charter proposal that describes in a
compressed way what you described below.  Would one offer such 3-line
text to add in the Charter proposal?  I just put this:
"Each moving vehicle may use disjoint and heterogeneous radio systems
(e.g. a vehicle employs two or more different radios which may be used
simultaneously, or alternatively)."

On another hand, an Item is typically a Working Group document (WG item)
to which effort could be committed.  For example, we may have an Item
document which contains Scenarios and Requirements text.  This has a
time tag associated - deliver this Item by that Month of Year.

Do you think we may have an Item about the use of multiple interfaces?
Do you think there would be people ready to put effort in authoring,
presenting such document?

I am interested in all that discussion about multiple interfaces.

That said, please see some replies below.

Le 18/03/2013 18:30, Rex Buddenberg a écrit :
> I contributed the 'instrumented ambulance' example to Alex's set of
> use cases, so let me expand it a bit.
>
> Ronald, you're right -- there should be a LAN (actually a good case
> for about three LANs) in an ambulance.  End systems such as casualty
> vital signs monitors, radionav receivers, voice handsets for both the
> EMT and the driver, etc, should be attached to the LAN.  Systems
> engineering sanity: keep them modularly separated from the radio
> issues.
>
> Next up the food chain is a router.  Nothing unusual about it,
> except that it has more ports than the el cheapo one you have in
> your residence;-).
>
> anywhere else and there are several possibilities.  Since the
> ambulance operation can be considered 'mission critical', high
> availability is a driving factor.  And since radio is much more iffy
> than wireline, and the ambulance can be expected to travel in and out
> of various footprints in its travails ... you want more than one link
> to the wired internet. - at least in the US, the most likely single
> link will be a cellphone one -- the emergency services and Congress
> are off in the direction of LTE over 700Mhz spectrum. - in rural
> areas, wider coverage is necessary so at least some ambulances may
> have satellite comms links. - at least some in [its] started with
> vehicle-vehicle daisy chains. - it's plausible, in urban and suburban
> areas, to exploit the WiFi access point density.  If we find one
> house in each city block with a public-accessible AP, we'd probably
> have pretty solid coverage.  Close cousin to the v2v above except
> it's handoffs instead.
>
> that in the ER/ICU. And never do with radio what you can do with
> wire (e.g. LAN in your question).  Multi-homed hosts make no scale
> sense to me.
>
>
> [its] does not seem to have defined its scope yet, you can't
> pooh-pooh these as out of scope ... at least not yet;-) - security.
> End-to-end authenticity and confidentiality are required. This is an
> application design issue (layer 6/7 and operating system). In the
> ambulance case, at least within the US, we have the law (HIPPA) on
> the same side of the issue. - manageability.  This is also an app
> design issue; need SNMP agents in end systems.  (the reason here, to
> be fair, has nothing to do with radio but with the high availability
> needs). - multicast.  A lot of emergency services data is multicast
> in nature (and the bandwidth-limited radio links are, after all,
> shared media, giving us multicast payoff exactly where we need it).
> Emergency services (and military) in general have much higher
> proportion of data that is multicast in nature than an all-civilian
> setting.  But all-civilian settings do need to be multicast-receive
> friendly (weather warnings, for example).
>
> Radio characteristics.  We can't re-engineer the media -- God did
> that, and as near as I can tell from Genesis, on the First Day.  It's
> four orders of magnitude less capacious than wireline and its four
> orders of magnitude noisier than wireline.  So we have to
> accommodate.  Thus my previous comments about transport protocols: -
> consider the case of the ambulance transporting a casualty from a
> reasonable to expect that packets will never arrive out of order?
> Or that we can jigger the underlying infrastructure so that doesn't
> happen ... without infringing on a bunch of other desiderata? - IEEE
> 802.16 (at least) has a layer 2 NAK-retransmit algorithm.  This would
> also cause packets to arrive in scrambled order ... over a single
> link. I picked these observations because they will cause
> out-of-order arrival, whether or not you have multiple-radio-hop
> daisy chains (which I would expect to exacerbate).  The robustness
> exemplified by TCP has been soft-pedaled as the internet migrated to
> lots more fiber optic, but it's all back with radio.
>
> Topology.  Vehicle-to-vehicle daisy chains appear to present the
> most volatile topology situations.  So I suspect that if we get that
> part right (and [MANET] has that issue within its scope) then we
> probably have a super-solution to the rest (rapid handoffs to fixed
> internet, for example).  (Org comment: if topology is the only issue
> in [its] scope, then some of the critics here are right -- [its] has
> nothing to do.)

No, topology is not the only  issue here.  We used topology as a
distinction to dissociate between n-hop and 1-hop protocols.  That may
lead to scenarios pertinent to MANET/RoLL vehicular protocols distinct
from scenarios pertinent to direct Vehicle-to-Vehicle communications.

For example, we may have an Item which says that we require MANET and/or
RoLL to deliver a n-hop routing protocol for V2V2V n-subnet topology -
that translates in a pure MANET/RoLL discussion.

And another Item which analyzes how to do 1-hop routing with IPv6 1-hop
protocols.  This is a pure non-MANET discussion.

But this deserves its separate thread.

I see beneficial to keep separate threads for: multiple interfaces
discussion, 1-hop routing discussion and n-hop routing discussion.

This is all open for discussion.

Alex

>>> -----Original Message----- From: [email-masked]
>>> [mailto:[email-masked]] On Behalf Of Wesley Eddy Sent:
>>> maandag 18 maart 2013 2:08 To: Kevin Lahey Cc: [email-masked]
>>> This requires the end-host to have multiple addresses visible to
>>> it, which could be a design constraint on the way the mobile
>>> network is setup.
>>
>> Right. Can I ask what model is being assumed here? Is this entity
>> with multiple radio interfaces supposed to be a router or a
>> multi-homed host? I tend to think it is the former, with hosts
>> residing on some in-vehicle (ethernet or CAN-bus based) link
>> attached to it. If that's the case, one would have to perform some
>> special tricks to make the hosts see multiple source addresses.
>>
>> Ronald in 't Velt This e-mail and its contents are subject to the
>> DISCLAIMER at http://www.tno.nl/emaildisclaimer
>>

---

From: Alexandru Petrescu <[email-masked]>
Date: Tue, 19 Mar 2013 18:15:58 +0100

> This is more of what I was getting at.  I was pretty active in mext,

YEs, I agree.  But Monami6 had a very specific problem - use multiple
addresses as CoAs (and not just one).  It was a very specific and
focused work.

If we do something about multiple interfaces here, I am trying to
identify what could be that very specific and focused work.  What
protocol should be extended?  Once I understand that, I think it's easy
to synthesize out the generic text to put in the Charter proposal.

> and a few other mobile-ip related groups working mobile networking.
> Some of the first code I used working the Cisco allowed you to use
> multiple egress interfaces, but only one at a time.  If the preferred
> link when down, the next best (as pre configured) link would be used.
> If I had 5 egress interfaces with 5 different radio systems, I could
> use any of these, but only one at a time.  I might have wired, wifi,
> 4G, UHF with priority in that order. Note, they all may be provided
> by different service providers.
>
> My question was do you want to be able to simultaneously use multiple
> link? If so, the problems becomes very interesting.  It also may be
> one wants specific traffic to use specific links.

Something along the lines of Mobile IPv6 Flow Mobility?  Use Mobile IPv6
messages to assign application flows to interfaces or to CoAs?

> Buy the way, I am currently working on spacesuit communications.
> Spacesuits are mini-spacecraft with full life support and such.  They
> really are an intelligent vehicle, but probably less intelligent than
> a car.  There is a current concept for the next generation to have
> two radios, a low-rate highly reliable radio and a wifi-type radio
> with large bandwidth.  Weather or no they would both be used
> simultaneously is an open question. Obviously, high-rate,
> high-resolution video would not be permitted over the UHF low-rate
> link.

Will - thank you.  The spacesuit is a relevant use-case.  I can see
interesting spacesuit-to-spacesuit needs of communication.

But, we already have two use-cases in the Charter proposal: ambulance
coupled to automating driving, and smart-city.  Do you think we should
add spacesuit use-case?  In the Charter proposal?  Or commit to
contribute to a draft about usecases and Scenarios?

Alex

> "Multi-Homed Mobile Networks," 67th IETF, nemo and monami working
> groups, November 9-10, 2006 San Diego, California ( Powerpoint ) file
> size 679,424 bytes
>
ami6-final.ppt

---

From: "Velt, R. (Ronald) in 't" <[email-masked]>
Date: Tue, 19 Mar 2013 17:32:21 +0000

Rex,  all,

(in-line)

>From: Rex Buddenberg [mailto:[email-masked]]
>Sent: maandag 18 maart 2013 18:31
>To: Velt, R. (Ronald) in 't
>Cc: Wesley Eddy; Kevin Lahey; [email-masked]
>Gee, I didn't think I'd have to defend this one.  Warning: this is a longer
>message, trying to roll up several questions you folks are asking.
>
>Ronald, you're right -- there should be a LAN (actually a good case for about
>three LANs) in an ambulance. 

I did not have your ambulance in mind. To me, ITS is more about improving traffic safety and traffic efficiency by means of data communications among vehicles and between vehicles and roadside infrastructure. Viewed that way, an interesting feature for an ambulance would its abilty to alert other road users that it is coming through before its audible / visible signals can be heard / seen. 

<snip>

>
>Ronald, does this answer your question?  

Erm... I was merely reacting to Wes Eddy's mention of MPTCP. It isn't obvious to me how MPTCP could exploit the availability of multiple radio paths, if the interfaces to those radios exist on the router as opposed to the MPTCP-capable host. I am aware of draft-wr-mptcp-single-homed-04. What is proposed there is what I would categorize under "special tricks". 

<snip>

Ronald

---

From: Rex Buddenberg <[email-masked]>
Date: Tue, 19 Mar 2013 09:38:27 -0800

Alex,

On Tue, 2013-03-19 at 18:06 +0100, Alexandru Petrescu wrote:
> trying to figure out whether the use of multiple interfaces is just
> radios.

The number of alternate routes is driven by the level of availability
required.  The more nines, the more interfaces.  That's a requirements
analysis and systems design issue, not a standards one.  

The standards issue is essentially a negative one -- don't do something
that would inhibit.  (requiring in-order packet delivery at layer 3
would probably do that).

---

From: Rex Buddenberg <[email-masked]>
Date: Tue, 19 Mar 2013 09:48:35 -0800

On Tue, 2013-03-19 at 17:32 +0000, Velt, R. (Ronald) in 't wrote:
> ITS is more about improving traffic safety and traffic efficiency by
> means of data communications among vehicles and between vehicles and
> roadside infrastructure. Viewed that way, an interesting feature for
> an ambulance would its abilty to alert other road users that it is
> coming through before its audible / visible signals can be heard /
> seen. 

I get to the same internetwork design using this (entirely legit)
variant of the use case.  

'Traffic efficiency' would include things like optimum routing and
traffic light control, which require highly available comms from the
platform through the terrestrial internet.  Security requirements are
similar -- authenticity of the data is critical (you should shudder to
consider spoofed traffic data implications).  

In internet-speak, same infrastructure, different app.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 19 Mar 2013 15:31:24 -0400

>>>>> "Rex" == Rex Buddenberg <[email-masked]> writes:
    Rex> The number of alternate routes is driven by the level of
    Rex> availability required.  The more nines, the more interfaces.
    Rex> That's a requirements analysis and systems design issue, not a
    Rex> standards one.

    Rex> The standards issue is essentially a negative one -- don't do
    Rex> something that would inhibit.  (requiring in-order packet
    Rex> delivery at layer 3 would probably do that).

Can we go back a bit to what is driving the need for so much data that
multiple radios are called for?

Would these radios be visible as seperate L3 interfaces? 

I'm asking because there are a number of candidate routing protocols
which would only use one interface for a given flow.  Many feel that
that might be a bug in the specification.

-- 
Michael Richardson <[email-masked]>, Sandelman Software Works

---

From: Rex Buddenberg <[email-masked]>
Date: Tue, 19 Mar 2013 12:47:23 -0800

Michael,

Here's the academic.  The first question to be addressed is 'what is the
availability requirement?'  Availability is defined as up time / total
time and is usually expressed as the 'number of nines'.  For instance,
99.99% availability pencils out to around 15 minutes of tolerable
downtime in a calendar quarter.  
  Since it's unlikely that you'll get 99.99% availability out of a
single radio-WAN (read interface to the router), you need altroutes.
The knee-jerk is to improve the number of 9s in a particular link. ....
but this is radio guys!

The second question is, once you've considered how much availability is
required, how to get it?  
  There are three principles of high availability engineering:
  - elimination of single points of failure.  In comms systems, this
principle usually means altroutes and backup power*.  
  - reliable crossover.  In multiply-threaded systems, the crossover
switch itself can become a single point of failure.
  - prompt detection of failures as they occur.  Or, in SNMP-speak,
fault management.  A user may never see a failure, but somebody needs to
in order to direct maintenance (or in the case of radio-WANs, fill in
coverage holes).

The first principle is a provisioning principle, it's technology-choice
independent.  As an example, it may mean stashing an emergency generator
in your garage for periods when grid power fails.  *in practice, once
you exceed 2 nines of Ao, you have to dual-thread the network because
nobody can fix equipment that fast. 

The second principle is at the heart of the questions you're asking.
First, it's moot if there are no altroutes.  In the internet, the basic
design of IP (RFC791, connectionless, stateless) meets this need.
Routers do reliable crossover all day, every day. Anything that
reintroduces state at layer 3 (or tries to do the crossover elsewhere,
such as layer 2 bridges) tends to defeat the purpose.  (for internet
history types, Paul Beran was right).  

The third principle is best exemplified by a failure scenario I observed
last summer: a city's emergency services (LMR) network failed because
one of its three towers failed.  The tower failed because an air
conditioner in the hut at the base of the tower failed, the comms gear
overheated and quit working.  It took hours for the city to triage the
failure, plus a couple more hours for the HVAC repairman to actually fix
the problem -- all system downtime.  

Most of the Ao problem is not a standards one.  The first principle is
provisioning, the third is instrumenting (you could also read that as
provisioning).  I'd contend that neither need standards that we don't
already have, but they do need follow-through and thorough
implementation (a 'best practices' issue?).  I'd also contend that the
second principle is one met in the internet, provided we don't do stupid
things (which include QoS controls that compromise the connectionless
state of IP).  

> > Would these radios be visible as seperate L3 interfaces? 
Yes.  Otherwise you wouldn't meet the second principle.  

> > that might be a bug in the specification.
I'd consider anything that chokes off perfectly usable altroutes as a
bug.  (At layer 2, the spanning tree algorithm does precisely that --
don't use it in high Ao situations).

---

From: Michael Richardson <[email-masked]>
Date: Wed, 20 Mar 2013 13:22:07 -0400

>>>>> "Rex" == Rex Buddenberg <[email-masked]> writes:
    Rex> The third principle is best exemplified by a failure scenario I
    Rex> observed last summer: a city's emergency services (LMR) network
    Rex> failed because one of its three towers failed.  The tower
    Rex> failed because an air conditioner in the hut at the base of the
    Rex> tower failed, the comms gear overheated and quit working.  It
    Rex> took hours for the city to triage the failure, plus a couple
    Rex> more hours for the HVAC repairman to actually fix the problem
    Rex> -- all system downtime.

So, why did it take hours to triage the failure?
My *bet* is that the failure wasn't visible at layer-3, so it couldn't
be seen.

    Rex> Most of the Ao problem is not a standards one.  The first
    Rex> principle is provisioning, the third is instrumenting (you
    Rex> could also read that as provisioning).  I'd contend that
    Rex> neither need standards that we don't already have, but they do
    Rex> need follow-through and thorough implementation (a 'best
    Rex> practices' issue?).  I'd also contend that the second principle
    Rex> is one met in the internet, provided we don't do stupid things
    Rex> (which include QoS controls that compromise the connectionless
    Rex> state of IP).

    >> > > Would these radios be visible as seperate L3 interfaces?

    Rex> Yes.  Otherwise you wouldn't meet the second principle.

Good, we agree.

    >> > I'm asking because there are a number of candidate routing
    >> protocols > which would only use one interface for a given flow.
    >> Many feel that > that might be a bug in the specification.

    Rex> I'd consider anything that chokes off perfectly usable
    Rex> altroutes as a bug.  (At layer 2, the spanning tree algorithm
    Rex> does precisely that -- don't use it in high Ao situations).

So, many of the routing protocols would give you the alternate route as
soon as one can see the primary has failed, but they will not, for a
single flow (for various definitions of flow and different protocols),
send packets out both radios, giving you increased bandwidth.

---

From: Rex Buddenberg <[email-masked]>
Date: Wed, 20 Mar 2013 10:53:43 -0700

Two issues in here:

in line...

> My *bet* is that the failure wasn't visible at layer-3, so it couldn't

First issue.  The failure was not visible at _any_ layer.  There was NO
fault monitoring system in place (other than, perhaps, some vendor
proprietary stuff for the comms gear itself).  

The solution that should suggest itself to you is to instrument
everything in the hut, a/c included, with SNMP agents, with an SNMP
console at the dispatcher (PSAP in public-safety-ese)

> send packets out both radios, giving you increased bandwidth.

Second issue.  From a high availability point of view, if an alternate
route exists in the routing table, that should meet the need ... whether
or not it is in use.  The outstanding question, though, is how does the
router know that the primary route has failed?  Traditionally by sensing
congestion -- buffer overflow.  This 'how' mechanism sticks capacity and
fault issues together -- it can't distinguish the two.  

I think it may be easy to get too cute.  One of my students a few years
ago, put an 802.16 transceiver on a boat, then sailed it away from it's
opposite number set up on the beach.  As the boat got offshore, it would
disappear into the trough, breaking the link for several seconds (enough
time for the .16 MAC to close the layer 2 connection).  When the boat
reappeared out of the trough, the pair would re-establish a layer 2
connection from scratch.  The traces that the student brought me clearly
showed the handshakes and dropouts.  
  The question that this trial should raise to you is at what point
would a router decide to kick in an altroute (and then kick it back
out)?  
  If the router is simply able to point traffic to any operating
interface that has buffer space to accept it, then the issue is moot.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 21 Mar 2013 10:16:32 -0400

>>>>> "Rex" == Rex Buddenberg <[email-masked]> writes:
    Rex> Second issue.  From a high availability point of view, if an alternate
    Rex> route exists in the routing table, that should meet the need
    Rex> ... whether 
    Rex> or not it is in use.  The outstanding question, though, is how
    Rex> does the 

So, I'm trying to distinguish the case where packets of the same
microflow arrive uninterrupted (this is usually the goal of internet
routing).

There is another form of uptime where a new TCP connection would
succeed, but it might involve a new set of prefixes, and a new route.

    Rex> The question that this trial should raise to you is at what point
    Rex> would a router decide to kick in an altroute (and then kick it back
    Rex> out)?  
    Rex> If the router is simply able to point traffic to any operating
    Rex> interface that has buffer space to accept it, then the issue is moot.  
There are bufferbloat (latency and jitter) issues with a naive approach
here, but it may well be the right answer.

Was the .16 system doing security, or just beacon/station-attachment.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Rex Buddenberg <[email-masked]>
Date: Thu, 21 Mar 2013 09:03:17 -0700

in line

> microflow arrive uninterrupted (this is usually the goal of internet

um...  what do you mean by microflow?  One TCP session, perhaps?  

If a vehicle is moving down the highway and shifting from one router to
another, but it's still the same awful youTube download, is that a
microflow?  By diverse routes.

> here, but it may well be the right answer.

One of the hardwon lessons in systems engineering (numerous backside
scars): Never discuss QoS issues before you 1) understand the
reach-to-mobile issues and 2) do the Ao homework.  

The internet, by default, maximizes the qos parameter of bandwidth
efficiency and trades off the qos parameters of latency, jitter,
interactivity to get it.  Any attempts to use QoS controls to change
this usually have unforeseen side effects that are usually all bad.
TANSTAFL. 

> Was the .16 system doing security, or just beacon/station-attachment.
The experiment was a simple how-far-does-it-go one.  But the obvious is
that when the link was up, it would haul packets.  None of the
non-default 802.16 security measures (e.g. keypair exchange) were turned
on.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 21 Mar 2013 12:41:14 -0400

>>>>> "Rex" == Rex Buddenberg <[email-masked]> writes:
    Rex> Second issue.  From a high availability point of view, if an
    Rex> alternate route exists in the routing table, that should meet
    Rex> the need ... whether or not it is in use.  The outstanding
    Rex> question, though, is how does the
    >> microflow arrive uninterrupted (this is usually the goal of
    >> internet routing).

    Rex> um...  what do you mean by microflow?  One TCP session,
    Rex> perhaps?

That's an example of a microflow, yes. 
Also includes a SIP, RTP, DNS request/reply, etc.

    Rex> If a vehicle is moving down the highway and shifting from one
    Rex> router to another, but it's still the same awful youTube
    Rex> download, is that a microflow?  By diverse routes.

Well.... some video downloaders actually notice TCP session failure,
implement happy eyeballs, and will restart a TCP connection, so it's a
bad example.

My point is that if initiating a new TCP session when the old one fails
succeeds, that may well resiliant enough for some application.

That's why I'm asking what the use case is here.

    >> beacon/station-attachment.

    Rex> The experiment was a simple how-far-does-it-go one.  But the
    Rex> obvious is that when the link was up, it would haul packets.
    Rex> None of the non-default 802.16 security measures (e.g. keypair
    Rex> exchange) were turned on.

okay, so this was just attachment/beacons work, not concern with trying
to rekey each time.