Subject: [ipwave] Adaptive RESTful Real-time Live Streaming for Things (A-REaLiST)'

From: Abhijan Bhattacharyya <[email-masked]>
Date: Thu, 24 Jan 2019 09:37:58 +0530

Dear list,
We submitted a draft in CoRE with a technical proposal for efficient transfer of time-series stream and presented in IETF Bangkok. The immediate use case that we considered was to transfer First Person View (FPV) video from a remote drone to a control unit over IP based infrastructure. We built it on CoAP leveraging some of CoAP's unique features. The ambition is to create a protocol stack that would be equally good in sending small sporadic bytes (sensor data) as well as streaming infinite time-series (like live video) while ensuring good QoE despite being low on resource consumption. 

The draft is available here: https://tools.ietf.org/html/draft-bhattacharyya-core-a-realist-00 

Probably, the use case that we had in mind and the technology we proposed, both can be extended to the purview of this WG.

Requesting your views on this draft please. If relevant then we can present this draft in Prague.

Here is the abstract for your ready reference:

   This draft presents extensions to Constrained Application Protocol
   (CoAP) to enable RESTful Real-time Live Streaming for improving the
   Quality of Experience (QoE) for delay-sensitive Internet of Things
   (IoT) applications. The overall architecture is termed ''Adaptive
   RESTful Real-time Live Streaming for Things (A-REaLiST)''. It is
   particularly designed for applications which rely on real-time
   augmented vision through live First Person View (FPV) feed from
   constrained remote agents like Unmanned Aerial Vehicle (UAV), etc.
   These extensions provide the necessary hooks to help solution
   designers ensure low-latency transfer of streams and, for contents
   like video, a quick recovery from freeze and corruption without
   incurring undue lag. A-REaLiST is an attempt to provide an
   integrated approach to maintain the balance amongst QoE, resource-
   efficiency and loss resilience. It provides the necessary hooks to
   optimize system performance by leveraging contextual intelligence
   inferred from instantaneous information segments in flight. These
   extensions equip CoAP with a standard for efficient RESTful
   streaming for Internet of Things (IoT) contrary to HTTP-streaming in
   conventional Internet.

Thank you. Looking forward.

P.S. The work has been accepted and presented in Wi-UAV workshop in Globecom, Dec,  2018. The public link to the paper is yet to be available.

With Best Regards
Abhijan Bhattacharyya
Consultant / Scientist,
{Internet Protocols | 5G | Standardization}, 
TCS Research,
Tata Consultancy Services
Building 1B,Ecospace
Plot -  IIF/12 ,New Town, Rajarhat,
Kolkata - 700160,West Bengal
India
Ph:- +91 33 66884691
Cell:- +919830468972 | +918583875003
Mailto: [email-masked]
Website: http://www.tcs.com
____________________________________________
Experience certainty.	IT Services
Business Solutions
Consulting
____________________________________________
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

---

From: Alexandre Petrescu <[email-masked]>
Date: Thu, 24 Jan 2019 12:17:48 +0100

Le 24/01/2019 à 05:07, Abhijan Bhattacharyya a écrit :
> Dear list,
> We submitted a draft in CoRE with a technical proposal for efficient 
> transfer of time-series stream and presented in IETF Bangkok. The 
> immediate use case that we considered was to transfer First Person View 
> (FPV) video from a remote drone to a control unit over IP based 
> infrastructure. We built it on CoAP leveraging some of CoAP's unique 
> features.

Hi,

Thank you for the message.

I have seen another drone from DLR that communicates with a van, but not 
using CoAP.  It uses TCP/IP family of protocols.

I understand the need to transfer video (First-Person View, FPV), but I 
do not understand how CoAP helps with video.

Video over IPv6 over OCB works fine between vehicles.

In the future, with the hopeful development of 802.11-NGV (Next 
Generation Vehicular), I expect latencies in the order of hundreds of 
micro-seconds, and 1Gbps bandwidths.  I think video over IP, and lidar 
streams over IP, would take advantage of that.

Why CoAP?

> The ambition is to create a protocol stack that would be 
> equally good in sending small sporadic bytes (sensor data) as well as 
> streaming infinite time-series (like live video) while ensuring good QoE 
> despite being low on resource consumption.

But such a multi-feature protocol is also claimed by Google with QUIC 
protocol.  Why not QUIC?

> The draft is available here: 
> https://tools.ietf.org/html/draft-bhattacharyya-core-a-realist-00
> 
> Probably, the use case that we had in mind and the technology we 
> proposed, both can be extended to the purview of this WG.
> 
> Requesting your views on this draft please.. If relevant then we can 
> present this draft in Prague.

I can look at the draft, but I need to understand why CoAP?

Video per se is not a sufficient reason to use CoAP, I think.

(and yes, I do agree there are strong IP requirements for video in 
drones, like multi-cameras, multiple laser, infrared, and others)

Alex

> P.S. The work has been accepted and presented in Wi-UAV workshop in 
> Globecom, Dec,  2018. The public link to the paper is yet to be available.
> 
> With Best Regards
> Abhijan Bhattacharyya
> Consultant / Scientist,
> Mailto: [email-masked] <mailto:[email-masked]>
> _______________________________________________
> its mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/its
>

---

From: Michael Richardson <[email-masked]>
Date: Thu, 24 Jan 2019 08:38:29 -0500

> I understand the need to transfer video (First-Person View, FPV), but I do
    > not understand how CoAP helps with video.

    > Video over IPv6 over OCB works fine between vehicles.

And SCTP should be easy to deploy too if there is losing frames is preferred
to winding up non-realtime.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Abhijan Bhattacharyya <[email-masked]>
Date: Thu, 31 Jan 2019 03:31:07 +0530

Hello Alex and Michael,
Thank you for your comments. May I please furnish my responses to both of you in this single mail.

<Alex>

>do not understand how CoAP helps with video.

I agree to your observation. Most of the FPV applications from drones are using TCP/IP as the transport.  Before we designed the solution (A-REaLiST) we had analysed the performance of some popular off-the-shelve drone for applications which needed real-time navigation decisions to be delivered to the drone from a controller base station sitting remotely. The controller decision engines used to work based on the FPV inputs from the drone. The drone uses some proprietary variation of H.264 encoded video that is delivered over TCP/IP. There had been several instances of long video freezes. We investigated. It seemed that TCP's state machine became a bottleneck. Also, the drone needs to manage the process memory (which is not too high) in order to buffer the contents in case of a lossy transmission so that TCP's reliable delivery mechanism can function properly.

It would not be really worthwhile to compare CoAP, which is a RESTful application layer protocol, with TCP/IP transport. Rather, our approach was a bit different as will be explained below.

>do not understand how CoAP helps with video.    

CoAP has been traditionally designed for lightweight transfer of small sensor data for constrained devices in LLN. So video may sound a bit wiered at first in the context of CoAP. However, video is nothing but a time-series information which needs to be rendered in real-time (and in strict real-time for FPV based remote decision engines). We found that CoAP has a unique feature that allows seamless switching of protocol semantics (incurring no additional cost on resources) between reliable and best-effort delivery while transmitting the information segments of the stream. So, depending on certain application driven criteria, you can classify some parts of your information as critical and some as non-critical. The critical segments are delivered reliably and the non-critical ones with best-effort. The "criticality" of an information segment is determined by the fact, whether that segment is essential to render the video at the receiver. A naive example: In a temporally compressed video stream, the I-frames may be considered as critical, while the p-frames and b-frames may be sent with best effort because the rendering engine essentially syncs with the I-frames. 

Moreover, our approach follows the tradition set by HTTP where video is considered just like any other web-traffic and HTTP-streaming  on TCP/IP (the de facto mechanism for OTT video delivery at this point) streams video over RESTful API calls. Same way, considering the requirements of LLN and constrained devices, we tried to extend CoAP so that we have a unified RESTful mechanism which is equally good in transferring small sensor information, control, telemetry and actuation information, as well as stream real-time time-series information like video. Being RESTful helps to seamlessly connect to other services on the web as well. 

>streams over IP, would take advantage of that.
 
As you may notice from the above, the proposed solution is agnostic of what you use at the lower layers. One thing that we kept in mind while conceiving this solution is that, you may have a radio link with a promised high bandwidth, but you may have unpredictable intermittent connectivity too due to random impairment / obstructions . As per our experience, drones flying within WiFi zone in an indoor environment also suffer from such unpredictable link behaviour due to sporadic zero-radio zones. This overpowers the promise of bandwidth. We tried to design the solution keeping those practical problems in mind.

>But such a multi-feature protocol is also claimed by Google with QUIC

I am not sure whether QUIC can cater the kind of flexibility we wanted at the application layer (please forgive my limited knowledge on this). Also, we wanted to create a solution which should be based on something that already has a strong foundation through standardization and wide adoption (may not be particularly in drones right now - but who knows!). Furthermore, while performing the state-of-the-art studies we encountered this paper: https://dl.acm.org/citation.cfm?id=3083175 ( Not so QUIC: A Performance Study of DASH over QUIC ). But, I do agree that further investigations is worth carrying out to check the performance over QUIC. But, perhaps that should not disqualify the proposed solution leveraging CoAP at this point.

>I can look at the draft, but I need to understand why CoAP?

It becomes a bit of a race-condition. :) We have explained this "why" in section 2 of the draft (https://tools.ietf.org/html/draft-bhattacharyya-core-a-realist-00#page-5) . But to look into the draft you keep the explanation as a pre-requisite. To address this deadlock, I tried to briefly explain the idea above in this mail. If you find it satisfactory, then may I please request you have a look at the link and scroll through the entire draft subsequently. You may also please go through page# 70-81 of this deck: https://datatracker.ietf.org/meeting/103/materials/slides-103-core-consolidated-slides-07 . We presented this in Bangkok.

>drones, like multi-cameras, multiple laser, infrared, and others)

Thank you. Probably there is an extensive scope to contribute.

</Alex>

<Michael>

>to winding up non-realtime.

First of all, we did not find much standard practices using SCTP for this kind of applications in recent implementations. Again, we are addressing the problem from the application-layer with a view of a generalized web infrastructure. Furthermore, if you may have noticed in the previous texts explaining the idea, losing frame is not the design concern. The concern is: That loss has to be an informed decision controlled by the application-layer. Also, if you may please have the opportunity to go through the draft, we are essentially proposing a rudimentary mechanism to marry application intelligence with protocol semantics.  

</Michael>

More views and comments will be more than welcome.

With Best Regards
Abhijan Bhattacharyya
Consultant / Scientist,

-----"its" <[email-masked]> wrote: -----

>To: [email-masked]
>From: Alexandre Petrescu 
>Sent by: "its" 
>Date: 01/24/2019 04:48PM
>Subject: Re: [ipwave] Adaptive RESTful Real-time Live Streaming for
>Things (A-REaLiST)'
>
>"External email. Open with Caution"
>
>
><mailto:[email-masked]>

---

From: Alexandre Petrescu <[email-masked]>
Date: Thu, 31 Jan 2019 12:31:51 +0100

Le 30/01/2019 à 23:01, Abhijan Bhattacharyya a écrit :
> [...] The drone uses some proprietary variation of H.264 encoded 
> video that is delivered over TCP/IP. There had been several
> instances of long video freezes. We investigated. It seemed that
> TCP's state machine became a bottleneck.

If TCP's state machine was a bottleneck, has one tried to use UDP
instead?

> Also, the drone needs to manage the process memory (which is not too
>  high) in order to buffer the contents in case of a lossy 
> transmission so that TCP's reliable delivery mechanism can function 
> properly.

Makes sense to have little memory on the drone because memory is heavy.

> It would not be really worthwhile to compare CoAP, which is a RESTful
> application layer protocol, with TCP/IP transport. Rather, our
> approach was a bit different as will be explained below.
> 
> />I understand the need to transfer video (First-Person View, FPV), 
> but/ />I / />do not understand how CoAP helps with video. /
> 
> CoAP has been traditionally designed for lightweight transfer of 
> small sensor data for constrained devices in LLN.

Does CoAP work on Ethernet?

Does CoAP work in an end-to-end manner or does it need protocol
conversion gateways?

I am asking because although a drone may carry just one computer, the
in-car receiver of the video is deep in the car network.  The gateway
that connects that car is end-to-end: it is a Router, it is IP-OBU.  Can
CoAP go through that router.

> So video may sound a bit wiered at first in the context of CoAP. 
> However, video is nothing but a time-series information which needs 
> to be rendered in real-time (and in strict real-time for FPV based 
> remote decision engines).

Video is very important in car networks.  Not only one stream, but
multiple streams.  There are at least 4 IP cameras in a small car.
There are other kinds of video-like streams, like lidar.

802.11-NGV can easily accomodate all of them, because it promisses more 
bandwidth than 802.11p.

> We found that CoAP has a unique feature that allows seamless
> switching of protocol semantics (incurring no additional cost on
> resources) between reliable and best-effort delivery while
> transmitting the information segments of the stream. So, depending on
> certain application driven criteria, you can classify some parts of
> your information as critical and some as non-critical. The critical
> segments are delivered reliably and the non-critical ones with
> best-effort. The "criticality" of an information segment is
> determined by the fact, whether that segment is essential to render
> the video at the receiver. A naive example: In a temporally
> compressed video stream, the I-frames may be considered as critical,
> while the p-frames and b-frames may be sent with best effort because
> the rendering engine essentially syncs with the I-frames.

The I-frames should be sent with a particular QoS characteristic on OCB.

OCB link will ensure the criticality that you need, not TCP.

The current text in the IPv6-over-OCB draft that may help I-frames is 
the following:>    The IPv6 packet transmitted on 802.11-OCB MUST be 
immediately
>    preceded by a Logical Link Control (LLC) header and an 802.11 header.
>    In the LLC header, and in accordance with the EtherType Protocol
>    Discrimination (EPD, see Appendix E), the value of the Type field
>    MUST be set to 0x86DD (IPv6).  In the 802.11 header, the value of the
>    Subtype sub-field in the Frame Control field MUST be set to 8 (i.e.
>    'QoS Data'); the value of the Traffic Identifier (TID) sub-field of
>    the QoS Control field of the 802.11 header MUST be set to binary 001
>    (i.e.  User Priority 'Background', QoS Access Category 'AC_BK').
---end-of-excerpt-from-draft.

> Moreover, our approach follows the tradition set by HTTP where video
>  is considered just like any other web-traffic and HTTP-streaming  on
>  TCP/IP (the de facto mechanism for OTT video delivery at this point)
>  streams video over RESTful API calls. Same way, considering the 
> requirements of LLN and constrained devices, we tried to extend CoAP
>  so that we have a unified RESTful mechanism which is equally good in
>  transferring small sensor information, control, telemetry and 
> actuation information, as well as stream real-time time-series 
> information like video. Being RESTful helps to seamlessly connect to
>  other services on the web as well.

I want to ask you: please use IPv6 for CoAP and RESTful.

> />Video over IPv6 over OCB works fine between vehicles.
>> 
>> micro-seconds, and 1Gbps bandwidths. I think video over IP, and 
>> lidar streams over IP, would take advantage of that./
> As you may notice from the above, the proposed solution is agnostic 
> of what you use at the lower layers. One thing that we kept in mind 
> while conceiving this solution is that, you may have a radio link 
> with a promised high bandwidth, but you may have unpredictable 
> intermittent connectivity too due to random impairment / obstructions
> . As per our experience, drones flying within WiFi zone in an indoor
> environment also suffer from such unpredictable link behaviour due to
> sporadic zero-radio zones. This overpowers the promise of bandwidth.
> We tried to design the solution keeping those practical problems in
> mind.

The solution looks interesting.

> />But such a multi-feature protocol is also claimed by Google with 
> QUIC
>> 
>> protocol. Why not QUIC?/
> 
> I am not sure whether QUIC can cater the kind of flexibility we 
> wanted at the application layer (please forgive my limited knowledge
>  on this). Also, we wanted to create a solution which should be based
>  on something that already has a strong foundation through 
> standardization and wide adoption (may not be particularly in drones
>  right now - but who knows!). Furthermore, while performing the 
> state-of-the-art studies we encountered this paper: 
> https://dl.acm.org/citation.cfm?id=3083175 ( Not so QUIC: A 
> Performance Study of DASH over QUIC ). But, I do agree that further 
> investigations is worth carrying out to check the performance over 
> QUIC. But, perhaps that should not disqualify the proposed solution 
> leveraging CoAP at this point.

I am not an expert of QUIC either.

I am not here to disqualify anything.  I appreciate the advantages of 
solution.

It would be good if CoAP trials run on IPv6, CoAP to not rely on 
protocol gateways, and CoAP to map to OCB QoS classes.

> />I can look at the draft, but I need to understand why CoAP?/
> 
> It becomes a bit of a race-condition. :) We have explained this "why"
> in section 2 of the draft 
> (https://tools.ietf.org/html/draft-bhattacharyya-core-a-realist-00#page-5)

Thank you, I clicked and looked over it quickly.  I may have some comments.

Then I searched for the keyword 'IPv6' in the draft.

 > But to look into the draft you keep the explanation as a
> pre-requisite. To address this deadlock, I tried to briefly explain 
> the idea above in this mail. If you find it satisfactory, then may I
>  please request you have a look at the link and scroll through the 
> entire draft subsequently. You may also please go through page# 70-81
> of this deck: 
> https://datatracker.ietf.org/meeting/103/materials/slides-103-core-consolidated-slides-07

If you add 'IPv6' considerations to it, then I will comment on it.

Alex

. We presented this in Bangkok.
> 
> />(and yes, I do agree there are strong IP requirements for video in
> /
> *</Alex>*
> *<Michael>*
> />And SCTP should be easy to deploy too if there is losing frames is
>> preferred to winding up non-realtime.
> /
> 
> First of all, we did not find much standard practices using SCTP for
>  this kind of applications in recent implementations. Again, we are 
> addressing the problem from the application-layer with a view of a 
> generalized web infrastructure. Furthermore, if you may have noticed
>  in the previous texts explaining the idea, losing frame is not the 
> design concern. The concern is: That loss has to be an informed 
> decision controlled by the application-layer. Also, if you may please
> have the opportunity to go through the draft, we are essentially
> proposing a rudimentary mechanism to marry application intelligence
> with protocol semantics.
> 
> *</Michael>*
> 
> With Best Regards Abhijan Bhattacharyya Consultant / Scientist, 
> {Internet Protocols | 5G | Standardization}, TCS Research, Tata 
> Consultancy Services Building 1B,Ecospace Plot -  IIF/12 ,New Town, 
> Rajarhat, Kolkata - 700160,West Bengal India Ph:- +91 33 66884691 
> Cell:- +919830468972 | +918583875003 Mailto: 
> [email-masked] <mailto:[email-masked]> 
> ____________________________________________ Experience certainty. IT
> Services Business Solutions Consulting 
> -----"its" <[email-masked] <mailto:[email-masked]>> 
> wrote: -----
> 
>> To: [email-masked] <mailto:[email-masked]> From: Alexandre Petrescu 
>> Sent by: "its" Date: 01/24/2019 04:48PM Subject: Re: [ipwave] 
>> Adaptive RESTful Real-time Live Streaming for Things (A-REaLiST)'
>> 
>>> Dear list, We submitted a draft in CoRE with a technical proposal
>>> for
>> efficient
>>> transfer of time-series stream and presented in IETF Bangkok.
>>> The immediate use case that we considered was to transfer First 
>>> Person
>> View
>> I have seen another drone from DLR that communicates with a van, 
>> but not using CoAP. It uses TCP/IP family of protocols.
>> 
>> I understand the need to transfer video (First-Person View, FPV), 
>> but I do not understand how CoAP helps with video.
>> 
>> micro-seconds, and 1Gbps bandwidths. I think video over IP, and 
>> lidar streams over IP, would take advantage of that.
>> 
>> Why CoAP?
>> 
>>> The ambition is to create a protocol stack that would be equally
>>>  good in sending small sporadic bytes (sensor data) as well
>> as
>>> With Best Regards Abhijan Bhattacharyya Consultant / Scientist, 
>>> {Internet Protocols | 5G | Standardization}, TCS Research, Tata 
>>> Consultancy Services Building 1B,Ecospace Plot -  IIF/12 ,New 
>>> Town, Rajarhat, Kolkata - 700160,West Bengal India Ph:- +91 33 
>>> 66884691 Cell:- +919830468972 | +918583875003 Mailto: 
>>> [email-masked]
>>> ____________________________________________ Experience 
>>> certainty. IT Services Business Solutions Consulting 
>>> =====-----=====-----===== Notice: The information contained in 
>>> this e-mail message and/or attachments to it may contain 
>>> confidential or privileged information. If you are not the 
>>> intended recipient, any dissemination, use, review, distribution,
>>> printing or copying of the information contained in this e-mail
>>> message and/or attachments to it are strictly prohibited. If you
>>> have received this communication in error, please notify us by
>>> reply e-mail or telephone and immediately and permanently delete
>>> the message and any attachments. Thank you
>>> 
>>> 
>>> _______________________________________________ its mailing list 
>>> [email-masked] <mailto:[email-masked]> 
>> _______________________________________________ its mailing list 
>> [email-masked] <mailto:[email-masked]>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 31 Jan 2019 13:41:42 +0100

> If TCP's state machine was a bottleneck, has one tried to use UDP

I think that is indeed one of the advantages CoAP brings go the table here.

> Does CoAP work on Ethernet?

CoAP was designed to be able to run on UDP, which is on IP which in turn works very well on Ethernet.

> Does CoAP work in an end-to-end manner or does it need protocol

I works end-to-end (as long as your network doesn’t break UDP).

> I want to ask you: please use IPv6 for CoAP and RESTful.

CoAP was designed to work well over IPv6 (but works as well over IPv4).

> Then I searched for the keyword ‘IPv6' in the draft.
> […]
> If you add ‘IPv6' considerations to it, then I will comment on it.

For a CoAP application such as A-REaLiST, IPv6 makes little difference (beyond being able to assign addresses to both ends in the first place), so I don’t know there is a lot to say.

Grüße, Carsten