Subject: Re: [Simple] Review of draft-ietf-simple-message-sessions-08 - Byte Ranges in REPORTs

From: Ben Campbell <[email-masked]>
Date: Fri, 1 Oct 2004 17:16:52 -0500

Oops, I just realized I missed one of Paul's comments.

Paul suggests there is no value saying what part of a message fails, 
only that the whole message fails. I disagree. If a sender learns that 
a particular byte-range failed, it may be able to send a new chunk 
containing that byte range.

On Oct 1, 2004, at 5:15 PM, Ben Campbell wrote:

> Sorry for the late response. Somehow my mail hid this entire thread 
> from me until a couple of days ago. Else I just forgot how to use 
> computers. Comments inline.
>
> On Sep 24, 2004, at 11:20 AM, Paul Kyzivat wrote:
>
>> I am really confused about how byte ranges are supposed to work in
>> reports, and I suspect we are permitting a bunch of nonsense.
>>
>> Fundamentally, a message can only be considered a success if all its
>> bytes are received successfully. If it is to be positively 
>> acknowledged
>>   with success reports, then there must be success reports covering
>> every byte. Is there any point in sending a success report for less 
>> than
>> the entire message? It doesn't help the receiver, and actually makes
>> more work because it then requires keeping track of what bytes have 
>> been
>> confirmed. And it doesn't ease the job of the sender, who could easily
>> just wait until all has been received before sending the success 
>> report.
>>
>> The flip side is that a message fails if a single byte fails. At that
>> point it does no good to know that other parts succeeded or failed. So
>> while it makes sense to send a failure report based on problems with a
>> single chunk, there is little value in saying which byte range failed.
>>
>> Bottom line: unless I have missed something, there is no reason for
>> Byte-Range in REPORT or responses. A success report should only be 
>> sent
>> after all the chunks of a message have been received. A failure 
>> report,
>> or an error status, can be sent at the time the failure of a single
>> chunk is known, but need not identify the byte range. (In that case,
>> maybe the Byte-Range could be optional for debugging purposes only.)
>>
>
> I agree you need reports that cover the entirety of the message for a 
> success. I further agree that endpoints should not send success 
> reports until they have the entire message. I wonder, however, if 
> there are situations where this is not possible. For example, if a 
> message crosses a gateway, and somehow different gateways send 
> different chunks. But, since said gateway will need to reconstruct the 
> entire message to pass on, this probably cannot happen.
>
> So I am willing to assert that you only send success reports on the 
> entire message. Do we think there is any need for the sender to be 
> able to accumulate success reports on subsets of a message?
>
> Just to make sure we are clear, though, I think _failure_ reports are 
> sent on a per-chunk basis. If an endpoint decides to send a failure 
> report because it is missing a data range, it should report on that 
> specific data range.
>
>>
>>
>>
>> _______________________________________________
>> Simple mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/simple
>>
>

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Ben Campbell <[email-masked]>
Date: Fri, 1 Oct 2004 17:15:12 -0500

On Sep 24, 2004, at 11:20 AM, Paul Kyzivat wrote:

> bytes are received successfully. If it is to be positively acknowledged
> Byte-Range in REPORT or responses. A success report should only be sent
> after all the chunks of a message have been received. A failure report,

I agree you need reports that cover the entirety of the message for a 
success. I further agree that endpoints should not send success reports 
until they have the entire message. I wonder, however, if there are 
situations where this is not possible. For example, if a message 
crosses a gateway, and somehow different gateways send different 
chunks. But, since said gateway will need to reconstruct the entire 
message to pass on, this probably cannot happen.

So I am willing to assert that you only send success reports on the 
entire message. Do we think there is any need for the sender to be able 
to accumulate success reports on subsets of a message?

---

From: Paul Kyzivat <[email-masked]>
Date: Sun, 03 Oct 2004 11:20:01 -0400

Ben,

Learning about failure of a chunk is only of value if there is something 
that you can and should do about it that you wouldn't do based on 
knowing about the failure of the whole message.

You suggest that you might retransmit the chunk. Question is whether 
there is any value/point in doing this. At least point to point there 
seems to be no reason, since the underlying protocol is reliable. Maybe 
it could make sense in the presence of relays, or in cases when a 
connection fails and is reestablished.

Expecting this kind of recovery will then impose added responsibilities 
on senders.

I guess I could go either way on this. A problem with using Byte-Range 
with REPORT is that the semantics of Byte-Range is highly variable 
depending on the message it appears in. In SEND it applies to the body 
of the SEND. In REPORT, even though REPORT can have its own body, the 
Byte-Range does not apply to that, but instead applies to the SEND the 
REPORT references. This is perhaps OK, but then I think we need to 
clarify the definitions to make this crystal clear. We then also ought 
to spell out clearly what we expect a sender to do when receiving a 
failure report with a Byte-Range.

	Paul

Ben Campbell wrote:
> only that the whole message fails. I disagree. If a sender learns that a 
> particular byte-range failed, it may be able to send a new chunk 
>>> every byte. Is there any point in sending a success report for less than
>>> more work because it then requires keeping track of what bytes have been
>>> just wait until all has been received before sending the success report.
>>>

---

From: Aki Niemi <[email-masked]>
Date: Tue, 05 Oct 2004 15:09:15 +0300

> containing that byte range.

I don't get this. We have a protocol that runs on top of a TCP
connection, or a chain of TCP connections. The only way a chunk can ever
get lost without anyone noticing it hop-by-hop is if someone's input
buffer or output buffer is seriously leaking bits. Do we really need to
design around that hypothetical condition?

Cheers,
Aki

---

From: Ben Campbell <[email-masked]>
Date: Tue, 5 Oct 2004 17:32:03 -0500

> design around that hypothetical condition?

On reflection, I think I am mostly convinced that you are right. The 
only case I can think of  is when you successfully send a bunch of 
chunks, the session fails, you signal a new session, and attempt to 
resend the content. One could be saved the task of sending all the 
parts that had already been delivered. But I think I agree that this 
benefit is not worth the additional complexity and statefulness 
necessary to make it work.

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 05 Oct 2004 18:58:43 -0400

Ben Campbell wrote:
> resend the content. One could be saved the task of sending all the parts 
> that had already been delivered. But I think I agree that this benefit 
> is not worth the additional complexity and statefulness necessary to 
> make it work.

The significant case now occurs to me:

- we are sending that 7gb Lord of the Rings as one message,
   and part way through the tcp connection fails somewhere
   along the way. (Maybe a relay crashed.)

- it would be good not to have to retransmit the part that
   got there. Would prefer to establish a replacement
   session and just send the part we aren't sure was received.

To make this work, the sender needs to have received success reports for 
fragments. Maybe this is worth supporting them.

Of course the recipient is still free to just send a single success 
report at the end, or it can periodically send reports for arbitrary 
byte ranges that it has received ok. These wouldn't have to correspond 
to the actual chunking of the message. One could simple be sent every so 
often, reflecting what has been received to date.

So I think I have talked myself back into liking this for success reports.

I'm still not convinced about the value in failure reports, but if the 
range can be there for one then why not the other?

	Paul

---

From: Ben Campbell <[email-masked]>
Date: Thu, 7 Oct 2004 20:07:45 -0700

On Oct 5, 2004, at 3:58 PM, Paul Kyzivat wrote:

> To make this work, the sender needs to have received success reports 
> for fragments. Maybe this is worth supporting them.
>
> to the actual chunking of the message. One could simple be sent every 
> so often, reflecting what has been received to date.
>

I had a conversation with Cullen, where he expressed the same opinion 
for failure reports. Imagine, in your LoTR example, you were using a 
relay, and it has a transport failure between itself and the next hop. 
It sends you one or more failure reports for the chunks for which it 
could not confirm delivery. You establish a new session, and continue; 
resending the failed chunks.

I'd like to close on this quickly, so I offer the following questions 
to anyone who cares. I would appreciate opinions asap, so I can get 
this into the next revision.

1) Should failure reports be sent per chunk or per complete message? (I 
think it is per chunk.)

2) Should success reports be sent per chunk or per complete message? 
Note that, if we send them per-chunk, then the sender must accumulate 1 
or more reports covering all the bytes in the message before it can 
consider the message successful. These reports may or may not 
correspond one-to-one with the chunks it sent, as a relay may re-chunk.

(Again, I vote per-chunk.)

3) Do we need to say anything about how long the _receiver_ holds onto 
the chunks of an incomplete message in hopes any remaining chunks 
arrive? In particular, do we need to talk about this for holding chunks 
after a session fails, in case the sender establishes a new session and 
sends the remaining chunks?

(I vote that we say no more than we already did in 08 concerning 
re-sending chunks on new sessions.)

4) Do we need to enable the receiver to send a failure report about 
_missing_ chunks? This does not make sense with success reports 
requested, but might make sense when only failure reports are 
requested.

(I vote no, as I think that, if failure reports are requested, any such 
failure will be discovered and reported by some up-stream device except 
in the most rare of corner cases. The exception is when the sender does 
not actually send all the chunks; but if that happens it doesn not need 
to be told about it by another device.)

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 08 Oct 2004 06:45:36 -0700

I think that the case where my notebook is on wireless then docks to wired
Ethernet connection and turns off it's wireless helps answer these. inline

> 1) Should failure reports be sent per chunk or per complete message? (I

sort of both - yes they have to be per chunk so the chunk had a fialure in
transmission but the message is fine. If the whole message is going to be
bad, because of something like the body type is not understood, then they
are sent on the whole message. You indicate they are on the whole message by
using a * in the byterange

> (Again, I vote per-chunk.)
I think we have to have a success for the whole message or there is no way
to know if everything has arrived - questions to me is do we also need to
have chunk by chunk acknowledgments along the way. (This is all assuming
that positive reports where requested). I'm not sure I see the reason that
these would be needed but I feel like I am forgetting something.

> sends the remaining chunks?
No, I think this is like how long the end to end timer should be - it is
totally situation dependent. Probably needs to say something along lines off
"for awhile"
 
> requested, but might make sense when only failure reports are
I don't think so - if the sender cared about the data, someone will sent a
negative at the appropriate time. The chunks can arrive out of order and
this makes it hard for the receiver to know when to request a missing block.

---

From: David R Oran <[email-masked]>
Date: Fri, 8 Oct 2004 10:00:54 -0400

Based on this exchange, I suggest we rename the protocol to 
MHRFTPWCBYFIM.

(Multi hop recoverable file transfer protocol which could be used for 
instant messaging).

I am getting creeping featurism vibes here, guys...

Dave.

On Oct 7, 2004, at 11:07 PM, Ben Campbell wrote:

>> byte ranges that it has received ok. These wouldn't have to 
>> correspond to the actual chunking of the message. One could simple be 
>> sent every so often, reflecting what has been received to date.
>>
>> I'm still not convinced about the value in failure reports, but if 
>> the range can be there for one then why not the other?
>>
>
> 1) Should failure reports be sent per chunk or per complete message? 
> (I think it is per chunk.)
>
> Note that, if we send them per-chunk, then the sender must accumulate 
> 1 or more reports covering all the bytes in the message before it can 
> arrive? In particular, do we need to talk about this for holding 
> chunks after a session fails, in case the sender establishes a new 
> session and sends the remaining chunks?
>
> (I vote no, as I think that, if failure reports are requested, any 
> such failure will be discovered and reported by some up-stream device 
> except in the most rare of corner cases. The exception is when the 
> sender does not actually send all the chunks; but if that happens it 
> doesn not need to be told about it by another device.)
>
>
>> 	Paul
>
>
David R. Oran
Cisco Fellow
Cisco Systems
7 Ladyslipper Lane
Acton, MA 01720 USA
Tel: +1 978 264 2048
Email: [email-masked]

---

From: Aki Niemi <[email-masked]>
Date: Fri, 08 Oct 2004 17:16:20 +0300

On Fri, 2004-10-08 at 06:07, ext Ben Campbell wrote:
...snip...
> resending the failed chunks.

I think you're assuming that the other end failed gracefully, and that
the previously sent chunks (and the context in which they were sent)
still exist in this new session.

If you were uploading the file to my device and it barfed because of a
faulty device driver, once the UA is back up again, it will probably
have no idea what to do with the chunks you are sending it.

So at a minimum, the "new session" in this case should be a re-INVITEd
one, so that if indeed the other end has lost the context, it can fail
with a non-existent dialog error.

> 1) Should failure reports be sent per chunk or per complete message? (I 

I think per chunk, but then again, there are already the hop-by-hop
reports being sent per chunk, so I'm confused...

> correspond one-to-one with the chunks it sent, as a relay may re-chunk.

Same as above. I don't understand the model here -- the three levels of
ACKs worry me: TCP, hop-by-hop, and end-to-end. Especially as they seem
to really only be ACKs -- I can't for example push back on the relay
before me if chuncks are coming in too fast...

> sends the remaining chunks?

No. If I "Save As" the transmitted file, and decide to "Save As" it
again, it could be there *very* long. 

I think it is important to document how the two ends find out whether
the context of the old session still exists (whatever that means).
Something like "4xx Byte-range unknown" to a SEND, or 481 to the
re-INVITE, etc. 

> requested, but might make sense when only failure reports are 

No. If a chunk goes missing, the hop-by-hop mechanisms MUST be able to
deal with it.

Cheers,
Aki

---

From: Aki Niemi <[email-masked]>
Date: Fri, 08 Oct 2004 17:18:25 +0300

On Fri, 2004-10-08 at 16:45, ext Cullen Jennings wrote:
...snip...
> this makes it hard for the receiver to know when to request a missing block.

How can they ever arrive out-of-order? Isn't TCP supposed to guarantee
in-order delivery?

Cheers,
Aki

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 08 Oct 2004 10:39:46 -0400

> using a * in the byterange

This is confusing. The failure reports are really for byte ranges, not 
chunks, since there is no consistent understanding by all parties of 
what the chunking is. And the failure reports aren't necessarily all 
sent by the same node.

It seems that if failure reports for byte ranges are supported at all, 
then the sender will need to in effect keep a separate status for every 
byte sent. The status for each byte is one of:
- unknown
- failure reported
- success reported

(Obviously there are optimizations on how to store this status for a 
message of many bytes.) It appears that there could be at least as many 
reasons for overlapping byte ranges in reports as in sends, so that 
needs to be handled as well.

Then the sender will need to decide what to do if failure is reported 
for one or more bytes. It could decide to just give up on the message 
and stop sending. Or it could just retransmit a chunk containing the bad 
byte(s). (For that to be useful, it would have to assume that some relay 
was at fault and will recover.) Or it could negotiate a replacement 
session and then send a chunk containing the bad byte(s).

This is starting to look very ugly.

> these would be needed but I feel like I am forgetting something.

Success reports only make sense end to end. Ultimately you need 
confirmation of success of the *whole* message. So if byte ranges are 
used then the sum total of them must cover the entire message.

This is complicated by the fact that REPORTs aren't confirmed, and so 
might be lost. (E.g. by a relay.)

If byte ranges in success reports are to be used, then I think the best 
strategy might be for the receiving node to send cumulative ones. For 
instance, send one for the largest range of bytes received that includes 
previously unconfirmed bytes. (With in-order delivery this would mean 
that each report would cover everything received so far.) These reports 
might not be sent for every chunk received - just every so often.

This is of course just as complicated for the sender as the failure 
reports above, and more complicated for the receiver.

> totally situation dependent. Probably needs to say something along lines off

I am inclined to agree.

> this makes it hard for the receiver to know when to request a missing block.

I agree this doesn't make sense.

Because of the complexity that is now apparent, I only see a couple of 
useful ways forward:

- Specify how to handle recovery, driven by byte ranges in responses.

- Put off recovery to a future draft. Remove support for byte ranges
   in reports. But figure out how we could gracefully incorporate that
   future draft and migrate to support of it.

- Entirely abandon any thought of recovery in MSRP. Remove support for
   byte ranges from reports. Leave it for a replacement protocol.

I think leaving byte ranges in reports without specifying how to use 
them for recovery will result in major interoperability problems.

I am of mixed thought about which of those alternative I prefer. Given 
the time pressures and priorities, I could easily be convinced to 
abandon any thought of recovery, though I am not happy about it.

	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 08 Oct 2004 10:45:58 -0400

> How can they ever arrive out-of-order? Isn't TCP supposed to guarantee

They could get re-ordered by a relay. We haven't forbidden that. (Maybe 
we should have.)

	Paul

---

From: Aki Niemi <[email-masked]>
Date: Fri, 08 Oct 2004 18:00:52 +0300

> They could get re-ordered by a relay. We haven't forbidden that. (Maybe 

I thought this was assumed already. But if not, I think we definitely
should forbid reordering. I don't think we gain anything by allowing it,
but introduce a lot of complexity instead. Further more, it makes no
sense to use TCP and still have to cope with reordering of packets,
retransmissions and such in the application. 

That'd clearly be "tårta på tårta" as they say here... ;-)

CHeers,
Aki

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 08 Oct 2004 09:44:27 -0700

> How can they ever arrive out-of-order? Isn't TCP supposed to guarantee

If one of the relays has more than one TCP connection to the next relay, the
chunk on one of these path's may get ahead of the chunk on the other path.

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 08 Oct 2004 09:57:43 -0700

I think Paul clarified this for me. The semantics of the protocol are that
reports are for the byterange specified in the report. No more or less is
implied on how this range lines up with any chunk or message.

If someone has asked for reports that a message succeeded, you can't do that
till you have all the message so you don't send a SUCCESS report until you
have all the bytes in the message.

If you know some bytes have failed, and the sender has asked for failure
reports, you send that as soon as you know which bytes have failed. You
might later send another one that other bytes have failed. The byte ranges
in a FAILURE report will correspond to some chunk somewhere but that may not
correspond to any chunk that the first sender sent since an intermediate
relay may have chunked differently.

The issue of tracking which bytes you have received and not received already
existed as soon as we did chunks even if there was no reporting. This is not
hard to implement with linked lists of chunks.

When you receive a negative report on a byterange. You renegotiate the
connection and resend that chunk.

As some level this is complicated - as soon as we said that we wanted a
system that multiplexed message onto one TCP transport, we choose this level
of complexity. None of this is that complicated - we have been around this
over and over for the last 1.5 years, if we remove any part of this system
we break requirements that people said were absolutely critical to them.

---

From: Aki Niemi <[email-masked]>
Date: Fri, 08 Oct 2004 20:06:58 +0300

> chunk on one of these path's may get ahead of the chunk on the other path. 

I have to say I hate this. I've also never seen a requirement for the
ability to demux messages from a single connection onto man connections.
Can you help me understand where this requirement is coming from?

Cheers,
Aki

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 08 Oct 2004 11:11:26 -0700

> Can you help me understand where this requirement is coming from?

I think it came up over the traffic between MSN and Yahoo is not all going
to flow from host to another, it is going to flow from a group of machines
to group of other machines so individual chunks of the same message may not
all end up following exactly the same set of TCP connections

You can also get in the same problem if you transmit a few chunks that work
to one relay, the session fails, then you start a new session on a relay
that is faster than the original one.

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 08 Oct 2004 18:51:04 -0400

> have all the bytes in the message.

You *can* do it on a per-byte-range basis. There is a question of *why* 
you would do that - it would only be because it was of some value in 
error recovery.

> relay may have chunked differently.

Right.

> hard to implement with linked lists of chunks.

Yes. But if you want to send partial success reports, there is some 
complexity in deciding when to do so, and what range to include in it.

We haven't mentioned the issue, but if we can lose chunks of message, we 
can probably lose REPORTs too. If there is known to be only one success 
report, and it fails to come, the sender can time and declare a failure.

If there are multiple success reports for disjoint byte ranges, and you 
don't get them all, then what? I guess you wait until you have sent 
everything once, and then timed out waiting for all the success reports, 
and then you could try retransmitting the ranges that haven't been 
confirmed. But then what? Do you go thru another timeout cycle?

> connection and resend that chunk.

That works for negative ones, but not the positive kind. But negative 
ones are even less reliable. If things aren't working well so that 
chunks are being lost, what is chance that the failure will be lost too?

Another question that comes to mind: Suppose I have sent bytes 
1-1,000,000. I get a failure report for bytes 500,000-501,000. If I 
renegotiate a new connection, right away, and close the old one, what do 
I retransmit? Just 500,000-501,000, or everything starting from 500,000? 
There could be failure reports for other byte ranges that I would 
receive if I kept listening to the old connection - even for earlier 
byte ranges than the one I received. If I kill the connection and start 
another then I won't get those. I can just retransmit starting from 
500,000 and hope that nothing earlier failed too.

> we break requirements that people said were absolutely critical to them.

I agree that some scheme like this is ultimately not that complicated.
But it isn't written down anywhere. The question is whether we want to 
take the time to write it all down.

The various combinations of success and failure reports with and without 
byte ranges at least makes a lot of cases to explain and for people to 
understand and implement correctly.

Forbidding byte ranges in success reports would at least prune away a 
few of the options.

	Paul

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 11 Oct 2004 17:28:08 +0200

> complexity in deciding when to do so, and what range to include in it.

I am not proposing the client do anything with success reports other than
perhaps be able to discard this data or know that it has been sent. I am not
proposing that they be used in any way to figure out what needs to be
retransmitted.
  
> report, and it fails to come, the sender can time and declare a failure.

Yes, but reports are in a single message and never fragmented. This is why
we need to ACK them

> 500,000 and hope that nothing earlier failed too.

You continue to transmit things that you have not yet transmitted AND you
retransmit things you get a negative report for.

> Forbidding byte ranges in success reports would at least prune away a

just saying "success reports are not used to decide what needs to be
retransmitted"  seems like a better plan

---

From: Ben Campbell <[email-masked]>
Date: Tue, 12 Oct 2004 20:30:25 -0500

> Can you help me understand where this requirement is coming from?

I think the real problem is not that we want to go out of our way to 
enable such behavior, but do not want to add the complexity necessary 
to ensure it never happens. I can think of some failure-recovery 
conditions that might cause out-of-order delivery.