Subject: [Taps] Review of draft-trammell-taps-post-sockets-03

From: Michael Welzl <[email-masked]>
Date: Sat, 11 Nov 2017 09:43:51 +0800

Dear all,

Because of the planned request for draft-trammell-taps-post-sockets-03 to become a WG item, I gave the draft another close read and decided to write a review based on this. I hope it's useful.

To me, post-sockets aims in the right direction - I like a number of things  about it: the introduction of higher-layer concepts such as the carrier and association; I suspect that the long-term association management could also be implemented one-sided, which is nice. I like the idea of "forking" a carrier, which would clearly translate into doing multi-streaming "under the hood" whenever that's possible (side note: from the list on pages 7/8, it seems that *only* "Listener" can be forked. Why not a Source? This seems odd to me). Finally, obviously I also agree that being message-oriented is the right way to go.

On the negative side, I still see several problems here (actually almost nothing has changed from --02 to -03), listed below:

The biggest problem I see is that this has very limited usage scenarios, as it requires a post-socket system on both sides (there's quite a bit of functionality in there that won't work otherwise - to give an illustrative example: "When a Carrier is forked, its corresponding Carrier at the remote endpoint receives a fork request, .."). I find it particularly disappointing because, in my opinion, it wouldn't take much to make this able to fall-back to TCP such that e.g. a post-sockets client could talk to an existing TCP server. Anyway, what IS this falling back to? If you add functionality both on the sender and receiver side, you require a post-sockets system on both sides - and then you could just as well consistently run over UDP or TCP or HTTP/2 or whatever else you want, without any fall-backs (and as this develops new transport functionality, we're then really moving away from the TAPS charter). I think we must better understand what types of fall-backs are envisioned here.

The document talks about "asynchronous reception", as if this was a feature that would only be attainable with a novel approach, and/or a message-based transport. I see having a callback-based API and a socket-style API as largely orthogonal from the protocol below: one can of course also write a callback-based API over TCP... the real problem is to know after how much received data the receiving app should be notified, by whatever means - that's just a general dilemma, and not a new problem, as the PUSH bit in RFC 793 shows. Messages may seem to make this easier, but then you have to start wondering how large can messages be, how we should handle partial messages... e.g., you may need a signal from the receiver-side TAPS system to the app to say "you have stored half a message, but please throw this data block away now, you'll never get the rest". You have some discussion of partial messages in this draft, and we've been around that block with SCTP. IMO, the right approach is to leave all of that out: define a short max limit, and let an application take care of splitting into smaller messages itself (since you have message dependencies, why do you need to have large messages too, and not just smaller sub-messages that all sequentially depend on another?). This also makes prioritization more meaningful.

Generally, the draft just leaves many questions unanswered at this stage, as it stays at a rather high level and lists most services just as examples, saying that more could be possible. If it's that vague, how should anybody implement such a system? Here's a list of transport features (from minset) that we will lose if we don't offer them to an application, and that I didn't see mentioned in post-sockets (sorry if I missed some!). Some of them may seem a bit odd and unimportant, others less so (this is from appendix A2, where "T" and "U" mean "can be supported with a fall-back to TCP and UDP, respectively):

  o  T,U: Specify number of attempts and/or timeout for the first
     establishment message
  o  T: Change timeout for aborting connection (using retransmit limit
     or time value)
  o  T: Suggest timeout to the peer
  o  T,U: Disable Nagle algorithm
  o  T,U: Notification of Excessive Retransmissions (early warning
     below abortion threshold)
  o  T,U: Specify DSCP field
  o  T,U: Notification of ICMP error message arrival
  o  T,U: Set Cookie life value
  o  T,U: Disable checksum when sending
  o  T,U: Disable checksum requirement when receiving
  o  T,U: Specify checksum coverage used by the sender
  o  T,U: Specify minimum checksum coverage required by receiver
  o  T,U: Specify DF field
  o  T,U: Get max. transport-message size that may be sent using a non-
     fragmented IP packet from the configured interface
  o  T,U: Get max. transport-message size that may be received from the
     configured interface
  o  T,U: Obtain ECN field
  o  T,U: Enable and configure a "Low Extra Delay Background Transfer"
  o  T,U: Request not to delay the acknowledgement (SACK) of a message
  o  T,U: Notification that the stack has no more user data to send

From this list, a few functions that I would consider important are:

- DSCP, and enabling and configuring LEDBAT-like behavior: these two could be lumped together in a higher-level request, but they're not just a priority. Sure, your message "niceness" could be translated into a DSCP choice, but I mean that an app may want to be able to control more (e.g. do a general per-carrier configuration for low latency or LBE, for example). Speaking of LEDBAT, post-sockets also doesn't say anything about whether the transport is congestion controlled or not (which translates into: does the app have to worry about it?). Another congestion control thing: access to the ECN flag in case of UDP should also be covered in some form.

- Notification that the stack has no more user data to send: this is the TCP_NOWAIT_LOWAT function. Hmmm... wait!  At first I thought the binding to an .OnAcked() event handler can't be done with TCP, but if the post-sockets layer would intelligently use TCP_NOWAIT_LOWAT, then you could pull this off.

- Get max. transport-message size that may be sent using a non-fragmented IP packet from the configured interface: this is useful to avoid fragmentation overhead, and

- Specify DF field => if we fall back to UDP, we need this in order to do PMTUD.

... and then there's also the need for early configuration of some things to guide protocol choice, which I wrote as a decision tree in draft-ietf-taps-minset-00. While I agree with your idea of specifying a limited set of protocols to use - we do the same with policy in NEAT - that can't be all: a TAPS system should probably prevent, e.g., first picking UDP and then getting a request for reliability. That brings us back to the question about what you're planning to fall back to, of course.

I hope this was helpful,

cheers,
Michael

---

From: Tommy Pauly <[email-masked]>
Date: Sat, 11 Nov 2017 10:06:04 +0800

Hi Michael,

Just a couple initial notes that may help:

- The version diff you should look at is between -01 and -03. -02 is the same as -03, but had a typo.

- You've mentioned previously that you thought that Post requires both peers to use that API. That is absolutely not the case in any way. Having implemented Post myself, I am only communicating to "legacy" servers that know nothing about Post. I think this fundamental understanding needs to be cleared up before coming to any conclusions. We can discuss during the WG.

- The point of the API is to give the shape of the application interaction, which is why you find it general. I believe that many of the protocol-specific options like you mention (disabling Nagle, retransmissions, etc) don't belong to this main abstract API draft, but to a different document that goes into how to configure options that can be used for setups like TCP/UDP. Again, in my implementation of Post, all of these options exist as part of the Configuration object mentioned in the draft. However, as I'll discuss in the WG, I believe that the general shape of the API needs to be defined to be more-forward looking than just what we've specified with the minset for TCP/UDP/SCTP. Essentially, these become a set of Configuration options that can be used, but as transport protocols continue to evolve (and when we need more protocol-specific options), we need to be able to expand. Certain aspects of the minset, like the connection state management, are of course general and common enough to be part of the highest level of API description.

Thanks,
Tommy

> On Nov 11, 2017, at 9:43 AM, Michael Welzl <[email-masked]> wrote:
> 
> Dear all,
> 
> The document talks about "asynchronous reception", as if this was a feature that would only be attainable with a novel approach, and/or a message-based transport. I see having a callback-based API and a socket-style API as largely orthogonal from the protocol below: one can of course also write a callback-based API over TCP... the real problem is to know after how much received data the receiving app should be notified, by whatever means - that's just a general dilemma, and not a new problem, as the PUSH bit in RFC 793 shows. Messages may seem to make this easier, but then you have to start wondering how large can messages be, how we should handle partial messages... e.g., you may need a signal from the receiver-side TAPS system to the app to say "you have stored half a message, but please throw this data block away now, you'll never get the rest". You have some discussion of partial messages in this draft, and we've been around that block with SCTP. IMO, the right approach is to lea
> ve all of that out: define a short max limit, and let an application take care of splitting into smaller messages itself (since you have message dependencies, why do you need to have large messages too, and not just smaller sub-messages that all sequentially depend on another?). This also makes prioritization more meaningful.
> 
> _______________________________________________
> Taps mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/taps

---

From: Michael Welzl <[email-masked]>
Date: Sat, 11 Nov 2017 10:36:48 +0800

> On Nov 11, 2017, at 10:06 AM, Tommy Pauly <[email-masked]> wrote:
> 
> Hi Michael,
> 

Oh?!  That would be fantastic!!  But I stumbled over a number of things that require a system on the other side, or they just couldn’t happen (like the “carrier forking” notification I quoted below - but there were more). If you say you can talk to a “legacy” server that knows nothing about Post, WHAT does that server speak? Legacy which protocol?  Maybe it’s just a matter of me thinking TCP, and you thinking TLS… dunno! Tell me   :)

> - The point of the API is to give the shape of the application interaction, which is why you find it general. I believe that many of the protocol-specific options like you mention (disabling Nagle,…

It’s clear to me that we want a higher abstraction level than what the list from minset has - e.g., rather than a DSCP value, it would be better to specify general requirements (low latency or such) for a carrier. Rather than saying “disable Nagle”, we could also say “ low latency, even if it comes at the cost of some overhead” - we do such things in the NEAT API too. You’re focused on the interaction with the application (e.g. callback-based instead of traditional socket-style) - which is fine, but I think doesn’t have much to do with the actual protocol choice.

As for being more forward-looking, I wonder what the new transport features are that we’re missing out on (things that apps really see). I follow QUIC from a slightly too large distance (I simply run out of cycles there :(  ), but so far I’m not sure there’s anything we’d be missing  (but that’s maybe also because I’m not an HTTP/2 expert either). Except security of course, but there the argument is perhaps that it’s enough to consider falling back to TLS?

As for how you represent Nagle etc. in your code, that sounds good to me, but I see an issue in that so much transport functionality is simply not covered by the draft. E.g., the packet sizes / fragmentation thing is important too … yes that’s lower level stuff, but UDP-based applications will only get inefficient if they don’t get this information exposed, which risks having app programmers move to the old ways again IMO.

So, the general design approach of post-sockets is IMO fine, but what I’m missing is a long section with the nitty-gritty details on how such a system would really be implemented (not the happy eyeballing bit, this is fine to be elsewhere - but “how would this work with current protocols”). If I was to implement it from the current draft, I’d have a ton of question marks - e.g. the draft convinced me that there must be a system on both sides (this issue is also not discussed in the draft).

Cheers,
Michael

---

From: Tommy Pauly <[email-masked]>
Date: Sat, 11 Nov 2017 11:58:03 +0800

> On Nov 11, 2017, at 10:36 AM, Michael Welzl <[email-masked]> wrote:
> 
> 

So, since Post (and TAPS in general) isn't defining any new protocol features, we need to of course be compatible with all existing protocols. Certainly, some functionality is a bit degenerate in some cases.

Whatever the server speaks, the client needs to conform to. So, we use our Post implementation for raw TCP, raw UDP, TLS over TCP, HTTP/2 Streams over TLS over TCP, QUIC streams over UDP, etc. These are all standard server configurations, but Post offers a client application a single set of APIs to talk over any of these protocols.

In the case of "Carrier Forking", that's essentially what you say to the protocol stack when you want to open a new stream to the same place. So, when I call "fork Carrier", it turns into:
- For raw UDP/TCP or TLS (or anything that is mono-streamed), open a new five-tuple to the same remote endpoint
- For HTTP/2 and QUIC, open a new stream on the same connection

> As for being more forward-looking, I wonder what the new transport features are that we’re missing out on (things that apps really see). I follow QUIC from a slightly too large distance (I simply r…

QUIC is using TLS 1.3, so it can give equivalent security properties to HTTP/2. Doing fallback between these is definitely an important feature. I think this flexibility is why the API needs to allow per-protocol configs.

> As for how you represent Nagle etc. in your code, that sounds good to me, but I see an issue in that so much transport functionality is simply not covered by the draft. E.g., the packet sizes / fra…

A Post API definitely allows access to IP-packet level configuration and metadata. However, you're right that the draft doesn't specify this in detail now. I still think that may be a separate doc, but yeah.

> So, the general design approach of post-sockets is IMO fine, but what I’m missing is a long section with the nitty-gritty details on how such a system would really be implemented (not the happy eye…

Getting feedback on how to clarify this for the implementer will be really important, so thanks for this feedback! I'd also like to discuss with the WG the scope of various documents that we want to produce. If you have one document that does the abstract connection-management and I/O API and all the protocol options and the racing/fallback strategy, that would be a very long and complex doc.

Best,
Tommy

---

From: Michael Welzl <[email-masked]>
Date: Sat, 11 Nov 2017 14:09:18 +0800

Hi,

In line:

> On Nov 11, 2017, at 11:58 AM, Tommy Pauly <[email-masked]> wrote:
> 
> 
> 
>> On Nov 11, 2017, at 10:36 AM, Michael Welzl <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>>> On Nov 11, 2017, at 10:06 AM, Tommy Pauly <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> Hi Michael,
>>> 

Oh, great!  But then you can’t guarantee that a message arrives as a message in case of raw TCP, or you assume application-level framing. This is what I’ve been proposing, but the post-sockets draft reads entirely different from that.

> - For HTTP/2 and QUIC, open a new stream on the same connection

Sure, that’s what I thought it would. Now, with SCTP, you wouldn’t get a “fork request” equivalent on the other side: the client would just start using a new stream, and that’s it.  (“request” also indicates the possibility to say no, which also doesn’t exist in SCTP AFAIK, at least not in the API - this would be a “stream reset”).  Now, in QUIC, I don’t know exactly how that is, but I would have thought it’s the same - you just go ahead and use a new stream. No “fork request” on the other side - just a new stream (in post, carrier) being used.

> QUIC is using TLS 1.3, so it can give equivalent security properties to HTTP/2. Doing fallback between these is definitely an important feature. I think this flexibility is why the API needs to all…

Sure, but I think (and that was my point) this doesn’t change anything about the stuff above (security is in your separate document) - so what exactly are the “more forward looking” requirements except for these security needs?

> A Post API definitely allows access to IP-packet level configuration and metadata. However, you're right that the draft doesn't specify this in detail now. I still think that may be a separate doc,…

I wouldn’t mind that (though it’s not up to me to decide), but —

> Getting feedback on how to clarify this for the implementer will be really important, so thanks for this feedback! I'd also like to discuss with the WG the scope of various documents that we want t…

Great - I’m happy that this is constructive!  Regarding the requests I’m making about post sockets, a reason I think more of these implementation level considerations need to be in the draft is that this gave me so many hiccups of the sort “but then you need it on both sides”, “but then you need to add a lot more protocol functionality”, “but do you think you could do that?” … etc.
I guess no matter if there are multiple docs or one, any document for charter item 3 should leave its reader with the warm feeling of understanding how this could be implemented.

To figure out these details, I’d like to mention that the minset is here to help  :-)    (that’s what it’s meant for). Yes it’s too low-level to be a good API proposal in its own right, but that’s also not the point - it tells you what can and can’t be done. Every decision in it can be traced back to specific reasoning based on the original RFCs; even if you end up disagreeing with reasoning X about a mechanism Y from a RFC Z, at least you know precisely what you’re doing in relation to the TCP, MPTCP, SCTP, UDP, UDP-Lite and LEDBAT specs).

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Mon, 13 Nov 2017 00:30:40 +0800

Hi,

Another thought related to the post-sockets draft:

Post-sockets lets an application programmer define dependencies. That’s good, because dependencies exist, but it comes at the cost of complexity.
My gut feeling tells me: if you have dependencies to take care of, it’s best to leave the data with the application and not hand it over to the transport at all until the last minute.

Controlling the buffer with something like TCP_NOTSENT_LOWAT would be a way for an application to only give the most recent, relevant data to the transport, and decide to throw away data on its own in case a preceding message was thrown away (which it might learn from transport).

I haven’t written code that uses TCP_NOTSENT_LOWAT myself, but I would guess that the trade-off in using this is: if you set this value low, you have more control over the buffer and can be more efficient in your decisions, but the disadvantage is that the transport permanently comes back to you and says “empty, empty, empty, ….” - which may be inconvenient / inefficient for the programmer.

If I’m right with all my musings here, then for an application that has block dependencies, the design should depend on what is easier to handle: defining priorities towards the transport layer on one side, or dealing with the transport calling us quickly all the time because its buffer is about to run empty, plus handling dependencies in the application on the other.

I don’t know - which one IS easier?

I’d love to see this discussed a little, and I’d also love to know if my interpretation of the trade-off in this email is even correct.

Cheers,
Michael

---

From: Tommy Pauly <[email-masked]>
Date: Mon, 13 Nov 2017 07:54:04 +0800

The code I work with does TCP_NOTSENT_LOWAT by default, so we have a fair amount of experience with it. If you're using sockets with TCP_NOTSENT_LOWAT, and you're doing asynchronous non-blocking socket operations, then you don't have the annoyance of getting these "empty" callbacks you're referring to—it just changes how aggressively the writable event fires, making it back off a bit.

With a Post-like API, having something directly like TCP_NOTSENT_LOWAT doesn't make much sense. Instead, the implementation may internally use that, but the equivalent application feedback is the completion that is returned based on the write messages. The timing of when the stack indicates when something has been written allows the application to understand the back-pressure properties of the transport, and if it able to generate or fetch data more slowly to match the back-pressure, it can. Otherwise, it can simply keep writing and the data will get enqueued within the library.

Dependencies between the messages that are being written, then, doesn't actually come into play much here. Dependencies are hints to the implementation and protocol stack of how to order work when scheduling to send (PRE-WRITE). TCP_NOTSENT_LOWAT or the completions to indicate that writes have completed are all about signaling back to the application to provide back-pressure (POST-WRITE).

Does that help clarify things?

Thanks,
Tommy

> On Nov 13, 2017, at 12:30 AM, Michael Welzl <[email-masked]> wrote:
> 
> Hi,
>

---

From: Tommy Pauly <[email-masked]>
Date: Mon, 13 Nov 2017 08:07:24 +0800

> On Nov 11, 2017, at 2:09 PM, Michael Welzl <[email-masked]> wrote:
> 
> Hi,
> 
> In line:
> 
>> On Nov 11, 2017, at 11:58 AM, Tommy Pauly <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>> 

Messages do indeed "always work", but they may be degenerate. There's no new protocols here, just ways of looking at them:

- UDP: One message per datagram
- TCP: Entire stream is one big message that only ever ends when that half of the stream closes
- TLS: Generally viewed like TLS, but can view one record as one message 
- HTTP/QUIC: One message per request/response (so, framing)
- Length-value framing on streams, as used by some protocols (like IKEv2 over TCP): one message per L-V frame
Etc, etc.

That is what the draft does say in Section 2.2. Note how a TCP stream is the message in some cases.

A Message is the unit of communication between applications.
   Messages can represent relatively small structures, such as requests
   in a request/response protocol such as HTTP; relatively large
   structures, such as files of arbitrary size in a filesystem; and
   structures of indeterminate length, such as a stream of bytes in a
   protocol like TCP.

> Sure, that’s what I thought it would. Now, with SCTP, you wouldn’t get a “fork request” equivalent on the other side: the client would just start using a new stream, and that’s it.  (“request” also…

The term "fork-request" is misleading, yes. It really should just say: "forking a carrier creates a new flow to the same Remote, which may be a new stream on a multiplexing protocol. The protocol will handle any required signaling to the remote to open a new stream, based on the protocol".

> Sure, but I think (and that was my point) this doesn’t change anything about the stuff above (security is in your separate document) - so what exactly are the “more forward looking” requirements ex…

I'm trying to distinguish certain aspects of the minset that are:
a) fundamental to all transports, that we have reason to believe will be the case for all future transports in some fashion
	CONNECT, SEND, RECEIVE, CLOSE, etc
b) derived from the protocols we happened to survey or based on current semantics (sockets), based on what exists
	Things I see in the NEAT draft like SET_MIN_CHECKSUM_COVERAGE, SET_TTL, SET_LOW_WATERMARK. Important to have, but often the values the application needs to set relate to which protocol might be used, and may have different requirements for future protocols.

So, there's a set of things in category (a) that should be stable as the main API go forward for quite a while, while things in category (b) may belong as configuration properties, in a list that may shrink or expand over time as new developments are made.

---

From: Michael Welzl <[email-masked]>
Date: Mon, 13 Nov 2017 10:08:38 +0800

> On Nov 13, 2017, at 7:54 AM, Tommy Pauly <[email-masked]> wrote:
> 
> The code I work with does TCP_NOTSENT_LOWAT by default, so we have a fair amount of experience with it.

I figured  :-)   AFAIK think Stuart invented it…

> If you're using sockets with TCP_NOTSENT_LOWAT, and you're doing asynchronous non-blocking socket operations, then you don't have the annoyance of getting these "empty" callbacks you're referring to—it just changes how aggressively the writable event fires, making it back off a bit.

Ah, ok, sure.

> With a Post-like API, having something directly like TCP_NOTSENT_LOWAT doesn't make much sense. Instead, the implementation may internally use that, but the equivalent application feedback is the c…

I mean, the way you describe it here, the application has no means to say how much it wants the layers below to buffer. I think that would be useful, no?
A big buffer is more convenient (based on the number write returns), a smaller buffer lets the application have control over the data until the last minute.
But then, doesn’t this amount to the same "nuisance vs. control of data until the last minute" trade-off that I described?

> Dependencies between the messages that are being written, then, doesn't actually come into play much here. Dependencies are hints to the implementation and protocol stack of how to order work when…

I understand the functionality is separate, but you can achieve the same with it: from an application’s point of view, if I have blocks with dependencies and if you allow me to tune the buffer below the write call, then I can decide until the last minute that I’d rather not send a certain data block.

I guess additionally offering to describe dependencies doesn’t hurt anyway, btw … what made me worried about the complexity is the possibility that these dependencies would change over time, and then the application would want to send an update to post-sockets.  I guess the easy way out is not to offer such dynamics (as indeed in this case an application might be better off handling it in the way I describe above?).

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Mon, 13 Nov 2017 10:16:26 +0800

> On Nov 13, 2017, at 8:07 AM, Tommy Pauly <[email-masked]> wrote:
> 
> 
> 
>> On Nov 11, 2017, at 2:09 PM, Michael Welzl <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi,
>> 
>> In line:
>> 

I read this; how is this efficient?  Are you suggesting to open and close connections to signal that a message is over?
Otherwise, how can you know it is? The length of a message is determined by the application...

> The term "fork-request" is misleading, yes. It really should just say: "forking a carrier creates a new flow to the same Remote, which may be a new stream on a multiplexing protocol. The protocol will handle any required signaling to the remote to open a new stream, based on the protocol”.

Ok; the remote must not have a chance to refuse it, then, because the protocol may not really signal anything all before you get your first byte of data.

> So, there's a set of things in category (a) that should be stable as the main API go forward for quite a while, while things in category (b) may belong as configuration properties, in a list that m…

Hmm. Not sure I buy that e.g. set_min_checksum_coverage has any dependency on the protocol.  As for set_low_watermark, see my other email about that matter: the point is, the application should have some means to tune the size of the buffer below it, I believe.

Cheers,
Michael

---

From: Tommy Pauly <[email-masked]>
Date: Mon, 13 Nov 2017 10:36:27 +0800

> On Nov 13, 2017, at 10:08 AM, Michael Welzl <[email-masked]> wrote:
> 
> 

It can control exactly how much is buffered, since it knows when the data has been actually sent. If you never fail the enqueue of data into the API, you can put as much or as little into the buffer as you want. Since you don't just have a single "writable" callback from a socket, you actually get much more control.
> I understand the functionality is separate, but you can achieve the same with it: from an application’s point of view, if I have blocks with dependencies and if you allow me to tune the buffer belo…

Since the app knows which blocks are outstanding, you can always wait to schedule the block, and you don't need to add dependencies. The dependencies are useful when you want to express that if there is internal re-ordering within a protocol, it can make sure that certain relationships between data are maintained.

---

From: Michael Welzl <[email-masked]>
Date: Mon, 13 Nov 2017 18:15:32 +0800

Hi,

This is me trying to conclude this with a text proposal fort he draft, for the record and for the group - in line:

> On Nov 13, 2017, at 10:36 AM, Tommy Pauly <[email-masked]> wrote:
> 
> 
> 

Tommy and I talked in person; Tommy explained to me that the application knows about the state of the buffer below because it gets a callback whenever the post sockets system is sending something. This may mean a lot of callbacks, but I guess that isn’t a real problem. This sounds good to me - you don’t need to adjust the buffer this way, you just keep it small by waiting for a callback if you want - but then, the text in the post-sockets draft needs to be very clear about this procedure in my opinion.

Specifically, now there’s only this text in there:
***
   Carriers also
   provide .OnSent(), .OnAcked(), and .OnExpired() calls for binding
   default send event handlers to the Carrier, and .OnClosed() for
   handling passive close notifications.
***
… which should be expanded to explain exactly when these events are fired. So, OnSent fires whenever post sockets sends a message (and returns what? Any parameters?)
And FWIW, from this text, OnAck could be a transport layer event, which may not even be visible… when is this expected to fire?

> Since the app knows which blocks are outstanding, you can always wait to schedule the block, and you don't need to add dependencies. The dependencies are useful when you want to express that if the…

Also for the record, from the conversation, we agree that:
- this is useful to have,
- but it should be static, i.e. not allowing the application to change dependencies it has once specified.

This should also be written in the draft.

Cheers,
Michael