Subject: [tcpm] We already support multiple versions of the TCP header

From: David Borman <[email-masked]>
Date: Fri, 31 Oct 2014 09:57:55 -0500

I won’t be at the Honolulu IETF, so I’d like some more discussion
going here on the list.  This message is to explicitly bring up
some issues that haven’t really been addressed yet, and to hopefully
bring some clarity to what decisions need to be made.  A key point
is that using expanded option space during connection establishment
should be separated from how that expanded option space is encoded.

TCP Header Versions
-------------------
The EDO option is really just another example of how do you
support a new version of the TCP header.  We’ve done it with
Window Scale; once negotiated, the TCP header has been modified
for the duration of the connection, in how the Window field is
interpreted.

TCP doesn’t have a version number, so the approach taken by
both WS and EDO is that both sides determine that the modification
to the TCP header is understood, and then it takes effect by mutual
agreement to change how the TCP header is interpreted.  The focus
has been on a single TCP connection to an unknown host, so no a priori
knowledge of whether or not the other side supports the new TCP header.

The mechanism provided by TCP is to use TCP options, the initial SYN
has to assume the legacy TCP header until it can find out otherwise
from the other system, but include everything needed in a TCP option.
Say we wanted to define an entirely new TCP header that was 40 bytes
long, and include it verbatim in a TCP option in the initial SYN, so
as to not introduce any latency or degradation for either legacy or
updated TCP implementations.  But it won’t fit, so we need the expanded
option space to include the new TCP header.  But by definition, the
expanded option space doesn’t fit in the existing option space.

Hence, the expanded option space is our chicken-and-the-egg problem.
By now I assume that everyone is convinced that there is no clever way
to include expanded option space in an initial SYN, such that a
legacy TCP will properly ignore the extra option space, and still
continue to process the packet as a normal SYN packet.  Hence the only
way to send the additional options are in a second packet.  Now the
question becomes how the legacy TCP deals with that second packet.

1) Ignore it: SYN-EOS
    Uses a normal SYN and a separate OOB packet to contain
    the extended option space.  The OOB packet does not have the SYN
    or the ACK bit set, which legacy implementations should
    silently drop.

2) Abort it: Inner-space
    Uses two SYN packets for two distinct TCP connections,
    one with the extended option space and one without.  The client
    terminates the extended option space connection before it becomes
    established if it gets a response indicating that it is a legacy
    TCP, else it terminates the first connection if the remote host
    indicates it supports extended option space.

3) Don’t send it: Four-way Handshake (4WHS)
    Uses a normal SYN, but does not make use of extended option
    space until it knows that the other side supports it.  It
    replaces the ACK with a SYN,ACK so that there is another
    SYN packet client->server than can use extended option space.

These can be classified into two categories:

1) Include the extended option space packet within a single TCP session.
      o SYN-EOS
      o 4WHS

2) Create 2 TCP sessions, one with and one without extended option
   space, then abort one of the connections.  This is a dual-SYN
   approach.
      o Inner-space

How you encode the extended option space is a separate issue.  There
are two techniques on the table, EDO and Inner-space.  Either one of
these techniques could be used with any of the three proposals.  So
the first question is whether to use a dual-SYN approach, or to
use a single TCP connection.

The TCP State Machine
---------------------
TCP is not just the format of the bits on the wire, it is also the
processing at each end of the connection, done through the TCP state
machine.  The introduction of a second packet to contain the extended
option space needs to be accounted for in the TCP state machine.  TCP
runs off of state transitions based on packets received and packets
sent.  Timers ensure that control and data packets are retransmitted
if they are not acknowledged.  (Yes, there are other timers, such as
for delayed ACKs, but that is an efficiency enhancement, TCP can
functionally operate without them).  I believe that this model has served
TCP very well over the years, and we should continue to stick to this
basic model.

The current SYN-EOS description does not follow this model due to:

   If an upgraded server has received only a SYN with the SYN-EOS
   option but no corresponding OOB segment, after a certain time it
   MUST proceed with the connection as if the SYN had been received
   without the SYN-EOS option. I.e. it processes all other TCP options
   and responds with a SYN/ACK without the SYN-EOS option.
and
   >> If a server, whether upgraded or not, receives only an OOB
   segment and no corresponding SYN, it MUST discard it and it MUST NOT
   ever respond (see Security Considerations).

Inner-space adds a layer above the standard TCP state machine, to
manage the dual connections and to decide which one to abort.  The
state machine has to be modified to have knowledge of the parallel
connection, so the added layer can hook in when determining which
connection to abort or to allow to continue.

The Four-way Handshake is defined with extensions to the existing
TCP state machine, so it already fits into the current model.

Delivering User Data
--------------------
Today, even if SYN packets have user data, the data is not delivered
to the application until the connection has entered ESTABLISHED state.

The cleanest way to extend the option space is with a five way
handshake.  An initial two packet exchange is used to discover that
the remote host supports extended option space, then a full three-way
handshake can ensue making use of the extended option space.  This
approach has been mostly dismissed because it adds a full RTT to
connection establishment, delaying when user data can be presented
to the application.

The Four-way handshake adds one extra packet to the connection
startup, the server still enters ESTABLISHED after the first 3
packets, but the client doesn’t enter ESTABLISHED until after all
4 packets, instead of only 2 with the three way handshake.  Depending
on the direction of the initial data, and whether or not it is
included in a SYN packet, there is either a 0 or 1 RTT change in
when the initial data can be delivered to the application.

The SYN-EOS approach uses the standard 3WHS, so the timing of
presenting user data is mostly unchanged.

The dual-SYN approach, for the connection that goes to
established, also completes the standard 3WHS, so the timing of
presenting user data is also mostly unchanged.

So at first blush, when thinking about presenting user data, the
SYN-EOS and Dual-SYN approaches appear to be better than the 4WHS.
But before making that quick decision, I’d like to consider what
is the real criteria for when it is safe to present user data to
the application.

The reason that data in the initial SYN is not presented to the
application is that given a single inbound SYN, we can not verify
verify that this is a genuine connection startup, i.e., it could
be an old duplicate SYN.  The criteria for allowing initial data
in the SYN to be presented to the application is that we have
sent a SYN packet to the other side, and we have received
acknowledgment of that SYN.  That takes a 3 packet exchange,
which happens to correspond exactly with the 3WHS entering
ESTABLISHED state.

If we take the same criteria and apply it to the 4WHS, then
we can see that delivering the initial data does not have to
be delayed due to the additional packet exchange; once the
first 3 packets have been exchanged both sides have verified
that this is a viable connection, and it should be safe to
deliver any user data to the application, making this no
different than the 3WHS when considering when data can be
delivered to the application.

The 4WHS added a conditional transition for the client from
SYN-SENT to SYN-RCVD, mirroring the transition that happens
during a simultaneous open.  Instead it would go to a parallel
state, say PRE-ESTABLISHED, that would be identical to
SYN-RCVD, with the exception that user data could be delivered
to the application, implying that the connect() call would return
without waiting for the transition to ESTABLISHED.

In closing...
-------------
Clearly I think that the 4WHS has some nice properties, since
it is what I have proposed.  My biggest concern has been with
the delay in presenting user data to the application, but I
now believe that that is no longer an issue, for the reasons
stated above.  I think that the 4WHS has applications beyond
just expanding the option space, that’s just the most immediate
issue that it can be used to address.  I also like that the 4WHS
continues to use single packets for each TCP state transition.

The SYN-EOS approach complicates initial connection startup
with two parallel packets, and rules need to be developed for
how to handle all combinations of reordering or either packet
getting lost.  This can be done, but there are more state
transitions than with the 4WHS.

The biggest downside of the Dual-SYN approach, in my mind,
is that it does not handle the case where an application
needs to specify the source port number for a connection.
The simultaneous open case becomes very tricky, if not
impossible, since by definition a simultaneous open requires
specification of the source port.  While the vast majority
of TCP connections are client->server with the client not
caring about the source port, I’m not ready to give up that
functionality to solve the extended option space problem.

I also believe that the way that the extended option space
is defined is orthogonal to how connections get established
with knowledge that the other side supports extended option
space.  You could do inner-space options using an OOB packet
or as part of a 4WHS, you could use EDO with a Dual-SYN approach.

It’ll be easier to make forward progress if we separate using
extended option space from how the extended option space is
encoded.

		-David Borman

---

From: John Leslie <[email-masked]>
Date: Fri, 31 Oct 2014 13:05:40 -0400

David Borman <[email-masked]> wrote:
> 
> I won't be at the Honolulu IETF,

   :^( :^(

> so I'd like some more discussion going here on the list.

   :^) :^)

> This message is to explicitly bring up some issues that haven't really
> been addressed yet, and to hopefully bring some clarity to what
> decisions need to be made.

   Alas, exchanges between the same few participants don't seem likely
to accomplish this. :^(

   Yet, if we can get the discussion on-list, everything can work
better...

> A key point is that using expanded option space during connection
> establishment should be separated from how that expanded option space
> is encoded.

   For discussion purposes, I entirely agree.

> support a new version of the TCP header.

   I'd much rather we used a different name than "EDO" here. EDO
has been adopted as a WG document; and holds the promise of solving
our problem after the SYN and SYN-ACK, and possibly for the SYN-ACK
itself. We should not abandon that promise.

   Nonetheless, there is great value in solving the problem of
expanding option space starting with the original SYN. I'd like to
see this actively explored.

> We've done it with Window Scale; once negotiated, the TCP header has
> been modified for the duration of the connection, in how the Window
> field is interpreted.

   That, alas, was a much simpler problem. There was no reason to
worry about window scaling on the initial SYN; and RFC 1323 happened
in 1992, before we had all those meddleboxes to deal with.

> TCP doesn't have a version number, so the approach taken by both WS
> and EDO

   (PLEASE, can we use a different term here?)

> is that both sides determine that the modification to the TCP header
> is understood, and then it takes effect by mutual agreement to
> change how the TCP header is interpreted.

   Indeed, mutual agreement seems to be the only way to proceed.

   Alas, in 2014 we _have_ all those meddleboxes. :^( :^( :^(

> The focus has been on a single TCP connection to an unknown host,
> so no a priori knowledge of whether or not the other side supports
> the new TCP header.

   (That, of course, is not the only possible scenario...)

> from the other system,

   Yes.

> but include everything needed in a TCP option.

   We like to believe that a TCP option will either tunnel through
any middleboxes, or be dropped by them. Alas, this may not always
be true...

> updated TCP implementations.  But it won't fit,

   Yes. :^(

> so we need the expanded option space to include the new TCP header.

   (This is a good way to think about our problem.)

> But by definition, the expanded option space doesn't fit in the
> existing option space.

   Thus, we must look elsewhere.

> continue to process the packet as a normal SYN packet.

   (Actually, that is _almost_ solvable; but there's no hope of
avoiding middlebox damage. So I _do_ agree we need to find something
else.)

> Hence the only way to send the additional options are in a second
> packet.

   I don't think that's quite correct. Parallel-SYN approaches can
send one packet with added options and one without. At the very least,
the one _with_ added options can be sent first.

   But also, there could be ways to ensure the _with_extra_options_
packet could never lead to a legacy connection -- and there will
certainly be some cases (in the future) where there's no point in
establishing a legacy connection at all.

   Nonetheless, I'm quibbling here...

> Now the question becomes how the legacy TCP deals with that second
> packet. 
> 1) Ignore it: SYN-EOS

   "should" is a big question here!

>    indicates it supports extended option space.

   This _does_ sound ugly; but it _is_ workable. IMHO we'd be foolish
to rule it out entirely at this point; but it seems to require a
significant period of experimental status (during which we would hope
to finish something better).

> 3) Don't send it: Four-way Handshake (4WHS)

   This is an interesting approach.

   While I very much want to consider it, I'm leaving the lengthy
explanation out of this response -- it deserves its own thread.

> How you encode the extended option space is a separate issue.

   "Separable," certainly; and I'm happy to separate them here.

> There are two techniques on the table, EDO and Inner-space.

   PLEASE, don't use the name EDO for this: it's not even clear what
David intend this term to refer to.

> Either one of these techniques could be used with any of the three
> proposals.

   As well as many other encodings...

> So the first question is whether to use a dual-SYN approach, or to

   That's exactly the kind of question I _don't_ want the WG to try
to solve.

   Nonetheless, David should get an opportunity to explain why his
solution is better.

   (Just not here: I'll start another thread for that.)

====

   We have three proposals for how to expand option space beginning
in the initial SYN. IMHO, all three deserve Experimental status.

   High among the issue needing to be resolved is middlebox damage.
WG participants have many different approaches to dealing with
middlebox damage. I tend to think so long as we detect it, the
specifics about routing around it don't need to be solved right now.
YMMV.

   I see no reason three experiments can't run it parallel: there
seems to be no obvious reason the three proponents can't find folks
to work on all three.

   What I wish this WG would do is give those three free rein to
experiment; but process EDO-as-adopted to RFC status. Myself, I
don't see how any of the three initial-SYN options can reach
substantial deployment in under five years, while EDO could.

--
John Leslie <[email-masked]>

---

From: John Leslie <[email-masked]>
Date: Fri, 31 Oct 2014 14:01:28 -0400

David Borman <[email-masked]> wrote:
> machine.

   I agree with David that nothing which messes with the state machine
should proceed to Proposed Standard without a clear description of
what changes to the state machine it proposes.

   (But that's not _as_ necessary for Experimental status...)

> The introduction of a second packet to contain the extended option
> space needs to be accounted for in the TCP state machine.  TCP
> if they are not acknowledged.

   (Those timers deserve mention; but I'm not convinced that particular
values are needed.)

> (Yes, there are other timers, such as for delayed ACKs, but that is
> an efficiency enhancement, TCP can functionally operate without them).

   (Actually, TCP-bis could operate without timers also.)

> I believe that this model has served TCP very well over the years,
> and we should continue to stick to this basic model.

   Agreed. But at this stage of discussion, eyes would tend to glaze
if we talk too much of the state machine.

>    and responds with a SYN/ACK without the SYN-EOS option.

   This indeed is hand-waving. :^(

>    segment and no corresponding SYN, it MUST discard it and it
>    MUST NOT ever respond (see Security Considerations).

   As is that. :^(

   But IMHO, even such obvious hand-waving could make its way into
an Experimental document by simply saying that determining the timer
values is part of the experiment.

> connection to abort or to allow to continue.

   It is described that way right now, and may or may not stay as a
layer above the TCP state machine. I'm not too concerned until we
try to progress it to Proposed Standard.

> TCP state machine, so it already fits into the current model.

   I thank David for doing this. (But I worry how many eyes are
glazed...

> Delivering User Data

   Very important. Hopefully we realize that TCP-Fast-Open will have
to be a special-case in whatever we propose.

> handshake can ensue making use of the extended option space.

   Indeed!

> This approach has been mostly dismissed because it adds a full RTT to

   David correctly states that few uses of TCP will happily wait an
extra RTT.

> packets, but the client doesn???t enter ESTABLISHED until after all
> 4 packets, instead of only 2 with the three way handshake.

   This is the critical think to understand. (And I agree with
David that it's best understood as a change to the state machine.)

> Depending on the direction of the initial data, and whether or not
> it is included in a SYN packet, there is either a 0 or 1 RTT change
> in when the initial data can be delivered to the application.

   (Note that an extra RTT to when a client may receive server data
is not considered trivial. Nonetheless, it is not necessarily fatal.)

> The dual-SYN approach, for the connection that goes to established,
> also completes the standard 3WHS, so the timing of presenting user
> data is also mostly unchanged.

   Neither of those necessarily add delay.

> But before making that quick decision, I'd like to consider what

   This gets a bit complicated; but is very worth considering.

> be an old duplicate SYN.

   (I'm biting my tongue here; but I'll let that pass.)

> The criteria for allowing initial data in the SYN to be presented
> to the application is that we have sent a SYN packet to the other
> side, and we have received acknowledgment of that SYN. That takes
> a 3 packet exchange, which happens to correspond exactly with the
> 3WHS entering ESTABLISHED state.

   (NB: TCP-Fast-Open uses a different paradigm of what constitutes
sufficient confirmation.)

> If we take the same criteria and apply it to the 4WHS, then we
> can see that delivering the initial data does not have to be
> delayed due to the additional packet exchange; once the first 3
> packets have been exchanged both sides have verified that this is
> a viable connection, and it should be safe to deliver any user
> data to the application, making this no different than the 3WHS
> when considering when data can be delivered to the application.

   (Alas, we will have to consider middlebox damage sometime.)

> state, say PRE-ESTABLISHED, that would be identical to SYN-RCVD,
> with the exception that user data could be delivered to the
> application, implying that the connect() call would return

   I'm not positive even I understand this paragraph. I think
David is talking about the sender of the original SYN -- moving to
SYN-RCVD state when it receives the SYN-ACK. By his 4-way handshake,
it goes to SYN-RECEIVED state instead of ESTABLISHED. He seems to be
saying that the original sender might instead enter a new
PRE-ESTABLISHED state upon sending its own SYN-ACK; and upon
entering that state it could deliver the data it received in the
first SYN-ACK to the originating application.

   My eyes are already a bit glazed...

   I don't understand how that could be particularly useful data.

> In closing...
> -------------
> Clearly I think that the 4WHS has some nice properties, since it
> is what I have proposed. My biggest concern has been with the delay
> in presenting user data to the application, but I now believe that
> that is no longer an issue, for the reasons stated above. I think
> that the 4WHS has applications beyond just expanding the option
> space, that's just the most immediate issue that it can be used to
> address.

   Indeed, it does seem that a 4-way handshake has potential beyond
just added option space. :^)

> I also like that the 4WHS continues to use single packets for each
> TCP state transition.

   I share some of David's concern about trying to relate OOB
packets to the TCP state machine.

> The SYN-EOS approach complicates initial connection startup with
> two parallel packets, and rules need to be developed for how to
> handle all combinations of reordering or either packet getting
> lost.  This can be done, but there are more state transitions
> than with the 4WHS.

   Perhaps Joe would like to respond here...

> The biggest downside of the Dual-SYN approach, in my mind, is that
> it does not handle the case where an application needs to specify
> the source port number for a connection.

   An interesting question. (There certainly are cases where the
originator of a TCP connection needs to use a known port: BGP comes
to mind first...)

   But, honestly, I doubt that any of those cases will need expanded
option space. (And I very much hope that we'll be able to continue
using legacy TCP where the option space is already sufficient.)

> The simultaneous open case becomes very tricky, if not impossible,
> since by definition a simultaneous open requires specification of
> the source port.

   This strikes me as a more serious issue.

   Of course, simultaneous-open is not expected to ever occur in
the client-server paradigm; and I'm not sure this is actually any
worse than the BGP case...

> While the vast majority of TCP connections are client->server
> with the client not caring about the source port, I'm not ready
> to give up that functionality to solve the extended option space
> problem.

   Good point, there.

   But I see no reason to object to Experimental status because
of that.

> I also believe that the way that the extended option space is
> defined is orthogonal to how connections get established with
> knowledge that the other side supports extended option space.

   It certainly could be...

> You could do inner-space options using an OOB packet or as part of
> a 4WHS, you could use EDO with a Dual-SYN approach.

   I see no reason to go there. The proposals are hard enough to
understand as it is.

> It'll be easier to make forward progress if we separate using

   I'm afraid I disagree.

   Our path forward will be far easier if we give the three proposals
free rein to experiment.

---

From: David Borman <[email-masked]>
Date: Fri, 31 Oct 2014 14:14:46 -0500

John,

Thanks for the well thought out response.

I’ll touch on just a few points, in hopes of less eyes glazing over. :-)

> On Oct 31, 2014, at 1:01 PM, John Leslie <[email-masked]> wrote:
> 
...
> try to progress it to Proposed Standard.

For me the key issue here is that currently a single TCP connection
is processed in isolation, it doesn’t really matter what is going on
in other parallel TCP connections.  With the dual connection model,
we now have pairs of TCP connections that need to be processed with
consideration for the state of the other TCP connection.

...

> first SYN-ACK to the originating application.

The point is the active (client) side of the connection sends a
SYN, and when the SYN,ACK comes back, if it contains user data,
it is safe to make that available to the application.

For the passive (server) side, it receives a SYN with some data,
but it has to buffer it and send back a SYN,ACK.  When it receives
an ACK for its SYN, it can now safely deliver the data to
the application.

This is exactly what happens with the 3WHS, and corresponds with
transitioning to ESTABLISHED state.

You can use the same criteria with the 4WHS for when it is safe
to deliver data to the application.  For the active side, this
happens when it transitions from SYN-SENT to SYN-RCVD instead
of to ESTABLISHED.  I am adding the PRE-ESTABLISHED state so
that it isn’t overloading the SYN-RCVD state, which is normally
only reached by the active side during a simultaneous open, when
it is *not* safe to make user data available, because its SYN
has not yet been ACKed.

...
> just added option space. :^)

What it really does is allow the passive (server) side of the connection
to initiate some aspects of the connection setup, rather than being
limited to just responding to the initiatives from the active (client)
side.  And, it does that with a packet containing a control bit (SYN),
which is reliably transmitted.

For example:

The MPTCP specification uses the final ACK as (an optional) part of
MP_CAPABLE negotiation, where it contains both keys, allowing the server
to act statelessly.  Today that ACK is not reliably transmitted, with
the 4WHS it would be part of the SYN,ACK and be reliably transmitted.
Today, for a stateless server, if that ACK is lost, the connection
will never complete, because the ACK is never retransmitted.

...

			-David Borman

---

From: Joe Touch <[email-masked]>
Date: Fri, 31 Oct 2014 13:07:43 -0700

On 10/31/2014 7:57 AM, David Borman wrote:
...
> The TCP State Machine

If a SYN-EOS is received without the OOB packet, the server has three
options, based on whether the server supports SYN-EOS:

	if SYN-EOS isn't supported, ACK without the EOS flag

	if SYN-EOS is supported:

		a) wait for retransmission
		(the client would retransmit both
		SYN-EOS and OOB)

			NB: during this time, the SYN-EOS
			could be cached for pairing with 	
			the OOB when it arrives, but doesn't
			have to be

		b) give up and act like a non-SYN-EOS
		server

I don't see any impact on the state machine.

...
> TCP state machine, so it already fits into the current model.

It fits into the model by *changing* the model to a new one. That's not
called "fitting", FWIW.

...
> delivered to the application.

I continue to believe this isn't accurate, and that the user data
shouldn't be delivered in 4WHS until the final ACK is exchanged (5WHS).
I agree that *part* of the 3WHS exchange is to coordinate the initial
SYN, but IMO it's also to coordinate all the options that have been
negotiated, and I still am unconvinced this happens for all use cases
for options in 4WHS.

...
> getting lost. 

Caching a limited amount of such packets solves that.

> This can be done, but there are more state

SYN-EOS uses exactly the same transitions as 3WHS, which is less than 4WHS.

...
> extended option space from how the extended option space is

+1, but I also believe that it isn't our job to "get around" all
possible NAT behaviors by - in effect - running TCP inside TCP, which is
how I view in-band encoding of the options in innerspace.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 31 Oct 2014 13:17:57 -0700

FWIW, IMO, there are four things on the "table"

	EDO - the current WG doc
		negotiated during the SYN, active after SYN-ACK
		and maybe for SYN-ACK too (depends on what WG decides
		vs. the potential for asymmetric middleboxes with
		currently unseen behaviors)

	4WHS - (as already known)

	Dual-SYN (I don't care if Bob wants to call this something else,
		but I don't want to call it "innerspace" because that
		includes the in-band encoding, and I agree with David
		that encoding is orthogonal, and it's basically a
		"dual-stack" approach)

	SYN+OOB	- Yeah, the name has changed a lot, but IMO that's the
		most accurate term that differentiates it from the
		others

On 10/31/2014 10:05 AM, John Leslie wrote:
...
>    (PLEASE, can we use a different term here?)

EDO is the correct term AFAICT; we're not talking about the SYN variants
for this statement, are we?

>> Hence the only way to send the additional options are in a second

For the SYN (and maybe the SYN-ACK too, if you believe the asymmetric
rogue middlebox case).

...
>> There are two techniques on the table, EDO and Inner-space.

EDO does also include a way of encoding where the new space is, and
these are the two variants that can be used by either SYN-extensions or
after-SYN extensions.

I'd prefer to call them "new header space vs. in-band encoding". I agree
calling the former "EDO" isn't useful.

> substantial deployment in under five years, while EDO could.

The only issue here is TCPCRYPT - if you beleive that group will do
something useful. They could deploy a SYN extension at the same time as
their system, which would help get experience faster, but I don't think
any of us wants or expects them to support multiple solutions.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 31 Oct 2014 13:20:10 -0700

>> > an efficiency enhancement, TCP can functionally operate without them).

I'm not sure I agree with this.

It seems like TCP would lockup and never make progress if packets were lost.

Joe