Subject: Re: status of T/TCP

From: "Charles M. Hannum" <[email-masked]>
Date: Tue, 8 Sep 1998 10:43:20 -0400 (EDT)

I think it's a shame that you have to get emotional about it, Bob.  My
draft was not an emotional piece of work; it simply (and quite dryly)
presents a number of ways to abuse T/TCP that are not possible with a
normal TCP implementation, and points out an interoperability issue.
Whether or not these concerns are truly significant wasn't the issue I
was presenting when I wrote it.

I do, however, find it interesting that one of the holes was actually
used to break into machines.  I believe this suggests that people
should not simply turn up their noses (as you did) when someone
publishes a paper that puts their pet project in a bad light.  Had you
actually read the paper carefully and reviewed your implementation,
that hole could have been fixed a long time before it was ever abused.
Your response instead seemed childish and irresponsible.

As for your particular points:

1) There *are* cases in which the r-utilities are secure with TCP,
   where they are not with T/TCP.  This is basically any environment
   where only `internal' addresses are permitted to establish
   connections.  With normal TCP, the responses would never be routed
   to the outside world, and so it would be impossible to do anything;
   with T/TCP, you only need one packet.  So saying `T/TCP is no more
   insecure than TCP is already' is simply fallacious.

2) Denial of service attacks are a serious issue and cannot simply be
   ignored.  If you really believe you can dismiss this out of hand,
   tell that to all the service operators who were completely screwed
   by SYN flood attacks a year or so ago, and imagine it 10x worse.
   (There are also some more subtle and much more devastating denial
   of service attacks that I didn't mention in the draft.)

I never once said `this performance improvement is undesirable'; I did
not in any way attack that (laudable) goal.  However, your arrogant
dismissal of the issues seems to me to cast doubt on your credibility.

---

From: "Charles M. Hannum" <[email-masked]>
Date: Tue, 8 Sep 1998 11:43:39 -0400 (EDT)

BTW, on review, I believe the `sequence number attacks' section is
actually irrelevant.  The comment about TIME-WAIT state is incorrect,
because TIME-WAIT can be assassinated even with normal TCP.  This
causes the entire logic of the argument to fall apart.  It's also the
case that this is only really relevant for applications using
host-based authorization, which should be disabling T/TCP for the
other reasons I mentioned.  (I actually noticed this once before, but
due to the distinct lack of feedback I never updated the draft.)

So, let's assume for a moment that there are correct implementations
resolving the concerns mentioned in sections 2 (`Host-Based
Authorization') and 4 (`Pre-SYN Queueing'), and that section 5
(`Sequence Number Attacks') is simply discarded.  That still leaves
the concerns in section 3 (`SYN flooding') and 6 (`Conclusions').

I could easily give on the interoperability problem mentioned in
section 6.  It's specifically the client using T/TCP and connecting to
an old server that causes the problem.  It doesn't matter whether or
not a server uses T/TCP.  So this problem isn't particularly relevant
to service operators.

However, the potential for denial of service attacks is still real,
and I don't see any way around it.

Now, I expect at this point someone will simply fire up a stupid SYN
flooding engine and claim that nothing happened.  So I will explain a
simple scenario that your SYN flooding tool didn't exercise, but which
will completely hose your server:

* Say you have a CGI script that takes 1 second of real time to load.
  Lots of sites -- especially ones with fancy shopping carts and
  whatnot -- have a plethora of these.

* Now say I flood you at a pretty slow rate, perhaps 11 packets every
  10 seconds, with requests that exercise this particular CGI script.
  Since we're using T/TCP, I only need 1 packet per probe to do this,
  and I never have to see the replies.

* In not too much time at all, I'll have hundreds of httpd processes
  on your machine stuck waiting for this damned CGI script, and not
  processing any real requests.

* Your customers get pissed off.

Scale the numbers to suit your CGI scripts, of course.  It doesn't
matter; the point is still the same.

---

From: "Charles M. Hannum" <[email-masked]>
Date: Tue, 8 Sep 1998 12:04:33 -0400 (EDT)

> * Your customers get pissed off.

In case it wasn't blindingly obvious to everyone, the punchline is:

Because you're using T/TCP, I can submit these 1-packet probes with
totally random (or very pointedly non-random) source IP addresses, so
you can't filter them out by source.  I can randomize the data in the
query a bit and make it very difficult -- perhaps even impossible --
for you to filter out the requests based on data.

In short, I can potentially make it impossible for you to prevent this
attack from happening.  And the reason I can do this is because you're
using T/TCP.  If you were using TCP, this wouldn't be possible.

I *think* we could actually fix this, by using some sort of exchange
in the first connection (which always does a full 3WHS to initialize
the connection count) and a hashing mechanism similiar to RFC-1948 to
ensure that future SYN-data-FIN packets came from the same source as
the initial connection request (or at worst a snooper in between).
This would effectively restore the `security' (in this case, really
the *accountability*) of normal TCP.

I haven't worked out the details, but it seems emminently doable.

And I think that's enough mail for this morning.

---

From: [email-masked]
Date: Tue, 8 Sep 1998 09:25:27 -0700

Charles,

Gosh, thanks.  It is not very often that I am honored by someone
from MIT calling me "arrogant".

Actually, the discussion I wanted to have concerns protocol
architecture, something like: "is it necessary, reasonable, and
desirable for the TCP 3 way handshake sequence number synchronization
mechanism to be an essential security mechanism?"  But as this is the
wrong list, I will say no more on this subject here.

Bob Braden

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 8 Sep 1998 11:08:33 -0600 (MDT)

> From: "Charles M. Hannum" <[email-masked]>

> ...

Yes, but why bother? Instead of inventing yet another mechanism to keep
state above T/TCP, why not simply modify the application protocol to use
TCP?  Why not arrange to pay the price of the 3-way handshake/A&A once,
and then re-use the TCP connection for several transactions?

I've long liked the idea of reducing the number of packets in the TCP
3-way handshake.  (It was one of the selling points of XTP).  However,
your point about the utility of delaying committing significant resources
until at least one round-trip of packets has at least somewhat verified
the identity of the peer is very interesting.

Besides your notion of some kind of hashing/pre-reservation, you might do
something in you HTTP server example such as start the HTTP query for
requests that arrived by T/TCP, but perpare to abandon it early.  Either:

  1. respond to the initial T/TCP packet by both starting the query
      and sending a window-probe-style TCP segment to the client.
      If the client does not respond quickly (say <1 sec), abort the
      query.  Yes, this sounds like it would require violations of
      various layering religions.

  2. limit the total number of processes doing work for T/TCP requests
      that have not been ACK'd.  If the system gets load, drop a random
      request, much like the random-drop defense to SYN attacks.  This
      could not be distinguished by clients from ordinary packet losses,
      and so would be harmless, except for the work wasted before request
      is dropped.  I think this should not offend the layering priests
      except in concept, since it would require no hacks in the stack.

Still, why bother?  Never mind newer versions of HTTP.  What benefit does
T/TCP give, other than a speed-up by one RTT of the initial response?
Since all HTTP clients now use more than one TCP connection, delays for
3-way handshakes after the first are insignificant, aren't they?  Given
commerical HTTP servers that can server more hits/second than you can buy
Internet bandwidth no matter what your budget, there doesn't seem much
that T/TCP can do for servers.

Vernon Schryver    [email-masked]

---

From: [email-masked] (W. Richard Stevens)
Date: Tue, 8 Sep 1998 13:22:28 -0700

[In your message of Sep  8, 12:04pm you write:]
> you can't filter them out by source.

But these 1-packet probes with random source addresses are not passed
to the application *until* the 3WHS is completed, assuming that these
random source addresses have not had previous connections with this
server that have initialized the CC for this client.  And if you use
random source address, the 3WHS will never complete.  So to launch
these attacks you minimally have to come from a host that supports
T/TCP and you must use your own source IP.

	Rich Stevens

---

From: "Charles M. Hannum" <[email-masked]>
Date: Tue, 8 Sep 1998 19:31:10 -0400 (EDT)

> T/TCP and you must use your own source IP.

That turns out not to be the case, for two reasons:

1) The existing implementation creates the TAO cache before the first
   3WHS has completed.

2) C.f. `very pointedly non-random'.  Say I pick source addresses of
   known customers, or known data providers, instead.  With T/TCP, I
   don't have to know the port numbers they used in previous
   connections, because the TAO cache applies to all connections from
   that particular machine; all I have to know is that they're likely
   to be in the TAO cache.

There are some very subtle aspects to #2.  For example, say you have
something that automatically detects this sort of attack and filters
out that client.  Besides the potential for annoying real customers,
this has other effects.  One such attack would be to get you to
automatically filter out known search engine robots, so your site
never appears on them.  It's likely that you wouldn't even notice
this, and you'd just lose hits (read: potential revenue) without ever
knowing why.  (And, of course, in most cases I could just keep
round-robining between a number of known addresses.  You can't filter
out everything; eventually you'd have to start doing aggregate
filtering or timing out old entries...)

The idea may be digusting to some people (and I confess that it churns
my own stomach a bit), but the 3WHS really does provide a certain
amount of accountability that is important for maintaining a clean
neighborhood.

I think a fix for this is as simple as:

* Send a random cookie in the CC.NEW.

* Return the cookie in the CC.

* If the received cookie doesn't match the cached (sent) cookie, fall
  back to a 3WHS and reinitialize the TAO cache.

This increases the size of the CC.NEW/CC options slightly, but should
not have any significant impact on performance.  It is incompatible
with the old CC.NEW/CC options, but it's not as if they're widely
deployed anyway.

---

From: David Borman <[email-masked]>
Date: Wed, 9 Sep 1998 09:59:50 -0500 (CDT)

I wasn't going to step into the T/TCP discussion, but it has gone
on long enough that if people are thinking at all seriously about
T/TCP, then I should.

Several years ago I studied the T/TCP protocol, and decided that it
was more complicated than it needed to be.  I'm attaching the message
that I sent out to the end2end-interest mailing list in 1996.

And before people start jumping up and down, yes, I know that this
doesn't address any of the spoofing issues.  But if people are
thinking about changing T/TCP to address security issues, I'd like
to change T/TCP to simplify the CC processing at the same time.

			-David Borman, [email-masked]

> From dab Wed Mar 13 15:08:54 1996
> To: [email-masked]
> Subject: T/TCP & CC processing
> Cc: [email-masked]
> 
> 
> Hello all.
> 
> This message is a follow up to what I brought up at the T/TCP
> BOF in LA about an alternate method for validating CC values.
> 
> This proposal does several things:
> 	1) It eliminates the need for the CC.NEW option, the
> 	   client only sends the CC option.
> 	2) Allows the first SYN/CC that is received from
> 	   any host to be accepted without waiting for the
> 	   3-way handshake to complete, which means that a
> 	   stream of in-sequence T/TCP requests can be processed
> 	   without forcing any of them to wait for the 3-way
> 	   handshake to complete.
> 	3) It simplifies the CC processing at both ends.
> 
> Background
> ----------
> 
> The goal of the cache of CC values is to be able to identify
> incoming SYN packets that are valid, so that the data in them
> can be processed immediately via the TCP Accelerated Open (TAO)
> mechanism, which passes the data up to the application without
> waiting for the three way handshake to complete.  Or, more precisely,
> we want to identify SYN packets that might be old SYN packets
> and not process the data until we have verified the connection
> by completing the 3-way handshake.
> 
> There are two ways that a duplicate SYN can arrive.  Either
> somewhere along the path the packet was duplicated, or else
> the source has retransmitted the SYN packet.  In the first
> case, the duplicate should be received within one MSL, of
> the first SYN, since after one MSL it should be discarded.
> As long as the connection created by the original MSL exists
> at least one MSL, it will catch any duplicated SYNs.  In the
> second case, if the response comes in just after the SYN is
> retransmitted, as long as the control block exists at least
> one MSL before going away, the retransmitted SYN will be
> caught by the control block.  Since the control block has
> to go through TIME-WAIT, it will be around at least 2 MSL.
> It should also be noted that while the source is retransmitting
> SYNs, it means that the connection at the receiver, once one
> of the SYNs has gotten through, will persist until the source
> has gotten back a response, meaning that the control block
> at the receiver will take care of any duplicate SYNs.
> 
> When the first SYN packet from host A arrives at host B, it will
> always be a valid SYN.  If it was not a valid SYN, then that means
> that host A has destroyed the connection in less than the MSL
> after sending out the SYN, which should not happen.
> 
> Additionally, when there have been no SYN packets from a host for
> more than an MSL, the next SYN will be a valid SYN, since all old
> duplicate SYN will have exited the network.
> 
> From this it can be implied that if the server remembers the time
> that the last SYN/CC was received, then if when the next SYN/CC
> arrives is more than an MSL, it should be a valid SYN/CC, whether
> or not the CC passes the check.
> 
> Proposal for processing the CC option
> -------------------------------------
> 
> The server maintains a table of HOST, CC and CLOCK values.  The CLOCK
> is when CC was received from HOST.  If a CC arrives for a HOST that is
> not in the table and there are empty slots in the table, TAO can be
> used, and HOST and CC are added to the table with CLOCK = CURRENT_CLOCK.
> If there is an entry, but it is more than 2 MSL old, the entry can be
> ignored and treated as if it wasn't in the table.
> 
> There is a table for cached CC information.  Each entry contains:
> 	TABLE[n].HOST
> 	TABLE[n].CC
> 	TABLE[n].CLOCK
> 
> In addition, there are two values:
> 	OLD_CLOCK
> 	CURRENT_CLOCK
> 
> At start up:
> 	OLD_CLOCK = CURRENT_CLOCK
> 
> SYN/CC arrives from <host>:
> 
> 
> if there exists a table entry TABLE[n] such that TABLE[n].HOST == SEG.SRC
> then
> 	if SEG.CC > TABLE[n].CC or CURRENT_CLOCK - TABLE[n].CLOCK > 2 MSL
> 	then
> 		TABLE[n].CC = SEG.CC
> 		TAO = OK
> 	else
> 		TAO = NOT_OK
> 	endif
> 
> 	TABLE[n].CLOCK = CURRENT_CLOCK.
> 
> else SEG.SRC is not in the table
> 
> 	if CURRENT_CLOCK - OLD_CLOCK > 2 MSL then
> 		TAO = OK
> 	else
> 		TAO = NOT_OK
> 	endif
> 
> 	Find an empty entry in the table.
> 
> 	if there is no empty entry, then
> 		Find the oldest entry in the table.
> 		OLD_CLOCK = TABLE[n].CLOCK
> 	endif
> 
> 	TABLE[n].CLOCK = CURRENT_CLOCK
> 	TABLE[n].CC = SEG.CC
> 	TABLE[n].HOST = SEG.SRC
> endif
> 
> 
> A few observations on the algorithm
> -----------------------------------
> 
> 	1) Setting OLD_CLOCK = CURRENT_CLOCK at startup time forces
> 	   all new connections to do a full 3-way handshake until the
> 	   host has been up long enough for any old SYNs that were
> 	   floating around in the network to trickle in.
> 
> 	2) You only need to maintain the CC cache for hosts from
> 	   which SYN/CCs have sent within the last MSL.
> 
> 	3) The timeout period must be between 1 MSL and 2 MSL.
> 	   1 MSL is the lower bound, because that is how long
> 	   duplicates can be floating around.  2 MSL is the
> 	   upper bound, because CC values can wrap anytime
> 	   after 2 MSL.
> 
> 	4) Once a SYN arrives and a control block is created, that
> 	   control block will exist until any retransmitted or
> 	   duplicate SYNs have either arrived or timed out in the
> 	   network.
> 
> 	5) Because of #4, the CC cache code is only used when a SYN
> 	   arrives and there does not already exist a connection for
> 	   that SYN.
> 
> 	6) The current RFC is not at-most-one, and this proposal
> 	   neither strengthens or weakens that situation.
> 
> 			-David Borman, [email-masked]
>

---

From: "Charles M. Hannum" <[email-masked]>
Date: Wed, 16 Sep 1998 14:20:57 -0400 (EDT)

> So I was, and still am, a fan of T/TCP.  I believe that HTTP/1.1 does NOT
> reduce the need for it, for the reasons given. I pose to this list: can it be
> fixed?  If it can be fixed, I think it is good stuff.

I believe solutions have been proposed (mostly by me!) to all of the
problems I've mentioned.  One of them would require changing the
protocol in an incompatible way -- but as I pointed out, T/TCP is not
widely deployed, so this shouldn't be a large burden.

---

From: der Mouse <[email-masked]>
Date: Wed, 16 Sep 1998 16:30:40 -0400 (EDT)

>> More commonly, with 28.8 modems, the RTT is in the 170ms range, to
>> the first hop; [...]

> BTW - some of this latency is the modem being too clever about
> buffering and compressing.  Shutting it down might decrease the
> latency.

Might?  Definitely does, in my experience.  Turn off error correction
and compression and I've always found latency goes down.  The
per-packet error rate goes up, of course; whether this is an acceptable
tradeoff depends on the application.

Of course, this particular side note is probably getting a little
off-topic for tcp-impl....

					der Mouse

			       [email-masked]
		     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Joe Touch <[email-masked]>
Date: Wed, 16 Sep 1998 14:39:12 -0700 (PDT)

> From: [email-masked] (Jim Gettys)
> To: Joe Touch <[email-masked]>
> Cc: [email-masked], [email-masked], [email-masked], [email-masked],
>         [email-masked], [email-masked], [email-masked],
>         [email-masked]
> Subject: Re: status of T/TCP
> 
...
> > > Another example: my machine at MIT to Inria's web server (in France):
> > > RTT is ~125 milliseconds.  (And we have our own line to Inria) 2X this
> > > is 250 milliseconds. 
> > 
> > BTW2 - why are you doubling the RTT?
...
> How do you anticipate a new user's first visit to a web site (without
> burning infinite bandwidth?

You don't - that's the one you have to just wait.

However, T/TCP won't help this either - if it really is the first
time you're there, you take a 3-way handshake to establish state.
T/TCP helps only for subsequent connections.

(just to round out the original discussion on T/TCP, which is where we started)

Joe

---

From: [email-masked]
Date: Wed, 16 Sep 1998 18:40:04 -0700

- I still want T/TCP for the additional round trip I'd get back, along with 
- all the other reasons I stated originally.

And I still remain convinced that security concerns, the added complexity
to TCP (as hashed out quite well by the original researcher in a BOF, what?,
two years back?) and the lack of serious deployment experience with T/TCP
mean it's a long-term, if ever, affair.

Look for latency reduction by reducing the size of content, and by
making the content arrive in a sensible order.  Let the document include
tips about preloading the most common next choice(s).

- In any case, I don't think this conversation should continue here,
- as it has little bearing on general TCP implementation.

Yup.  Especially considering T/TCP is only experimental at best.

Cheers,

	~sparker

---

From: "Charles M. Hannum" <[email-masked]>
Date: Thu, 17 Sep 1998 12:56:32 -0400 (EDT)

> Are you saying that a T/TCP client actually *breaks* a TCP that
> "implements RFC793 to the letter"?

Yes, that's what I've been saying for years.  The processing rules in
RFC793 have at least one interesting condition that conflicts with
T/TCP.

On page 75, near the middle:

    eighth, check the FIN bit,

      Do not process the FIN if the state is CLOSED, LISTEN or SYN-SENT
      since the SEG.SEQ cannot be validated; drop the segment and
      return.

This is very clear.  It says `drop the segment', not `drop the FIN' or
`send a RST'.  So if your T/TCP client sends a SYN-data-FIN to my TCP
server, and I implement exactly what the spec says, your packets will
be blackholed and you won't be able to talk to me.

In practice, most servers just drop the FIN, but I for one
specifically chose that behaviour to be compatible with T/TCP clients.
(I've been specifically asked at least twice to change the behaviour
to what RFC793 specifies.)

I believe this is also the issue with talking to KA9Q.

(And while we *could* update RFC793, I don't think it's valid to say
that a vast pile of systems which correctly implemented the spec are
now incorrect.  That effectively requires a flag day.)

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 18 Sep 1998 09:25:32 -0600 (MDT)

> ...
> > > Sure.  The right thing to do (if HTTP did not have such verbose request
> > > response headers) would be to turn off all modem compression, and
> > > send the HTML content compressed.  Deflate compression does very
> > > much better than anything in modems.
> > 
> > Modems tend to use LZW, always (I trust) with at least 10-bit and at least
> > sometimes more.  Judging from what Paul Mackerras has told me of his
> > comparisons between BSD Compress (RFC 1977) and Deflate (RFC 1979) in his
> > widely used `pppd`, the difference between Deflate and your typical modem
> > on the data that is compressed should be about the difference you see
> > between `compress` and `gzip`, or around a 10% improvement in compression
> > ratio but at a ~2X cost in CPU cycles, and mostly on the side that does
> > not have cycles to waste, the HTTP server.
> > 
>
> The server can cache the compressed object, and only compress it once.
>
> Our measurements show a much greater effect of compression than 10%;
> see: http://www.w3.org/Protocols/HTTP/Performance/#Compression
>
> This are measurements over real modems.

That difference is much larger than 10%, but you neglected to mention the
brand or even variety (e.g. v.34 vs. v.32) of the modems.  Your modems
did about 20 kbit/sec, so they could not have been v.32, but they have
been either v.32bis or v.34.  You also omitted TCP window sizes and mention
of the CTE-DTE speeds, both of which have large effects.  With the high
latencies of modem links, you really want a TCP window of more than 1
or 2 segments, despite the fact that some TCP/IP/PPP stacks for PCs
think 2 segments is a lot.

That your modems appeared to move data at the same speed regardless of
whether data was compressable (i.e. precompressed or not) suggests that
v.42bis was turned off.  Please do the obvious sanity check.  If modem
compression were as absolutely useless as your numbers say, then why would
anyone bother?

That your compressed and uncompressed HTML differed 11K vs. 42K means that
your HTML is over the line into embarassingly compressable territory.  (I
could easily believe that most image-free HTML is over that line.)  Your
test obviously, as you note, did not have involved any images, which makes
it a less than especially interesting test.

Over the years, I've made a bunch of measurements of precompressed, random,
ordinary, and embarrassingly compressable data (e.g. `ttcp` or `ping`
without -P) through SLIP/modems and PPP/modems.  At my previous major UNIX
vendor employer, I maintained the UUCP/PPP/cu/SLIP/etc modem configuration
scripts in my spare time, along with doing PPP, and cajoled the loan of
a pair of modems from everyone who wanted another modem supported so that
I could tune the scripts and make some measurements.  There is no such
thing as a "standard modem" as mentioned in your report.  All modems claim
to support various standards, but their performance varies radically,
whether you are concerned with analog issues (i.e. phone lines) or bit
moving.  I've seen variations up to a factor of 4 just by switching modems
at a single modem-to-modem bit rate; never mind the modems that simply do
not work on some phone lines.  For example one ome major brand of v.32bis
modem seemed to be unable to send at the same time it was receiving, or
vice versa, and so its performance with ordinary data with TCP/IP/PPP/modem
is other than impressive because of TCP ACKs.  (v.32bis is the old,
pre-"56K" standard.)

You can compare for yourself the effects of modem compression by itself
with Deflate compression by itself comparing `gzip` with `compress` on
your HTML pages.  (You might want `compress -b10` to make LZW as poor as
the worst v.42bis modems.)  I bet you will rarely see more than 10% and
will never more than 25% difference between LZ78 (or is it LZ77?) used in
gzip and Deflate compared to the LZW used in v.42bis (modems) and
`compress`.   (This obvious test is the standard counter to modem and PPP
box salescritters that claim that their boxes get "8:1" or "9:1" on
"typical" data.)

Note that the v.42bis modem compression standard says that when a modem
notices that the data did not compress, it should send it in the clear.
That means that once you've paid latency that v.42 packetizing costs,
v.42bis compression is free--well, only in good modems.  There are of junk
modems that can move data or compress but not both, and not very well.

Lest you say that the junk modems prove your point, I've found the USR
Couriers, among some others, to be other than junk.  The last time I bought
one, it cost me about $250.

Never mind the effects of PPP compression.  I have clocked my favorite
flavor of CCP at 3-digit compression ratios over real links, with data
patterns used in `ttcp` or `ping` without -P.  (I added -P to the `ping`
program seen in some systems precisesly to foil modem and PPP compression.)