Subject: Stateless SFTP server and READDIR race condition.

From: Michael Closson <[email-masked]>
Date: Thu, 13 Apr 2006 15:44:02 -0600

This is kind of a nit-pick, but I seems to me that the SFTP server
cannot be implemented as a stateless server because the server must
track the directory offset to support multiple READDIR calls.  Also,
there is a potential race condition when there are multiple READDIR
calls.

NFS READDIR(at least version 2 and 3) returns a cookie to the client,
and the client passes the cookie back to the server so the stateless
server knows where it left off.  NFS version 3 also includes a cookie
verifier to detect directory changes that could occur between multiple
readdir calls.

I'm wondering if this is an issue that should be considered, both the
statelessness and detecting directory changes that happen in between
SFTP READDIR calls.

This is kind of a nit pick for a couple of reasons.  First,
statelessness is useful for crash recovery, but since NFS is UDP
(mostly) and SSH is TCP (always), it is easy for the client to detect a
server crash.  Second, since SFTP returns multiple directory entries
instead of one (as is the case with NFS) the window of vulnerability for
directory changes is only an issue for very large directories.

Thanks!

Mike Closson

---

From: der Mouse <[email-masked]>
Date: Thu, 13 Apr 2006 20:17:55 -0400 (EDT)

> track the directory offset to support multiple READDIR calls.

I don't think you'd want to anyway, because the key exchange and
authentication overhead would be insane.  Rememebr that SFTP is
intended to be used as an SSH subsystem, and it's designed for that
environment.  While it doesn't necessarily have to be done there, it
does mean that its design tradeoffs are partially optimized for that
environment.

> Also, there is a potential race condition when there are multiple
> READDIR calls.

Could you detail this?  I don't see any race condition unless there are
directory-modifying calls involved.

> Second, since SFTP returns multiple directory entries instead of one
> (as is the case with NFS)

...huh?  There's no inherent reason a single NFS READDIR can't return
multiple directory entries.  A server that refuses to do so when there
is space available I would be tempted to call defective, unless there
is some unusual justification (such as its running on a microcontroller
or some such).

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 13 Apr 2006 21:24:29 -0500

> there is a potential race condition when there are multiple READDIR

SFTP needs to decide whether it's a filesystem protocol or a file
transfer protocol.  Then we can figure out what to do about state
recovery across reboots, partitions, etc.  A file transfer protocol
shouldn't really care about such failures, whereas a filesystem protocol
kinda has to.

Nico
--

---

From: der Mouse <[email-masked]>
Date: Tue, 18 Apr 2006 14:14:18 -0400 (EDT)

> I guess that stateless doesn't necessarily mean reconnect on every
> request.

I don't see how it could help it, since the connection is state.

> I guess the question becomes, why would one ever want to implement a
> stateless server?  I don't know.

It greatly simiplifies many aspects of crash recovery.

>> Could you detail this?  I don't see any race condition unless there
>> are directory-modifying calls involved.
> 1. SFTP OPENDIR causes an opendir() on the server.
> 2. 1st SFTP READDIR causes 100 call to readdir().
> 3. Third party modifies directory on server in some arbitrary way.
> 4. 2nd SFTP READDIR causes at most 100 additional calls to readdir().
> Entries could be missed.

True.  But the same applies to any other way of reading a directory on
most systems, even including local ls or moral equivalent.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 18 Apr 2006 13:38:19 -0500

On Tue, Apr 18, 2006 at 12:01:13PM -0600, Michael Closson wrote:
> On Thu, 2006-04-13 at 21:24 -0500, Nicolas Williams wrote:
> Yes, I see your point.
> Readers of this group may be interested in what people are doing with
> ssh and sftp.
> 
> My research group at the U of Alberta is involved in a Grid-like project
> (i.e., Globus/OGSA).  Basically, we want to be able to easily aggregate
> multiple HPC (High Performance Computing) systems at the user-level
> (also non-root level).  A typical use case goes something like this:  A
> researcher has ssh access to multiple clusters (probably administered by
> different groups and there fore no shared file system), and they want to
> load-balance some jobs across these clusters.  So, we need a nice way to
> ship job data back and forth.  We have an overlay file system that
> presents global view of these multiple systems to application programs.
> This overlay file system uses SFTP on the back end.  On the front end
> (i.e., the interposition agent) we have NFS, Samba, FUSE and a library
> that exports a POSIX api (open(), read(), etc.)

Just because NFS is typically implemented in kernel-land doesn't mean
you can't have one in user-land.  In fact, I think there may one or two
user-land NFSv4 implementations out there...

The same applies to CIFS.  And the reverse applies to FTP, and probably
to SFTP.

The point here is: don't get stuck with the wrong protocol because of
any misperceptions you may have.  Code availability probably means that
it is easier for you to use protocols traditionally implemented for your
purposes, but not necessarily, since one of the things you need is
thread-safety, and I doubt much off-the-shelf SFTP code is...

Nico
--

---

From: Michael Closson <[email-masked]>
Date: Tue, 18 Apr 2006 12:01:13 -0600

Yes, I see your point.

Mike

---

From: der Mouse <[email-masked]>
Date: Tue, 18 Apr 2006 14:54:38 -0400 (EDT)

> you can't have one in user-land.  In fact, I think there may one or
> two user-land NFSv4 implementations out there...

Back in the '80s, I wrote an NFS server that largely runs in user-land.
(Not v4, of course, not back then.)  It's not totally userland; it has
a handful of syscalls to help do things by inumber instead of path -
but they're not essential, merely helpful.  I could have kept track of
paths instead.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Michael Closson <[email-masked]>
Date: Tue, 18 Apr 2006 11:37:33 -0600

> does mean that its design tradeoffs are partially optimized for that

Yes, you are correct.  I guess that stateless doesn't necessarily mean
reconnect on every request.  Although ssh connection sharing (openssh)
could be used.  I guess the question becomes, why would one ever want to
implement a stateless server?  I don't know.

I made this comment because in the current sftp draft section 8.1 second
paragraph it states that sftp open/close allows for the implementation
to be either stateful or stateless.  open/close may allow this, but
readdir doesn't, so there is a small inconsistency with the spec.

> directory-modifying calls involved.

Yes, a third party could modify the directory on the server side.

The race condition comes down to this:  (I'm using the OpenSSH
implementation as a reference).

So, I guess it comes down to the implementation of opendir()/readdir()
on the server.  Maybe it isn't a problem at all.

> is some unusual justification (such as its running on a microcontroller

Opps, looks like my recollection of the NFS protocol is a little hazy.
You are correct.  I guess the two protocols are similar in this aspect,
they each return multiple entries, but maybe not all of them (due to
packet size limitations, either an application layer limit or a
transport layer limit).

Mike Closson

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 18 Apr 2006 14:40:19 -0500

> but they're not essential, merely helpful.  I could have kept track of

A client wouldn't need anything special...

---

From: der Mouse <[email-masked]>
Date: Tue, 18 Apr 2006 16:23:33 -0400 (EDT)

>> Back in the '80s, I wrote an NFS server that largely runs in
>> user-land.  [...]

No, this was just a server.  There was nothing special client-side.
(Indeed, if the server conforms to the protocol, it *must* work with
nothing-special clients.)

This also is rather far from anything ssh-related, so unless something
brings it back on-topic, this is my last message on the thread to the
list.  I'll be happy to discuss it off-list with anyone interested.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 18 Apr 2006 18:20:48 -0500

> implement a stateless server?  I don't know.

Statelessness has got to mean that you can recover from having to
reconnect (e.g., because the server or the client rebooted or what have
you [in NFSv4 because of network partitions that lasted longer than time
periods associated with leases]).

> readdir doesn't, so there is a small inconsistency with the spec.

I don't believe what you say about readdir.  If a client's connection is
reset in the middle of a readdir the client can reconnect and continue,
possibly by redoing the readdir completely and comparing to what it had
read up to the point where the connection was dropped.

In all cases if the protocol lacks any support for file leases and/or
directory change notifications then clients may see slightly different
file/directory contents.

When we talk about stateless filesystem protocols we mean server-side
statelessness, not client-side statelessness.  I've not combed through
it recently, but, at the very least version 3 of SFTP is indeed
stateless, to the best of my recollection.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 18 Apr 2006 13:30:27 -0500

> It greatly simiplifies many aspects of crash recovery.

Until you realize that you want locking, and better caching semantics,
and...   And you realize that none of those things can be done in an
entirely stateless manner, that you need to figure out how to deal with
client and server crashes as well as network partitions.

> most systems, even including local ls or moral equivalent.

I suspect that what the above quoted text was getting at is some sort of
directory change notification system; go check out the NFSv4 WG list
archives for why this is not trivial.  BTW, notifications == more state.

Nico
--