Subject: [ftpext] Soliciting comments/reviews on draft-yevstifeyev-ftp-uri-scheme

From: Mykyta Yevstifeyev <[email-masked]>
Date: Sun, 22 May 2011 08:07:36 +0300

Hello all,

I'm writing to ask people to comment my draft 
draft-yevstifeyev-ftp-uri-scheme, that may be found here:

http://tools.ietf.org/html/draft-yevstifeyev-ftp-uri-scheme-00

This document removes uncertainty with the 'ftp' URI scheme, currently 
specified by obsolete RFC 1738 (see my message: 
http://www.ietf.org/mail-archive/web/uri-review/current/msg01442.html).  
RFC 4395 says that 4 weeks review-and-comment period is fine for 
permanent registration, so please feel free to provide your comments on 
my document during these 4 weeks.

{I understand that ftpext list isn't the most appropriate list, but I 
think posting this message there will allow FTP experts read and express 
their thoughts on this document.}

Please, when responding, keep all "To:" and "Cc:" addresses in the message.

Thanks,
Mykyta Yevstifeyev

---

From: Daniel Stenberg <[email-masked]>
Date: Sun, 22 May 2011 19:45:44 +0200 (CEST)

> draft-yevstifeyev-ftp-uri-scheme, that may be found here:

Hi

I think the document looks fine and basically repeats what RFC1738 says. It 
would be really helpful if it would mention where or how it differs from 
RFC1738.

My concern is "The <ftp-path> Part", section 2.2.3, and the problems aren't 
really introduced by you but repeating existing problems in a brand new 
document seems a bit unnecessary.

The original problem I have with this exists already in RFC1738 as it mandates 
a CWD command for each <cwd> part of the path. There are lots of servers out 
in the wild who are configured to not allow this, but yet allows the client to 
do a single CWD to the entire path. That means "CWD dir1/dir2" works while 
"CWD dir1" fails. We can of course just consider such servers as non-compliant 
but clients may still need to consider this (and I know lots of clients only 
do the CWD <full path> approach).

The next problem in the same section is what to do with empty <cwd> parts. As 
in "ftp://example.com/path1//path2". RFC1738 acknowledges that the <cwd> parts 
may be empty. It continues to say that each <cwd> should result as an argument 
to CWD. But lots of FTP servers will treat "CWD" with a blank argument similar 
to what unix machines does with 'cd' without a directory: it changes directory 
to a pre-determined location (home directory) that certainly is not at all 
what the client wants for this. A "normal" FTP client will therefore not issue 
a separate CWD for empty parts.

In draft-yevstifeyev-ftp-uri-scheme there's no mention of the fact that <cwd> 
can be empty (which isn't really solving how to deal with them) but there's a 
similar mention that each <cwd> should be passed on to CWD.

-- 

  / daniel.haxx.se

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Mon, 23 May 2011 16:19:22 +0300

Hello Daniel, and thanks for comments.

See my responses in-line.

22.05.2011 20:45, Daniel Stenberg wrote:
> I think the document looks fine and basically repeats what RFC1738 
> says. It would be really helpful if it would mention where or how it 
> differs from RFC1738.
OK, I'll add some words on it (probably in a separate Appendix).
>
> My concern is "The <ftp-path> Part", section 2.2.3, and the problems 
> aren't really introduced by you but repeating existing problems in a 
> brand new document seems a bit unnecessary.
>
> The original problem I have with this exists already in RFC1738 as it 
> mandates a CWD command for each <cwd> part of the path. There are lots 
> of servers out in the wild who are configured to not allow this, but 
> yet allows the client to do a single CWD to the entire path. That 
> means "CWD dir1/dir2" works while "CWD dir1" fails. We can of course 
> just consider such servers as non-compliant but clients may still need 
> to consider this (and I know lots of clients only do the CWD <full 
> path> approach).
I think the following text would be OK:

(the algorithm of handling <ftp-path>):
>  (1a) each of <cwd> parts are consistently supplied as arguments
>       to the CWD (change working directory) FTP command after
>       establishing the FTP connection to the server identified
>       by the <host-port> part of the URI; or
>
>  (1b) the whole path (the <cwd> parts together) is supplied
>       as an argument to the aforementioned command;
>
>       [ . . . ]
It addresses your concern, I believe.  Does it look fine?
> The next problem in the same section is what to do with empty <cwd> 
> parts. As in "ftp://example.com/path1//path2". RFC1738 acknowledges 
> that the <cwd> parts may be empty. It continues to say that each <cwd> 
> should result as an argument to CWD. But lots of FTP servers will 
> treat "CWD" with a blank argument similar to what unix machines does 
> with 'cd' without a directory: it changes directory to a 
> pre-determined location (home directory) that certainly is not at all 
> what the client wants for this. A "normal" FTP client will therefore 
> not issue a separate CWD for empty parts.
>
> In draft-yevstifeyev-ftp-uri-scheme there's no mention of the fact 
> that <cwd> can be empty (which isn't really solving how to deal with 
> them) but there's a similar mention that each <cwd> should be passed 
> on to CWD.
I'll correct this issue as well, mentioning the ability of null <cwd>s 
as well as rules for their handling, proposed by you.

Mykyta Yevstifeyev

---

From: John C Klensin <[email-masked]>
Date: Mon, 23 May 2011 13:53:22 -0400

--On Sunday, May 22, 2011 19:45 +0200 Daniel Stenberg
<[email-masked]> wrote:

> I think the document looks fine and basically repeats what
> RFC1738 says. It would be really helpful if it would mention
> where or how it differs from RFC1738.
> 
> My concern is "The <ftp-path> Part", section 2.2.3, and the
> problems aren't really introduced by you but repeating
> existing problems in a brand new document seems a bit
> unnecessary.
> 
> The original problem I have with this exists already in
> RFC1738 as it mandates a CWD command for each <cwd> part of
> the path. There are lots of servers out in the wild who are
> configured to not allow this, but yet allows the client to do
> a single CWD to the entire path. That means "CWD dir1/dir2"
> works while "CWD dir1" fails. We can of course just consider
> such servers as non-compliant but clients may still need to
> consider this (and I know lots of clients only do the CWD
> <full path> approach).
> 
> The next problem in the same section is what to do with empty
> <cwd> parts. As in "ftp://example.com/path1//path2". RFC1738
> acknowledges that the <cwd> parts may be empty. It continues
> to say that each <cwd> should result as an argument to CWD.
> But lots of FTP servers will treat "CWD" with a blank argument
> similar to what unix machines does with 'cd' without a
> directory: it changes directory to a pre-determined location
> (home directory) that certainly is not at all what the client
> wants for this. A "normal" FTP client will therefore not issue
> In draft-yevstifeyev-ftp-uri-scheme there's no mention of the
> fact that <cwd> can be empty (which isn't really solving how
> to deal with them) but there's a similar mention that each
> <cwd> should be passed on to CWD.

Daniel,

I think we really need to be careful about your line of
reasoning here (whether 1738 is correct or not).  CWD was
defined for environments in which not only was the path
separation identifier unknown (take your pick among "/", "\",
".", and ">" at least) but in which there was a real possibility
that 
   CWD a
   CWD b
would have different semantics and lead to a different place
than 
   CWD a<delim>b (e.g., "CWD a/b").

I vaguely recall a discussion about a possible "set path
hierarchy delimiter" command a long time ago; nothing came of it.

And, while some systems (as you point out) construe the
equivalent of "CWD" (no arguments) as, e.g., "cd $HOMEDIR$",
others construe it as "return name of current directory", e.g.,
a synonym of "pwd".  Having a URL with elements that have
semantics that different, depending on what the implementation
does, is just looking for trouble.

By contrast, the semantics of the FTP protocol when no CWD
commands are transmitted (presumably what happens when no
<ftp-path> appears in the URL as specified in 1738) are
perfectly well defined: an FTP server implementation must have a
way to bind a default (or "home" or "startup") directory to the
user/pass [/acct] triple and that is where one ends up and stays.

It seems to me that, if we are doing to do an updated FTP URI
scheme, we need to either:

(1) Clean this mess up and generate a scheme that really
reflects the protocol.   That might well mean deprecating
(although probably retaining for backward-compatibility)
ftp-path in ftp://host/path-element1/path-element2/.../filename
form and moving toward something more like

ftp://host/filename;cwd=path-element1;cwd=path-element2;...;typecode=type;...
(or using query syntax for some or all of those elements).

Cleaning up the mess almost certainly also requires permitting
additional parameters or other mechanisms for ACCT, PASV, and
the sundry extensions that have come along, including those
under consideration by the FTPEXT WG.  Having a WG to extend FTP
(the second one, with several extensions accepted by the earlier
WG) and then approving a URL whose syntax covers only a small
subset of the required commands (and one optional one) from RFC
959 makes no sense to me at all.

As one of many example, this community is on record as not
considering protocols that need authentication mechanisms and
whose only mechanism is cleartext names and passwords acceptable
any more.  We not only have a few mechanisms available for
better FTP authentication, but this draft mentions them in its
Security Considerations section.  But neither of those
mechanisms are accessible via the proposed revised URI.

The spec probably also has to have a serious discussion of what
happens when something is specified that the server rejects.  If
one's model is a URI processor that calls on an FTP client, it
would be useful to figure out and specify what the URI processor
should tell the user when the FTP client gets a 5xx (or worse)
reply back from the FTP server.  I suppose "you lose" could be
an answer but, if we don't think that is acceptable, some
discussion of how one maps from the vocabulary of FTP to the
vocabulary of URIs seems necessary.

I note that this document not only does not contain any
provision for extensions to accommodate additional
commands/parameters (those that have been approved already and
those that might be approved by the FTP WG) either.  That, to
me, is just broken.

(2) Deprecate the FTP URI entirely, name what this document
covers something that conveys
"Restricted-FTP-for-Unix-like-systems:", and move on, hoping
that someone will do a real FTP and FEAT-compatible URI at some
future data.

There is obviously a high-level issue in all of this, which is
whether it either acceptable to the community or a good use of
IETF time to take an old spec and update it in some minor ways,
ignoring known issues because the new spec doesn't make things
any worse.  My position is that it is not a good use of time and
may be irresponsible.   Mykyta and I have discussed that issue
in other contexts and may ultimately need to agree to disagree,
but I worry about the marginal opportunity costs for the
community of trying to review specs like this one.

regards,
     john

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Mon, 23 May 2011 22:19:30 +0300

23.05.2011 20:53, John C Klensin wrote:
> ".", and">" at least) but in which there was a real possibility
> that
>     CWD a
>     CWD b
With respect to this issue, I think mentioning that null <cwd>s must 
cause no commands generated and dent out is OK and suits to the common 
sense.  Defining such parts causing sending "CWD" with no arguments 
really looks wrong; so I think specifying "noting to do if <cwd> is 
null" should be fine.
> 959 makes no sense to me at all.
Complete re-specifying of 'ftp' URI scheme seems quite interesting; yet, 
it's too radical.  The scheme in its existing view has been used in the 
Internet for years.  I don't think folks will be encouraged to change 
their apps to suit new syntax/semantics and will continue using the old 
one, even if it is formally deprecated.  RFC 1738 was written at the end 
of 1994, RFC 1630 - in the middle.  Paul's draft, as I remember, was 
written at the end of 2005; these three documents make no change to the 
'ftp' scheme (maybe 1630 isn't a good example of this, but...).  So, I 
should repeat that even though making up a completely new spec of 'ftp' 
URI is a good deed, it isn't really worth the energy which would be put 
in it since it is very likely to get ignored.
> mechanisms are accessible via the proposed revised URI.
Indeed.  But the current situation is left "as is" is for historical 
purposes as well.  If we mention, eg. user-pass part of 'ftp' URI should 
have an optional "auth-part" of ";AUTH=<auth-type>" (as in many URI 
schemes, such as 'pop' or 'imap'), define a set of initial tokens for 
<auth-types>, establish a registry for them, describe everything related 
to this and then pass the specification to implementators, they will 
probably ignore this new possibility and will continue to make the FTP 
clients compatible with "user:pass" userinfo part, not the new one.  So 
my opinion is just the same as above.
> vocabulary of URIs seems necessary.
On this I can't disagree; so this issue needs wider community discussion.
> those that might be approved by the FTP WG) either.  That, to
See my response #2, it concerns this as well.
> that someone will do a real FTP and FEAT-compatible URI at some
Considered your first proposal, would be a good idea, but considering it 
separately it's probably a bad one.  The 'ftp' URIs in their curent form 
are widely used; so deprecating it won't reault in anything good, in my 
opinion.  I think such redical actions won't be accepted by the 
community; so "updating old spec. correcting minor mistakes" seems to be 
a good approach for now.

Mykyta Yevstifeyev

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 23 May 2011 22:51:23 +0200 (CEST)

On Mon, 23 May 2011, John C Klensin wrote:

> I think we really need to be careful about your line of reasoning here 
> (whether 1738 is correct or not).

I said it was incomplete or possibly most servers are non-compliant. I tried 
to state facts. Please tell me/us where I'm wrong.

> And, while some systems (as you point out) construe the equivalent of "CWD" 
> (no arguments) as, e.g., "cd $HOMEDIR$", others construe it as "return name 
> of current directory", e.g., a synonym of "pwd".  Having a URL with elements 
> that have semantics that different, depending on what the implementation 

I'm just pointing out that the way RFC1738 is laid out and the way 
draft-yevstifeyev-ftp-uri-scheme describes things, it is not working on a not 
insignificant amount of existing servers.

I don't have the answer of how to deal with it in a failsafe way but I'm not 
at all convinced that just adding a "method B" into the spec is a very good 
idea without careful thoughts.

> It seems to me that, if we are doing to do an updated FTP URI scheme, we 
> need to either:

Yes, perhaps. The question is then: are we doing an updated FTP URI scheme?

If we're not, what's the point of just repeating RFC1738 with its flaws once 
again? Isn't there a middle-ground that at least maintains most of what 
RFC1738 says but that clarifies/corrects the biggest mistakes?

If we *are* updating the FTP URI scheme, then surely we have more work to 
do...

> There is obviously a high-level issue in all of this, which is whether it 
> either acceptable to the community or a good use of IETF time to take an old 
> spec and update it in some minor ways, ignoring known issues because the new 
> spec doesn't make things any worse.  My position is that it is not a good 
> use of time and may be irresponsible.

I agree with you John.

I think repeating old known mistakes in a new spec is a very bad idea and I 
would be very much opposed to that. Even if it doesn't "make things worse" in 
the spec, a fresh spec kind of makes the old mistakes more wrong in my view.

-- 

  / daniel.haxx.se

---

From: John C Klensin <[email-masked]>
Date: Mon, 23 May 2011 23:03:51 -0400

--On Monday, May 23, 2011 22:19 +0300 Mykyta Yevstifeyev
<[email-masked]> wrote:

>...

> Complete re-specifying of 'ftp' URI scheme seems quite
> interesting; yet, it's too radical.  The scheme in its
> existing view has been used in the Internet for years.  I
> don't think folks will be encouraged to change their apps to
> suit new syntax/semantics and will continue using the old one,
> even if it is formally deprecated.  RFC 1738 was written at
>...

Actually, modulo some very small differences in syntax and
semantics, the concept of a one-line FTP invocation goes back at
least to TENEX.  That one-line approach has been more or less
   ftp <host> <path>
and, modulo syntactic sugar and the user and password
identifiers (which, as you point out, aren't used very often
with the URI form), the URI just reflects that model.

But this is exactly where we have a philosophical difference.  I
hope I've got this right, but it seems to me:

-- You say that if we make any substantive changes, then no one
will pay any attention to what we say and will keep using what
was specified in 1738.

-- I respond by saying, "Ok, if that is the case, then updating/
replacing the definition in 1738 has no value other than dealing
with the half-obsolete status of 1738 itself.  And dealing with
that doesn't have nearly enough value to justify the effort of
producing a new document and getting it right."

-- Daniel says that a lot of implementations don't do what 1738
says anyway, so repeating what it says is silly.  (I hope that
interpretation isn't too far off the mark.)

-- I respond by saying "Ok, if that is the case, we have a mess
on our hands and it is irresponsible to produce a new spec that
does not clean up the mess.    So, either no new spec or a
cleanup.

-- If you respond to that by saying that you are convinced that
no one would pay any attention to a cleaned up and complete
specification either, then my answer would be that we are
finished: there is no value in spending any time on a new
specification that we are convinced, a priori, that no one is
going to pay any attention to.

And that is where we probably agree to disagree, since neither
of us is likely to convince the other.

Daniel wrote...

> If we're not, what's the point of just repeating RFC1738 with
> its flaws once again?

Up to this point at least, I think we are 100% in agreement.

> Isn't there a middle-ground that at
> least maintains most of what RFC1738 says but that
> clarifies/corrects the biggest mistakes?

Probably.  But then we need to get consensus about what is a
"biggest mistake" and, in particular, whether some of the
interpretations of 1738 which you describe are the desired
behavior or just broken.  As one trivial example, Mykyta and I
seem to think that interpreting "ftp://servername.example.com/"
as an instruction to send "CWD" without an argument is broken;
the implementations you cite seem to differ.   So there isn't
100% agreement on that point.

So, even if we are trying to just "maintain[s] most of what
RFC1738 says but that clarifies/corrects the biggest mistakes",
we have issues with getting agreement, not just copying out and
editing some text.   That, in turn, makes the "clarify and
correct" job less different from "updating" than might be
assumed.

> If we *are* updating the FTP URI scheme, then surely we have
> more work to do...

Indeed.   And, despite my comments above, I'm painfully aware
that figuring out how to accommodate required commands that are
not in the 1738 definition, to think about paths and parameters,
and to allow to extension for new commands without having to
write a new definition of the URI each time is much more work
than trying to copy and clarify the 1738 material.    While I'd
be opposed to it, I can see the sense in a "copy and clarify but
don't specify anything new or different" model, especially if we
hope that an update and revision would come along in the future.

>> There is obviously a high-level issue in all of this, which
>> is whether it  either acceptable to the community or a good
>> use of IETF time to take an old  spec and update it in some
>> minor ways, ignoring known issues because the new  spec
>> doesn't make things any worse.  My position is that it is not
>> a good  use of time and may be irresponsible.
> 
> I agree with you John.
> 
> I think repeating old known mistakes in a new spec is a very
> bad idea and I would be very much opposed to that. Even if it
> doesn't "make things worse" in the spec, a fresh spec kind of
> makes the old mistakes more wrong in my view.

Concur.  And, fwiw, it is procedurally almost impossible because
the requirements for Proposed Standard include the "no known
technical omissions" rule.  An "old known mistake" is certainly
a known technical omission with regard to that rule.  While the
IESG can waive the rule, concluding that a new spec is useful
and necessary enough to justify doing so when there is an
existing published spec would seem to me to require a trip
beyond the plausible.

    john

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Tue, 24 May 2011 17:30:01 +0300

Hi John, Daniel, all,

Thanks for your responses once more.  See my comments in-line.

24.05.2011 6:03, John C Klensin wrote:
>     ftp<host>  <path>
Yes, I've said this.
> producing a new document and getting it right."
See below.
> interpretation isn't too far off the mark.)
Agreed with this as well.
> does not clean up the mess.    So, either no new spec or a
Also see below.
> going to pay any attention to.
I didn't want to say that no one would care attention to an updated 
specification if it was published.  Some would probably do; and the new 
'ftp' URI scheme might be Ok enough to abandon the old one.  However I 
do think it makes sense to document the 'ftp' URI *as it is currently 
used* on the Standards Track.  What I really deem fine to clean up the 
mess, as you say, is:

(1) specifying the scheme borrowing the 1738 basis on Standards Track as 
it is currently used correcting major and minor flaws/uncertainties and 
allowing 1738 to approach its "obsolete" status;

(2) making up the Experimental document proposing a new, experimental 
form of the 'ftp' URI scheme, cleaning up most of the mess in existing 
specification;

(3) observing whether people are satisfied by new proposed syntax or are 
Ok with the old one;

(4) deciding whether to change the official spec using the new form or 
leaving the old one.

This, I think, will allow to propose a new form without having made 
radical changes to Standards Track, but see if it is really fine.  So I 
should agree that updating (in fact, re-specifying) 'ftp' URI scheme is 
certainly necessary, but it should be done in other way, as I proposed 
above.

Now for Daniel's comments.
> Up to this point at least, I think we are 100% in agreement.
I agree as well.
> correct" job less different from "updating" than might be
My personal opinion is that this way is the most satisfactory, following 
the golden mean principle.  And I think it will be possible to reach an 
agreement on the necessary changes to 1738 specification.
> don't specify anything new or different" model,
I'm really glad to hear this :-)  That's what my draft tries to do.  As 
for the expectations for more radical changes, I think they will really 
appear soon (see my comment #1).
>   especially if we
By the way, another requirement for PS is (citing 2026):

>     A Proposed Standard specification is generally stable, has resolved
>     known design choices, [ . . . ]
and, thus, changing the spec radically will certainly fail to suit this 
rule.  But, the way I proposed at the beginning of the message is 
certainly fine to allow the new ftp URI scheme syntax receive these 
"design choices".

So, as far as I can see, we all have reached the following consensus:

(1) repeating 1738 is certainly a bad idea;

(2) repeating 1738 and correcting all known major and minor issues 
should be fine;

(3a) 'ftp' URI scheme really needs more fundamental work; but

(3b) /am not sure if all agree/ it should be done in other way.

Mykyta Yevstifeyev

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 24 May 2011 10:09:33 -0600

> form of the 'ftp' URI scheme, cleaning up most of the mess in existing

What is the scope of the experiment, how will the success of the
experiment be determined, and what will happen if it is a success?

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Wed, 25 May 2011 11:21:16 +0300

> experiment be determined, and what will happen if it is a success?
I hoped I had explained this.  The success of such experiment will be
determined if the new experimental form of the scheme gets widely
implemented and satisfies people enough to replace the existing one
(this is also the answer to "what will happen if it is a success").

I don't clearly understand what you mean under the "scope", but I
suppose I am right saying that the scope is general; but the
experiment is intended to concern application implementators mostly.

Mykyta Yevstifeyev