Subject: RE: [saag] draft-kwatsen-reverse-ssh-01 submission for review

From: Kent Watsen <[email-masked]>
Date: Tue, 5 Jul 2011 15:32:09 -0700

Allowing the SSH server open channels on the client has a lot of merit.

In trying to extend the generic SSH server (listening to port 22), would the client start an application-specific subsystem having the business logic knowing, for instance, how many channels to open, etc?  This solution would likely require a PAM module auth the client (i.e. the device) in the app's database, since its surely not a system-level user.  This solution would use more system resources (processes, domain sockets, etc.) than our current implementation.

A less resource intensive option would be for the "server" to be the application itself, linked to a SSH server library.   Of course, this server wouldn't be able to listen on port 22, since it wouldn't be generic.  Would not being generic defeat the purpose of trying to do this inside the SSH protocol?

In both cases, the solution would rely on the developers of the various SSH apps and libraries to support this ability.  This might be difficult sell given the use-case is somewhat limited to network management; not that it couldn't be used for other purposes, I just can think of any other.  Assuming we're able to update the SSH implementations, we'd then have to convince all the devices to use this new version of the SSH client - the role-out could take awhile...

I'm still a bit partial to my original proposal, bootstrapping the standard SSH protocol, so the device is always running `sshd`.  This solution is easy to implement and works with all existing SSH apps and libraries.  If there is any interest in going back to this approach, we could greatly improve the -00 draft by enabling the MAC and host-key algorithms to be negotiated, exactly like SSH, instead of mandating the use of hmac-sha256.  Any interest in going back to this approach? 

Thanks,
Kent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Kent Watsen
Sent: Monday, June 13, 2011 6:00 PM
To: Jeffrey Hutzelman
Cc: [email-masked]; [email-masked]
Subject: Re: [saag] draft-kwatsen-reverse-ssh-01 submission for review

Hi Jeff,

Thank you for sending out your comments!

> So, let's make sure I have this right.  A device "calling home" is going
> to connect to some server on port 22, and expect the server to
> immediately pick up the role of being an SSH client?

Yes, this is the proposal

> How is the server to distinguish between such a device and a normal ssh
> client being used by an admin who wants to log in to the server?  Or by
> some other protocol that runs over ssh?

It can't, the server has to *know* that it's having this purpose.  As mentioned before, the devices can't connect to a generic SSH server, since application-level business logic needs to be linked-in to open channels to the device and what not.  This is why is can't be a generic SSH server.  It's also why a distinct IANA-assigned port might be good, so as to discourage standard SSH clients from trying to connect to it.

> I'm pretty sure I recall the discussion on this point involving some
> sort of negotiation, rather than a requirement that both sides know a
> priori which protocol variant they're going to speak.  In fact, I seem
> to recall der Mouse suggesting use of a pre-kex extension packet to
> indicate this.

True.  Not that it's a good excuse, but it I couldn't find a way to have a pre-kex extension packet that wouldn’t also necessitate versioning the SSH Transport Layer protocol (SSH 2.1).  I wasn't sure if this was going to be OK.  I neglected to mention anything about this when I sent out the update - thanks for catching it!  If desired, I'll certainly update the draft to complete this item...  [PS: I like more your idea to let the SSH server open channels on the SSH client, more on that below]

> This is not the way to achieve what you're trying to do.  Nor is
> specifying use of only particular host key algorithms, which rather
> badly breaks modularity.  Further, what you've done doesn't actually buy
> anything.  If the HMAC is precomputed before the device that will act as
> SSH server is deployed, then there is no operational advantage over
> X.509 certificates.  On the other hand, if the HMAC is computed at the
> time of the connection, then both the client and server must know the
> key _and_ there must be a different key for each client/server pair (so
> one device cannot impersonate another).  In which case, you may as well
> preshare per-device RSA keys instead of per-device HMAC keys.
>
> Except in cases like X.509 (where a host key is associated with a
> long-term signature by some trusted third party), SSH proves server
> identity and authenticates host keys as part of key exchange.  If none
> of the already-defined KEX methods are suitable for your needs, then it
> might be possible to define a new one along the lines of what you've
> described.  However, I seriously doubt its utility, for the reasons
> described above.

I expected this to be challenged, as it's quite unique.   Let me start by stating that we (Juniper) have been using an HMAC to secure reverse-ssh connections for a few years now.  We use the HMAC in a bootstrapping message similar to that described by the -00 version of this draft.  Creating the "hmac-*" family of algorithms was only to preserve this functionality in the SSH protocol itself.

The way it works is as follows.  The NMS creates an entry in its database for the device it expects will connect sometime later.  The entry is keyed by a "device-id" and a unique HMAC value is stored for it.  Later, Juniper will drop-ship the device (with factory-default image) directly to its destination (i.e. a retail outlet store), where it is expected that the *store manager* will be able to get it up and running.  Being not very technical, the store manager follows simple instructions sent from HQ (plug phone cable here, plug power cable here, wait for light to blink, call this number, etc.).  Ultimately, the device needs to get enough config to initialize its network stack and securely "call home" to its NMS.  The device MAY get this information from a DHCP server and/or USB pen-drive plugged into it and/or via its web/cli interfaces.  For an HMAC, we'd never put the HMAC-key on a DHCP server to be retrieved via option 43 (as that would be insecure), we either expect the HMAC-key to be read off a USB drive or manually keyed in by the store manager.

I'm not sure what you mean by modularity, but regarding operational advantage over x.509 certificates, consider the following: 1) there's no opportunity in the above description for the store manager to get a x.509 certificate for the device's dynamically-generated private-key, 2) the HMAC is much simpler to key in than a PEM file (for when cli/web interface is used to do that)

> It sounds to me like you really _don't_ want the device to be the SSH
> server at all; you only think you do.  I suspect you really want the
> device to be the SSH _client_, except that once the connection is up,
> you want the server to open some sort of session to the client to
> execute commands, transfer files, run netconf or SNMP, or whatever.
>
> Interestingly, this role reversal -- allowing the SSH server to open
> sessions and run commands on the SSH client, requires no change to the
> SSH protocol at all.  It simply requires that the client sit around
> waiting for a channel-open request from the server and accept it when it
> comes around.  This is exactly the sort of situation in which it is
> appropriate to behave contrary to the SHOULD in RFC4254 section 6.1.

This is a very interesting idea; it would be *much* simpler to allow the server to open channels on the client.  Presumably the client would have to be configured to indicate which subsystems are allowed (tty, sftp, netconf).

Of course, as mentioned above, we still couldn't use a generic SSH server.  Instead, it would have to be a SSH server linked to some application code to do the domain-specific business logic.  For this reason, it would be good to run it on another port, but now it'd be hard to justify for an IANA-assigned port, since it really is the SSH protocol and there already is a port for that.  Instead, applications would likely need to use the dynamic/private port range.

Thinking this through, using the store-manager example from above, let's say the device SSH's to the application.  The device authenticates the server's SSH host key, which could either be a x.509 cert or an HMAC-* key.  Then the device logs into the application (userauth) again using either x.509 certificate or HMAC-* key.  Then the application (the SSH server) opens channels on the device (the SSH client).  

A couple thoughts about how identities can be determined:

1. the device can be configured to know which user it should run the SSH client as.  We need to ensure a user is specified since Common Criteria requires the device to identify a "user" for each auditable event.  Note that we can't leverage RFC 6125 (Service Identity), since section 1.7.2 specifically left client or end-user identities out of scope.

2. the application can use the username the device passes as the "device-id" value.   This seems obvious, but it's different then before and hence worth stating explicitly.  Also, since the username value is passed inside the SSH tunnel, it would be OK to pass a serial-number, something the other solutions couldn't recommend for security reasons.

It would be interesting to review a few SSH Server library implementations (i.e. libssh) to see if they provide the hooks an application would need to implement its business logic...

> FWIW, I'm fairly concerned about the hmac-* algorithms described in this
> document, which don't seem to provide any guarantee of freshness.

I agree that it's not in the draft.  I think that a "Security Consideration" would suffice though.  Our NMS does replace the HMAC on devices, using a cryptographic PRNG.  That said, I don't believe entropy is lost over time, since the message that is signed changes only when the host-key changes...

> I'm concerned that there is no description in this document as to how
> the SSH client is expected to authenticate itself to the device.  This
> is fairly important, particularly since in the described deployment
> scenario, the device is likely about to give nearly unlimited power to
> the SSH client.

True, and it also left out how the device's networking stack would be configured.   FWIW, section 6 says:

  "This RFC does not attempt to define any strategy for how an initial
   deployment might obtain its bootstrapping "call home" configuration
   (address to connect to, signature algorithm to use, authentication
   credentials to use, etc.).  That said, implementations may consider
   use of a DHCP server or a USB pen drive as viable options for these
   kinds of deployments."

We primarily use a USB drive, for the device to read on its first boot.  The USB drive can contain everything the device needs to "call home" from a factory-default configuration.  I didn't think it was important to the protocol change being discussed, though it's clearly important to the overall solution.

> Finally, I'm very concerned about the statements in the first paragraph
> of section 7, which assert that the proposed role reversal does not
> introduce any new security concerns.  On the contrary, the proposed
> protocol is very nearly akin to calling someone up and saying "Give me
> your password", later followed by "by the way, I might be your bank".

That's not true.  Section 4 states:

  "in order to enable the Reverse SSH server to identify the
   Reverse SSH client and automatically authenticate its SSH host key,
   each peer SHOULD only advertise support for one of the following host
   key algorithms:"

and then goes on to list just the x.509 and the hmac-* algorithms, both of which emphasize associating an identity with the public key.  So it's not "I might be your bank", it's "you know I'm your bank because my identity is signed by something you trust" (either a CA or an HMAC-key).

> We've discussed "reverse SSH" ideas before*, and they haven't gained any
> traction.  Part of the problem is that server and client authentication
> is not symmetric in the SSH protocol, and there is a basic assumption
> that a client knows a priori what server it intends to talk to, while
> the server does not know what client will be talking to it.  Attempting
> to falsify this assumption and still have things work typically results
> in warping the protocol rather badly.

Exactly.  This is why my original draft stressed that the device would *always* be the SSH server, regardless which peer initiated the underlying TCP connection.  I wasn't trying to "falsify this assumption", but merely enable to device to initiate the underlying TCP connection, everything else remains standard SSH protocol, with the device as the SSH server.

> I'm not saying a reverse SSH is not possible, and with a few more
> iterations this document might well become one.  However, as mentioned
> above, I think it's probably not the right approach to solving the
> stated problem.

I'm motivated to do whatever it takes to get a solution standardized.  Juniper doesn’t want to have a proprietary solution and other NETCONG WG members have asked for it as well.   I think that, with this round of Q&A, that we've gotten down to the core of the issue.  Hopefully we'll iterate to something soon now. 

Thanks again,
Kent

_______________________________________________
saag mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/saag

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 07 Jul 2011 14:51:23 -0400

On Tue, 2011-07-05 at 15:32 -0700, Kent Watsen wrote:
> In trying to extend the generic SSH server (listening to port 22),
>  would the client start an application-specific subsystem having the
>  business logic knowing, for instance, how many channels to open, etc? 

Either the client would have to do something explicit, like starting a
shell or a subsystem, or the server would have to recognize that client
(say, based on the username it sent or how it authenticated or both) and
do something appropriate.

>  This solution would likely require a PAM module auth the client (i.e.
>  the device) in the app's database, since its surely not a system-level
>  user.  This solution would use more system resources (processes,
>  domain sockets, etc.) than our current implementation.

Now you're talking about implementation details.  The SSH protocol
doesn't know anything about PAM or "system users" or such things.

> A less resource intensive option would be for the "server" to be the
>  application itself, linked to a SSH server library.   Of course, this
>  server wouldn't be able to listen on port 22, since it wouldn't be
>  generic.

You could do that, too.

> Would not being generic defeat the purpose of trying to do
>  this inside the SSH protocol?

You mean, as opposed to trying to make the SSH protocol turn around and
operate in reverse halfway through, wedge authentication mechanisms into
the wrong role, etc?  No, I don't think so.  Even if you run the
application on its own port, you still have a much cleaner protocol this
way.

> In both cases, the solution would rely on the developers of the various
>  SSH apps and libraries to support this ability.  This might be
>  difficult sell given the use-case is somewhat limited to network
>  management; not that it couldn't be used for other purposes,

Frankly, you will probably have a hard time getting upstream OpenSSH to
accept patches to implement this or any other approach.  They're _very_
conservative about changes.  On the other hand, I would expect an SSH
server library already to provide most of what you need.  Servers
already have to be able to open channels for things like TCP port
forwarding.

Incidentally, if you're a little bit less picky about how much
flexibility you allow, there may be approaches you can implement today
without any changes to client or server software.  For example:

1) The client (device) connects to the SSH server in the usual fashion.
   It then starts a subsystem which is your field setup application.
   On the client side, the subsystem's channel is connected to a shell:

   ssh -s <server> fieldsvc | sh

   ... except that you need to connect both the shell's stdin and stdout

2) On the client, start something that listens on some local port (
   loopback interface only) and connects each new client to a shell.
   Start an SSH connection with port forwarding back to that port,
   and run the provisioning app on the server as a command or subsystem:

   ssh -R 0:127.0.0.1:rawshell -s <server> fieldsvc

   again, it's a little more complicated, because you need to collect
   the port number allocated by the server and somehow feed it to the
   fieldsvc subsystem (perhaps on stdin?)

>  I just
>  can think of any other.  Assuming we're able to update the SSH
>  implementations, we'd then have to convince all the devices to use
>  this new version of the SSH client - the role-out could take awhile...

Well, obviously if you have things in the field already using the
"reverse SSH" approach, you'll have to support them for a while.  That's
going to be true no matter how this ends up, and is simply part of the
cost of shipping something before relevant standards work is finished.

-- Jeff