SSH has no Host header

We have a challenge with ssh. Each VM has a standard URL that we use for both HTTPS and SSH, for example undefined-behavior.exe.xyz. Just as you can type the domain into a web browser (and the TLS and auth are taken care of for you), you can run:

ssh undefined-behavior.exe.xyz

To get a shell in your VM.

This is much easier to implement if you give each machine its own IP address, but exe.dev gives you multiple VMs on a flat rate subscription.

We cannot issue an IPv4 address to every machine without paying the subscription cost. We can’t just use IPv6 because that means some parts of the Internet can’t reach the VM on the web. This means we have to share IPv4 addresses between VMs.

For the Web, this is a long-solved problem. Many sites can and do have the same IP address. Web browsers send the domain they use to reach the server in an HTTP request. Host Topic. The Exe.dev proxy switches on this header and sends the request to the appropriate VM.

SSH, on the other hand, has no equivalent to a host header. If we reuse IPv4 addresses between VMs, we have no way to send the SSH connection to the correct VM.

How we solved it: ssh ip sharing

Instead of using one IP address for all VMs, we have a pool of public IPv4 addresses. Each VM is assigned a unique address relative to its owner.

So instead of an A record, you’ll get

$ dig undefined-behavior.exe.xyz

; <<>> DiG 9.10.6 <<>> undefined-behavior.exe.xyz
...
;; ANSWER SECTION:
undefined-behavior.exe.xyz. 230 IN      CNAME   s003.exe.xyz.
s003.exe.xyz.           230     IN      A       16.145.102.7

relative to its owner This means that while the IP represented by s003 is used by multiple VMs, it is only used by a single VM owned by this user.

This is all the additional information we need to route the SSH connection. When SSH connects, it presents a public key and comes in through a special IP address. The public key tells us the user, and {user, IP} The tuple uniquely identifies the VM they are connecting to. In diagram form:
A diagram of ssh'ing into the exe.dev VM
Building a proxy that does this requires some cross-system communication: when we create a VM we have to carefully assign it an IP based on the user (or in the near future: team) that owns it. Our SSH proxy needs to be able to determine the local IP the request came from, which is easy on bare metal, harder in a cloud environment where the public IP is NATed to a private VPC address. All this requires specialized management software, so we can’t recommend it as a general solution for people who want to multiplex VM SSH access over a single IP. But consistent, predictable domain name behavior is important to us, so we took the time to build it for exe.dev.



<a href

Leave a Comment