Persistent Reverse SSH Tunnels with autossh and systemd

Christopher Hobbs
Software Development

Once in a while, you may have a system you need to access remotely
but want to avoid the overhead of setting up remote access yourself (or you might not have permission to make those changes at all). Some examples may be remote sensors collecting data or temporary hosts used during prototyping on networks you don't control. A simple way to handle this problem is with reverse SSH tunnels.

If you need the tunnel to be persistent for a period of time and
gracefully handle occasional connectivity issues, creating a system
service for [autossh]( might be the
way to go. Note that if the network is jittery or unreliable, a solution
like [mosh]( may be a better option.

Let's take a look at how we might set up autossh. To get started, you'll
need a pair of machines:

- A remote server running an SSH server that you control
- Your local target for the tunnel with autossh installed and a listening ssh server

In our case we'll be using Ubuntu 22.04 LTS but most any Linux distro
running systemd should work.

Ensure your remote server has a user account set up that you'd like
to use. It's wise to set `PermitRootLogin` to `no` in your
`sshd_config`, especially since this machine will be used as a jump box
of sorts, giving you access to foreign networks.

On the local target, generate an SSH key with `ssh-keygen -t rsa`. You
may use any key type that you wish. RSA is specified here for
uniformity. Once that key is generated, copy the new public key to the
remote server:  
`scp ~/.ssh/ user@remote_server:.ssh/authorized_keys`.  This
assumes that you haven't added any other keys to the remote server yet.
If you happen to already have an `authorized_keys` file, simply append
the contents of your local `` file to the remote
`authorized_keys` file.

Now you can make sure that you can establish a reverse tunnel
before proceeding. From the local target, run:
`ssh -R 2002:localhost:22 user@remote_server`. `-R` tells SSH to
create a reverse tunnel. `2002` can be any valid port number, and it
represents the additional port that'll be open on the remote machine
and it'll be forwarded to `22` on the local target. If all goes well,
you should be dropped into a shell on the remote server.

Go ahead and open a new terminal from any machine with access to the
remote server and run `ssh -p2002 user@remote_server`. Rather than
being dropped into the remote server, you should get a shell on the
local target. If this works, you should be ready to proceed with
setting up autossh and a systemd service to handle an automatic
persistent connection. Go ahead and close the two terminals you used
to test this out. Note that the use of `-N` will cause SSH not to
execute any remote commands, and you can simply add it to your reverse
shell command if you need the connection temporarily.

Start by creating a service file for autossh. It should look something
like the following. Be sure to replace all instances of `USER` and
`SERVER` with your usernames and remote server address respectively.
The first instance of `USER` is the local target's user while the second
instance is the remote server's user account.

Description=persistent autossh tunnel


ExecStart=/usr/bin/autossh -M 0 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o "ExitOnForwardFailure=yes" -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=false -N USER@SERVER -R 2002:localhost:22

# kill process
ExecStop=/usr/bin/killall -s KILL autossh


Once the file is properly edited and saved, copy it to
`/etc/systemd/system/autossh.service` and reload your systemd manager
configuration with `sudo systemctl daemon-reload`.

You can now start and enable the new service with:
`sudo systemctl start autossh.service && sudo systemctl enable autossh.service`

Check `sudo systemctl status autossh.service` or
`sudo service autossh status` to see if everything went well. If there
are no errors, you should be able to run `ssh -p2002 remoteuser@localhost`
from your local server and be dropped into a shell on your remote host.  
If the local target restarts or if its network connection drops occasionally,
it should automatically re-establish the tunnel for you.

Continue reading