In a previous article we talked about how to use ssh keys and an ssh agent.
Unfortunately for you, we promised a follow up to talk about the security implications of using such an agent. So, here we are.
If you are the impatient kind of reader, here is a a few rules of thumb you should follow:
Never ever copy your private keys on a computer somebody else has root on. If you do, you just shared your keys with that person.
If you also use that key from that computer (why would you copy it, otherwise?), you also shared your passphrase. I generally go further and only keep my private keys on my personal laptop, and start all ssh sessions from there.
Never ever run an ssh-agent on a computer somebody else has root on.
Just as with the keys, I generally don't run ssh-agents anywhere but my laptop. And when I say "has root on", consider that you are both trusting that person to not abuse his privileges, and to do a good job at keeping the system safe, up to date, and without other visitors.
Only forward your agent connection to machines you trust.
As you will see further down in this article, forwarding an agent is equivalent to sharing your keys with anyone who managed to get root on that machine. And this is not theoretical: getting access to your keys takes at most a few lines of a shell script.
Make sure your keys and your agent are unloaded when you log off your machine.
If you are one of the old school guys that simply starts his agent with something like:
if [ -z "$SSH_AUTH_SOCK" ] ; then eval `ssh-agent -s` ssh-add fi
in his .bashrc, don't forget that every time you open a terminal you are creating a new agent
that nobody will ever kill. It will remain happily hanging there forever with all your
keys ready for anyone to use.
(the snippet of code is the one suggested on various threads, including vairous stackoverflow answers)
To protect my agent forwarding, I personally follow a 5th rule:
Use different keys for different purposes, and keep them in different agents.
The reason for this rule is a direct consequence of the other rules, and is best explained with an example: let's say in order to connect to the servers at work you must use ssh keys. But these servers are on a private network, so you must first use agent forwarding and connect to some sort of "gateway".
Every time you connect to the gateway with agent forwarding you give the ability to anyone on that machine with root to use any and every key loaded in your agent.
If any of those people gains access to any other server at work, well, that's life. Something my employer will need to worry about. At the same time, I really don't want those people to gain access to my home server, personal github account, or to the VPS I use to backup my family photos.
What I do: one key for work, one key for home, one key for backup server, "one key per customer", or "per security domain". And I do the same for agents, as otherwise agent forwarding will expose my keys: one agent for work, one for home, and so on.
If I end up forwarding the agent to a compromised machine, the attacker will gain access only to machines within that domain.
Sounds like a giant pain to manage and use? Not really, if you use something like ssh-ident [disclaimer: I'm one of the authors].
Now that we have covered the bases, let's try to cover some of the reasons behind those recommendations...
If you do a quick search on how to use an ssh-agent, most pages will tell you to start an agent by using something like:
$ eval `ssh-agent`
simple and fast, isn't it? But annoying to do every time you log in. Go back on google and search "how to automatically start ssh-agent", and you'll find many suggestions to add something like:
if [ -z "$SSH_AUTH_SOCK" ] ; then eval `ssh-agent -s` ssh-add fi
to your .bashrc. Problem solved? Not really. Now for every console you open, you
end up with a new agent. So back on google, and after some time you will find
some variation of:
SSH_ENV="$HOME/.ssh/environment" function start_agent { echo "Initialising new SSH agent..." (umask 066; /usr/bin/ssh-agent > "${SSH_ENV}") . "${SSH_ENV}" > /dev/null /usr/bin/ssh-add; } # Source SSH settings, if applicable if [ -f "${SSH_ENV}" ]; then . "${SSH_ENV}" > /dev/null ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || { start_agent; } else start_agent; fi
(from one of the most voted answers on stackoverflow)
Which in short keeps the details of the agent in a file, tries to load it, checks if that agent is still running (after a reboot or similar), and if not, it starts another one.
I personally don't like grepping for ssh-agent and checking pids, and I don't like the fact that the script above may break agent forwarding, as it does not detect any agent already available.
So I much prefer versions of the script like the one here:
ssh-add -l &>/dev/null if [ "$?" == 2 ]; then test -r ~/.ssh-agent && \ eval "$(<~/.ssh-agent)" >/dev/null ssh-add -l &>/dev/null if [ "$?" == 2 ]; then (umask 066; ssh-agent > ~/.ssh-agent) eval "$(<~/.ssh-agent)" >/dev/null ssh-add fi fi
which just queries the agent for available keys. If none can be found, it will try to load the agent config from a file, and if still can't connect to the agent, it will start a new one. This version has the added benefit that if your window manager has an agent already running, you will use it. Easy peasy, right?
Well, there are a few problems with this approach:
Your agent will run forever! And keep your keys with it.
You have one agent for all your keys, which violates the 5th rule at the top of the document.
Let's try to solve one problem at a time, so let's try with problem #1 first.
You could:
Specify a maximum key lifetime with the -t parameter. For example,
-t 3600 will keep your keys in memory for at most one hour.
But what happens after an hour of inactivity? Well, your key will disappear, and the next time you try to use ssh it will simply prompt you for your password. That's right, as the key is gone, it doesn't know there was a key in the first place. It will not tell you "look, we need to reload your key" or "ay yo, one of your keys has expired, give me your passphrase again, and I'll happily try to reload it".
This is generally taxing on my brain, as every time this happened to me, I had to reconcile the password prompt with the fact I always use the agent, and come up with "oh drat! my keys expired, let's run ssh-add again".
Annoying, isn't it? You can make it simpler with a few tweaks to ~/.ssh/config, but it still is pretty annoying.
What I ended up doing in the past, was well, never use ssh directly: instead, use a shell script that would check if my keys were still there, and if not, call ssh-add first magically. Complicated? that is another thing that ssh-ident can do for you.
Kill the ssh-agent when you are done using it. Easy peasy, no?
You could try using .bash_logout. But if you do, and your shell 'execs' another
command, crashes, your ssh terminal dies, or you use screen or tmux,
well, it won't work very reliably, your ssh-agent will not be killed.
Feeling brave? Maybe you can trap EXIT 'killall ssh-agent' or something
similar. But this still has many of the same drawbacks.
The most reliable method I found was the exec support in ssh-agent,
that by looking around the .net, seems also the least mentioned?
After ssh-agent you can specify a command to run. That command will be
started with the rigth environment variables set, and ssh-agent will keep running for as long as that command is alive.
For example, if I type something like:
$ exec /usr/bin/ssh-agent /bin/bash
from my shell prompt, I end up in a bash that is setup correctly with
the agent. As soon as that bash dies, or any process that replaced
bash with exec dies, the agent exits. Simple enough, I could add it
to my .bashrc, no? Watch out for loops, and well, you'll be disappointed
to find out that in each and every terminal, you will end up with a
different ssh-agent, needing to run ssh-add every time.
If you use a graphical interface, you can probably use this approach to load your window manager, so all your terminals will have an agent, which leads us straight into the 3rd approach...
The third method is to just rely on your distribution.
Given how many people are using ssh-agent today, many distributions just
start your window manager with ssh-agent or some equivalent above.
That way, you have a nice ssh-agent tied to your session, which is killed
when you log off. Some distributions even use dbus to start and manage
an agent, which I have not dug into yet.
This has worked on and off for me as I upgraded laptops, changed window
managers, login managers, and various versions or the graphical interfaces
I use felt entitled to replace ssh-agent with something else for the sake
of annoying their users.
To this day, the method I found most reliable and comfortable with is to 1) wrap my ssh around a
script, that 2) load agents keys as necessary, and 3) expires them after a certain timeout.
Now that we have determined that running and killing an agent is not as easy as it might seem, let's look at what someone can do with root access on a machine running your agent.
First, he may try to get your keys out of it. This is not as hard as it seems, you can find many tutorials online on how to do it.
It boils down to dumping the memory of the ssh-agent, and looking for the keys in memory.
Second, he may try to just use your agent. This literally requires no skill or tool whatsover. Let me give you an example, let's start by loading an agent, a key, and verifying it works:
$ eval `ssh-agent` $ ssh-add ~/.ssh/my-private-key Enter passphrase: $ ssh-add -l 4096 0a:3c:c9:f7:d0:7a:6d:d2:c0:13:c6:0f:15:12:39:1d my-private-key
Now let's act as another evil user who has access as root to the machine:
$ su # ssh-add -l Could not open a connection to your authentication agent. # ps aux |grep bash ... myself 32684 0.0 0.0 18028 2008 pts/5 S 17:17 0:00 /bin/bash ... # . <(cat /proc/32684/environ |xargs -0 -i echo {} |grep SSH) # ssh-add -l 4096 0a:3c:c9:f7:d0:7a:6d:d2:c0:13:c6:0f:15:12:39:1d my-private-key
All the attacker had to do was find the PID of one or my processes, import the right environment variables, and well, profit! The magic was a single line of shell.
Turns out that the same exact trick used above works with agent forwarding: find a process your victim is running, look at his environment, and well, configure yours to use his agent forwarding socket. Total time to use your keys: < 1 minute.
The only improvement here is that the attacker can't steal your keys. Also, he can only authenticate for as long as you are logged in, both of which sound like a win. But is this such an improvement?
Keep in mind that the attacker can write a 2 line shell script to, for
example, scan all the hosts nearby with nmap, and automatically run ssh-copy-id
to install his keys on your machine while you are logged in.
Or keep watching what you connect to, and install his key on every such host. Hard? Not really:
while :; do servers=`pgrep -u victim -a ssh |sed -ne 's/.*ssh //p'`; \ test -z "$servers" && { sleep 1; continue; }; \ ssh-copy-id -i ~/.my-evil-key.pub $servers; done;
will basically intercept any ssh command you run, and install the attacker's
keys on your remote server.
In short: even a few minutes of access to your agent will enable an attacker to do a lot of damage, escalate the number of machines it has access to, and install backdoors to access your system at the most convenient times.
There is one more problem with the naive approach to ssh-agents. Let's say you go the route of having at least one key per customer, or per "security domain", but still use a single agent.
One thing to keep in mind is that when you try to login into a remote host, ssh will try authentication with all the keys you have loaded, one at a time, one after the other.
This works ok for as long as you have a few keys. As soon as you start having many keys, with many being like more than 5, the remote server will kick you out even before you are able to prove your identity.
That's right: most ssh servers allow a maximum number of authentication attempts before killing your connection. Each key you have loaded counts as an attempt, and if you have more than a handful of keys, you will never be able to use your last ones.
Sites like github.com or gitorious also use your key to verify your identity.
If you have a work account and home account, for example, you will always submit
patches or login as the first key you have loaded in your agent, fancy, not?
I probably sound like a broken record by now, but something like ssh-ident allows you to keep different keys in different agents, easily, while loading agents and keys on demand, keep your identities separated, and easily set a timeout while reloading all keys as necessary.
It is not for everyone to use, but it has served me well so far, and addresses most of the issues discussed in this document with no effort on your side.