Multihost SSH Wrapper
MUltihost SSH Wrapper, or mussh, is truly an old utility that you can still find to be of good use for today. At its core, mussh is just a shell script wrapper around SSH that allows you to execute the same command across multiple hosts either in sequence or parallel. The script has been filling a gap that until recently most CM tools have ignored: running one-off commands across your distributed systems.
When we have configuration management, why would we possibly want to do something outside of that? While it is true that you should be enforcing system state with a bigger hammer, sometimes you only need to perform a task once, or infrequently.
Take for example, NTP clock skew. I recently ran into an issue where I found that a service had a dramatically increased network queue time, after a hardware upgrade. NTP configuration and service state were being enforced through CM, so I knew that it was running. I wanted to verify that all the system clocks were synchronized, and I didn’t want to manually ssh to each of them.
$ mussh -l pi -m 2 -h rpi-1 rpi-2 -c ‘sudo ntpdate -u ntp.home’ pi@rpi-1: ntpdate: adjust time server ntp.home offset 0.0151 sec pi@rpi-2: ntpdate: adjust time server ntp.home offset 0.0006 sec
Whoa, that’s a long command. So let’s break it down.
Sync NTP across multiple hosts concurrently
- mussh \
-l pi \# Set the ssh username
|-m||2 \||#||Run on two hosts concurrently|
|-h||rpi-1 rpi-2 \ #||Hostnames for the command|
-c ‘sudo ntpdate -u ntp.home’ # Sync ntp with this peer
The resulting output is in the format of hostname: output of the command, which is useful when you run commands that you expect a single line from. You can optionally pass the -b flag to have output buffered by mussh, otherwise the output from all systems will display interwoven with each other, if you are expecting multiple lines of output per host.
We also make three assumptions in this example. The first is that you have a key-based ssh authentication working between your computer and the list of hosts (you are using keys, right?). You also need to have ssh-agent running and working, so you’re not prompted for your private key password for every host. Finally, this example assumes that the pi user is able to execute sudo on ntpdate without a password prompt, which works well for demos in a lab environment but is not a best practice for production.
Well, that’s all fine and dandy, but I don’t want to type all the server names in for every command! Luckily, mussh supports reading the hosts one line at a time out of a file with the -H option. So instead, grab them from whatever storage mechanism you have (flat-file, database, RESTful endpoint) and pipe them in!
$ grep “rpi-” servers.txt | mussh -l pi -H – -c ‘uptime’ pi@rpi-1: 21:12:54 up 4 days, 23:18, 1 user, load average: 0.00 pi@rpi-2: 21:12:54 up 4 days, 23:18, 1 user, load average: 0.00
In this snippet, I’m searching through a text file for my Raspberry Pi “servers” and sending those through mussh. The secret is in the -H –, which tells mussh that the file to read from is stdin.
Enroll in DevOps Training, for getting better insight into its present day scenarios.
We still have a lot to learn from the past about where we can go in the future. The tools of our past live on and inspire innovation, daring us to replace them with the next generation. Some of the tools discussed here have successors that are still in infancy. Some of them are still in active development and could use an influx of motivated developers to push them to the next level. All of them have contributed to getting us where we are today and deserve — if nothing else — a tip of the hat.