Bolt + Wash

If you’re not using Puppet Bolt, check it out.

If you’re using Bolt, then we created the Boltwash plugin that lets you navigate your inventory. Let’s check it out

wash . > stree bolt
bolt
└── [group]
    └── [target]
        └── fs

Exploring Boltwash

As you can see, the Bolt plugin organizes Bolt targets as groups. It also mounts something called fs (a view of the filesystem on that target). Let’s explore it a bit more.

wash . > cd bolt
wash bolt > ls
all/
my-docker-app/
my-linux-nodes/
my-windows-nodes/

I have an inventory setup organizing some machines by where they run. Your groups would be organized however you like. The all group will appear for everyone, it contains all targets in your inventory (it’s an implicit group that can be used in Bolt). Let’s explore a particular target.

wash bolt > tree -L 2 my-linux-nodes
my-linux-nodes
├── node1
│   └── fs
└── node2
    └── fs
wash bolt > cd my-linux-nodes/node1/fs
wash bolt/my-linux-nodes/node1/fs > ls
bin/
boot/
dev/
etc/
home/
...

Great, I can see the files on that system, and I didn’t have to do any additional setup. I can go look at config or watch logs

wash bolt/my-linux-nodes/node1/fs > cat etc/hosts
127.0.0.1 localhost
...
wash bolt/my-linux-nodes/node1/fs > tail -f var/log/syslog
Feb  7 18:39:51 pe-agent01 systemd[18116]: Startup finished in 23ms.
Feb  7 18:39:51 pe-agent01 systemd[1]: Started User Manager for UID 1000.
...

We can also explore Windows targets using WinRM

wash bolt/my-windows-nodes/node5/fs > ls Users
Administrator/
Public/

Boltwash includes target configuration as metadata, so you can also view that

wash bolt/my-linux-nodes > meta node1
alias: []
config:
  ssh:
    connect-timeout: 10
    disconnect-timeout: 5
    load-config: true
    run-as: root
    tty: false
    user: ubuntu
  transport: ssh
facts: {}
features: []
name: node1
plugin_hooks: {}
uri: node1
vars:
  datacenter: west coast

and use find to search for nodes based on that metadata

wash bolt > find . -meta .vars.datacenter 'west coast'
all/node1
all/node5
my-linux-nodes/node1
my-windows-nodes/node5

Debugging with Bolt + Wash

Wash can be helpful debugging all sorts of failures. As an example, suppose you have a cluster of machines (in the webservers group) that are having problems

wash bolt/webservers > ls
server1/
server2/
server3/

You’ve noticed an issue in one of their logs, and want to see if it’s present everywhere

wash bolt/webservers > grep OutOfMemory */fs/var/log/syslog
server1/fs/var/log/syslog:Jan 28 12:09:34 server1 my-server[435]: ...java.lang.OutOfMemoryError...
server2/fs/var/log/syslog:Jan 28 12:09:43 server2 my-server[435]: ...java.lang.OutOfMemoryError...
server3/fs/var/log/syslog:Jan 28 12:09:01 server3 my-server[435]: ...java.lang.OutOfMemoryError...

Seeing that they’re all experiencing these issues, let’s see where the server process is still running

wash bolt/webservers > wps * | grep bin/java
server1     9027   12:10.02   /usr/bin/java -Xmx2048m -cp /opt/my-server/server.jar ...
server2    21204   12:13.02   /usr/bin/java -Xmx2048m -cp /opt/my-server/server.jar ...

Looks like they’ve restarted recently, and the server’s not running on server3. Let’s reconfigure the servers to allow more memory (using Bolt or by directly modifying config/service files) and restart them

wash bolt/webservers > bolt task run service -t webservers action=restart name=my-server
Started on server1...
Started on server2...
Started on server3...
Finished on server1:
  {
    "status": "MainPID=751,LoadState=loaded,ActiveState=active"
  }
Finished on server2:
  {
    "status": "MainPID=2927,LoadState=loaded,ActiveState=active"
  }
Finished on server3:
  {
    "status": "MainPID=21936,LoadState=loaded,ActiveState=active"
  }
Successful on 3 targets: server1,server2,server3
Ran on 3 targets in 1.91 sec

You can imagine other scenarios as well. Suppose you ran a Bolt plan across several targets and want to get some files back from them. You can use Wash to directly copy them with something like

wash bolt > for f in group/*/fs/path/to/file; do cp -v "$f" ~/dest/"${f//\//_}"; done

The Bash string manipulation (${f//\//_}) replaces all slashes in the original path to the file with underscores to make it unique.

Configuring Boltwash

Boltwash loads the default Boltdir at ~/.puppetlabs/bolt, but you can configure it to load any Boltdir in Wash’s config. To use Boltwash, you add it to the external-plugins key in ~/.puppetlabs/wash/wash.yaml; adding a custom Boltdir location looks like

external-plugins:
- script: '/path/to/boltwash/bolt.rb'
bolt:
  dir: /path/to/boltdir

The Bolt plugin for Wash provides an accessible, interactive means of investigating multiple systems. Combined with Bolt inventory, it’s natural to then take action with Bolt. Try it out!