Multi-master SaltStack setup

Initial setup

First off, install salt-master package on two servers that are designated to be master servers.

RHEL/CentOS:

sudo yum install salt-master

FreeBSD:

cd /usr/ports/sysutils/py-salt
sudo make install clean

Set file_roots in master configuration file:

file_roots:
- /srv/salt

Let's also run salt-master as a nonroot user. Change user option in the master configuration file:

user: salt

Change permissions of all directories:

sudo chown -R salt:salt /etc/salt /var/cache/salt /var/log/salt /srv/salt /var/run/salt
sudo chmod 664 /var/log/salt/master

Add your regular user to the salt group so your regular user can execute salt commands against the minions.

RHEL/CentOS:

sudo usermod -G salt username

FreeBSD:

sudo pw usermod username -G salt

We'll be using a default config for everything else.

Multi Master configuration

Let's jump to the multi master configuration.

Make sure your master keys are the same on all the masters. This needs to be done only once or whenever you update the keys. You can sync the keys from one of the servers. Master keys are located in /etc/salt/pki/master/master.{pem,pub}

Then, sync the minion public keys (obviously it would be beneficial to setup ssh keys for the salt user before doing this). Make sure this directory is synced whenever you add a new minion.

sudo -u salt /usr/bin/rsync -avz -e ssh /etc/salt/pki/master/{minions_pre,minions,minions_rejected,minions_denied,minions_autosign} salt@${masterhostB}:/etc/salt/pki/master/

Next, sync file_roots.

sudo -u salt /usr/bin/rsync -avz -e ssh /srv/salt salt@${masterhostB}:/srv/salt

We're done with salt master setup. We should have two identical masters now. Let's setup the minions now.

Minion setup

Install salt-minion package.

RHEL/CentOS:

sudo yum install salt-minion

FreeBSD:

cd /usr/ports/sysutils/py-salt
sudo make install clean

Edit the minion config file to point to the masters. We will use failover type scenario in which the minion will connect to only master a time but will failover to another master if the first one goes down:

sudo cat /etc/salt/minion
master:
- masterhostA
- masterhostB
master_type: failover
master_alive_interval: 60

Master type is set to failover thus all minions will only connect to one master server. If we remove that setting, minions will connect to all master servers but will not be able to determine which one is live until they to contact them. We can adjust the availability check using the master_alive_interval setting (which in this example is set to 60 seconds).

If you want to connect to a single, random master server, use the "random_master: True" setting.

Start the salt-minion service:

sudo service salt-minion start

Watch the first salt master for minion's keys to pop up:

sudo salt-key -L

Since salt-master is running as salt user, we still need root perm's to view keys that are available to access. Word of caution: whenever you run salt-key, the daemon changes the permissions of /etc/salt/pki/master directory to whatever the daemon is running as. If we ran the salt-master daemon as root, the permissions of that directory would change to root:root which would be problematic when rsyncing as salt user (you should never be ssh-ing as root user for security's sake). Running salt-master as salt user will keep these permissions as salt:salt after running salt-key.

You should see the minion's public key pending under "Unaccepted Keys":

sudo salt-key -L
Accepted Keys:
Unaccepted Keys:
webserver01
Rejected Keys:

Accept the key with "sudo salt-key -A". You will see this minion's public key in /etc/salt/pki/master/minions directory on only one salt master server. Now that's a problem so let's sync that directory using one of the rsync scripts that we came up with:

sudo -u salt /usr/bin/rsync -avz -e ssh /etc/salt/pki/master/{minions_pre,minions,minions_rejected,minions_denied,minions_autosign} salt@${masterhostB}:/etc/salt/pki/master/

This will keep the directories in sync on both master servers and when you run the "sudo salt-key -L" command, you will see the minion's key on both servers. You could also employ something like glusterfs, but rsync scripts are simple and perfectly fine.

Test salt state modules or execution modules against a minion from one of the salt masters. You will be able to call salt commands from only one master because the minion connects to only one server (consequence of master_type: failover setting in master configuration file).

salt 'webserver01' test.ping
webserver01:
True

You should see a successful ping test.