Remote-Url: https://arslan.io/2019/07/03/how-to-write-idempotent-bash-scripts/
Retrieved-at: 2022-02-12 08:47:58.511256+00:00

July 3, 2019Photo by Callum Wale on UnsplashIt happens a lot, you write a bash script and half way it exits due an error. You fix the error in your system and run the script again. But half of the steps in your scripts fail immediately because they were already applied to your system. To build resilient systems you need to write software that is idempotent.What is idempotency?Idempotent scripts can be called multiple times and each time it’s called, it will have the same effects on the system. This means, a second call will exit with the same result and won’t have any side effects. From thedictionary:Idempotent: denoting an element of a set which is unchanged in value when multiplied or otherwise operated on by itself.Good software is always written in an idempotent way, especially if you’re working in distributed systems, where operations might be eventually consistency and you might end up calling functions multiple times because of duplicate requests (such as inqueueswithAt-Least-Oncedelivery guarantee).Bash idiomsLet me show a couple of bash tips and idioms you can use to change your scripts to be idempotent. You’re probably using most of them without being aware of the side effects:Creating an empty fileThis is an easy one. Touch is by default idempotent. This means you can call it multiple times without any issues. A second call won’t have any effects on the file content. Note though it’ll update the file’s modification time, so if you depend on it be careful.Creating a directoryNever usemkdirdirectly, instead use it with the-pflag. This flag make sure mkdir won’t error if the directory exists:Creating a symbolic linkWe create symbolic links with the following command:But this will fail if you call it again on the same target. To make it idempotent, pass the-fflag:The-fflag removes the target destination before creating the symbolic link, hence it’ll always succeed.When linking a directory, you need to pass-ntoo. Otherwise calling it again will create a symbolic link inside the directory.mkdir a
ln -sf a b
ln -sf a b
ls a
aSo to be safe, always useln -sfn source target.Removing a fileInstead of removing a file with:Use the-fflag which ignores non-existent files.Modifying a fileSometimes you’re adding a new line to an existing file (i.e:/etc/fstab). This means, you need to make sure not to add it the second time if you run your script. Suppose you have this in your script:echo"/dev/sda1 /mnt/dev ext4 defaults 0 0"|sudo tee -a /etc/fstabIf this is run again, you’ll end up having duplicate entries in your/etc/fstab. One way of making this idempotent is to make sure to check for certain placeholders viagrep:if! grep -qF"/mnt/dev"/etc/fstab;thenecho"/dev/sda1 /mnt/dev ext4 defaults 0 0"|sudo tee -a /etc/fstabfiHere the-qmeans silent mode and-Fenablesfixed stringmode. Grep will silently fail if/mnt/devdoesn’t exist so the echo statement will never be called.Check if variable, file or dir existsMost of the time you’re writing to a directory, reading from a file or doing simple string manipulations with a variable. For example you might have a tool that creates a new file based on certain inputs:echo"complex set of rules"> /etc/conf/foo.txtCalculating the text might be an expensive operation, hence you don’t want to write it every time you call the script. To make it idempotent you check if the file exists via the-fflag of the inbuilttestproperty of the shell:if[! -f"/etc/conf/foo.txt"];thenecho"complex set of rules"> /etc/conf/foo.txtfiHere-fis just an example, there are many other flags you can use, such:-d: directory-z: string of zero length-p: pipe-x: file and has execute permissionFor example suppose you want to install a binary, but only if it doesn’t exist
in your host, you can use the-xlike this:# install 1password CLIif![-x"$(command-v op)"];thenexportOP_VERSION="v0.5.6-003"curl -sS -o 1password.zip https://cache.agilebits.com/dist/1P/op/pkg/${OP_VERSION}/op_linux_amd64_${OP_VERSION}.zip
  unzip 1password.zip op -d /usr/local/bin
  rm -f 1password.zipfiThis installs theopbinary to /usr/local/bin. If you re-run your script, it
won’t install it anymore. Another benefit is, you can easily upgrade the binary
to a new version by just removing it from your system, update theOP_VERSIONenv and re-run your script.For a list of complete flags and operators checkoutman test.Formatting a deviceTo format a volume, say with anext4format, you can use a command like the following:Of course, this would fail immediately if you call it again. To make this call idempotent, we prepend it withblkid:blkid"$VOLUME_NAME"||mkfs.ext4"$VOLUME_NAME"This command prints attributes for a given block device. Hence prepending basically means to proceed with formattingonlywhenblkidfails, which is an indication that the given volume is not formatted yet.Mounting a deviceTrying to mount a volume to an existing directory can be done with the following example command:mount -o discard,defaults,noatime"$VOLUME_NAME""$DATA_DIR"This will fail however if it’s already mounted. One way is to check the output ofmountcommand and see if the volume is already mounted. But there is a better way to do it. Using themountpointcommand:if! mountpoint -q"$DATA_DIR";thenmount -o discard,defaults,noatime"$VOLUME_NAME""$DATA_DIR"fiThemountpointcommand checks whether a file or directory is a mount point. The-qflag just makes sure it doesn’t output anything and silently exits. In this case, if the mount point doesn’t exist, it’ll go forward and mount the volume.VerdictMost of these tips and tricks are already known, but when we write Bash scripts
those can be easily neglected without even thinking about it. Some of these
idioms are very specific (such as mounting or formatting), but as we saw,
creating idempotent and resilient software is always beneficial in the long
term. So knowing them is useful nevertheless.I used all of the above tips and tricks recently in mybootstrap.shscript that I use to create and provision myremote development machine. I know that I could use more sophisticated tools to provision a VM from
scratch, but sometimes a simple bash script is the only thing you need.