◀ Back to blog
Linux

Shell scripting for DevOps

Published on 01 May 2024· 8 min read
#Linux#Bash#Automatisation

The art of shell scripting

Shell scripting remains a fundamental tool for every DevOps engineer. Despite the rise of Ansible, Terraform and declarative CI pipelines, there is always a moment when you write a few lines of Bash: a deployment hook, a nightly backup, a container entrypoint, a pipeline step. These scripts are often written quickly and rarely reviewed, which makes them a classic source of incidents. Here are the essential patterns and best practices for writing scripts that are reliable, readable and easy to maintain.

When to choose Bash (and when to avoid it)

Bash excels at orchestrating existing commands: chaining git, rsync, docker, systemctl, checking an exit code, writing a log line. It is available on virtually every Linux server and requires no dependencies.

However, as soon as a script manipulates data structures (JSON, nested associative arrays), performs non-trivial calculations or grows beyond a few hundred lines, a language like Python or PHP becomes safer and more readable. In production, I recommend a simple rule: if you start parsing JSON with grep and sed, it is time to switch tools (or at the very least to use jq).

Structure of a robust script

A good script always follows the same skeleton: an explicit shebang, strict mode, constants at the top, functions, guaranteed cleanup and a main function called at the end.

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

# Variables
readonly SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
readonly LOG_FILE="/var/log/deploy.log"
readonly TEMP_DIR="$(mktemp -d)"

# Functions
log() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

cleanup() {
    log "Cleaning up..."
    rm -rf "$TEMP_DIR"
}
trap cleanup EXIT

# Main script
main() {
    log "Starting deployment"
    # ...
    log "Deployment finished"
}

main "$@"

Let's go through each element:

  • set -e: the script stops as soon as a command fails, instead of carrying on in an inconsistent state.
  • set -u: any undefined variable raises an error. A typo in $TEMP_DIR no longer turns into rm -rf /.
  • set -o pipefail: a pipeline fails if any of its commands fails, not just the last one. Without it, failing_command | tee file is considered a success.
  • IFS=$'\n\t': word splitting no longer happens on spaces, which avoids nasty surprises with file names that contain spaces.
  • readonly: constants cannot be accidentally overwritten further down the script.
  • mktemp -d: creates a unique temporary directory, instead of a fixed path like /tmp/deploy that may collide with another run.
  • trap cleanup EXIT: the cleanup function runs however the script ends (success, error, explicit exit). Temporary files are never left behind.
  • main "$@": all the code runs inside a function, which keeps the flow readable and guarantees that Bash has read the whole file before starting.

The limits of set -e

Strict mode is not magic. set -e is ignored in some situations: in the condition of an if, on the left-hand side of && or ||, and in a function called from one of those contexts. Likewise, local var=$(command) hides the command's exit code, because the one that counts is the exit code of local. Declare the variable and assign it on two separate lines:

# Bad: the curl failure is hidden by "local"
local body=$(curl -fsS "$url")

# Good: the curl failure stops the script
local body
body=$(curl -fsS "$url")

Another classic trap: ((counter++)) returns an error code when the value before the increment is 0. With set -e, the script stops without any message. Prefer counter=$((counter + 1)) when the counter can start at zero.

Useful patterns

Some functions show up in almost every operations script. The following two are good examples: a service status check and a retry with progressive delay.

# Check whether a service is active
check_service() {
    if systemctl is-active --quiet "$1"; then
        echo "$1 is active"
    else
        echo "$1 is inactive" && return 1
    fi
}

# Retry with backoff
retry() {
    local max_attempts=$1; shift
    local attempt=1
    while [ $attempt -le $max_attempts ]; do
        if "$@"; then return 0; fi
        echo "Attempt $attempt/$max_attempts failed"
        sleep $((attempt * 2))
        ((attempt++))
    done
    return 1
}

The retry function takes the number of attempts as its first argument, then the command to run. shift removes the first argument, and "$@" runs the rest as is, preserving spaces and quoting. The wait grows after each failure (2, 4, 6 seconds…), giving a database or an API time to come back. Here attempt starts at 1, so ((attempt++)) is safe with set -e. Example usage:

retry 5 curl -fsS https://example.com/health
check_service nginx || systemctl restart nginx

Validating arguments and prerequisites

A script started with wrong arguments must fail immediately, with a clear message, before it has changed anything. Likewise, check that the required tools are installed:

usage() {
    echo "Usage: $(basename "$0") <environment> [version]" >&2
    exit 1
}

require() {
    local cmd
    for cmd in "$@"; do
        command -v "$cmd" >/dev/null 2>&1 || {
            echo "Required command not found: $cmd" >&2
            exit 1
        }
    done
}

[ $# -ge 1 ] || usage
readonly ENVIRONMENT="$1"
readonly VERSION="${2:-latest}"

case "$ENVIRONMENT" in
    staging|production) ;;
    *) echo "Invalid environment: $ENVIRONMENT" >&2; exit 1 ;;
esac

require git rsync curl

Note the use of ${2:-latest} for a default value, which works with set -u, and the redirection of error messages to standard error (>&2), so they do not get mixed with output that another program might consume.

Preventing concurrent runs

A script started by cron every five minutes can overlap with the previous run if that one falls behind. Two backups or two deployments in parallel are a guaranteed inconsistent state. The flock command (util-linux package) solves this cleanly:

exec 9>/var/lock/deploy.lock
if ! flock -n 9; then
    echo "Another run is already in progress" >&2
    exit 1
fi

The lock is released automatically by the kernel when the script ends, even if it crashes: no orphaned lock file to delete by hand.

Quotes and variables

Most Bash bugs come from unquoted variables. Always write "$variable" in double quotes, unless you explicitly want word splitting. Prefer $(command) over backticks, which nest poorly, and [[ ... ]] over [ ... ] in Bash scripts: it handles empty strings better and supports regular expressions with =~.

Best practices

  • Always use set -euo pipefail
  • Document variables and functions
  • Use functions for readability
  • Handle errors with trap
  • Test with shellcheck
  • Always put variables in double quotes
  • Write errors to stderr and return a non-zero exit code
  • Make scripts idempotent: a second run must not break anything

ShellCheck deserves a special mention: this static analyser detects unquoted variables, incorrect comparisons, local pitfalls and dozens of other mistakes. It integrates with most editors and is easy to run in CI:

shellcheck scripts/*.sh
bash -n scripts/deploy.sh   # syntax check only

When not to write a script

Before writing a new script, check whether a dedicated tool already exists. To schedule a task, a systemd timer provides logging and failure handling. To configure a fleet of servers reproducibly, Ansible is a better fit than an SSH loop. A good shell script is short, does one thing, fails loudly and can be reviewed in two minutes.