BASH Scripting: Automate Your Life with Pipes

One of the best things about using Unix-like operating systems such as Linux, Linux Subsystem for Windows, and MacOS, is the ability to use shell scripts to automate so much.

I do, however, realize that the thing that gives the shell so much power also makes it feel impenetrable for many people more used to mousing around.

This is why whenever I have the opportunity, I try to (gently) introduce people to some simple but incredibly powerful aspects of Shell Scripting.

Namely, Pipes.

What are pipes and why should I care?

Pipes (|) in Unix-like systems let you use the output of one command directly as the input to another. They allow chaining commands together to process data in steps without creating intermediate files.

This takes simple command line instructions and allows you to think of them almost like programming functions instead.

Pipes eliminate the need for temporary files or intermediate variables.

How this works is a feature called "Standard In/Standard Out". The textual output that ordinarily is sent to your screen can be instead sent to the next command in a chain of commands.

Although there are a few ways to string applications together in a graphical environment, there are none as simple and as powerful as this simply because there are as yet few standard ways to grab the output from a desktop app and plumb it in as input to the next.

Example - Count Your Photographs

The easiest way to get your head around this is by example.

Say you want to count how many .JPG files you have in your photos directory and all its subdirectories. There are a number of ways to do this, but one way might be to use find command:

find -type f -name "*.jpg"

There is one problem, this just lists all the files that match, we need to use wc (Word Count) to do the counting. This is where pipes come in - we send the output of find into wc and tell wc we want to count lines instead of words!

find -type f -name "*.jpg" | wc -l

More Useful Example - Backing Up All Your Websites

How does this help with automation?

Consider a case where you host many websites, the number of which changes reasonably often, but you still need daily website backups.

A couple of options spring to mind, including having a database of clients that is read by a Python script, but instead here is my actual solution that makes use of pipes and loops:

#!/bin/bash

# Base directory where your WordPress sites are stored
BASE_DIR="/path/to/sites"

# Backup destination directory
BACKUP_DIR="$HOME/backups"
mkdir -p "$BACKUP_DIR"

# Find all website directories but exclude those with 'staging.' in the path
find "$BASE_DIR" -type d -path "*/app/public" -not -path "*/staging.*/*" | while read -r WP_PATH; do
    # Extract the domain name from the path (e.g., "example.com")
    DOMAIN=$(basename "$(dirname "$(dirname "$WP_PATH")")")

    # Set the output tar.gz file path
    BACKUP_FILE="$BACKUP_DIR/${DOMAIN}_$(date +%F).tar.gz"

    # Compress the directory into a tar.gz file
    tar -czf "$BACKUP_FILE" -C "$(dirname "$WP_PATH")" "$(basename "$WP_PATH")"

    echo "Backup created for $DOMAIN: $BACKUP_FILE"
done

My script uses the fact that each website has a standard structure

Here’s how it works:

1. The find command outputs a list of paths, with one path per line.
2. -not -path "*/staging.*/*" Ensures that directories containing staging. in their paths are excluded from the search results.
3. The while read -r WP_PATH loop reads this output line-by-line.
4. Each line (delimited by \n) is assigned to the variable WP_PATH.
5. The -r flag prevents backslash escapes () from being interpreted.
6. The loop processes each path individually within the WHILE block.
7. DOMAIN=$(basename "$(dirname "$(dirname "$WP_PATH")")") extracts the domain name (e.g., example.com) from the directory path stored in the variable WP_PATH.
   • dirname command removes the last component of a path.
   • basename command extracts the last component of a path.
   • Another way to do this would be with pipes! DOMAIN=$(echo "$WP_PATH" | dirname | dirname | basename)

Using this script I can run a command on every website that is NOT a staging site, without having to copy and paste unique paths into the script.

Set and forget!