Bash Scripting & Capstone Project

This final lesson ties everything together: bash scripting to automate tasks, and a capstone project building a real reproducible pipeline.

Bash Scripting Fundamentals

The Shebang

Every bash script starts with a shebang line that tells the system which interpreter to use:

#!/bin/bash
chmod +x myscript.sh
./myscript.sh

Variables

#!/bin/bash
NAME="simulation_v2"
CORES=8
OUTPUT_DIR="./results/${NAME}"
echo "Running $NAME with $CORES cores"
mkdir -p "$OUTPUT_DIR"

Arguments

#!/bin/bash
CONFIG_FILE=$1
OUTPUT_DIR=$2
if [ -z "$CONFIG_FILE" ]; then
    echo "Usage: $0 <config_file> <output_dir>"
    exit 1
fi

Conditionals

if [ -f "$FILE" ]; then
    echo "File exists"
elif [ -d "$FILE" ]; then
    echo "Is a directory"
else
    echo "Not found"
fi

CORES=$(nproc)
if [ $CORES -ge 8 ]; then
    echo "Enough cores"
fi

Common test flags: -f (file exists), -d (directory exists), -z (string is empty), -n (string is not empty), -eq/-ne/-lt/-gt (numeric comparisons).

Loops

for CONFIG in configs/*.ini; do
    echo "Running: $CONFIG"
    python3 simulate.py "$CONFIG"
done

for CASE in baseline mesh_fine mesh_coarse; do
    mkdir -p "results/$CASE"
done

while [ ! -f "results/done.flag" ]; do
    echo "Waiting..."
    sleep 10
done

Functions

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"; }

run_simulation() {
    local config=$1
    local output=$2
    log "Starting: $config"
    python3 simulate.py "$config" --output "$output"
    if [ $? -eq 0 ]; then
        log "SUCCESS: $config"
    else
        log "FAILED: $config"
        exit 1
    fi
}

Capstone Project: Reproducible Engineering Pipeline

Time to put it all together. You'll build a complete, reproducible simulation pipeline using every tool from this course.

Project Structure

~/capstone-pipeline/
├── .gitignore
├── Dockerfile
├── docker-compose.yml
├── Makefile
├── requirements.txt
├── configs/
│   ├── baseline.ini
│   └── refined.ini
├── scripts/
│   ├── run_pipeline.sh
│   └── post_process.py
├── simulate.py
└── systemd/
    └── simulation-pipeline.service

Step 1: Initialize the Repository

Start with a Git repo and a proper .gitignore:

mkdir ~/capstone-pipeline && cd ~/capstone-pipeline
git init
cat > .gitignore << 'EOF'
results/
__pycache__/
*.pyc
.env
venv/
EOF
git add .gitignore && git commit -m "Initial commit with .gitignore"

Step 2: Add a Feature via Git Worktree

Create a branch, develop the feature, and merge:

git worktree add ../capstone-feature feature/pipeline-script
cd ../capstone-feature
# develop your feature here
git add . && git commit -m "Add pipeline script"
cd ../capstone-pipeline
git merge feature/pipeline-script

Step 3: The Pipeline Script

This is the heart of the project — a bash script with logging, error handling, and a loop over configurations:

#!/bin/bash
set -eu

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
RESULTS_DIR="results/${TIMESTAMP}"
LOG_FILE="${RESULTS_DIR}/pipeline.log"

log() { echo "[$(date '+%H:%M:%S')] $1" | tee -a "$LOG_FILE"; }

mkdir -p "$RESULTS_DIR"
log "Pipeline started"

for CONFIG in configs/*.ini; do
    CASE=$(basename "$CONFIG" .ini)
    log "Running: $CASE"
    python3 simulate.py --config "$CONFIG" --output "$RESULTS_DIR/$CASE" >> "$LOG_FILE" 2>&1
    log "Done: $CASE"
done

log "Pipeline complete"

Step 4: Dockerize

Dockerfile:

FROM python:3.12-slim
WORKDIR /app
RUN apt-get update && apt-get install -y gcc && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["bash", "scripts/run_pipeline.sh"]

docker-compose.yml:

version: '3.8'
services:
  pipeline:
    build: .
    volumes:
      - ./configs:/app/configs
      - ./results:/app/results

docker compose up --build

Step 5: systemd Service for Nightly Runs

Create systemd/simulation-pipeline.service:

[Unit]
Description=Nightly Simulation Pipeline
After=network.target

[Service]
Type=oneshot
User=engineer
WorkingDirectory=/home/engineer/capstone-pipeline
ExecStart=/usr/bin/docker compose up --build
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Pair it with a systemd timer for nightly scheduling, or use cron:

# crontab -e
0 2 * * * cd /home/engineer/capstone-pipeline && docker compose up --build

Step 6: Deploy to Remote with rsync + SSH

rsync -av --exclude='results/' --exclude='.git/' \
    ~/capstone-pipeline/ myserver:~/capstone-pipeline/

ssh myserver "cd ~/capstone-pipeline && docker compose up --build -d"

Quick Quiz

What does set -e do in a bash script?

Answer

B) It exits the script immediately when any command returns a non-zero exit code, preventing cascading failures from unnoticed errors.

Congratulations

You've completed the entire Dev Environment for Engineers course. Here's what you now know:

• Terminal & Shell Fundamentals — navigating the command line with confidence.
• Linux Filesystem & Permissions — understanding where things live and who can access them.
• Text Processing & Terminal Editors — manipulating data and editing files without a GUI.
• Shell Configuration & Environment — customizing your shell for productivity.
• Package & Dependency Management — installing software and managing project dependencies.
• Version Control with Git — tracking changes, branching, and collaborating.
• Remote Access & SSH — connecting to servers, clusters, and cloud instances.
• Docker & Containers — packaging environments for reproducibility.
• Build Tools & System Monitoring — automating builds and watching system resources.
• Bash Scripting — automating workflows end to end.

AI writes code. You now control the environment that runs it.