scripts/docker-stacks-backup/README.md

# Docker Compose Stack Backup & Restore

A bash script for backing up and restoring Docker Compose stacks with full permission preservation, container lifecycle management, and running-stack discovery.

By default, `--all` discovers stacks relative to the **script's location** (so drop the script in your stacks directory or add it to PATH and `cd` to your stacks). Backups go into `$PWD/backups/`. External stacks anywhere on the filesystem are discoverable via `--running` or explicit `--stack /path`.

## How It Works

A **stack** is any directory containing a compose file (`compose.yaml`, `compose.yml`, `docker-compose.yaml`, or `docker-compose.yml`).

### Backup Flow

```
1. Resolve target stacks (--all, --running, or --stack)
2. For each stack:
   a. Elevate to root via sudo (if not already)
   b. Load x-backup config from compose.yaml
   c. Run pre-backup hook (if configured)
   d. If x-backup.stop != false → `docker compose down`
   e. Dump named Docker volumes into _volumes/ (unless opted out)
   f. Create a timestamped tar.gz archive:
      tar -czf <backup-dir>/<stack>_<timestamp>.tar.gz \
           --same-permissions --preserve-permissions \
           [--exclude=<path>...] \
           -C <archive-root> <stack-name>
   g. Write .meta file (path, sha256, size, volumes flag)
   h. Run post-backup hook (if configured)
   i. Apply retention (prune old backups if configured)
   j. If was stopped → `docker compose up -d`
```

The archive contains the **entire stack directory** — compose file, `.env` file, and all data subdirectories (bind mounts). Permissions and ownership are preserved via `tar --same-permissions --preserve-permissions`, which is critical for bind mounts that run as non-root UIDs inside containers. Named Docker volumes are also dumped by default (see `backup-volumes`).

### Restore Flow

```
1. Resolve target stacks (same logic as backup)
2. For each stack:
   a. Find the most recent backup matching <stack>_*.tar.gz
   b. Remove any stale pre-restore directories from previous failed restores
   c. Load x-backup config from the backup archive
   d. `docker compose down` (always — we're about to replace the directory)
   e. Run pre-restore hook (if configured)
   f. If stack directory exists → rename to <stack>.pre-restore-<timestamp>
   g. Extract backup:
      tar -xzf <backup-file> \
           --same-permissions --preserve-permissions \
           -C <stacks-dir>
   h. Remove the pre-restore copy (backup archive is the authoritative source)
   i. Restore named Docker volumes from _volumes/ in the archive
   j. Run post-restore hook (if configured)
   k. `docker compose up -d` (only if the stack was running before restore)
```

## Discovery

The script finds stacks in three ways:

| Mode | What it finds |
|------|---------------|
| `--all` | Every subdirectory of the **script's directory** containing a compose file |
| `--running` | All running compose projects **anywhere on the filesystem** by reading `ConfigFiles` from `docker compose ls --format json` |
| `--stack <spec>` | A specific stack by name (relative to the **current directory**) or by absolute path |

Add `--dir <path>` to any command to override the default scan directory (script dir for backup/restore, current dir for init).

### External Stacks

When `--running` discovers a compose project living outside the stacks directory, the script uses its absolute path directly. During backup, it archives from the correct parent directory so the stack's directory structure is preserved. Restore writes back to the same absolute path.

You can also explicitly target external stacks by passing an absolute path to `--stack`:

```bash
./docker-stack-backup.sh backup --stack /srv/docker/nextcloud
```

This is useful for ad-hoc backups of stacks that aren't under your main stacks directory.

## Usage

### Init

```bash
# Inject default x-backup config into all stacks in the current directory
./docker-stack-backup.sh init

# Or specify a directory to scan (no cd needed)
./docker-stack-backup.sh init --dir /mnt/data/stacks

# Or target specific stacks
./docker-stack-backup.sh init --running
./docker-stack-backup.sh init --stack random_stack
./docker-stack-backup.sh init --stack /srv/docker/nextcloud
./docker-stack-backup.sh init -s stack_a -s stack_b
./docker-stack-backup.sh init --dry-run --verbose   # preview changes
```

### Backup

```bash
# Interactive menu — shows all available stacks with running status
./docker-stack-backup.sh backup

# All stacks found in the script's directory (drop it in your stacks folder)
./docker-stack-backup.sh backup --all

# Only stacks currently running in Docker (wherever they live)
./docker-stack-backup.sh backup --running

# Specific stacks (by name or absolute path)
./docker-stack-backup.sh backup --stack random_stack
./docker-stack-backup.sh backup --stack /srv/docker/nextcloud
./docker-stack-backup.sh backup -s stack_a -s stack_b
./docker-stack-backup.sh backup --stack stack_a,stack_b

# Scan a specific directory for stacks instead of the script's directory
./docker-stack-backup.sh backup --dir /srv/docker --all

# With a custom backup destination
./docker-stack-backup.sh backup --all --backup-dir /mnt/nfs/backups

# Dry run to see what would happen
./docker-stack-backup.sh backup --all --dry-run --verbose
```

Running `backup` without flags opens an interactive menu:

```
Available stacks:
-----------------
   1) external_stack       (running, /tmp/external_stack)
   2) random_stack         (running)
  all) Back up all stacks
   q)  Cancel

Select stacks to back up (numbers, 'all', or 'q'):
```

After selecting stacks, the script re-executes via `sudo` with your selections as `--stack` flags so root elevation happens only when needed. Local stacks and running stacks discovered elsewhere are listed together. Select by index, ranges (`1-3`), or `all`.

### Restore

```bash
# Interactive menu — shows all available backups with readable dates
./docker-stack-backup.sh restore

# Restore a specific stack from its latest backup
./docker-stack-backup.sh restore --stack random_stack

# Restore an external stack by path
./docker-stack-backup.sh restore --stack /srv/docker/nextcloud

# Restore to a different location (overrides the original path)
./docker-stack-backup.sh restore --stack random_stack --to /new/location/random_stack

# Restore from a specific backup directory
./docker-stack-backup.sh restore --stack random_stack --backup-dir /mnt/nfs/backups
```

Running `restore` without `--stack` opens an interactive menu:

```
Available backups:
------------------
   1) external_stack        2026-07-11 19:49:49
   2) random_stack          2026-07-11 19:49:48
  all) Restore all stacks
   q)  Cancel

Select stacks to restore (numbers, 'all', or 'q'):
```

After selecting backups, the script re-executes via `sudo` with the resolved stack paths so the actual restore runs as root. You can enter space-separated numbers, ranges like `1-3`, or `all`. Stacks that were originally external (backed up from an absolute path) use a stored `.meta` file to restore to the correct location automatically.

When restoring over an existing stack directory, the current directory is briefly renamed to `<stack>.pre-restore-<timestamp>` as a safety net during extraction. Once the restore completes successfully, the pre-restore copy is removed — the backup archive is the authoritative restore point. Any stale pre-restore directories from previous failed restores are cleaned up automatically.

### Init (Inject Default Config)

```bash
# Inject default x-backup config into all stacks in the current directory
./docker-stack-backup.sh init

# Or target running/external stacks
./docker-stack-backup.sh init --running
./docker-stack-backup.sh init --stack /srv/docker/nextcloud
```

Scans each compose file (relative to the **current directory** by default, or use `--dir` to point elsewhere) and injects a default `x-backup` block at the top:

```yaml
x-backup:
  stop: true
  backup-volumes: true
  # pre-hook: ""
  # post-hook: ""
  # pre-restore-hook: ""
  # post-restore-hook: ""
  # exclude:
  #   - path/to/exclude
  # retention: 7
```

Active defaults (`stop: true`, `backup-volumes: true`) are uncommented. Optional fields are commented out as a reference — uncomment and adjust as needed.

If the compose file already has an `x-backup:` block, it is removed first so the result is always a clean standard config. This makes `init` idempotent and safe to re-run at any time.

Does not require root by default — reads and writes compose files as the current user. If a compose file is owned by another user (e.g. root), the script automatically escalates with `sudo` for the write.

### Verify

```bash
# Check integrity of all backups
./docker-stack-backup.sh verify

# Check a specific stack's backups
./docker-stack-backup.sh verify --stack random_stack
```

Each backup has a SHA256 hash stored in its `.meta` file. The `verify` command recalculates it and reports any corruption:

```
  OK    random_stack_20260711_204621.tar.gz   (4.0K)
  FAIL  badstack_20260711_120000.tar.gz       (sha256 mismatch)
```

Does not require root — read-only operation.

### List Stacks & Backups

```bash
./docker-stack-backup.sh list
```

Shows all discovered stacks — local ones and running ones found elsewhere — with their x-backup config tags. Backups display a truncated SHA256 hash, original path for external stacks, and a `[volumes]` tag if named volumes were included.

```
=== Stacks ===
  - random_stack (running) [x-backup: pre-hook post-hook exclude retention=7]
  - nextcloud (/srv/docker/nextcloud) (running)

=== Backups ===
  random_stack_20260711_204621.tar.gz  (4.0K)  -> random_stack sha256:e3b0c44298fc..
  nextcloud_20260711_193245.tar.gz    (128M)  -> nextcloud sha256:1a2b3c4d5e6f.. [volumes]
```

External stacks show their full path in parentheses.

## x-backup Extensions

You can control per-stack backup behavior by adding an `x-backup` top-level key to your `compose.yaml`. This follows Docker Compose's [extension mechanism](https://docs.docker.com/compose/compose-file/10-extension/).

```yaml
services:
  app:
    image: nginx
  db:
    image: postgres:16

x-backup:
  stop: false
  exclude:
    - app_data/cache
    - db_data/wal_archive
  pre-hook: pg_dumpall -U postgres > /tmp/db_dump.sql
  post-hook: rm -f /tmp/db_dump.sql
  pre-restore-hook: echo "About to restore..."
  post-restore-hook: echo "Restore complete"
  retention: 14
```

### Configuration Options

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `stop` | boolean | `true` | Stop containers during backup. Set to `false` for zero-downtime backups of databases that handle snapshots internally. |
| `exclude` | list | — | Paths to exclude from the archive, relative to the stack directory. Useful for large cache directories, temp files, or generated data. |
| `pre-hook` | string | — | Shell command run **before** the backup archive is created (after container stop). |
| `post-hook` | string | — | Shell command run **after** the backup archive is created (before container start). |
| `pre-restore-hook` | string | — | Shell command run **before** a restore (after container stop). |
| `post-restore-hook` | string | — | Shell command run **after** a restore (before container start). |
| `retention` | integer | — | Maximum number of backups to keep. Older backups are pruned after each successful backup. |
| `backup-volumes` | boolean | `true` | Dump named Docker volumes alongside the file backup. Set to `false` to skip. Volumes are archived into a `_volumes/` directory inside the archive and restored automatically. |

### Hook Examples

**Database-safe backup** — dump the database before tarring, then clean up:

```yaml
x-backup:
  pre-hook: docker exec -t $(docker ps --filter name=db -q) pg_dumpall -U postgres > /tmp/db.sql
  post-hook: rm -f /tmp/db.sql
```

**Notification on completion**:

```yaml
x-backup:
  post-hook: curl -fsS -m 10 --retry 3 -o /dev/null "https://hc-ping.com/my-uuid"
```

### Named Docker Volumes (`backup-volumes`)

Compose files can define **named volumes** — data stored inside Docker's storage area rather than on the host filesystem. By default the script detects them from your compose file and dumps them into `_volumes/` inside the archive.

Variable substitution in volume paths (`${VAR:-/default}:/container/path`) is handled correctly — parameterised bind mounts are not mistaken for named volumes.

To skip named volumes (e.g. they're ephemeral or backed up separately), set `x-backup.backup-volumes: false`:

```yaml
services:
  app:
    image: nginx
    volumes:
      - ./app_data:/app/data         # bind mount — always backed up
      - app_cache:/app/cache         # named volume — skipped if opted out

volumes:
  app_cache:

x-backup:
  backup-volumes: false
```

When opted out, a warning is shown listing the skipped volumes:

```
  [!] Named volumes detected:
      - app_cache
      - pgdata
      These store data inside Docker's storage area, not on the host filesystem.
      Set 'x-backup.backup-volumes: false' to skip them.
```

When enabled (default), each named volume is dumped via `docker run ... alpine tar` into a `_volumes/` directory inside the backup archive:

```
random_stack_20260711_204621.tar.gz
  random_stack/
    compose.yaml
    ...
    _volumes/
      app_cache.tar.gz
      pgdata.tar.gz
```

During `restore`, the volumes are recreated and their data is extracted back. The `_volumes/` directory is automatically removed after restore.

## Metadata Files

Each backup archive has an accompanying `.meta` file with key-value metadata for identification and verification:

```
stack=random_stack
path=/home/jeremy/Projects/docker/random_stack
date=20260711_204621
compose=compose.yaml
volumes=true
sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
size=4096
```

| Field | Description |
|-------|-------------|
| `stack` | Stack name (directory basename) |
| `path` | Original full path — used by interactive restore to place files correctly |
| `date` | Backup timestamp (`YYYYMMDD_HHMMSS`) |
| `compose` | Compose filename found in the stack |
| `volumes` | Whether named Docker volumes were included |
| `sha256` | SHA256 hex digest of the archive file |
| `size` | Archive size in bytes |

The `verify` command reads `sha256` and `size` to check integrity. Legacy `.meta` files (just a bare path string) are still supported.

### Detecting x-backup Config in `list`

Running `list` shows active x-backup settings for each stack:

```
=== Stacks ===
  - random_stack (running) [x-backup: no-stop pre-hook post-hook exclude retention=7]
```

Non-default settings (anything other than `stop: true`) are displayed as tags.

## Directory Layout

Stack discovery (`--all`) looks relative to the **script's location**. Backups default to `$PWD/backups/` so archives land where you ran the script, not where the script lives. Use `--backup-dir` to override.

```
/data/stacks/                   # put the script here, or cd here and use it from PATH
├── random_stack/               # A compose project
│   ├── compose.yaml
│   ├── .env
│   ├── app_data/               # Bind-mounted volume dir
│   └── db_data/                # Bind-mounted volume dir
├── nextcloud/                  # Another compose project
│   ├── compose.yaml
│   └── ...
└── backups/                    # Created automatically in $PWD
    ├── random_stack_20260711_191731.tar.gz
    └── nextcloud_20260711_193245.tar.gz

/srv/docker/
└── grafana/                    # External running stack (discovered via --running
    ├── compose.yaml            # or targeted with --stack /srv/docker/grafana)
    ├── .env
    └── data/
```

## Running in a Container (Cron)

The script is designed to be trivially containerized. Run the container as root (or use `user: root`) to preserve file ownership.

Example Docker Compose entry:

```yaml
services:
  stack-backup:
    image: alpine:latest
    user: root
    entrypoint: |
      sh -c "
        apk add --no-cache docker-cli tar &&
        /data/docker-stack-backup.sh backup --running --backup-dir /backups
      "
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /path/to/stacks:/data:ro            # stacks dir for discovery
      - /path/to/backups:/backups
      # Mount any external stack directories that --running may discover:
      - /srv/docker:/srv/docker:ro
    restart: no
```

Or as a cron job on the host (in root's crontab, or use `sudo`):

```cron
0 3 * * * /path/to/stacks/docker-stack-backup.sh backup --running >> /var/log/stack-backup.log 2>&1
```

## Requirements

- **Root access** — backup and restore require root because `tar --same-permissions --preserve-permissions` only preserves UID/GID ownership when running as root. The script automatically re-executes via `sudo` when needed. Read-only commands (`list`, `verify`, and the interactive menus) run without elevation. The `init` command also runs without root by default, but auto-escalates with `sudo` if the compose file isn't writable by the current user.

  When run via `sudo`, the backup archives (`.tar.gz` and `.meta` files) are automatically `chown`ed back to the original user so you can list, copy, or delete them without root.

- **bash** (no bashisms beyond `set -euo pipefail` and arrays)
- **tar** (for archive creation/extraction)
- **docker** (for compose lifecycle — `docker compose`)

If Docker is not available or not running, the script will still back up and restore files; it simply skips the container stop/start steps with a warning.