scripts/proxmox-backup/README.md

# Proxmox VE Backup & Restore Tool

A self-contained bash script for backing up and restoring all critical Proxmox VE configuration files. Archives are stored locally (never on the remote host) and can be used to rebuild a PVE node after boot drive failure.

## Overview

The script connects to a Proxmox node (locally or over SSH), collects configuration files from across the filesystem, packages them into a timestamped `.tar.gz` archive, and stores it on the machine running the script. Restore extracts a backup archive and pushes files back.

## Requirements

- **Source node:** Proxmox VE (any supported version)
- **Machine running the script:** Linux with `bash`, `tar`, `gzip`
- **Remote access:** SSH access to the Proxmox node (passwordless key recommended)
- **Permissions:** Root or passwordless sudo on the Proxmox node
- **rsync (optional):** Installed on both machines for incremental transfers — falls back to tar/cat/pipe if absent

## Usage

```
Usage: pve-backup.sh <command> [options]

Commands:
  backup        Create a backup of Proxmox configuration
  restore       Restore Proxmox configuration from a backup
  list          List available backups
  verify        Verify a backup archive's integrity (SHA256)

Backup options:
  -d, --dir DIR         Local backup directory (default: ./pve-backups)
  -h, --host HOST       Remote Proxmox host (SSH)
  -u, --user USER       SSH user (default: root)
  -i, --identity FILE   SSH identity file

Restore / Verify options:
  -d, --dir DIR         Directory containing backup archive
  -f, --file FILE       Specific backup archive to restore/verify
  -y, --yes             Skip confirmation prompts (DANGEROUS)

List options:
  -d, --dir DIR         Directory to list backups from
```

### Examples

```bash
# Local backup (must be root on the Proxmox node)
./pve-backup.sh backup

# Remote backup with custom directory
./pve-backup.sh backup -h 192.168.1.100 -d /mnt/backups

# Remote backup as non-root user (uses sudo -n automatically)
./pve-backup.sh backup -h 192.168.1.100 -u admin -i ~/.ssh/id_rsa

# Verify a backup archive
./pve-backup.sh verify -f pve-backups/pve-backup-prox1-20250101_120000.tar.gz

# Restore the latest backup
./pve-backup.sh restore

# Restore a specific backup archive
./pve-backup.sh restore -f /path/to/pve-backup-prox1-20250101_120000.tar.gz

# Non-interactive restore (for automation)
./pve-backup.sh restore -f /path/to/backup.tar.gz -y

# List all local backups
./pve-backup.sh list
```

### Defaults

| Setting | Default |
|---|---|
| Backup directory | `./pve-backups` (relative to script location) |
| SSH user | `root` |
| Archive name format | `pve-backup-{hostname}-{YYYYMMDD}_{HHMMSS}.tar.gz` |

## Transfer Method

The script uses one of two transfer methods depending on what's available:

### rsync (preferred)

When connecting to a remote host, the script checks if `rsync` is installed on the remote side. If yes:

- File-based configs (`/etc/pve/`, `/etc/network/interfaces`, APT sources, etc.) are transferred via `rsync -a`
- Subsequent backups only transfer changed files, reducing bandwidth and time
- Permissions and ownership are preserved natively by rsync
- If sudo is needed, `--rsync-path="sudo -n rsync"` is used automatically

### tar/cat/pipe (fallback)

If rsync is not available on the remote host, the script falls back to the original method:

- `/etc/pve/` is tarballed on the remote, piped through SSH, and extracted locally
- Individual files are fetched with `cat > file` over SSH
- Directories (APT sources, sysctl, cron) are tarballed with `cp -a`, then piped through SSH
- Slightly slower on repeated runs since everything is transferred every time

## What Gets Backed Up

### /etc/pve/ (pmxcfs snapshot)

The core Proxmox cluster filesystem. Transferred via `rsync -a` when available, otherwise tarballed and piped over SSH. Either way, runtime-only data is excluded:

| Included | Description |
|---|---|
| `datacenter.cfg` | Datacenter-wide settings |
| `storage.cfg` | Storage pool definitions |
| `user.cfg` | Users, groups, and permissions |
| `domains.cfg` | Authentication realms (LDAP, AD, etc.) |
| `vzdump.cfg` | Default backup job settings |
| `nodes/{node}/qemu-server/*.conf` | VM configuration files |
| `nodes/{node}/lxc/*.conf` | LXC container configuration files |
| `nodes/{node}/pve-ssl.pem` | Web UI SSL certificate |
| `nodes/{node}/pve-ssl.key` | Web UI SSL private key |
| `nodes/{node}/pveproxy-ssl.pem` | Custom SSL certificate (if set) |
| `nodes/{node}/pveproxy-ssl.key` | Custom SSL private key (if set) |
| `sdn/` | Software Defined Network configs (VLANs, VNets, subnets) |
| `ha/` | High Availability resources and groups |
| `firewall/` | Datacenter and node-level firewall rules |
| `priv/tfa.cfg` | TFA tokens (restore with caution) |
| `priv/shadow` | User password hashes |

| Excluded | Reason |
|---|---|
| `.rrd/` | Round-robin database (runtime stats, regenerated) |
| `priv/known_hosts` | SSH host keys (regenerated on boot) |
| `*.lock` | Runtime locks (ephemeral) |

### Network Configuration

| File | Description |
|---|---|
| `/etc/network/interfaces` | Bridge, bond, VLAN, and NIC definitions |
| `/etc/hostname` | System hostname |
| `/etc/hosts` | Static hostname resolution |
| `/etc/fstab` | Filesystem mount table (ZFS, NFS, CIFS mounts, etc.) |
| `/etc/resolv.conf` | DNS nameservers and search domains |

### APT Sources

| Path | Description |
|---|---|
| `/etc/apt/sources.list` | Main APT repository list |
| `/etc/apt/sources.list.d/` | Third-party and Proxmox enterprise/no-subscription repos |
| `/etc/apt/auth.conf` | APT authentication credentials |

### Cluster Configuration

| File | Description |
|---|---|
| `/etc/corosync/corosync.conf` | Corosync cluster ring and membership config |

### Ceph Storage

| File | Description |
|---|---|
| `/etc/ceph/ceph.conf` | Ceph cluster connection settings |

### System Tuning

| Path | Description |
|---|---|
| `/etc/sysctl.d/` | Kernel parameter overrides (network buffers, etc.) |
| `/etc/cron.d/` | System cron jobs |
| `/etc/cron.daily/` | Daily cron scripts |

### Package Manifest

| File | Contents |
|---|---|
| `packages.txt` | Full list of installed packages (`dpkg-query -W`) |
| `packages-manual.txt` | Manually installed packages (`apt-mark showmanual`) |
| `pveversion.txt` | Proxmox VE version string |

### Firewall Rules

| File | Description |
|---|---|
| `iptables.rules` | IPv4 iptables ruleset (via `iptables-save`) |
| `ip6tables.rules` | IPv6 ip6tables ruleset (via `ip6tables-save`) |

### Metadata (META file inside archive)

```
hostname:   proxmox1
created:    2025-01-01T12:00:00Z
pveversion: pve-manager/8.2.4/123456789abc
pve_nodes:  proxmox1 proxmox2 proxmox3
```

## Restore Process

1. Extracts the backup archive to a temporary directory
2. **Creates a rollback snapshot** of the current `/etc/pve` at `/var/tmp/pve-restore-rollback-{timestamp}/` (and also a tarball at `/etc/pve-rollback.tar.gz` on the remote node)
3. Restores `/etc/pve` by piping the archived tarball into `tar -x -C /etc/pve` on the target node
4. Restores network configs (`/etc/network/interfaces`, `hostname`, `hosts`, `resolv.conf`) by piping each file through SSH
5. Restores APT sources, sysctl, and cron overrides
6. Does **not** restore the package manifest — this is for reference only

After restore:
- **Reboot recommended** — network and hostname changes apply at boot
- **Restart `pveproxy`** if the web UI behaves oddly after `/etc/pve` restore

## SSH Non-Root Access (sudo)

If you connect with a non-root SSH user (`-u admin`), the script automatically detects this and prefixes all remote commands with `sudo -n`. The remote user must have passwordless sudo configured:

```bash
echo "admin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/admin
```

If passwordless sudo is not available, the script exits with a clear error.

## Archive Format

Each backup is a gzipped tarball with this internal structure. The `/etc/pve/` data and sysconfig may be stored either as raw directories (when rsync was used) or as nested tarballs (when the fallback method was used):

```
pve-backup-proxmox1-20250101_120000.tar.gz
├── META                          # Metadata (hostname, date, version, nodes)
└── etc/
    ├── pve/
    │   ├── datacenter.cfg        # (raw files — rsync) or
    │   └── pve-config.tar.gz     # (nested tarball — fallback)
    ├── apt/
    │   ├── sources.list          # (raw files — rsync) or
    │   └── apt-sources.tar.gz    # (nested tarball — fallback)
    └── system/
        ├── interfaces            # /etc/network/interfaces
        ├── hostname              # /etc/hostname
        ├── hosts                 # /etc/hosts
        ├── resolv.conf           # /etc/resolv.conf
        ├── fstab                 # /etc/fstab
        ├── corosync.conf         # /etc/corosync/corosync.conf
        ├── ceph.conf             # /etc/ceph/ceph.conf
        ├── sysctl.d/             # (raw dir — rsync) or
        ├── sysconfig.tar.gz      # (nested tarball — fallback)
        ├── packages.txt          # Full package manifest
        ├── packages-manual.txt   # Manually installed packages
        ├── pveversion.txt        # pveversion output
        ├── iptables.rules        # iptables-save output
        └── ip6tables.rules       # ip6tables-save output
```

The restore logic handles both layouts transparently.

## Verification

Every backup automatically generates a `.sha256` checksum file alongside the archive. You can verify integrity at any time:

```bash
# Verify the latest backup
./pve-backup.sh verify -f pve-backups/pve-backup-prox1-20250101_120000.tar.gz
```

The verify command checks:
1. The archive is a valid gzip'd tar file (no corruption)
2. The SHA256 checksum matches the stored `.sha256` file (no tampering)

If no `.sha256` file exists (e.g., from older backups), one is generated on the spot.

## Security Notes

- **Backup over SSH:** All data is transferred over the encrypted SSH channel. No data is stored on the remote host.
- **Sensitive data:** The archive contains SSL private keys, password hashes (`/etc/pve/priv/shadow`), and TFA tokens. Treat backup archives as sensitive credentials.
- **SSH keys:** Use `-i` to specify an SSH identity file. Agent forwarding is not required.

## Limitations

- Package manifest is for reference only — the script does not install/restore packages
- Firewall rules are exported via `iptables-save` — restoration requires manual `iptables-restore` (PVE firewall is handled via `/etc/pve/firewall/` in the pve-config archive)
- Ceph config is backed up, but the Ceph monitor maps and OSD tree are not
- VM/CT disk images and backups are **not** included — use the built-in Proxmox Backup Server or `vzdump` for those

## Future Enhancements (not yet implemented)

These are left as exercises for the reader or future development:

- **Retention / rotation** — `--keep N` flag to auto-prune backups older than N days (or keep only N most recent)
- **Cron / systemd timer** — drop-in unit for daily automated backups
- **Encryption** — GPG or `age` encrypt the archive before writing it to disk, decrypt on restore
- **Email / notification** — alert on backup failure or successful completion
- **Restore dry-run** — show what would be restored without actually pushing files