Self-Hosted S3 Object Storage for a Homelab: SigV4, Object Lock

Self-hosted S3 compatible object storage is a server you run on your own machines that answers the same HTTP API as Amazon S3, so backup tools, apps and scripts that speak S3 can write to your disks instead of a cloud bucket. For a homelab it works well as a backup target if you get four things right: request signing (SigV4) and clocks, deletion resistance (versioning and Object Lock, where the server supports them), data protection across disks and nodes (replication or erasure coding), and hardening (per application keys, TLS, no public admin surface, and restore tests).

This guide goes one level deeper than our posts on homelab backups with restic and Borg and the home NAS with ZFS. Those posts treat object storage as one possible backup target; here we open it up. Every protocol fact comes from the Amazon S3 and IAM documentation, and every server fact comes from that project's own repository or documentation, checked on 25 October 2026.

Key takeaways

• "S3 compatible" means a server implements some subset of the S3 REST API. Basic object reads, writes, lists and multipart uploads are near universal; versioning, Object Lock, bucket policies and lifecycle rules vary a lot. Check each project's own compatibility table before you design around a feature.
• SigV4 signs every request with a key derived from your secret, the date, the region and the service. A wrong clock, a wrong region string or a proxy that rewrites the Host header all produce signature errors that look alike.
• Object Lock only works on versioned buckets and protects object versions, not keys. It stops a stolen backup credential from destroying old data, but only if the server implements it and the attacker cannot reach an admin path that bypasses it.
• On three nodes, three way replication gives you one third of raw capacity as usable space. Erasure coding with two data and one parity chunk gives you two thirds, tolerates one node loss, and costs more CPU and more complex repair.
• The MinIO GitHub repository is archived and marked as no longer maintained, and its community edition moved to source only distribution before that. Garage (AGPL 3.0), SeaweedFS (Apache 2.0) and Ceph RGW (LGPL 2.1 or 3) are active.
• A single disk running an S3 server is not a backup. A backup is a copy you have restored from recently, on separate hardware, that one compromised key cannot erase.

What does S3 compatible actually mean?

Amazon's S3 API reference describes S3 as a REST API that uses standard HTTP headers and status codes, so any toolkit that speaks HTTP can use it. "S3 compatible" is not a certification. It is a claim by another server's authors that their server answers enough of those same requests, with the same request and response shapes, that S3 clients work against it.

The core building blocks

The model has only a few nouns:

• Bucket. A named container with configuration attached: versioning state, Object Lock configuration, lifecycle rules, policies, CORS and so on.
• Object. A key (a string that often looks like a path, such as backups/host1/data/ab/abcd...), a body of bytes and metadata. There are no real directories. A "folder" is just a shared key prefix, and listing with a delimiter fakes the hierarchy.
• Version. When versioning is on, writing the same key again creates a new version, and deleting a key adds a delete marker instead of removing data. Old versions remain until something explicitly deletes them by version ID.
• Access key and secret key. The credential pair a client uses to sign requests. The access key ID travels in the request; the secret never does.

Multipart uploads

Large objects are uploaded in parts. A client starts a multipart upload, uploads numbered parts in any order and in parallel, then sends a completion request listing the parts. Amazon's multipart documentation says part numbers run from 1 to 10,000 and do not need to be consecutive. The S3 quick facts table lists the limits that clients are built around: a maximum object size of 48.8 TiB, at most 10,000 parts per upload, and parts between 5 MiB and 5 GiB, with no minimum for the last part.

What varies between servers is cleanup. An upload that is started but never completed leaves orphan parts that consume disk. On AWS you clean them up with a lifecycle rule (AbortIncompleteMultipartUpload). Garage's compatibility page notes that its lifecycle support covers exactly two actions, AbortIncompleteMultipartUpload and Expiration, which happens to be the one that matters most for a backup target.

Presigned URLs

A presigned URL puts the SigV4 signature in the query string instead of a header, so someone without credentials can perform one specific operation (usually a GET or PUT on one key) until the URL expires. Amazon's documentation says the console allows expiry between 1 minute and 12 hours, the CLI and SDKs allow up to 7 days with long term credentials, and a URL signed with temporary credentials dies when those credentials expire. It also expires early if the credential that signed it is revoked or deleted. That last rule is a useful lever on your own server: if you ever hand out presigned URLs, sign them with a dedicated key you can delete.

Which parts self-hosted servers usually implement

Read the compatibility page of the specific server, for the version you will run. Garage, for example, publishes a full table that marks PutBucketVersioning, ListObjectVersions, PutObjectLegalHold and PutBucketPolicy as missing, and says plainly that it does not yet support object versioning. That is a design choice for small geo distributed clusters, and it means Garage alone is not where you get deletion resistance.

How does SigV4 request signing work?

Every S3 request to a server that enforces authentication carries a Signature Version 4 signature. The IAM user guide lays out the steps, and every S3 compatible server reproduces them because otherwise standard clients would fail.

The five steps

1. Build the canonical request. The client arranges the request into a fixed format: HTTP method, the canonical URI (the path, URI encoded), the canonical query string (parameters sorted by name), the canonical headers (lowercased names, trimmed values, sorted), the list of signed header names, and the hash of the payload. For S3 the payload hash is also sent in the x-amz-content-sha256 header, or the literal value UNSIGNED-PAYLOAD when the client chooses not to hash the body.
2. Hash the canonical request with SHA-256 and hex encode it in lowercase, as the IAM guide specifies.
3. Build the string to sign. This joins the algorithm name (AWS4-HMAC-SHA256), the request timestamp, the credential scope and the hash from step 2. The credential scope is date/region/service/aws4_request, for example 20261025/garage/s3/aws4_request.
4. Derive the signing key. The client never signs with the secret directly. It runs a chain of HMAC-SHA256 operations: first the date keyed with AWS4 plus the secret, then the region, then the service name, then the string aws4_request. The IAM guide notes that this makes the key, and therefore the signature, different for each region and each day.
5. Calculate the signature and attach it. The final HMAC over the string to sign, keyed with the derived signing key, goes into the Authorization header (or into X-Amz-Signature for a presigned URL), together with the access key ID, the credential scope and the signed header list.

The server repeats steps 1 to 5 with its own copy of the secret and compares. If anything that went into the canonical request changed in transit, or either side used a different date, region or service, the results differ and the server answers 403 with SignatureDoesNotMatch.

Worked example: what goes into the scope

Suppose restic on your laptop backs up to a Garage cluster configured with s3_region = "garage" at 23:58 UTC on 25 October 2026.

• Timestamp: 20261025T235800Z.
• Credential scope: 20261025/garage/s3/aws4_request.
• Signing key: HMAC chain over 20261025, then garage, then s3, then aws4_request.

Now suppose the client was configured with region us-east-1 because that is the SDK default. The scope says us-east-1, the server derives its key with garage, and every request fails even though the access key and secret are correct. Garage's quick start sets AWS_DEFAULT_REGION='garage' in its example client configuration for exactly this reason. Ceph RGW uses the zonegroup name as its region. SeaweedFS documents its own default. Always set the region your server expects.

Why clock skew breaks SigV4

The timestamp is part of the signed material, and S3 servers refuse requests whose timestamp is too far from their own clock. This is replay protection: a captured request cannot be resent hours later because its signature commits to the time it was made. Amazon returns the error code RequestTimeTooSkewed for this case, and S3 compatible servers implement the same check with their own tolerance.

In a homelab, skew usually comes from three places:

• A Raspberry Pi or other board without a real time clock that boots with the wrong time and has not synced yet.
• A virtual machine restored from a snapshot or resumed after the host slept, whose clock jumped back.
• A container host where NTP is blocked by a firewall rule you wrote for something else.

The fix is boring and permanent: run an NTP client (chrony or systemd-timesyncd) on every node and every client, point them at the same sources, alert when offset grows, and order your backup timer after time-sync.target on systemd so it does not run before the clock is set.

Other ways SigV4 fails behind a proxy

Because the Host header and path are signed, a reverse proxy that rewrites either, adds a path prefix such as /s3/, or re-encodes the body breaks every request. Path style URLs (https://s3.lab.example/bucket/key) avoid the wildcard DNS and certificate that virtual host style (https://bucket.s3.lab.example/key) needs, and most self-hosted servers and clients support them.

Which open source S3 compatible servers exist, and what changed?

There are many, but four projects come up in almost every homelab discussion. We checked each one's repository on 25 October 2026 for licence and maintenance status. This section describes projects, not rankings.

MinIO (community repository archived)

MinIO's GitHub repository is licensed AGPL 3.0, and GitHub reports it as archived. The README now opens with a note in capitals that the repository is no longer maintained. Before that, the README had already announced that the community edition is distributed as source code only, with no more pre-compiled binary releases, and that historical binaries "will not receive updates". Its install instructions point to go install github.com/minio/minio@latest or building your own container image from the Dockerfile.

What this means for a homelab: an existing deployment keeps working, but you now own every rebuild, and there is no upstream line of fixes to follow for the community code. If you stay on it, treat it like any other unmaintained dependency: pin it, track advisories for it and for its Go dependencies, and plan a migration. The repository's erasure code guide is still a good description of the model it uses: Reed-Solomon sharding of each object into data and parity blocks across the drives of an erasure set, with half data and half parity as the documented default, so that up to half the drives in a set can fail.

Garage

Garage is developed by the Deuxfleurs collective, licensed AGPL 3.0, with its main repository on the project's own Forgejo instance and a GitHub mirror that was updated on 24 October 2026. Its mirror describes it as an S3 compatible object store for small self-hosted geo distributed deployments, which is an accurate summary of its niche.

Garage protects data by replication, set with replication_factor in garage.toml. Its documentation's full example uses replication_factor = 3 and a consistency_mode setting. Nodes are placed in zones (garage layout assign -z dc1), and Garage spreads copies across zones, which suits three machines in three different houses. It exposes the S3 API on port 3900 in the quick start, an RPC port on 3901 secured with a shared 32 byte rpc_secret, an optional static website endpoint on 3902, and an admin API on 3903 that also serves Prometheus metrics.

The trade offs are documented on its compatibility page: no versioning, no Object Lock, no bucket policies, partial ACL reporting, and lifecycle limited to aborting incomplete uploads and expiration. Access control is Garage's own model: keys are granted read, write and owner rights per bucket with garage bucket allow.

SeaweedFS

SeaweedFS is licensed Apache 2.0 and was very active when we checked, with release 4.50 published on 25 October 2026. It is a broader system than an S3 server: a blob store with volume servers, a filer for metadata, and an S3 gateway on top. Its README says hot data is replicated and erasure coding is applied to warm data in the background, so writes never pay the encoding cost, and that replication is rack and data centre aware.

Its README lists versioning, Object Lock with retention and legal hold, lifecycle rules, presigned URLs and multipart uploads, and says the S3 compatibility suite runs in CI. The project wiki page on Object Lock describes governance and compliance modes, legal holds, the s3:BypassGovernanceRetention permission, and two prerequisites that match AWS: versioning must be enabled, and Object Lock can only be turned on at bucket creation. One SeaweedFS specific note from that page: admin users can always bypass governance retention, so governance mode does not protect you from someone holding admin credentials.

Ceph RADOS Gateway (RGW)

Ceph is licensed LGPL 2.1 or LGPL 3 for most of its code, per the COPYING file in its repository, and the v21 release line was current when we checked. RGW is Ceph's object gateway: an HTTP service that translates S3 (and Swift) requests into operations on RADOS, Ceph's distributed object layer. Data protection comes from the pool underneath, which is either replicated or erasure coded.

Ceph's S3 feature table lists bucket policies, versioning, lifecycle, bucket notifications, public access block and multipart uploads as supported, with bucket replication marked partial. Its bucket operations reference documents the x-amz-bucket-object-lock-enabled header on bucket creation and PUT /{bucket}?object-lock for the default retention rule. Ceph is the heaviest option here: several nodes, monitors, managers and OSDs.

Replication versus erasure coding: how much raw capacity do you need?

Both models survive disk or node loss. They differ in how much raw disk they need and what a failure costs.

Replication stores N full copies of each object on N different failure domains (disks, nodes, racks or sites). Usable capacity is raw capacity divided by N. Reads are cheap, and repair is a straight copy.

Erasure coding splits each object into k data chunks and computes m parity chunks, then stores the k plus m chunks on k plus m different failure domains. Any k of them can rebuild the object, so it survives m failures. Usable capacity is raw times k divided by (k plus m). Repair needs to read k chunks to rebuild one, which costs network and CPU.

Worked example 1: three nodes, 4 TB each

You have three mini PCs, each with one 4 TB disk, so 12 TB raw. You want to survive the loss of any one node.

That last row is the trap. Ceph's erasure code documentation says the default erasure coded pool profile is k=2 m=2, which needs four chunks on four distinct failure domains. With a failure domain of host and only three hosts, the chunks cannot be placed. The same page warns that a pool's profile cannot be changed after creation; to change it you create a new pool and move every object. Decide k, m and the failure domain before you load data.

Applied to a real backup: a 1.5 TB restic repository with 50 percent growth room needs about 2.25 TB usable. Replication factor 3 gives 4 TB, which fits. Erasure coding k=2, m=1 gives 8 TB, but leaves you one failure from data loss during every rebuild.

Worked example 2: six nodes, 8 TB each

Six nodes with 8 TB each is 48 TB raw.

• Replication factor 3: 48 / 3 = 16 TB usable, survives 2 node losses.
• Erasure coding k=4, m=2: 48 x 4 / 6 = 32 TB usable, survives 2 node losses.

Same fault tolerance, twice the usable space. The catch: every one of the six nodes holds a chunk of every object, so you cannot lose a node during a rebuild of a second one without hitting the limit, and a rebuild after one 8 TB disk dies reads roughly four times the lost data across the network. On a 1 Gbit/s homelab network, reading 32 TB worth of chunks to rebuild 8 TB takes most of a few days. Check your network before you pick a wide profile; our 10 gigabit homelab network guide covers the upgrade.

Choosing for a homelab

On one machine with several disks, run the server on a ZFS mirror or RAIDZ pool and replicate to a second site. On three small nodes, replication factor 3 across zones is the simple choice. Erasure coding with m=2 makes sense from about six nodes on a fast network.

Whatever you pick, remember that redundancy protects against hardware failure, not against deletion. A DELETE replicates to every copy just as faithfully as a PUT.

How do Object Lock and versioning make backups resist deletion?

Amazon describes Object Lock as a write once, read many (WORM) model that blocks deletion or overwrite of an object version for a fixed time or indefinitely. The details matter, because a lot of homelab setups enable it and then discover what it does not do.

What the AWS documentation says

• Object Lock works only in buckets with versioning enabled.
• It protects specific object versions. Retention periods and legal holds do not prevent new versions from being written or delete markers from being added on top of a key. The protected version stays in place underneath.
• There are two retention modes. In compliance mode, no user can overwrite or delete the protected version until retention expires, including the account root user, and the retention period cannot be shortened or the mode changed. In governance mode, most users are blocked, but users with the s3:BypassGovernanceRetention permission can remove protection or delete the version, if they explicitly ask to bypass.
• A legal hold protects a version with no expiry until someone with the right permission removes it. It is independent of retention.
• A bucket can carry a default retention mode and period that applies to every new version written into it.

How this changes a ransomware or stolen key scenario

Suppose an attacker gets the access key your backup client uses and runs a delete of every key in the bucket.

• Without versioning: the objects are gone.
• With versioning, no lock: the deletes add delete markers. The data is still there as noncurrent versions, until the attacker uses the same key to delete each version by ID, which a write key with delete permission can do.
• With versioning and Object Lock in compliance mode, default retention 30 days: the deletes add delete markers, and attempts to delete specific versions fail until each version is 30 days old. You can list versions and restore the state from before the attack.

Note what restic forget --prune does here: its deletes of unneeded pack files become delete markers, and the locked versions remain until retention ends. Restic sees success, but disk usage does not drop until versions age out and a lifecycle rule or admin cleanup removes them. Size storage for the retention window.

The limits on a server you own

On your own server, anyone with root on the storage nodes can delete files on disk, wipe a volume, or replace the server binary. Object Lock on a self-hosted server protects against a compromised client credential, not against a compromised storage host.

So make the storage host hard to reach from the hosts it protects, keep admin credentials and the admin API off client machines, and keep a second copy on a different system.

Which servers support it

The "practical minimum" column describes what each project's documentation shows, not a hard floor. Feature support changes between versions, so re-check against the release you deploy.

Step by step: a small cluster with TLS, per application keys and a tested restore

This walkthrough builds a three node Garage cluster in containers, puts a TLS reverse proxy in front, creates separate keys for separate apps, points restic at it, and proves a restore. Garage is used because its documented setup is compact. Where Garage lacks a feature (bucket policies, Object Lock), the walkthrough shows the equivalent S3 calls for a server that has them.

Adjust host names, paths and sizes. Run commands as a user with Docker access on each node.

Step 1: prepare three nodes

Each node needs Docker (or Podman), a data disk mounted at, for example, /srv/garage, and working time sync.

# On every node: confirm the clock is synchronised before anything else
timedatectl show -p NTPSynchronized --value     # expect: yes
chronyc tracking 2>/dev/null | head -5           # if you use chrony

# Confirm the data disk is the one you think it is before using it
lsblk -o NAME,SIZE,MODEL,MOUNTPOINT
df -h /srv/garage

Give the nodes stable addresses on a storage or management VLAN. In this example they are 10.20.0.11, 10.20.0.12 and 10.20.0.13.

Step 2: write the configuration

Generate one RPC secret and one admin token, and share them across all three nodes through a secure channel. The RPC secret lets nodes join the cluster, so treat it like a root password.

openssl rand -hex 32      > rpc_secret       # 32 bytes hex, as Garage requires
openssl rand -base64 32   > admin_token
openssl rand -base64 32   > metrics_token
chmod 600 rpc_secret admin_token metrics_token

On each node, create /srv/garage/garage.toml, changing rpc_public_addr to that node's address:

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "lmdb"
metadata_auto_snapshot_interval = "6h"

replication_factor = 3
consistency_mode = "consistent"

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "10.20.0.11:3901"
rpc_secret_file = "/etc/garage/rpc_secret"

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"

[admin]
api_bind_addr = "10.20.0.11:3903"
admin_token_file = "/etc/garage/admin_token"
metrics_token_file = "/etc/garage/metrics_token"

The admin API binds to the management address only, not to every interface. The metrics endpoint requires a token, because Garage's monitoring page notes that without metrics_token anyone who can reach the admin port can scrape metrics. And there is no [s3_web] section, because a backup cluster does not need a public static website endpoint.

Step 3: run the containers

Pin the image to a specific release tag from the Garage releases page. Do not use a floating tag for a storage system; Garage's upgrade guide distinguishes minor upgrades from major ones that change protocols or data structures, and you want to choose when a major one happens.

docker run -d --name garage --restart unless-stopped \
  --network host \
  -v /srv/garage/garage.toml:/etc/garage.toml:ro \
  -v /srv/garage/secrets/rpc_secret:/etc/garage/rpc_secret:ro \
  -v /srv/garage/secrets/admin_token:/etc/garage/admin_token:ro \
  -v /srv/garage/secrets/metrics_token:/etc/garage/metrics_token:ro \
  -v /srv/garage/meta:/var/lib/garage/meta \
  -v /srv/garage/data:/var/lib/garage/data \
  dxflrs/garage:PINNED_RELEASE_TAG

Garage's quick start notes that on Linux you can use host networking instead of publishing ports one by one. With host networking, your host firewall decides who reaches each port. Allow 3901 only between the three nodes, 3900 only from the proxy, and 3903 only from your management network and Prometheus.

Step 4: form the cluster and assign the layout

alias garage='docker exec -ti garage /garage'

garage node id              # on each node; note the id@address string
# From node 1, connect the other two
garage node connect <node2-id>@10.20.0.12:3901
garage node connect <node3-id>@10.20.0.13:3901
garage status

# One zone per node, so the three copies land on three machines
garage layout assign -z zone-a -c 3T <node1-id-prefix>
garage layout assign -z zone-b -c 3T <node2-id-prefix>
garage layout assign -z zone-c -c 3T <node3-id-prefix>
garage layout show
garage layout apply --version 1

Set capacities a little below real disk size to leave headroom.

Step 5: put TLS in front

Clients should never send SigV4 traffic, or the data it protects, over plain HTTP across your network. A reverse proxy such as Caddy or nginx terminates TLS and forwards to the three S3 ports. Use a certificate from your own internal CA if the endpoint is internal; our homelab private CA guide covers ACME issuance from a private root.

A Caddyfile for path style access:

s3.lab.example {
    tls /etc/caddy/certs/s3.crt /etc/caddy/certs/s3.key
    reverse_proxy 10.20.0.11:3900 10.20.0.12:3900 10.20.0.13:3900 {
        lb_policy round_robin
        health_uri /health
        health_port 3903
    }
    request_body {
        max_size 6GB
    }
}

Caddy passes the Host header through by default, which SigV4 needs. Do not add uri strip_prefix or similar rewrites. The active health check uses Garage's admin API /health endpoint, which its reference says returns 200 when enough nodes are up to form a quorum and 503 otherwise, so the proxy stops sending traffic to a node that cannot serve. The body size limit must be larger than your biggest multipart part, or uploads fail partway.

Test from a client:

curl -sS -o /dev/null -w '%{http_code}\n' https://s3.lab.example/
# 403 is expected: unauthenticated request, but TLS and routing work

Step 6: create per application keys

One key per application, one bucket per application, and only the rights it needs.

garage bucket create restic-laptop
garage bucket create restic-server1
garage bucket create nextcloud-data

garage key create restic-laptop-key
garage key create restic-server1-key
garage key create nextcloud-key

garage bucket allow --read --write restic-laptop  --key restic-laptop-key
garage bucket allow --read --write restic-server1 --key restic-server1-key
garage bucket allow --read --write nextcloud-data --key nextcloud-key

garage bucket info restic-laptop      # shows exactly which keys have which rights

Do not grant --owner to application keys. Owner rights allow a key to change bucket settings, and an application never needs that. Garage's own quick start grants --owner in its example for convenience; in production keep owner rights for an admin key that lives off the application hosts.

Note that restic needs delete rights even for plain backups, because it creates and removes lock files in the repository's locks directory. That is why key permissions alone cannot make a restic repository deletion proof. On Garage, deletion resistance has to come from a second copy that the client key cannot reach. On servers with Object Lock, it comes from the lock.

The same intent as a bucket policy, for servers that support them (SeaweedFS, Ceph RGW). The JSON follows the AWS policy grammar. The principal format for a local user differs between servers, so copy it from your server's bucket policy documentation rather than from this example.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ResticLaptopObjects",
      "Effect": "Allow",
      "Principal": { "AWS": ["arn:aws:iam:::user/restic-laptop"] },
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": ["arn:aws:s3:::restic-laptop/*"]
    },
    {
      "Sid": "ResticLaptopList",
      "Effect": "Allow",
      "Principal": { "AWS": ["arn:aws:iam:::user/restic-laptop"] },
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::restic-laptop"]
    }
  ]
}

aws --endpoint-url https://s3.lab.example s3api put-bucket-policy \
  --bucket restic-laptop --policy file://restic-laptop-policy.json

What this policy does not include matters as much as what it does: no s3:PutBucketPolicy, no s3:BypassGovernanceRetention, no s3:PutObjectRetention, no wildcard principal, no s3:*.

Step 7 (Object Lock servers only): create a locked bucket

On a server that implements Object Lock, the bucket must be created with it. Amazon and the SeaweedFS wiki both say you cannot add it to an existing bucket.

export AWS_ENDPOINT_URL=https://s3.lab.example
export AWS_DEFAULT_REGION=your-server-region

aws s3api create-bucket --bucket restic-locked \
  --object-lock-enabled-for-bucket

aws s3api put-object-lock-configuration --bucket restic-locked \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 30 } }
  }'

aws s3api get-object-lock-configuration --bucket restic-locked
aws s3api get-bucket-versioning --bucket restic-locked   # expect Enabled

Start with governance mode and a short period while you test, because compliance mode cannot be shortened, and on a self-hosted server a mistake costs disk until it expires. Move to compliance once you have sized storage for the retention window.

Step 8: point restic at the bucket

Restic's repository documentation shows the pattern for S3 compatible servers: set the credentials in environment variables and use an s3: repository URL with the server address.

export AWS_ACCESS_KEY_ID='GK...restic-laptop-key-id'
export AWS_SECRET_ACCESS_KEY='...'
export RESTIC_REPOSITORY='s3:https://s3.lab.example/restic-laptop'
export RESTIC_PASSWORD_FILE="$HOME/.config/restic/password"

restic -o s3.region=garage init
restic -o s3.region=garage backup ~/Documents ~/Projects --exclude-caches
restic -o s3.region=garage snapshots

Keep the key, secret and repository password in a password manager or a self-hosted secrets server rather than a dotfile. Without the restic password the repository cannot be decrypted, however redundant the cluster.

Two restic options matter on self-hosted servers. Restic's documentation says that if no region is given it uses us-east-1, which will not match a server configured with a different region, so pass -o s3.region= with your server's value on every command (or wrap restic in a small script). And -o s3.bucket-lookup selects path style or virtual host style; its default, auto, uses path style for any endpoint that is not Amazon or Google, which is what most homelab setups want.

Step 9: run a restore test

A backup that has never been restored is a hope.

# 1. Verify repository structure and a slice of the data
restic check --read-data-subset=1/10

# 2. Restore the latest snapshot to a scratch directory
mkdir -p /tmp/restore-test
restic restore latest --target /tmp/restore-test --include ~/Documents

# 3. Compare against the source
diff -rq ~/Documents /tmp/restore-test$HOME/Documents | head

# 4. Clean up the scratch copy
rm -rf /tmp/restore-test

Then stop one storage node (docker stop garage on node 3) and restore again; with replication factor 3 the cluster should keep serving with two nodes. Start it again and watch it catch up with garage status. Finally, restore on a different machine using only what you would have after losing the original laptop: the repository URL, the key, and the password from your password manager. If any of those lived only on the laptop, you have found your real single point of failure.

For a locked bucket, add one more test with a throwaway object: try to delete a specific version with the application key and confirm it is refused.

echo test > lock-test.txt
aws s3api put-object --bucket restic-locked --key lock-test.txt --body lock-test.txt
aws s3api list-object-versions --bucket restic-locked --prefix lock-test.txt
aws s3api delete-object --bucket restic-locked --key lock-test.txt \
  --version-id <VersionId-from-above>
# Expect AccessDenied while retention is active

How do you monitor and upgrade it?

Metrics and alerts

Every server in this guide exposes Prometheus format metrics. Garage serves them on the admin port at /metrics, protected by the bearer token you set as metrics_token. A Prometheus scrape job:

scrape_configs:
  - job_name: garage
    scheme: http
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/garage_metrics_token
    static_configs:
      - targets: ['10.20.0.11:3903', '10.20.0.12:3903', '10.20.0.13:3903']

Alert on what predicts data loss, not on everything:

Keep the monitoring stack on the management network. Our homelab monitoring guide covers safe exposure of Prometheus and Grafana.

Disk failure

With replication factor 3, a failed disk on one node is a capacity event, not a data event. Identify the disk by serial number, not device name, replace it, remount, restart the node and let the cluster resync, then run the server's repair checks (garage repair on Garage). On Ceph, wait until degraded placement groups recover before taking another host down.

Rolling upgrades

Read the release notes for each upgrade. Garage's guide is explicit:

• A minor upgrade keeps protocols and data structures, and does not imply cluster downtime. Upgrade one node at a time, waiting for it to rejoin and report healthy before moving on.
• A major upgrade changes protocols or data structures, and Garage signals one by bumping the first non zero component of the version (for example from v0.7.2 to v0.8.0). Major upgrades are supported only between contiguous versions, so a jump of two majors needs an intermediate step.

The same principle holds for SeaweedFS and Ceph: back up metadata before the upgrade, upgrade in the documented order, and do not mix major versions for longer than the procedure requires. Take a metadata snapshot first. Garage's metadata_auto_snapshot_interval setting creates periodic snapshots of its metadata database, which is the part of the cluster you can least afford to lose.

Common mistakes

1. One root key for everything. The admin key goes into every app's config "just for now". One leaked config file then gives an attacker full control of every bucket, including the power to change settings and delete everything. Create one key per application, never give application keys owner or admin rights, and keep the admin key on a separate machine. Our post on object storage keys and tenant isolation explains why a bucket name is not an access boundary.
2. Public bucket policies. A policy with "Principal": "*" and s3:GetObject makes every object in the bucket readable by anyone who can reach the endpoint. Combined with port forwarding, that is a public file server for your backups. Audit policies after every change with get-bucket-policy, and enable public access block where the server supports it (Ceph lists it as supported).
3. A single disk deployment called a backup. An S3 server on one disk in the same machine, or the same rack, as the data it protects is a cache with an API. A disk failure, a power event, theft or a fire takes both. A backup needs a second device and ideally a second location.
4. No restore tests. Most backup failures are discovered during the restore you needed. Schedule a monthly restore of a sample and a yearly full restore on different hardware.
5. Exposing the admin console or admin API. Management ports belong on a management network behind a VPN or SSO, never on a port forward. Our homelab single sign on guide covers putting web consoles behind an identity provider.
6. Object Lock assumed, not verified. The server accepts the create-bucket flag but does not enforce retention, or the bucket was created without it. Test a version delete with the application key, as in step 9, on every new bucket.
7. Governance mode with an admin key on the client. Governance mode can be bypassed by anyone with the bypass permission, and on SeaweedFS by admin users. If the backup client holds an admin key, governance mode protects nothing.
8. Unpinned images. A latest tag on a storage node can pull a major version that needs a migration. Pin versions and upgrade on purpose.

Your object storage cluster is built from open source code: the server itself, its Rust or Go dependencies, a TLS stack and a container base image, and with the MinIO community repository now archived, some homelabs are already running a server with no upstream fixes coming. The steps below apply to those components and to the apps that write into your buckets; they do not touch your disks, your backup data or the storage protocol itself.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Gold Registry provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. Ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

Safeguard does not manage disks, enforce Object Lock or check your backups. Where it fits is the software layer: the container images you run for the storage server and its proxy, and the repositories of the apps that store data in your buckets. The Safeguard CLI can scan a directory, a container image or an SBOM file (safeguard scan --dir ., --image <name>, --sbom <file>), with results written as JSON under ./output.

The first demo is a real Safeguard CLI session. After install, safeguard opens the interactive terminal, you ask it to fix the lodash CVE in a repository, and it reads the finding and advisory, asks before any command that changes something, edits package.json to lodash 4.17.21, runs npm install and npm test with your approval, prints a summary and pushes nothing. The same loop applies to a small upload service or dashboard in your lab that talks to the S3 endpoint.

Click through it at your own pace, or open it full screen.

Storage nodes sit on a private network and should stay there. The local runner demo shows how to set up the runner on your own machine, approve it with its device code, and then scan from the terminal. A runner on a management host can scan the pinned server images before you roll them out node by node.

Click through it at your own pace, or open it full screen.

If you work with Claude, the third demo shows how to connect Claude to the Safeguard MCP server with OAuth and choose which tools Claude can use. You can then ask about vulnerabilities in the projects you have scanned, such as the images behind your storage cluster, without giving the assistant anything more than the tools you picked.

Click through it at your own pace, or open it full screen.

To try it on your own repositories and images, download Safeguard and follow the CLI documentation.

FAQ

Is MinIO still open source?

The MinIO GitHub repository is still licensed AGPL 3.0, but GitHub shows it as archived, and its README says the repository is no longer maintained. Earlier, the community edition had moved to source only distribution with no new pre-compiled binaries. You can still build and run the code, but you are on your own for fixes.

Does Garage support Object Lock?

Not as of its current compatibility page, which lists versioning, Object Lock and legal hold operations as missing and says Garage does not yet support object versioning. Get deletion resistance from a second copy that the client key cannot reach.

Why does my S3 client say SignatureDoesNotMatch?

The server computed a different signature from the same request. The usual causes are a wrong secret, a region string that does not match the server's (for example us-east-1 against a server expecting garage), a reverse proxy that changed the Host header or path, or a modified body. Fix the region and proxy first.

What does RequestTimeTooSkewed mean?

The request's signed timestamp is too far from the server's clock, so the server rejects it as a possible replay. Synchronise time on both client and server with NTP, and make backup jobs start after time sync on boot.

Is replication or erasure coding better for three nodes?

Replication factor 3 is simpler and survives two node losses with one third of raw space usable. Erasure coding with k=2 and m=1 gives two thirds usable but survives only one loss. Ceph's default k=2, m=2 profile cannot be placed on three hosts with a host failure domain.

Can Object Lock protect my backups from ransomware?

It can protect object versions from deletion through the S3 API by a stolen client key, which is the most common way backups are destroyed. It cannot protect against someone with root on the storage nodes. Keep a second copy elsewhere and keep admin access off the hosts you back up.

Do I need TLS if the storage is on my LAN?

Yes. SigV4 protects the integrity of signed requests but does not encrypt their bodies. Without TLS, anyone on the network path can read objects in transit. Restic encrypts its data client side, but many other apps do not.

How many keys should I create?

One per application or per backup client, each limited to its own bucket, plus one admin key kept off application hosts. That way a single leaked config cannot affect other buckets, and you can revoke one key without breaking everything.

Quick check

Sources

• Amazon S3 API Reference: Welcome
• AWS IAM User Guide: AWS Signature Version 4 for API requests
• AWS IAM User Guide: Create a signed AWS API request
• Amazon S3 User Guide: Locking objects with Object Lock
• Amazon S3 User Guide: Uploading and copying objects using multipart upload
• Amazon S3 User Guide: Amazon S3 multipart upload limits
• Amazon S3 User Guide: Download and upload objects with presigned URLs
• Garage documentation: S3 compatibility status
• Garage documentation: Quick start
• Garage documentation: Configuration file format
• Garage documentation: Monitoring Garage
• Garage documentation: Upgrading Garage
• Garage repository mirror on GitHub
• SeaweedFS repository and README
• SeaweedFS wiki: S3 Object Lock and Retention
• Ceph documentation: Ceph Object Gateway
• Ceph documentation: Ceph Object Gateway S3 API
• Ceph documentation: S3 bucket operations
• Ceph documentation: Erasure code
• Ceph repository: COPYING
• MinIO repository and README
• MinIO repository: Erasure Code Quickstart Guide
• restic documentation: Preparing a new repository
• restic documentation: Removing backup snapshots

Last reviewed: 25 October 2026.