go tooling for s3 backups and maintenance
  • Go 93.7%
  • Makefile 5.5%
  • Dockerfile 0.8%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Michael Flemming f5c314efa7
Merge pull request #1 from opencloud-eu/add_ci
added Dockerfile and ci-pipeline
2025-11-20 17:16:42 +01:00
.woodpecker added Dockerfile and ci-pipeline 2025-11-20 17:15:30 +01:00
test use ListObject to verify tombstone cleanup 2025-10-09 16:27:59 +02:00
.env.example feat: add docker compose 2025-10-14 09:07:46 +02:00
.gitignore feat: add gitignore 2025-10-14 08:57:31 +02:00
docker-compose.yml feat: add docker compose 2025-10-14 09:07:46 +02:00
Dockerfile added Dockerfile and ci-pipeline 2025-11-20 17:15:30 +01:00
go.mod initial commit 2025-10-08 15:59:36 +02:00
go.sum initial commit 2025-10-08 15:59:36 +02:00
LICENSE Create LICENSE 2025-10-13 16:57:22 +02:00
main.go update tools and tests 2025-10-09 16:09:13 +02:00
Makefile feat: add makefile 2025-10-14 08:56:13 +02:00
README.md docs: reformat README 2025-10-14 09:27:56 +02:00

s3ctl

A tool for managing S3 bucket versions and tombstones. It can restore a bucket to a specific point in time and clean up stale delete markers.

Contributing

We welcome contributions! This quick guide shows how to build and run the integration-style tests locally.

  • Prerequisites: Go 1.25+ and access to an S3-compatible endpoint (use the MinIO stack below for local dev).

Build

  • Using Makefile (recommended):
    • Build: make build (outputs bin/s3ctl). You can override GOOS/GOARCH, e.g. make build GOOS=linux GOARCH=amd64.
    • Run from source: make run (equivalent to go run .).
    • Install to GOPATH/bin: make install.
    • Clean artifacts: make clean.
  • Plain Go: go build -o bin/s3ctl .

Test

Helpers under test/main.go exercise restore and tombstone-cleanup flows.

  1. Start a local S3-compatible service (optional): see "Local MinIO with Docker Compose". Export env vars for the endpoint: S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, S3_USE_SSL.
  2. Prepare test data and capture a restore timestamp: go run test/main.go setup (note the printed bucket and RESTORE_TIMESTAMP).
  3. Build, then run commands against the test bucket:
    • Restore: export S3_BUCKET="<bucket>" RESTORE_TIMESTAMP="<timestamp>" && bin/s3ctl restore
    • Cleanup tombstones: bin/s3ctl cleanup-tombstones
  4. Optional verifications: go run test/main.go verify-restore and go run test/main.go verify-tombstone.

Please update this README if you change the flow.

Overview

This tool provides two main functions:

  1. Restore a bucket to a specific point in time by deleting object versions created after that timestamp.
  2. Remove stale delete markers (tombstones) that have no live versions.

Prerequisites

  • An S3-compatible storage system (e.g., MinIO, AWS S3)
  • Properly configured environment variables (see below)

Configuration (Environment variables)

Variable Description
S3_ENDPOINT S3 endpoint (e.g., localhost:9000)
S3_ACCESS_KEY S3 access key
S3_SECRET_KEY S3 secret key
S3_BUCKET Target bucket name
S3_USE_SSL Use SSL (true/false, default: false)

Usage

Commands

  • restore — Restore a bucket to a specific point in time.
    • Options: -dry-run (default: true). Example to actually apply changes: -dry-run=false.
  • cleanup-tombstones — Delete stale delete markers (no live versions).
    • Options: -dry-run (default: true).

Examples

  • Restore to a timestamp:
    export S3_BUCKET="your-bucket" RESTORE_TIMESTAMP="2023-01-01T00:00:00Z"
    bin/s3ctl restore -dry-run=false
    
  • Cleanup tombstones (preview only):
    bin/s3ctl cleanup-tombstones
    

CI hints

A minimal CI can:

  • Run go run test/main.go setup to create a versioned test bucket and produce a RESTORE_TIMESTAMP.
  • Execute bin/s3ctl restore -dry-run=false with the printed timestamp, then go run test/main.go verify-restore.
  • Execute bin/s3ctl cleanup-tombstones and then go run test/main.go verify-tombstone.

Local MinIO with Docker Compose

A basic MinIO server is included for local testing.

Quick start

  1. Copy the environment template and adjust credentials if desired:
    cp .env.example .env
    
  2. Start MinIO:
    docker compose up -d
    
  3. Access endpoints:

Using with this project

Export variables so the tools/tests can talk to MinIO:

export S3_ENDPOINT=localhost:9000
export S3_ACCESS_KEY=${MINIO_ROOT_USER:-minioadmin}
export S3_SECRET_KEY=${MINIO_ROOT_PASSWORD:-minioadmin123}
export S3_USE_SSL=false

Then follow the steps in the Usage section (e.g., go run test/main.go setup).

To stop/remove the stack:

docker compose down