seaweedfs/seaweedfs: SeaweedFS is a distributed storage system for object storage (S3), file systems, and Iceberg tables, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.

SeaweedFS is a distributed storage system for object storage (S3), file systems, and Iceberg tables, designed to handle billions of files with O(1) disk access and effortless horizontal scaling. - ...

Summary

SeaweedFS is a Go-based distributed storage system that serves an S3 object store, a POSIX file system, and an Iceberg or Lance lakehouse from a single weed binary. It targets workloads with billions of files by storing small files as single blobs in append-only volume files, giving each read O(1) disk seek complexity. The project is Apache-2.0 licensed and exposes master, volume server, filer, and S3 components that can be run together or scaled out across machines.

What it is useful for

SeaweedFS is useful when you need an S3-compatible object store on your own hardware, including for Kubernetes clusters via the provided Helm chart, CSI driver, and Operator, and for bare-metal deployments. The S3 gateway implements 73 object and bucket operations, 36 S3 Tables operations, 39 IAM operations, and STS, so existing AWS SDKs, the AWS CLI, rclone, restic, Spark, and Trino work without modification. Object features documented in the README include versioning, Object Lock with retention and legal hold, lifecycle rules, SSE-S3, SSE-KMS, and SSE-C encryption, presigned URLs, multipart uploads, and bucket policies with conditions.

The filer layer adds directories, FUSE mounts on Linux, macOS, and Windows, WebDAV, SFTP, HDFS, and HTTP, with metadata stored in a database you already operate, such as PostgreSQL, MySQL, Redis, Cassandra, MongoDB, or etcd, among others. SeaweedFS can also act as a cache or write-back tier in front of S3, GCS, Azure, or Backblaze B2 through the Cloud Drive feature, and supports rack- and data-center-aware replication, background erasure coding, and active-active async replication between clusters.

How engineers can use it

The fastest path is downloading the binary and running one process:

curl -fsSL https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/install.sh | bash
AWS_ACCESS_KEY_ID=admin AWS_SECRET_ACCESS_KEY=secret S3_BUCKET=my-bucket ./weed mini -dir=./data

This brings up master, volume server, filer, WebDAV, the Iceberg REST catalog, and the Admin UI on one node, with the S3 endpoint at http://localhost:8333. A docker run with the chrislusf/seaweedfs image and matching environment variables provides the same behavior, while docker compose -f seaweedfs-compose.yml runs the components as separate services. For Kubernetes, helm install seaweedfs seaweedfs/seaweedfs plus the example values.yaml deploys a production-shaped cluster. To grow capacity, start additional weed volume -dir=/data -master=<host>:9333 processes; the README notes rebalancing only happens on demand via weed shell. Documented caveats include a macOS quarantine step (xattr -d com.apple.quarantine ./weed) and that weed mini is auto-tuned for a single node.