< See latest news & posts
Introducing S3lect: A Go package for leader election using S3
TL;DR
- S3lect is a new Apache 2.0 licensed Go package for leader election that uses S3 (or S3-compatible storage) as the primary coordination mechanism.
- It requires no additional infrastructure beyond S3-compatible object storage, and supports custom adaptors too.
- S3lect has a dual-interval optimisation and optional peer-polling mode to reduce S3 read costs during stable periods.
- Please show your support by starring the GitHub repository: github.com/nadrama-com/s3lect
Today we’re releasing S3lect, a Go package for leader election that uses S3 as its coordination mechanism. It’s being released under the Apache 2.0 license as part of our broader commitment to Open Source.
Why S3 for leader election?
When building distributed systems, leader election is often necessary for coordinating work across multiple instances. Traditional solutions like etcd, Consul, or ZooKeeper are excellent but require running additional infrastructure - which adds operational overhead and cost.
We’re building an Open Source PaaS and to make deployment as fast and cost-effective as possible, we wanted something with minimal operational footprint. S3 provides strong consistency guarantees and atomic operations through ETags, which makes it possible to perform safe leader election.
S3lect leverages these primitives to provide reliable leader election without requiring any additional services beyond S3-compatible object storage.
How it works
S3lect’s algorithm is straightforward:
- Leaders maintain a lockfile in S3 with their ID, address, and last updated timestamp.
- Followers monitor the lockfile to detect if the leader has failed to update within the timeout period.
- Leadership changes use conditional writes with ETags to ensure only one instance can claim leadership at a time.
- Resilience through retries handles transient network failures automatically.
The package includes a dual-interval system that balances fast failover with cost efficiency:
- Frequent interval (5 seconds by default): Used during leadership transitions for fast failover detection.
- Infrequent interval (30 seconds by default): Used during stable periods to reduce S3 operations.
Peer-polling mode for cost optimisation
One unique feature is the optional peer-polling mode. When enabled, followers check the leader’s health via HTTP rather than polling S3 during stable periods. This can significantly reduce S3 read operations and costs, especially with larger follower counts.
If peer communication fails, instances automatically fall back to direct S3 polling and the frequent interval, ensuring reliability and consistency isn’t compromised.
When to use S3lect
S3lect is a great fit when you:
- Need leader election in a cloud-native application that already uses S3
- Want to minimize operational overhead and avoid running additional coordination services
- Are building cost-sensitive systems where S3 read optimisation matters
Integration
S3lect is designed to be simple to integrate. You provide an S3-compatible storage implementation (or use the built-in AWS SDK v2 implementation), configure your intervals and timeouts, and register callbacks for leadership change notifications.
The package handles all the complexity of leader election, health checking, and failover logic.
What’s next
We built S3lect for use in two components we’re working on - more on this soon, stay tuned!
We’re releasing it as a standalone package because we believe it’s broadly useful for cloud-native Go applications that need simple, reliable leader election without additional infrastructure.
Check out the S3lect repository for more info. And please join our Discord server if you’d like to discuss S3lect!
Published: .