Why build a database on object storage?

A database built on object storage keeps its durable data in an object store, such as a service that implements the S3 API, and treats the memory and local disks of its servers as caches. This separates storage from compute: data can grow without adding servers, compute can scale with query load, and losing a server loses only cache. The cost is slower reads on a cache miss and a floor on write latency, so these systems depend on good caching and batched writes.

​

How do traditional databases store data?

In the traditional model, each database server owns its data on local disk, and some systems keep the whole dataset in memory. Durability comes from writing to that disk and replicating it to other servers’ disks. This couples storage and compute:

• Storing more data means bigger disks or more memory per machine, or splitting the data into shards across more machines.
• Serving more reads means adding replicas, and each replica keeps its own copy of the data (or of its shard).
• Replacing or adding a server means copying data to it before it can serve traffic.

This works well while the dataset fits comfortably on a few machines. It gets expensive when data grows faster than query load, because you pay for compute and fast storage to hold data that is rarely read. Graph databases often feel this sharply: many systems keep a large graph in memory or on local SSD for fast traversal, so the cost of fast storage grows with the whole graph, and in some systems its size is limited by what one machine can hold.

​

How does a database on object storage work?

A database on object storage writes all durable state to the object store and runs queries on compute nodes that cache the data they need in memory and on local disk.

• Object storage is the source of truth. Data files and index files live in the object store. The metadata that says which files make up the current version is typically stored there too, or in a separate strongly consistent metadata service.
• Compute nodes are close to stateless. They hold caches in memory and on local NVMe or SSD, and can be added, removed, or replaced without moving data.
• Reads go through the cache hierarchy: memory first, then local disk, then object storage on a miss.
• Writes are batched and published atomically. A writer typically uploads new immutable files, then updates the metadata to make them part of the current version. Object stores typically do not support updating part of an object in place; an object is written or replaced as a whole, so many designs use log-structured or immutable-file layouts.

Correctness depends on a few storage guarantees. The system needs newly written objects to be readable immediately (read-after-write consistency), so a reader that follows new metadata finds the files it points to. It also needs an atomic way to advance the current version, such as a conditional write (“create only if absent” or “replace only if unchanged”) or a separate coordination service, so two writers cannot both believe they committed the same version.

​

How does an object-storage database compare with a local-disk database?

The difference is where the durable copy lives, and that changes durability, scaling, recovery, and latency. Because the servers that run queries do not own the durable copy, storage and compute are separated and each side can be sized, scaled, and replaced on its own.

​

What are the tradeoffs of building a database on object storage?

The main costs are slower reads on a cache miss, a floor on write latency, and a heavy dependence on caching and request-efficient data layout.

• Cold-read latency. A cache miss pays an object-store round trip, which is much slower than reading local NVMe or memory. Latency depends on how much of the working set the caches hold.
• A write latency floor. When the object store is the only durable tier, a commit is durable only after the object store acknowledges it. Batching many writes into one upload keeps throughput high, but a single write cannot finish faster than that round trip. Some designs acknowledge a commit once it reaches a write-ahead log replicated across several servers, and upload it to the object store later; in those designs, losing one server still loses no committed data.
• Caching becomes central. Cache sizing, warming, and eviction policy decide latency, so many systems tune caching to the workload and warm caches after a restart.
• Request-oriented pricing and access. Object stores work best with fewer, larger requests, so data layout and batching matter more than on local disk.

​

When does a database on object storage cost less at scale?

It usually costs less when the bulk of a large dataset is cold: most of the data is read rarely, and a smaller working set serves most queries. An object-storage design matches cost to that shape:

• Bulk data is priced at object-storage rates, which are typically far lower per byte than provisioned SSD or memory, and the object store provides durability.
• Compute is sized for the working set and the query load, not for the total dataset.
• Compute scales with demand without copying data. New compute nodes share the same stored data and fill their caches as they serve queries, and nodes can be removed after a traffic spike because no data lives only on them.

Workloads with many small writes or frequent cache misses pay more in per-request charges, which can offset the per-byte savings.

​

When is object storage a poor fit?

A local-disk or in-memory database may be the better choice when every read, including the first, needs guaranteed sub-millisecond latency; when individual writes need the lowest possible commit latency; or when the dataset is small and stable enough that one well-provisioned machine holds it comfortably.

​

How does HelixDB use object storage?

HelixDB is built on object storage:

• Object storage is the source of truth. NVMe or SSD and memory are caches, and storage scales independently of compute.
• The system can recover from full cache loss by reading object storage. Cache misses fall through to object storage, so caching affects latency, not results.
• Helix Cloud runs a gateway, a single writer process, and readers that scale automatically. The writer uses MVCC, runs write transactions concurrently, and resolves conflicts at commit.
• Readers see new commits after a snapshot refresh. Reads served by the writer alone give read-after-write consistency.

Cold reads pay object-storage latency. See Architecture for the read and write paths and Tradeoffs for where the design fits and where another system may fit better.

​

Frequently asked questions

​

What is the cheapest way to store a large graph?

At scale, it is usually to keep the full graph in object storage and cache only the frequently traversed part. Keeping the whole graph in memory or on provisioned SSD means paying fast-storage prices for data that is rarely read. The object-storage approach trades that cost for slower traversals that touch uncached data.

​

Is a database on object storage slower?

It can be: reads that miss the cache and individual commits are slower, while cache hits perform like a traditional database. A miss pays object-storage latency, and a commit typically waits for a durable object-store write. Whether that matters depends on how well your working set fits in cache and how latency-sensitive each write is.

​

What happens if a server fails?

In an object-storage design, a failed compute node loses only its cache and any requests it was serving. A replacement reads the current version from object storage and warms its cache as it serves queries, so there is no data to re-replicate before it can start.

​

Is this the same as a serverless database?

Not exactly, but they are related. Separating storage from compute is one of the main things that lets serverless databases add and remove compute quickly, because no data lives only on a server and scaling needs no long data migration. A database can use object storage without being offered as a serverless service.

​

Related topics