Resumable uploads, and why they are hard

By Hein de Wilde··4 min read

A 100 GB upload will be interrupted before it finishes, so the only question that matters is whether it continues or starts again. Resumable protocols solve this by uploading in chunks and remembering the last committed offset — and the detail that catches everyone out is that the client's chunk size and the server's commit boundary have to agree, or the client retries the same bytes forever.

Why plain uploads fail at scale

A single HTTP request holding 100 GB has to survive for hours. In that window a laptop sleeps, a router restarts, a mobile connection changes cell, or a proxy times out. Any of those kills the request, and with a plain upload there is nothing to resume from — the server has bytes it cannot identify and the client starts from zero.

At 1 GB this is a nuisance. At 100 GB it means the file cannot be sent at all, because the probability of an uninterrupted multi-hour connection is low and the cost of each failure is the entire upload.

How resumable protocols work

The general shape, whatever the specific protocol:

  1. The client creates an upload and gets back a URL identifying it.
  2. The client sends bytes in chunks, each stating its offset.
  3. The server records how much it has durably committed.
  4. If interrupted, the client asks the server for the current offset and continues from there.

The tus protocol is the widely-implemented open standard for this. S3 multipart upload is the same idea at the storage layer. Both work; the important thing is that one of them is present.

The part that bites: chunk size is not a free choice

This is the failure that looks like a client bug and is actually a protocol mismatch.

The server can only tell you an offset it has durably committed. If the server is streaming your bytes into object storage as multipart parts, it can only commit at part boundaries — because a partial part is not a thing storage will acknowledge.

Now suppose the server's part size is 64 MB and your client sends 16 MB chunks. The client sends 16 MB and asks where it is. The server has not completed a part, so it honestly answers: offset 0. The client sends the same 16 MB again. Forever.

Nothing errors. Throughput looks fine. The upload never progresses.

The rule: the client's chunk size must be at least the server's part size. Providers that get this right either publish the number or make the client library derive it.

There is a second-order version of the same trap. Object storage limits the number of parts in a multipart upload — commonly around 10,000 — so a service handling very large files has to increase the part size as the file grows. Which means the correct chunk size is a function of file size, not a constant. Hardcode a constant and uploads work until someone sends a file above the threshold, at which point they hang.

What to test before you trust a provider

Five minutes, and it is worth more than any comparison table:

  1. Start uploading a file large enough to take a few minutes.
  2. Disable your network. Genuinely — turn off wifi, do not just close the tab.
  3. Wait thirty seconds.
  4. Reconnect.
  5. Watch what happens.

Three outcomes. It resumes near where it stopped: good. It restarts from zero: the advertised limit is not real for large files. It hangs at the same percentage: you have found the chunk-size mismatch above, and it will do that on every upload of that size.

Then repeat with the file closed and reopened, or from a fresh process. Resuming within one session is much easier than resuming after the client restarted, and only the second one saves you when a laptop sleeps.

Implementing the client side

Use an existing client. tus has implementations in most languages. Rolling your own means reimplementing offset negotiation, retry backoff and chunk sizing, and getting one of them subtly wrong.

Retry at the chunk level, not the job level. A job-level retry starts the whole upload again, which is what you were trying to avoid.

Store the upload URL somewhere durable if the client can restart. An upload you cannot address is an upload you cannot resume.

Back off on failures, and distinguish a network error (retry) from a 4xx (do not).

Do not compute a checksum by re-reading the whole file on every resume, unless you want to spend an hour reading 100 GB from disk to save re-sending 64 MB.

The determinism requirement, if you also encrypt

If the client encrypts before upload, resumption gets sharper teeth. To resume, the bytes you produce for a given region of the file must be identical to the ones you produced the first time.

That rules out random nonces generated per session. If a nonce prefix is regenerated on resume, the object becomes two incompatible halves — and it only shows up on a dropped upload of a large file, which is the hardest possible thing to reproduce deliberately.

Derive the nonce from something stable, like the file id and the master key, so a resumed upload produces byte-identical output.

Where this shows up in practice

The Yungle API uses tus for exactly this reason, and the CLI derives the chunk size from the file size rather than hardcoding it. The API reference is at /developers/uploads, and the API guide covers the surrounding pipeline.

Hein de Wilde

I build and run Yungle, and I write everything here. Comparisons name competitors and credit them, every claim about another company comes from that company’s own documentation, and where we fall short it says so. More about who is behind this.

Read next