For developers

Resumable uploads, and why they are hard

By Yungle·Updated ·5 min read

Key takeaways

  • A resumable protocol uploads in pieces and asks the server where it left off, so an interruption costs seconds rather than the whole file.
  • A chunk smaller than the server’s commit boundary makes the client re-send the same bytes forever, with no error anywhere.
  • Test any provider by cutting the network mid-upload, then again after restarting the client entirely.
Contents
  1. Why a plain upload fails at scale
  2. How resumable protocols work
  3. The part that bites: chunk size is not a free choice
  4. A five-minute test before you trust a provider
  5. Implementing the client side
  6. If you also encrypt: output must be deterministic
  7. Where this shows up

A 100 GB upload will be interrupted before it finishes, so the only question that matters is whether it carries on or starts again. Resumable protocols solve this by uploading in pieces and remembering the last offset the server has safely stored — and the detail that catches everyone out is that the client's chunk size and the server's commit boundary have to agree, or the client re-sends the same bytes forever.

Why a plain upload fails at scale

A single HTTP request carrying 100 GB has to survive for hours. In that window a laptop sleeps, a router restarts, a phone changes cell, or a proxy times out. Any of those kills the request, and with a plain upload there is nothing to resume from: the server holds bytes it cannot identify, and the client starts from zero.

At 1 GB that is a nuisance. At 100 GB it means the file cannot be sent at all, because the odds of an uninterrupted multi-hour connection are poor and each failure costs the whole upload.

How resumable protocols work

The general shape, whichever protocol you use:

  1. The client creates an upload and gets back a URL that identifies it.
  2. The client sends bytes, each request stating its offset.
  3. The server records how much it has durably committed.
  4. After an interruption, the client asks for the current offset and continues from there.

tus is the widely implemented open standard for this. S3 multipart upload is the same idea at the storage layer. Either works; what matters is that one of them is there. Here is where that fits in the life of a file on Yungle:

A file on Yungle: encrypted as it uploads, stored in Germany, scanned, and delivered only through Yungle — so a link can really expire.

The part that bites: chunk size is not a free choice

This failure looks like a client bug and is actually a protocol mismatch.

A server can only report an offset it has durably committed. If it streams your bytes into object storage as multipart parts, it can only commit at part boundaries — storage will not acknowledge half a part.

Now suppose the server's parts are 64 MB and your client sends 16 MB chunks. The client sends 16 MB and asks where it is. No part has completed, so the server honestly answers: offset 0. The client sends the same 16 MB again. Forever.

Nothing errors. Throughput looks healthy. The upload never moves.

The rule: a client's chunk must be at least the server's part size — or simply stream the whole file in one request and let the server commit parts as they complete, resuming from the last one if the connection drops. Providers that get this right either publish the number or make their client do the right thing by default.

There is a second trap behind the first. Object storage caps the number of parts in one upload — commonly around 10,000 — so a service that accepts very large files has to raise its part size as files grow. The correct chunk size is therefore a function of the file size, not a constant. Hardcode a constant and uploads work until someone sends a file above the threshold, at which point they hang.

A five-minute test before you trust a provider

Worth more than any comparison table:

  1. Start uploading a file big enough to take a few minutes.
  2. Turn the network off — properly, not by closing the tab.
  3. Wait thirty seconds.
  4. Reconnect.
  5. Watch.

Three outcomes. It resumes near where it stopped: good. It restarts from zero: the advertised limit is not real for large files. It hangs at the same percentage: you have found the chunk-size mismatch, and it will happen on every upload of that size.

Then repeat after restarting the client entirely. Resuming within one session is far easier than resuming after the process has died — and only the second saves you when a laptop sleeps.

Implementing the client side

Use an existing client. tus has maintained implementations in most languages. Writing your own means re-implementing offset negotiation, retry back-off and chunk sizing, and getting one subtly wrong.

Retry per request, not per job. A job-level retry restarts the whole upload, which is exactly what you were avoiding.

Store the upload URL somewhere durable if the client can restart. An upload you can no longer address is one you cannot resume. The Yungle CLI writes it to disk the moment the server assigns it, so killing the process and running the same command again picks up from the last committed part.

Back off on failure, and tell a network error (retry) from a 4xx (do not).

Do not checksum by re-reading the whole file on every resume, unless you enjoy spending an hour reading 100 GB from disk to avoid re-sending 64 MB.

If you also encrypt: output must be deterministic

If the client encrypts before uploading, resuming gets sharper teeth. The bytes you produce for a given region of the file must be identical to the ones you produced the first time.

That rules out random nonces generated per session. Regenerate a nonce prefix on resume and the stored object becomes two incompatible halves — something that only appears on a dropped upload of a large file, the hardest thing there is to reproduce on purpose. Derive the nonce from something stable, such as the file id and the key, so a resumed upload produces byte-identical output.

Where this shows up

The Yungle API uses tus for exactly these reasons.

The upload reference is at docs.yungle.co, the API guide covers the surrounding pipeline, and the API feature page shows how the CLI and MCP server fit around it.

Yungle

We build and run Yungle, and we write everything here. Comparisons name competitors and credit them, every claim about another company comes from that company’s own documentation, and where we fall short it says so. More about Yungle.

Read next