Large file uploads in the browser

Uploading a large file from the browser means sending it in pieces, retrying per piece, and using a protocol that can resume — but the bugs that actually ship are rarely in the upload code. They are in the folder handling: browsers hand over a dropped directory 100 entries at a time, and the obvious implementation silently drops everything past the first hundred.

The upload, briefly

Use a resumable protocol rather than one big POST. A fetch with a 40 GB body has to survive for hours, and it will not.

The shape: create an upload, send chunks with offsets, ask the server for the current offset after an interruption, continue. tus has a maintained browser client and is the least work. Why chunk size is not a free choice is the one piece of protocol detail worth reading first — a chunk smaller than the server's commit boundary makes the client re-send the same bytes forever, without an error.

Three things specific to the browser:

Never read the whole file into memory. A File is already a Blob. Hand it (or file.slice(start, end)) to the upload and let the browser stream it. Reading 40 GB with FileReader ends badly.

Keep the tab alive. Background tabs are throttled and mobile browsers suspend them. There is no reliable fix — tell people to keep the tab open, because the alternative is a silent stall they will blame on you.

Cap concurrency. Two to four parallel uploads saturate most connections. More does not go faster; it just makes everything slower and harder to reason about.

The folder traps

This is where the real bugs are, and all three are silent.

readEntries returns at most 100 at a time

When someone drops a directory, you walk it with FileSystemDirectoryReader. The obvious reading of the documentation:

// WRONG — returns at most 100 entries
const reader = dirEntry.createReader()
reader.readEntries((entries) => {
  for (const entry of entries) handle(entry)
})

readEntries returns at most 100 entries per call, and you must keep calling it until it returns an empty array. Call it once, and a folder of 400 photographs uploads 100 of them and reports success.

// RIGHT — drain until empty
async function readAll(dirEntry) {
  const reader = dirEntry.createReader()
  const out = []
  for (;;) {
    const batch = await new Promise((res, rej) => reader.readEntries(res, rej))
    if (batch.length === 0) break
    out.push(...batch)
  }
  return out
}

webkitGetAsEntry only works during the drop event

DataTransferItem.webkitGetAsEntry() is valid only while the drop event is being dispatched. The moment you await anything, the items are invalidated and you get null. So collect every entry synchronously first, then go async:

function onDrop(e) {
  e.preventDefault()
  // Synchronous: grab every entry before yielding.
  const entries = [...e.dataTransfer.items]
    .map((item) => item.webkitGetAsEntry())
    .filter(Boolean)
  // Now it is safe to go async.
  void traverse(entries)
}

Get this wrong and single files work while folders fail — a confusing bug report to receive.

Identity must include the path

If you de-duplicate staged files by name, size and modified time, two IMG_001.jpg from different subfolders collapse into one. A shoot organised in per-day folders loses most of itself. Include the relative path in whatever identity you compute.

Related: match rows the server created to your local files by position, never by name plus size. Exports collide constantly, and a mismatch puts the wrong bytes in the wrong row.

Hidden files

A folder opened on a Mac has a .DS_Store in every subfolder; Windows leaves Thumbs.db. Skip dotfiles when walking a directory — but keep them when someone picked one deliberately. A person uploading a .env on purpose should get it.

Progress that is not a lie

• Report bytes, not files. A 40 GB file and a 2 KB file are not each "one of two".
• Do not update state on every progress event. A state update per chunk per file makes the page unusable. Throttle to a few times a second, and write to the DOM through a ref if you animate anything.

Cancelling

Give people a cancel button and make it really abort the in-flight requests, via AbortController. Then tell the server, if the protocol allows, so the partial upload is cleaned up instead of counting against a quota until it expires.

What to test

1. A folder with more than 100 files. Count what arrives. It is the whole reason for this article.
2. A folder with subfolders — and check the structure survived.
3. Two files with the same name in different folders.
4. A deliberate disconnect mid-upload, then reconnect.
5. A file big enough to cross your provider's part-size threshold. Resuming can work at 1 GB and hang at 300 GB, for the part-count reason.

If you would rather not build any of this: the Yungle uploader was built around exactly these traps, with tests for the paged folder read, and the same tus endpoint is open to your own code:

The API guide covers the full request sequence, and the upload guide is the reference for the tus endpoint.