Large file uploads in the browser

Uploading a large file from the browser means chunking it, retrying per chunk, and using a protocol that can resume — but the bugs that actually ship are not in the upload code. They are in the folder handling: browsers hand a dropped directory over in pages of 100 entries, and the obvious implementation silently truncates any folder with more files than that.

The upload part, briefly

Use a resumable protocol rather than a single POST. fetch with a 40 GB body has to survive for hours, and it will not.

The shape: create an upload, send chunks with offsets, ask the server for the current offset after an interruption, continue. tus has a maintained browser client and is the least work. Why chunk size is not a free choice is the one piece of protocol detail worth reading before you pick a number — if your chunk is smaller than the server's commit boundary, the client re-sends the same bytes forever without erroring.

Three browser-specific things:

Do not read the whole file into memory. File is already a Blob; use file.slice(start, end) to get each chunk and let the browser stream it. Reading a 40 GB file with FileReader will end badly.

Keep the tab alive. Backgrounded tabs get throttled and mobile browsers suspend them. There is no reliable fix; tell the user to keep the tab open, because the alternative is silent stalling they blame on you.

Cap concurrency. Two to four parallel chunks saturates most connections. Six streams from a second drop while the first is still running does not go faster, it just makes everything slower and harder to reason about.

The folder traps

This is where the real bugs are, and all three are silent.

readEntries is paged at 100

When a user drops a directory, you traverse it with the FileSystemDirectoryReader API. The obvious reading of the documentation is:

// WRONG — returns at most 100 entries
const reader = dirEntry.createReader()
reader.readEntries((entries) => {
  for (const entry of entries) handle(entry)
})

readEntries returns at most 100 entries per call, and you must keep calling it until it returns an empty array. Call it once and a folder of 400 photographs uploads 100 of them and reports success.

// RIGHT — drain until empty
async function readAll(dirEntry) {
  const reader = dirEntry.createReader()
  const out = []
  for (;;) {
    const batch = await new Promise((res, rej) => reader.readEntries(res, rej))
    if (batch.length === 0) break
    out.push(...batch)
  }
  return out
}

Nothing errors in the wrong version. The user sees a successful upload with three quarters of their shoot missing.

webkitGetAsEntry only works during the drop event

DataTransferItem.webkitGetAsEntry() is valid only while the drop event is being dispatched. The moment you await anything, the DataTransfer items are invalidated and you get null.

So you must drain the items into an array synchronously first, then do the async traversal:

function onDrop(e) {
  e.preventDefault()
  // Synchronous: grab every entry before yielding.
  const entries = [...e.dataTransfer.items]
    .map((item) => item.webkitGetAsEntry())
    .filter(Boolean)
  // Now it is safe to go async.
  void traverse(entries)
}

Get this wrong and it works for a single file and fails for a folder, which is a confusing bug report to receive.

Fingerprinting must include the path

If you deduplicate staged files by name, size and modification time, two copies of IMG_001.jpg from different subdirectories collapse into one entry. A shoot with per-day folders loses most of itself.

Include the relative path in whatever identity you compute.

Related: match created server rows to local files by position, never by name plus size. Exports collide constantly, and a mismatch means the wrong bytes go into the wrong row.

Hidden files

A folder that has been opened on a Mac has a .DS_Store in every subdirectory; Windows leaves Thumbs.db. Skip dotfiles when walking a directory, but keep them if the user picked one deliberately — someone uploading a .env on purpose should get it.

Progress that is not a lie

Two rules:

Report bytes, not files. A 40 GB file and a 2 KB file are not each "one of two".

Do not update state on every progress event. A setState per chunk per file will make the browser unusable. Throttle to a few times a second, and write directly to a DOM node via a ref if you are animating anything.

Aborts

Give the user a cancel button and make it actually abort the in-flight requests via AbortController. Then tell the server, if the protocol supports it, so the partial upload is cleaned up rather than counting against a quota until it expires.

Testing what matters

1. A folder with more than 100 files. Count what arrives. This is the whole reason for this article.
2. A folder with subdirectories, and check structure survived.
3. Two files with the same name in different folders.
4. A deliberate disconnect mid-upload, then reconnect.
5. A file large enough to cross whatever part-size threshold your provider has — resumability can work at 1 GB and hang at 300 GB for the part-count reason.

If you would rather not implement any of this, the API guide covers using a service that has already done it, and /developers/uploads is the reference for the tus endpoints.