WhyS3UploadsDieat5GB(AndHowMultipartUploadsBetrayYou)

The 5GB Wall Nobody Talks About

If you've ever tried to upload a large file to Amazon S3 and watched it fail at almost exactly 5 gigabytes, you're not alone. I've seen this trip up senior engineers, confuse entire DevOps teams, and cause genuine production outages. The maddening part isn't that it fails — it's that it fails quietly, often with an error message that tells you almost nothing useful, like a generic EntityTooLarge or a vague network timeout that makes you blame your internet connection before you blame AWS.

Here's the core truth: S3 has a hard limit of 5GB for a single PUT object request. That's it. One shot, one upload, 5 gigabytes maximum. If your file is 5.1GB, 10GB, or 100GB, a standard upload will never work. Ever. The fix is something called multipart upload — a system where you split your file into chunks, upload each chunk separately, and then tell S3 to stitch them back together. Sounds elegant. In practice, it's a minefield of silent failures, abandoned uploads, and surprise AWS bills that show up weeks later.

This isn't a complaint about S3. It's a genuinely powerful service. But the multipart upload system has sharp edges that the documentation glosses over, and most tutorials skip straight to the happy path without warning you about what happens when things go wrong. Let me walk you through the whole picture.

What Multipart Upload Actually Is (And Why It Exists)

Think of multipart upload like moving furniture into a skyscraper. You can't fit a king-sized bed frame into an elevator in one piece. So you disassemble it, take each piece up separately, and reassemble it in the apartment. S3 works the same way. Your 50GB video file gets sliced into pieces — AWS calls them parts — and each part travels to S3 independently. Once all parts arrive, you send a final "complete" command and S3 reassembles the object.

The minimum part size is 5MB (except for the last part, which can be anything). The maximum is 5GB per part, and you can have up to 10,000 parts. That gives you a theoretical maximum object size of 5TB. The system exists because large files are inherently risky to send in one go. If a 20GB upload fails at 19.9GB on a single PUT request, you start completely over. With multipart, you only re-upload the failed chunk. For anyone building media platforms, data pipelines, or backup systems, this is critical.

Most AWS SDKs — the tools developers use to talk to AWS from their code — will automatically switch to multipart upload when your file crosses a certain threshold. In the JavaScript SDK v3, that threshold is 5MB by default for the Upload utility. In Python's boto3, the upload_file method kicks into multipart mode at 8MB. This automatic behavior is helpful right up until it isn't, because "automatic" also means "happening silently in the background" and that silence is where the problems live.

The Betrayal: What Actually Goes Wrong

Here's where I want to be brutally honest about what "multipart upload betrays you" actually means. The upload process has three phases: initiate, upload parts, and complete. If anything goes wrong after initiation but before completion — a network hiccup, a timeout, a crash in your application, a Lambda function hitting its 15-minute limit — you end up with what AWS calls an incomplete multipart upload. The parts are sitting on S3's servers. You're paying for that storage. And unless you explicitly clean them up, they stay there forever.

I've found S3 accounts that were accumulating hundreds of gigabytes of zombie multipart parts over months. The objects never existed — you'd never see them in your bucket listing — but the underlying storage costs were very real. It's like paying rent on a storage unit full of disassembled furniture that you'll never finish putting together. AWS doesn't automatically clean these up by default. You have to configure a lifecycle rule to handle it, and most people never do because nobody warns them to.

The second betrayal is more subtle. When you're uploading many files concurrently using multipart upload, part numbers can get mixed up if your concurrency logic isn't careful. Each part gets a number from 1 to 10,000, and S3 assembles the final object in part number order, not upload order. If you accidentally send part 3's data with part number 5's label, your reassembled file is silently corrupted. No error. The upload completes successfully. You only discover the corruption when someone tries to use the file.

The Code That Looks Fine But Isn't

Let me show you a common pattern that engineers write when they first discover multipart uploads in Python. This is the manual approach — not using boto3's built-in transfer manager — and it has a critical flaw.

import boto3
import os

s3_client = boto3.client('s3')
BUCKET = 'my-bucket'
KEY = 'large-file.zip'
FILE_PATH = '/data/large-file.zip'
PART_SIZE = 5 * 1024 * 1024  # 5MB per part

# Initiate the multipart upload
response = s3_client.create_multipart_upload(Bucket=BUCKET, Key=KEY)
upload_id = response['UploadId']

parts = []
part_number = 1

with open(FILE_PATH, 'rb') as f:
    while True:
        data = f.read(PART_SIZE)
        if not data:
            break
        
        # Upload each part
        part_response = s3_client.upload_part(
            Bucket=BUCKET,
            Key=KEY,
            PartNumber=part_number,
            UploadId=upload_id,
            Body=data
        )
        
        parts.append({
            'PartNumber': part_number,
            'ETag': part_response['ETag']
        })
        part_number += 1

# Complete the multipart upload
s3_client.complete_multipart_upload(
    Bucket=BUCKET,
    Key=KEY,
    UploadId=upload_id,
    MultipartUpload={'Parts': parts}
)

This code will work most of the time. That's what makes it dangerous. There's no error handling. If the upload_part call fails on part 47 out of 200, the exception will bubble up, your script will crash, and the upload stays in a permanently incomplete state. The upload_id is gone with your crashed process. Those 46 parts you already uploaded? They're sitting on S3, billing you, forever. The fix is to wrap the entire operation in a try/except block and call abort_multipart_upload when anything goes wrong.

import boto3

s3_client = boto3.client('s3')
BUCKET = 'my-bucket'
KEY = 'large-file.zip'
FILE_PATH = '/data/large-file.zip'
PART_SIZE = 5 * 1024 * 1024

response = s3_client.create_multipart_upload(Bucket=BUCKET, Key=KEY)
upload_id = response['UploadId']
parts = []

try:
    part_number = 1
    with open(FILE_PATH, 'rb') as f:
        while True:
            data = f.read(PART_SIZE)
            if not data:
                break
            
            part_response = s3_client.upload_part(
                Bucket=BUCKET,
                Key=KEY,
                PartNumber=part_number,
                UploadId=upload_id,
                Body=data
            )
            
            parts.append({
                'PartNumber': part_number,
                'ETag': part_response['ETag']
            })
            part_number += 1

    s3_client.complete_multipart_upload(
        Bucket=BUCKET,
        Key=KEY,
        UploadId=upload_id,
        MultipartUpload={'Parts': parts}
    )
    print(f"Upload complete: {KEY}")

except Exception as e:
    print(f"Upload failed: {e}. Aborting multipart upload...")
    s3_client.abort_multipart_upload(
        Bucket=BUCKET,
        Key=KEY,
        UploadId=upload_id
    )
    raise

This version properly cleans up after itself. The abort_multipart_upload call tells S3 to throw away all the parts associated with that upload ID. Your storage stays clean, your bill stays honest.

The Lifecycle Rule That Should Be Your First S3 Task

Even with perfect abort handling in your code, you should always configure an S3 lifecycle rule to expire incomplete multipart uploads. Network kills, server crashes, and out-of-memory errors don't wait for your try/except block. Hardware dies. Containers get killed. Lambda functions timeout mid-upload. A lifecycle rule is your safety net when your code can't clean up after itself.

Here's a Terraform snippet that adds this rule to an existing bucket, but you can do the same thing from the AWS Console or CLI:

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-bucket \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "CleanupIncompleteMultipartUploads",
        "Status": "Enabled",
        "Filter": {
          "Prefix": ""
        },
        "AbortIncompleteMultipartUpload": {
          "DaysAfterInitiation": 7
        }
      }
    ]
  }'

This tells S3: if a multipart upload has been sitting incomplete for more than 7 days, delete all its parts automatically. Seven days is a reasonable window — long enough for legitimate slow uploads to finish, short enough to keep your costs from ballooning. In my experience, I'd actually set this to 1-2 days for most applications and only increase it if you're dealing with extremely long-running upload jobs. The key insight is that this should be the first thing you configure on any bucket that accepts uploads. Before you write a single line of upload code.

Using the Transfer Manager Instead of Doing It Yourself

Here's the thing most tutorials skip: you almost never need to implement multipart upload manually. Both the Python boto3 library and the JavaScript AWS SDK v3 ship with a high-level transfer utility that handles chunking, concurrency, retries, and cleanup automatically. The boto3 upload_file method is backed by the TransferManager class, which will automatically choose between single-part and multipart uploads based on the file size.

import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "fs";

const client = new S3Client({ region: "us-east-1" });

const upload = new Upload({
  client,
  params: {
    Bucket: "my-bucket",
    Key: "large-video.mp4",
    Body: createReadStream("/data/large-video.mp4"),
    ContentType: "video/mp4",
  },
  // Upload 3 parts concurrently
  queueSize: 3,
  // Each part is 10MB
  partSize: 10 * 1024 * 1024,
  // Don't leave parts behind if we abort
  leavePartsOnError: false,
});

upload.on("httpUploadProgress", (progress) => {
  console.log(`Uploaded: ${progress.loaded} of ${progress.total} bytes`);
});

try {
  const result = await upload.done();
  console.log("Upload complete:", result.Location);
} catch (err) {
  console.error("Upload failed:", err);
  // The Upload class handles abort automatically when leavePartsOnError is false
}

Notice the leavePartsOnError: false option. This is critical. When set to false, the Upload class will call AbortMultipartUpload if the upload fails for any reason. When set to true (which is actually the default in some versions), it leaves the parts on S3 for potential resumption — but if you never actually resume, you're back to paying for ghost storage. Unless you have a genuine resumption strategy built into your application, set this to false.

The queueSize option controls how many parts upload simultaneously. Higher concurrency means faster uploads on high-bandwidth connections, but it also means more memory usage since each in-flight part is buffered in RAM. On a server with limited memory, setting queueSize to 10 with a partSize of 50MB means you could be holding 500MB in memory at once just for upload buffers. Tune these numbers for your actual environment.

When You're Behind a Proxy or Using a CDN

A whole separate class of 5GB failures comes from network infrastructure between your application and S3 that doesn't know there's a 5GB limit on single uploads. If you're routing S3 uploads through an API Gateway, a load balancer, or a reverse proxy like Nginx, that intermediary might have its own upload size limits — often much lower than 5GB — and it might also buffer the entire request body before forwarding it, turning your nicely chunked multipart upload into a single large request from S3's perspective.

API Gateway, for instance, has a hard payload limit of 10MB. If you're proxying S3 uploads through API Gateway, you will hit this wall fast, long before you even approach the 5GB limit. The standard solution here is to use presigned URLs — temporary, signed URLs that let your client upload directly to S3 without routing through your backend infrastructure. Your server generates the URL, hands it to the client, and the client uploads straight to S3. No proxy, no gateway limits, no bottleneck.

The caveat with presigned URLs and multipart upload is that you need to generate separate presigned URLs for each part, not just one URL for the whole operation. Your backend generates a multipart upload ID, generates presigned URLs for each part number, sends them all to the client, the client uploads each part using those URLs, and then notifies your backend to complete the multipart upload. It's more orchestration, but it completely sidesteps the proxy problem and scales beautifully for browser-based uploads of large files.

Auditing What's Already Stuck in Your Bucket

If you have a bucket that's been accepting uploads for a while without lifecycle rules, you probably have ghost parts accumulating right now. Here's how to find out, and how to clean them up:

# List all incomplete multipart uploads in a bucket
aws s3api list-multipart-uploads --bucket my-bucket

# If there are many, you can filter by prefix
aws s3api list-multipart-uploads \
  --bucket my-bucket \
  --prefix "uploads/"

# Abort a specific incomplete upload
aws s3api abort-multipart-upload \
  --bucket my-bucket \
  --key "path/to/object.zip" \
  --upload-id "your-upload-id-here"

The output of list-multipart-uploads will show you the object key, the upload ID, the date the upload was initiated, and the storage class. If you see uploads that were initiated weeks or months ago, those are almost certainly orphans. You can write a script to loop through all of them and call abort-multipart-upload on each one, or you can add the lifecycle rule and let S3 handle the cleanup automatically over the next few days.

Running this audit is often eye-opening. I've seen teams discover that 30-40% of their S3 storage costs were coming from incomplete multipart uploads — not from actual objects anyone could access or use. It's the kind of thing that hides in plain sight because the AWS Cost Explorer shows you "S3 storage" as one line item and doesn't break out multipart parts from real objects.

The Real Lesson Here

The 5GB wall isn't the real problem. The real problem is that multipart upload is a stateful, three-phase protocol masquerading as a simple file upload. Most upload APIs are stateless — you send data, the server either accepts it or rejects it, and you're done. Multipart upload is different. State accumulates on S3's side from the moment you call CreateMultipartUpload, and that state persists until you either complete it or abort it. If your application crashes, loses its connection, or simply forgets about an in-progress upload, that state remains. The bill runs.

Understanding this changes how you architect around S3. You think about abort handlers, lifecycle rules, and upload auditing as first-class concerns, not afterthoughts. You choose the SDK's transfer manager over rolling your own unless you have a specific reason to control the low-level behavior. You configure lifecycle rules before writing your first upload handler, not after a finance team asks why the S3 bill is mysteriously climbing.

The five-gigabyte limit is just the trigger. Multipart upload is the mechanism. How you handle that mechanism's failure modes is what separates a production-ready file handling system from one that slowly, silently accumulates storage debt in the background. Get the lifecycle rules in place, use the transfer manager utilities the SDKs provide, and when you do go low-level, make sure your abort logic is as carefully written as your upload logic. That's the whole game.