Mastering File Uploads in S3 with NestJS: Normal and Multipart Uploads

Introduction

File uploads are at the heart of many modern applications, enabling users to upload images, videos, documents, and more. Implementing file uploads effectively involves understanding their types, challenges, and solutions for scalability.

This guide delves into:

Normal file uploads to Amazon S3.
Multipart uploads for large files using NestJS.
Advanced features like progress tracking, resumable uploads, and cancellation.
Code examples and best practices.

By the end, you'll master file upload techniques and be equipped to handle everything from profile pictures to video uploads.

Types of File Uploads

1. Normal File Upload

Single file is uploaded in one request.
Ideal for small files (e.g., <100MB).

2. Multipart File Upload

Large files are split into chunks (parts), uploaded individually, and combined on the server.
Suitable for large files (e.g., >100MB), especially on unstable networks.

3. Streaming Upload

Files are streamed directly from client to server without fully loading into memory.
Useful for real-time video/audio recording and streaming scenarios.

Why Use Amazon S3 for File Uploads?

Amazon S3 (Simple Storage Service) is a highly scalable, secure, and cost-effective storage solution for file uploads.

Key Benefits

Scalability: Handles growing storage needs seamlessly.
Durability: Offers 99.999999999% durability.
Security: Built-in encryption and access control.
Integration: Works with AWS services like Lambda and CloudFront.

Normal File Upload to S3

Backend: NestJS Service Implementation

import { Injectable, InternalServerErrorException } from '@nestjs/common';
import * as AWS from 'aws-sdk';
import * as path from 'path';

@Injectable()
export class UploadService {
  private s3 = new AWS.S3({
    accessKeyId: process.env.AWS_ACCESS_KEY,
    secretAccessKey: process.env.AWS_SECRET_KEY,
    region: process.env.AWS_REGION,
  });
  private bucketName = process.env.S3_BUCKET_NAME;

  async uploadFile(file: Express.Multer.File) {
    const fileExtension = path.extname(file.originalname);
    const fileName = `${path.basename(file.originalname, fileExtension)}-${Date.now()}${fileExtension}`;
    const params = {
      Bucket: this.bucketName,
      Key: fileName,
      Body: file.buffer,
      ContentType: file.mimetype,
      ACL: 'public-read',
    };

    try {
      const s3Response = await this.s3.upload(params).promise();
      return {
        url: s3Response.Location,
        key: s3Response.Key,
      };
    } catch (error) {
      throw new InternalServerErrorException('Error uploading file');
    }
  }
}

Controller for Single File Upload

import { Controller, Post, UploadedFile, UseInterceptors } from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { UploadService } from './upload.service';

@Controller('upload')
export class UploadController {
  constructor(private readonly uploadService: UploadService) {}

  @Post('single')
  @UseInterceptors(FileInterceptor('file'))
  async uploadSingleFile(@UploadedFile() file: Express.Multer.File) {
    return this.uploadService.uploadFile(file);
  }
}

Frontend File Upload Example

export const uploadFile = async (file: File) => {
  const formData = new FormData();
  formData.append('file', file);

  const response = await fetch('/upload/single', {
    method: 'POST',
    body: formData,
  });

  return response.json();
};

Multipart Upload to S3

Why Multipart Upload?

Reliable for large files.
Allows resuming interrupted uploads.
Parallel uploads improve speed.

Step 1: Initiating Multipart Upload

async initiateMultipartUpload(fileName: string) {
  const params = {
    Bucket: this.bucketName,
    Key: fileName,
  };
  const { UploadId } = await this.s3.createMultipartUpload(params).promise();
  return UploadId;
}

Step 2: Uploading File Parts

async uploadPart(uploadId: string, partNumber: number, partBuffer: Buffer, fileName: string) {
  const params = {
    Bucket: this.bucketName,
    Key: fileName,
    UploadId: uploadId,
    PartNumber: partNumber,
    Body: partBuffer,
  };
  const { ETag } = await this.s3.uploadPart(params).promise();
  return { ETag, PartNumber: partNumber };
}

Step 3: Completing Multipart Upload

async completeMultipartUpload(uploadId: string, fileName: string, parts: { ETag: string, PartNumber: number }[]) {
  const params = {
    Bucket: this.bucketName,
    Key: fileName,
    UploadId: uploadId,
    MultipartUpload: { Parts: parts },
  };
  await this.s3.completeMultipartUpload(params).promise();
}

Controller for Multipart Upload

import { Controller, Post, Body } from '@nestjs/common';
import { UploadService } from './upload.service';

@Controller('multipart')
export class MultipartUploadController {
  constructor(private readonly uploadService: UploadService) {}

  @Post('initiate')
  async initiate(@Body('fileName') fileName: string) {
    return this.uploadService.initiateMultipartUpload(fileName);
  }

  @Post('upload-part')
  async uploadPart(@Body() { uploadId, partNumber, partData, fileName }) {
    const partBuffer = Buffer.from(partData, 'base64');
    return this.uploadService.uploadPart(uploadId, partNumber, partBuffer, fileName);
  }

  @Post('complete')
  async complete(@Body() { uploadId, fileName, parts }) {
    return this.uploadService.completeMultipartUpload(uploadId, fileName, parts);
  }
}

Client-Side Implementation

Upload with Progress

export const uploadFileWithProgress = async (file: File, progressCallback: (percentage: number) => void) => {
  const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB
  const totalChunks = Math.ceil(file.size / CHUNK_SIZE);
  const fileName = file.name;

  // Step 1: Initiate upload
  const { uploadId } = await fetch('/multipart/initiate', {
    method: 'POST',
    body: JSON.stringify({ fileName }),
  }).then((res) => res.json());

  const parts = [];
  for (let partNumber = 1; partNumber <= totalChunks; partNumber++) {
    const start = (partNumber - 1) * CHUNK_SIZE;
    const end = Math.min(start + CHUNK_SIZE, file.size);
    const partBuffer = await file.slice(start, end).arrayBuffer();

    // Step 2: Upload part
    const { ETag } = await fetch('/multipart/upload-part', {
      method: 'POST',
      body: JSON.stringify({
        uploadId,
        partNumber,
        partData: btoa(String.fromCharCode(...new Uint8Array(partBuffer))),
        fileName,
      }),
    }).then((res) => res.json());

    parts.push({ ETag, PartNumber: partNumber });
    progressCallback((partNumber / totalChunks) * 100);
  }

  // Step 3: Complete upload
  return fetch('/multipart/complete', {
    method: 'POST',
    body: JSON.stringify({ uploadId, fileName, parts }),
  }).then((res) => res.json());
};

Additional Features

1. Resumable Uploads

Use multipart upload IDs to resume incomplete uploads.

2. Cancelling Uploads

Use S3's abortMultipartUpload API to cancel incomplete uploads.

3. Streaming Uploads

Implement streams for real-time uploads (e.g., video/audio streaming).

Additional Tools for Advanced File Uploads

When building sophisticated file upload systems, leveraging the right tools can drastically improve performance, reliability, and user experience. Here are two excellent resources:

1. tus.io: Resumable File Upload Protocol

What It Is:

tus.io is an open protocol for resumable file uploads designed to overcome challenges like unstable networks and large file transfers. It ensures uploads can be paused and resumed, even after network interruptions.

Why Use tus.io?

Resumable Uploads: Upload progress persists, preventing restarts on failure.
Large File Handling: Handles multi-GB file uploads seamlessly.
Cross-Platform Support: Works on web, mobile, and desktop clients.

If your application requires reliability in file uploads over unstable connections, tus.io is a go-to solution.

2. Uppy.io: File Uploader for Modern Applications