S3 Multipart Upload Optimization¶

ObjectFS includes comprehensive support for S3 multipart uploads with intelligent chunking, progress tracking, and resume capability.

Overview¶

Multipart uploads allow large files to be uploaded to S3 in chunks, enabling:

• Parallel uploads for improved throughput
• Resume capability for interrupted uploads
• Intelligent chunking based on file size
• Progress tracking and metrics
• Automatic optimization via CargoShip integration

Features¶

1. Configurable Thresholds¶

ObjectFS allows you to configure when multipart uploads are used and how they're chunked:

storage:
  s3:
    multipart:
      threshold: "32MB"   # objects larger than this are uploaded in parts
      chunk_size: "16MB"  # size of each part
      concurrency: 8      # parts uploaded at once

Sizes are strings with a unit, not byte counts. Leave a key out and the backend's own default applies; threshold: "32MB", chunk_size: "16MB", and concurrency: 8 are those defaults, so the block above changes nothing and exists to name the keys.

chunk_size has a floor rather than a range: S3 rejects any non-final part below 5 MB with EntityTooSmall, and a smaller value is raised to 5 MB rather than being passed through to fail at upload time.

2. Intelligent Chunking¶

ObjectFS automatically adjusts chunk sizes based on file size for optimal performance:

Example Usage¶

import "github.com/scttfrdmn/objectfs/internal/storage/s3"

cfg := s3.NewDefaultConfig()

// Check if a file should use multipart
fileSize := int64(100 * 1024 * 1024) // 100MB
if cfg.ShouldUseMultipart(fileSize) {
    // Get optimal chunk size for this file
    chunkSize := cfg.GetOptimalChunkSize(fileSize)
    // chunkSize will be 16MB for a 100MB file
}

3. Upload State Tracking¶

ObjectFS tracks the state of multipart uploads for monitoring and resume capability:

Multipart Upload States¶

• initiated - Upload has been started with S3
• in_progress - Parts are being uploaded
• completed - All parts uploaded successfully
• failed - Upload failed
• aborted - Upload was aborted

State Management¶

// Create a state manager
manager := s3.NewMultipartStateManager()

// Track a new upload
state := s3.NewMultipartUploadState(
    uploadID,
    bucket,
    key,
    totalSize,
    chunkSize,
)
manager.TrackUpload(state)

// Update part status as uploads complete
manager.UpdatePartStatus(uploadID, partNumber, size, etag, nil)

// Check progress
state, _ := manager.GetUploadState(uploadID)
progress := state.GetProgress() // Returns 0-100

Upload State Features¶

• Progress tracking: Real-time upload progress (0-100%)
• Part tracking: Individual part status with ETags
• Retry tracking: Number of retry attempts per part
• Error tracking: Last error for each failed part
• Resume support: Get list of remaining parts to upload

4. Metrics and Monitoring¶

ObjectFS collects comprehensive metrics on multipart uploads:

Multipart Metrics¶

mc := s3.NewMetricsCollector()

// Record multipart operations
mc.RecordMultipartUploadStart()
mc.RecordMultipartUploadPart(partSize int64)
mc.RecordMultipartUploadComplete(totalBytes, duration)
mc.RecordMultipartUploadFailed()

// Get metrics
metrics := mc.GetMetrics()

Available Metrics¶

Calculated Metrics¶

// Get usage rate (percentage of requests using multipart)
usageRate := mc.GetMultipartUsageRate()

// Get success rate
successRate := mc.GetMultipartSuccessRate()

// Get average parts per upload
avgParts := mc.GetAveragePartsPerUpload()

5. There is one upload path¶

use_cargoship selected a second PutObject implementation, routed through CargoShip's transporter for its BBR/CUBIC congestion control. It was removed in v0.15.0 (#362), and the loader decodes strictly, so a config still setting the key fails at startup naming it.

It was removed rather than tuned because it could not express what the direct path expresses. A cargoships3.Archive has no field for a Content-Encoding, for the configured encryption headers, or for a per-object storage class, and the upload path had already grown three bypasses saying so. The fourth was not bypassed but silently wrong: Content-Type was written into Archive.Metadata, which is S3 user metadata rather than the header, so every small object was stored as application/octet-stream.

Nothing was measured on the other side. The transporter was only reachable below multipart.threshold — an object at or above it returned into ObjectFS's own multipart path, which never consulted a transporter — so the 64 MiB multipart buffer it installed served an upload shape a mount cannot produce, and sync.Pool handed that buffer back at every GC cycle. Throughput against a local endpoint was 35% slower at 4 KiB, 8% slower at 1 MiB, and a wash at 8 MiB; the sizes where congestion control could plausibly help never reached it.

There is also no throughput target or optimization level to set. Earlier versions of this page showed target_throughput: 800.0 and optimization_level: "standard"; neither key ever existed in the schema, and strict decoding now fails a config containing them rather than ignoring it.

Configuration Examples¶

High-Performance Configuration¶

For environments with high bandwidth and large files:

storage:
  s3:
    multipart:
      threshold: "50MB"
      chunk_size: "32MB"
      concurrency: 16

Conservative Configuration¶

For environments with limited bandwidth or small files:

storage:
  s3:
    multipart:
      threshold: "100MB"  # higher threshold: fewer objects go multipart
      chunk_size: "8MB"   # smaller parts
      concurrency: 4      # fewer at once

Development/Testing Configuration¶

For local development against MinIO or another S3-compatible endpoint:

storage:
  s3:
    endpoint: "http://localhost:9000"
    force_path_style: true
    multipart:
      threshold: "10MB"
      chunk_size: "5MB"   # the S3 floor; anything smaller is raised to it
      concurrency: 2

ObjectFS's own tests do not use a container for this. internal/testaws runs a substrate endpoint in-process over real HTTP — no network, no credentials, no AWS account — which is both faster and closer to the real thing than a mock. See the Testing section of CONTRIBUTING.md.

Performance Considerations¶

Optimal Chunk Size Selection¶

The optimal chunk size depends on several factors:

1. Network Latency: Higher latency benefits from larger chunks
2. Bandwidth: Higher bandwidth can handle larger chunks
3. File Size: Larger files should use larger chunks to reduce part count
4. S3 Limits: S3 allows up to 10,000 parts per upload

S3 Multipart Limits¶

• Minimum part size: 5MB (except last part)
• Maximum part size: 5GB
• Maximum parts: 10,000
• Maximum object size: 5TB

ObjectFS automatically respects these limits with intelligent chunking.

Performance Tips¶

1. Use appropriate concurrency: Match your network capacity
2. 1 Gbps: 8-16 concurrent uploads

3. 10 Gbps: 16-32 concurrent uploads

4. Consider CargoShip optimization: routes uploads through CargoShip's chunking and part scheduling instead of this package's. Whether it is faster for your object sizes and network is a question for benchmarks/, not a figure this document can supply

5. Monitor metrics: Use the metrics API to identify bottlenecks

6. Tune chunk size: Larger chunks reduce overhead, but smaller chunks improve parallelization

Resume Capability¶

ObjectFS tracks upload state, enabling resume of interrupted uploads:

// Get remaining parts for an interrupted upload
state, exists := manager.GetUploadState(uploadID)
if exists && !state.IsComplete() {
    remaining := state.GetRemainingParts()
    for _, partNum := range remaining {
        // Resume upload for this part
    }
}

Cleanup Old Uploads¶

Clean up completed or failed uploads after a certain time:

// Remove uploads completed/failed more than 1 hour ago
removed := manager.CleanupOldUploads(1 * time.Hour)

API Reference¶

Configuration Functions¶

// Calculate optimal chunk size for a file
chunkSize := s3.CalculateOptimalChunkSize(fileSize, threshold, baseChunkSize)

// Calculate number of parts needed
partCount := s3.CalculatePartCount(fileSize, chunkSize)

// Check if multipart should be used
shouldUse := cfg.ShouldUseMultipart(fileSize)

// Get optimal chunk size from config
chunkSize := cfg.GetOptimalChunkSize(fileSize)

State Management Functions¶

// Create state manager
manager := s3.NewMultipartStateManager()

// Create upload state
state := s3.NewMultipartUploadState(uploadID, bucket, key, totalSize, chunkSize)

// Track upload
manager.TrackUpload(state)

// Update part status
manager.UpdatePartStatus(uploadID, partNum, size, etag, err)

// Mark upload complete/failed
manager.MarkUploadCompleted(uploadID)
manager.MarkUploadFailed(uploadID)

// Query uploads
state, exists := manager.GetUploadState(uploadID)
allUploads := manager.GetAllUploads()
inProgress := manager.GetInProgressUploads()

// Cleanup
removed := manager.CleanupOldUploads(maxAge)
manager.RemoveUpload(uploadID)

Metrics Functions¶

// Create metrics collector
mc := s3.NewMetricsCollector()

// Record operations
mc.RecordMultipartUploadStart()
mc.RecordMultipartUploadPart(size)
mc.RecordMultipartUploadComplete(totalBytes, duration)
mc.RecordMultipartUploadFailed()

// Get metrics
metrics := mc.GetMetrics()
usageRate := mc.GetMultipartUsageRate()
successRate := mc.GetMultipartSuccessRate()
avgParts := mc.GetAveragePartsPerUpload()

Testing¶

ObjectFS includes comprehensive tests for multipart functionality:

# Run multipart tests
go test ./internal/storage/s3/... -run TestMultipart

# Run with coverage
go test ./internal/storage/s3/... -cover -coverprofile=coverage.out

# Run benchmarks
go test ./internal/storage/s3/... -bench=BenchmarkMultipart

Troubleshooting¶

Common Issues¶

1. Parts too small: S3 requires minimum 5MB per part (except last)

2. Solution: Increase multipart_chunk_size to at least 5MB

3. Too many parts: S3 limits uploads to 10,000 parts

4. Solution: ObjectFS automatically uses larger chunks for large files

5. High latency: Small chunks increase overhead

6. Solution: Increase chunk size for high-latency connections

7. Low throughput: Not enough parallelization

8. Solution: Increase multipart_concurrency and pool_size

Debugging¶

Enable debug logging to see multipart upload details:

logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{
    Level: slog.LevelDebug,
}))

backend := s3.NewBackend(bucket, cfg, logger)

Future Enhancements¶

Planned improvements for multipart uploads:

• Adaptive chunking: Dynamically adjust chunk size based on network conditions
• Bandwidth throttling: Limit upload speed per multipart upload
• Persistent state: Save upload state to disk for recovery across restarts
• Progress callbacks: User-defined callbacks for upload progress
• Compression: Optional compression of parts before upload
• Encryption: Client-side encryption of parts

Related Documentation¶

• Configuration reference — every key the loader accepts, including the S3 block
• S3 Transfer Acceleration — the other half of the large-object path
• AWS S3 Multipart Upload Documentation

../configuration/s3.md, ./cargoship.md, and ./performance.md were linked here and none was written; docs/configuration/ does not exist as a directory.

See Also¶

• Transfer Acceleration support (automatic fallback to standard endpoints)
• Connection pooling for efficient resource usage
• Intelligent tiering for cost optimization
• Circuit breaker pattern for resilience