Skip to content

CargoShip S3 Optimization Modularization Request

A request sent to another project in 2025, kept as a record. Not a description of either codebase. What was asked for and what exists have diverged, in both directions:

  • The proposed pkg/s3optimization/ tree below — network/, adaptive/, connection/, performance/ — was not built to this shape. CargoShip does now export pkg/s3optimization, but it holds access-pattern analysis, a request predictor, a predictive prefetcher, and an adaptive scheduler. Different components, same name.
  • ObjectFS imports none of it. What it actually consumes is pkg/aws/s3 (the upload transporter) and pkg/aws/config, imported directly — no shared-module layer, and the S3Optimizer / OptimizedS3Client types sketched below do not exist in either repository.
  • The BBR/CUBIC congestion control this asks CargoShip to extract, ObjectFS ended up writing itself in internal/network: a per-socket TCP_CONGESTION option on Linux, ignored on macOS.

The "4.6x performance improvement" this document repeated six times was CargoShip's figure for CargoShip's workload. It is removed from the body below, because restating it here is how it came to be quoted throughout ObjectFS's own documentation as though ObjectFS had measured it.

Overview

To enable integration between ObjectFS and CargoShip, extract CargoShip's S3 optimization components into reusable modules, eliminating the code both projects would otherwise each write.

Current CargoShip S3 Architecture

Based on analysis of /pkg/aws/s3/, CargoShip contains extensive S3 optimization components:

High-Value Components for ObjectFS Integration

1. Network Optimization Algorithms

// Congestion control and RTT estimation
pkg/aws/s3/bbr_bandwidth_probing.go        // Google's BBR algorithm
pkg/aws/s3/cubic_congestion_control.go     // Linux CUBIC implementation  
pkg/aws/s3/rtt_estimation_system.go        // Multi-algorithm RTT estimation
pkg/aws/s3/loss_detection_recovery.go      // Advanced loss detection
pkg/aws/s3/bandwidth_delay_product.go      // Dynamic BDP calculation

2. Adaptive Transfer Management

pkg/aws/s3/adaptive_transporter.go         // Adaptive upload strategies
pkg/aws/s3/realtime_parameter_optimizer.go // Real-time optimization
pkg/aws/s3/realtime_network_monitor.go     // Network condition monitoring
pkg/aws/s3/dynamic_parameter_adjuster.go   // Parameter adjustment

3. Connection and Memory Management

pkg/aws/s3/coordinator.go                  // Connection coordination
pkg/aws/s3/loadbalancer.go                 // Load balancing
pkg/aws/s3/memory_buffer.go                // Buffer management
pkg/aws/s3/parallel.go                     // Parallel operations

4. Performance Intelligence

pkg/aws/s3/pipeline_optimizer.go           // Pipeline optimization
pkg/aws/s3/predictive_adaptation_engine.go // Predictive adaptation
pkg/aws/s3/streaming_compressor.go         // Streaming compression

Proposed Modularization Strategy

Phase 1: Extract Core Network Algorithms

Create pkg/s3optimization/ Module Structure

pkg/s3optimization/
├── network/
│   ├── bbr.go                    // BBR bandwidth probing
│   ├── cubic.go                  // CUBIC congestion control
│   ├── rtt.go                    // RTT estimation system
│   ├── loss.go                   // Loss detection and recovery
│   └── bdp.go                    // Bandwidth-delay product
├── adaptive/
│   ├── transporter.go            // Adaptive transport logic
│   ├── monitor.go                // Real-time network monitoring
│   ├── optimizer.go              // Parameter optimization
│   └── predictor.go              // Predictive adaptation
├── connection/
│   ├── pool.go                   // Connection pooling
│   ├── coordinator.go            // Multi-connection coordination
│   ├── loadbalancer.go           // Load balancing
│   └── health.go                 // Health monitoring
└── performance/
    ├── pipeline.go               // Pipeline optimization
    ├── buffer.go                 // Memory buffer management
    ├── compression.go            // Streaming compression
    └── metrics.go                // Performance metrics

Phase 2: Create Shared Interface

Common S3 Optimization Interface

// pkg/s3optimization/optimizer.go
package s3optimization

import (
    "context"
    "io"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

type S3Optimizer struct {
    // Network optimization components
    bbrProber     *network.BBRBandwidthProber
    cubicControl  *network.CubicCongestionControl
    rttEstimator  *network.RTTEstimationSystem
    lossDetector  *network.LossDetectionRecovery
    bdpCalculator *network.BandwidthDelayProduct

    // Adaptive components
    transporter   *adaptive.Transporter
    monitor       *adaptive.RealtimeNetworkMonitor
    optimizer     *adaptive.RealtimeParameterOptimizer

    // Connection management
    pool          *connection.Pool
    coordinator   *connection.Coordinator
    loadBalancer  *connection.LoadBalancer
}

type OptimizedS3Client interface {
    GetObjectOptimized(ctx context.Context, input *s3.GetObjectInput) (*s3.GetObjectOutput, error)
    PutObjectOptimized(ctx context.Context, input *s3.PutObjectInput) (*s3.PutObjectOutput, error)
    GetMetrics() PerformanceMetrics
}

// Usage in ObjectFS
func (b *Backend) GetObject(ctx context.Context, key string, offset, size int64) ([]byte, error) {
    // Use optimized S3 client with BBR/CUBIC algorithms
    result, err := b.optimizedClient.GetObjectOptimized(ctx, input)
    // ... existing logic
}

Phase 3: Backward Compatibility

Maintain CargoShip Functionality

// Existing CargoShip code continues to work
// pkg/aws/s3/transporter.go becomes a wrapper
type Transporter struct {
    optimizer *s3optimization.S3Optimizer // Use shared optimizer
    // ... existing fields
}

func (t *Transporter) Upload(ctx context.Context, archive *Archive) (*UploadResult, error) {
    // Delegate to shared optimizer while maintaining CargoShip-specific logic
    return t.optimizer.PutObjectOptimized(ctx, convertToS3Input(archive))
}

Implementation Benefits

1. Code Reuse

  • Eliminate Duplication: one implementation of the congestion and transfer logic, not two
  • Shared Maintenance: Bug fixes and improvements benefit both projects
  • Consistent Behavior: Same optimization algorithms across platform

2. Development Efficiency

  • Faster ObjectFS Development: Leverage existing CargoShip optimizations
  • Unified Testing: Shared test suites for optimization components
  • Coordinated Evolution: Algorithm improvements deployed to both projects

3. Performance Guarantees

  • Proven Algorithms: BBR/CUBIC already tested and validated in CargoShip
  • Consistent Metrics: Same performance measurement across platform
  • Regression Prevention: Shared optimization ensures no performance loss

Migration Strategy

Step 1: CargoShip Refactoring

  1. Extract network algorithms into pkg/s3optimization/network/
  2. Create shared interfaces for S3 optimization
  3. Maintain backward compatibility with existing CargoShip APIs
  4. Add comprehensive tests for extracted modules

Step 2: ObjectFS Integration

  1. Add dependency on CargoShip's pkg/s3optimization
  2. Replace basic S3 client with optimized version
  3. Integrate BBR/CUBIC algorithms into ObjectFS transfers
  4. Validate performance with a benchmark naming bucket, region, and object size

Step 3: Platform Unification

  1. Shared metrics collection across both projects
  2. Unified monitoring dashboards for platform performance
  3. Coordinated optimization improvements deployed to both projects

Specific Extraction Priorities

High Priority (Phase 1 - Q4 2025)

  1. BBR Bandwidth Probing - Core performance algorithm
  2. CUBIC Congestion Control - Proven network optimization
  3. Connection Pooling - Essential for ObjectFS scaling
  4. Real-time Monitoring - Required for adaptive behavior

Medium Priority (Phase 2 - Q1 2026)

  1. RTT Estimation System - Enhanced network intelligence
  2. Loss Detection Recovery - Robust error handling
  3. Adaptive Transport Logic - Dynamic optimization
  4. Pipeline Optimization - Advanced performance tuning

Future Enhancements (Phase 3 - Q3 2026)

  1. Predictive Adaptation - ML-powered optimization
  2. Advanced Load Balancing - Multi-region coordination
  3. Streaming Compression - Real-time data optimization

Expected Outcomes

ObjectFS Benefits

  • Transfer optimization: inherit CargoShip's congestion control and part scheduling instead of reimplementing them
  • Reduced Development Time: 60% less implementation effort
  • Production-Ready Algorithms: Battle-tested network optimization
  • Consistent User Experience: Same performance across platform

CargoShip Benefits

  • Cleaner Architecture: Better separation of concerns
  • Reusable Components: Optimizations available to other projects
  • Enhanced Testing: Broader test coverage through dual usage
  • Strategic Positioning: Core technology becomes platform foundation

Platform Benefits

  • Unified Performance Stack: Shared optimization across all components
  • Competitive Advantage: Best-in-class S3 performance optimization
  • Faster Innovation: Coordinated algorithm development
  • Market Differentiation: Integrated platform vs point solutions

Conclusion

Modularizing CargoShip's S3 optimization components is essential for the unified ObjectFS + CargoShip platform strategy. This approach shares one implementation of the transfer optimizations instead of two, and lets improvements to them reach both projects.

The extracted modules will serve as the foundation for ObjectFS's high-performance S3 integration while maintaining CargoShip's existing functionality and providing a clear path for future platform unification.