Buffered Read: Accelerate Large Sequential Reads

In this release, we are introducing the Buffered Read feature, designed to accelerate performance of applications that perform large, sequential reads on files stored in Google Cloud Storage. This is helpful for reading model weights during inference or large scale media processing applications that read data sequentially.

This feature improves throughput (2-5x) by asynchronously prefetching parts of a GCS object parallelly into an in-memory buffer, serving subsequent reads from this buffer instead of making network calls. This asynchronous and parallel buffering approach improves throughput by saturating network bandwidth without requiring additional application-side parallelism.

• Feature Enablement: This feature is disabled by default and can be enabled with --enable-buffered-read flag or read:enable-buffered-read config. Buffered reads are ignored if file cache is enabled. Over time, we will work towards getting this enabled by default.
• Use Cases: Single-threaded applications reading large (> 100MB) files sequentially. E.g. For reading models during inference (prediction) of an AI/ML model.
• Memory usage:
  • Buffered readers will use CPU memory for storing the prefetched blocks.
  • Memory usage is capped at 320 MB per file-handle and 640 MB (40 x 16MB) globally. The global memory limit is configurable via the --read-global-max-blocks (default: 40).
  • This memory is automatically released when a file handle is closed or a random read access pattern is detected.
• CPU Usage: The CPU overhead is typically proportional to the performance gain achieved.
• Known Limitations:
  • Workload combining sequential and random reads (which also includes some model serving techniques such as Pipeline parallelism that do random reads at the start before switching to large sequential reads) may not benefit and could automatically fallback to default reads. We plan to improve buffered read for such scenarios in the future releases. Please reach out to us for improving the performance in such scenarios.
  • Please consider available system memory when enabling buffered reads to avoid out of memory(OOM) issues as we can go up to 640MB of memory usage by default. Please reduce --read-global-max-blocks(default:40) to avoid Out of Memory(OOM) issues.

Bug Fixes

• Resolved sporadic mount failures and enhanced stability by improving the retry mechanism for stalled API calls to the backend. (#3561, #3684)
• Fix for GCSFuse not returning errors in unmount of bucket (e.g. resource busy error) when mounted with gcsfuse (#3768) introduced in 3.2.0.

Improvements

• Streaming Writes now support retrying stalled write operations.
• Improved stability for writes(#3710)
• Logging/monitoring improvements
  1. Efficient log collection: Metrics are more efficient now with CPU usage reduced from 15% to <2% for small single threaded reads and memory allocation is reduced from 34% to 0%.
  2. Mount logs display block size used in streaming writes in MiB instead of bytes for improved readability.
  3. Added error log for unsupported values of flag: log-format(#3751)
  4. Made logs efficient by downgrading the logging level of some unimportant logs to trace instead of higher logging level. (#3746. #3749)

Dependency Upgrades / CVE fixes

• Dependency upgrades (#3740)

Full Changelog: v3.2.0...v3.3.0