
Choosing where to store training data and model checkpoints on Google Cloud comes down to how your training jobs access files, how large the cluster is, and where the data already resides. For most TensorFlow and machine learning workflows, Cloud Storage is the default foundation. When a workload requires shared low-latency file access, high-throughput parallel reads and checkpointing at large cluster scale, direct batch reads from warehouse tables, or built-in dataset management, the right choice shifts to Filestore, Managed Lustre, BigQuery, or Gemini Enterprise Agent Platform (formerly Vertex AI) managed datasets. On the Professional Machine Learning Engineer exam, storage questions usually test whether you can match each of these services to its specific access pattern and scale.
Cloud Storage is the primary storage choice for most TensorFlow workflows on Google Cloud. It is well suited for holding training data such as images, TFRecords, and CSV files, as well as saved models for deployment and versioning. Compared with managed file systems, Cloud Storage is less expensive and effectively unlimited in capacity, which makes it practical for large training datasets.
TensorFlow reads directly from Cloud Storage using tf.data.TFRecordDataset with gs:// URLs. Cloud Storage also supports streaming reads, which matters whenever a dataset is too large to fit in memory and must be read continuously during training.
Exam scenarios sometimes present HDFS alongside Cloud Storage for TensorFlow training. HDFS is a distributed file system commonly used on-premises within Hadoop ecosystems. Although TensorFlow can technically work with HDFS, adopting it in a cloud-native architecture adds unnecessary complexity when Cloud Storage is available. Cloud Storage delivers distributed, scalable storage without the operational overhead of running and maintaining a Hadoop cluster.
Filestore is a fully managed file storage service on Google Cloud, so teams do not provision disks, format volumes, or patch a file server. Its defining behavior is concurrent shared access: multiple VM instances can mount the same file share and read and write to the same file system at the same time. When one instance writes a file, that file is immediately visible to the other instances.
That concurrency relies on NFS, the Network File System protocol. A client mounts the Filestore share into its own directory tree and treats it like a local disk. Application code opens file paths and seeks into files normally while the operating system translates those operations into network calls behind the scenes.
This behavior fits distributed training jobs that need shared storage. In a multi-worker training job where every worker reads the same training data and writes checkpoints to the same place, all workers mount the same Filestore share and work against a single consistent copy. Filestore is also a better choice than Cloud Storage when a training loop reads files continuously and needs fast, low-latency file access. Cloud Storage offers lower cost and effectively unlimited capacity, while Filestore keeps access latency much lower.
Filestore handles shared file access for most workloads, but very large training clusters run into throughput demands that require a different architecture. Managed Lustre is a fully managed file storage service built on the open source Lustre parallel file system. Because Google Cloud operates it as a managed service, teams gain parallel file system performance without standing up and tuning the underlying storage servers themselves.
Managed Lustre is designed for high throughput and low latency at large scale. Files are spread across many storage servers at once, so when hundreds of clients read the same training data, they do not queue behind a single machine. Throughput scales with the size of the deployment rather than flattening out.
Checkpointing in large distributed training jobs shows why a parallel file system matters. During a checkpoint, many workers write their slice of model state at the same moment, and training does not resume until the slowest write finishes. A parallel file system absorbs that burst across many servers instead of handling writers one at a time, preventing checkpoint writes from becoming a bottleneck in the training loop. As a rule of thumb, Filestore serves general shared file access, while Managed Lustre is the choice when scale is the primary constraint.
Storage selection also depends on where data currently resides, how much preprocessing a pipeline requires, and whether the workflow benefits from higher-level management features. BigQuery is useful for tabular data that can be loaded in batches, especially when the data already lives in BigQuery and moving it elsewhere for training would add unnecessary work. TensorFlow connects directly to BigQuery using tfio.experimental.bigquery, allowing pipelines to read data straight from the data warehouse.
Gemini Enterprise Agent Platform also offers managed datasets, which function as a managed layer on top of Cloud Storage. They retain the underlying scalability and performance of Cloud Storage while adding capabilities beyond raw object storage, including data labeling, versioning, and integration with other Agent Platform services.
When evaluating an architecture for the Professional Machine Learning Engineer exam, we start by checking whether Cloud Storage already covers the requirement before introducing a managed file system or another data layer:
gs:// URLs and none of the operational overhead of HDFS.tfio.experimental.bigquery.Our Professional Machine Learning Engineer course covers ML training storage across Cloud Storage, Filestore, and Managed Lustre alongside TensorFlow data pipelines and Agent Platform managed datasets, with practice questions that drill these distinctions.