Fixing Out-of-Memory (OOM) Errors in Machine Learning Training

GCP Study Hub
September 18, 2026

An out-of-memory (OOM) error, often reported as a resource exhausted error, is an infrastructure failure that occurs when a machine learning training job tries to allocate more memory than the available hardware can provide. This error appears most frequently with GPU memory because GPUs have far less memory than system RAM. On the Professional Machine Learning Engineer exam, troubleshooting scenarios around resource exhausted errors focus on identifying which training changes actually reduce memory consumption and which ones either degrade model quality or have no meaningful effect on memory use.

What causes out-of-memory errors during training

During every training step, the hardware must hold three categories of data in memory at the same time: the current batch of training examples, the model weights, and the intermediate calculations generated during the forward and backward passes. When the combined footprint of these items exceeds available memory, the system cannot fit the workload and stops with a resource exhausted error.

Three common factors push a training job past its memory limit:

  • Large batch sizes that require the system to process too many training examples at the same time.
  • Very large models containing millions of parameters that require a substantial portion of available memory.
  • Large input data, such as high-resolution images, where each individual example requires a large amount of memory.

Reducing batch size as the primary fix

The most straightforward way to resolve an out-of-memory error is to reduce the batch size. Batch size controls how many training examples are processed together during a single forward and backward pass through the model. Larger batches consume more GPU memory because the hardware must retain all examples in the batch simultaneously along with every intermediate calculation required for backpropagation.

Lowering the batch size breaks the training data into smaller chunks that fit more comfortably within the hardware limits. In practice, we would step the batch size down incrementally, moving from 64 to 32, or from 32 to 16, until the job runs without hitting a memory ceiling. The main trade-off is that training can take longer because fewer examples are processed per step. Even with that extra runtime, reducing batch size is usually a reasonable compromise to keep training running reliably.

Why reducing image size is a last resort

When a training job works with image data, shrinking the input image dimensions can look like an appealing fix for an out-of-memory error. Smaller images do consume less memory, but resizing inputs is generally a mistake as a first response.

High-resolution images carry fine-grained details, textures, and structural cues that a model relies on to learn accurate representations. Shrinking those images causes fine features to blur or disappear entirely. When a model needs to detect subtle patterns or textures, lowering input resolution makes the learning task harder and can directly harm model performance.

Adjusting the batch size first avoids this trade-off. It keeps the full detail of the input images intact while still bringing memory usage within hardware constraints. Reducing input image size should only be considered as a last resort when other memory adjustments are not enough.

Why optimizer choice and learning rate do not fix OOM errors

Another distinction tested on the Professional Machine Learning Engineer exam involves hyperparameters that govern convergence rather than memory capacity. Changing the optimizer, such as switching between SGD and Adam, or adjusting the learning rate has a negligible effect on how much data needs to be placed into memory.

These settings shape learning dynamics and how quickly a model converges. The optimizer determines the path the training process takes toward the minimum, and the learning rate controls how large each step is along that path. Neither setting meaningfully changes the size of the batch of examples, the model weights, or the intermediate calculations stored during forward and backward passes. Because the underlying memory requirements remain essentially the same, tuning the optimizer or learning rate is unlikely to resolve an out-of-memory error.

Our Professional Machine Learning Engineer course covers troubleshooting out-of-memory errors alongside training infrastructure and hyperparameter tuning, with practice questions that drill these distinctions.

Get tips and updates from GCP Study Hub

arrow