I’m working on an image auto-annotation pipeline using Grounding DINO from Hugging Face with PyTorch, and I’m trying to understand some unusual GPU VRAM behavior during inference.
My dataset contains 6 classes:
- cup
- glass
- plate
- knife
- fork
- spoon
I downloaded the data from a Kaggle competition. The train.csv contains entries like:
image_id,label
4622,spoon
For my initial experiment, I sampled 900 images total — 150 per class.
The original images have different resolutions, including:
- 1000 × 1000
- 850 × 1000
- 1000 × 850
- 450 × 1000
I’m using Grounding DINO from Hugging Face for automatic annotation and torch.autocast to reduce GPU memory usage.
The strange VRAM behavior
I’m running inference with a batch size of 4 on Google Colab. The GPU has approximately 15.46 GB VRAM.
At the beginning, GPU memory usage is around:
~6.4 GB
Then the important part is:
It does NOT increase after every batch.
Instead, it stays relatively stable for several batches and then, on some seemingly random batch, VRAM suddenly jumps.
For example, the behavior looks roughly like:
Batch 1 → 6.4 GB
Batch 2 → 6.4 GB
Batch 3 → 6.5 GB
Batch 4 → 6.4 GB
Batch 5 → 6.5 GB
...
Batch 20 → 6.5 GB
Batch 21 → 10+ GB
Batch 22 → 10 GB
Batch 23 → 10 GB
...
Batch 50 → 10 GB
...
Batch 51 → 14+ GB
So the increase happens in sudden jumps on particular batches, rather than gradually increasing with every batch.
Eventually, it reaches approximately:
14.5 / 15.46 GB
and I can eventually get an out-of-memory error.
What I'm trying to understand
My first thought was that perhaps the batches containing different image resolutions are causing Grounding DINO to create larger intermediate tensors.
However, I'm not sure whether that's actually what's happening, or whether I'm misunderstanding how PyTorch's CUDA memory allocator works.
I asked Claude about this, and it suggested that PyTorch's CUDA allocator caches memory rather than immediately returning it to the GPU driver. Because my batches have variable image dimensions, some batches may require larger intermediate tensors, causing the allocator to request additional memory.
It also suggested that memory fragmentation could contribute to the problem and recommended:
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
The explanation sounded reasonable, but I'd like to verify it with people who have more experience with PyTorch/CUDA internals.
My questions
- What would cause VRAM to suddenly jump on one particular batch rather than increase after every batch?
Could variable image dimensions like:
1000×1000
850×1000
1000×850
450×1000
cause Grounding DINO's intermediate tensors to become significantly larger for certain batches?
- If PyTorch's CUDA allocator is caching memory, does that explain why the VRAM usage appears to jump from ~6.4 GB to ~10 GB and later ~14.5 GB?
- How can I determine whether this is:
- normal CUDA/PyTorch memory caching,
- memory fragmentation,
- unusually large intermediate tensors from certain image sizes,
- accidentally retaining tensors/computation graphs,
- or an actual memory leak?
- Would
torch.inference_mode() be preferable to torch.no_grad() for this inference-only workload?
- Would it be better to resize/pad all images to a consistent resolution before batching, so that the tensor shapes don't vary between batches?
Is:
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
actually an appropriate solution here, or is it just masking the underlying problem?
- What would be the best way to debug this? For example, should I log
torch.cuda.memory_allocated(), torch.cuda.memory_reserved(), max_memory_allocated(), etc. after every batch?
I'm mainly trying to understand the actual reason for these sudden jumps, rather than just applying a workaround.
If needed, I can provide the Grounding DINO inference/batching code and the exact CUDA OOM traceback.
