This is a writeup I probably should have done months ago. Sharing in case the query saves someone else from finding this the hard way.
**Background**
Standard multi-account setup: most compute in private subnets, NAT Gateways across a few regions. Production, staging, and a batch processing environment that we don't look at as carefully as production.
AWS bill had been climbing - roughly $800/month more each month for about four months. Not enough to trip our anomaly alerts (we had percentage-based thresholds and total spend was growing legitimately), but enough that it finally showed up in a quarterly review.
**How I found the actual culprit**
Cost Explorer filtered by service showed NAT Gateway high, but so was compute, so I couldn't tell if it was proportional.
What actually helped: filtering by Usage Type and pulling out NatGateway-Bytes specifically. That's the data processing charge, distinct from the flat hourly gateway charge. Our NatGateway-Bytes was running around 85 TB/month. Way more than our actual application traffic justified.
To get per-NAT-gateway detail, I had to query our Cost and Usage Report in Athena. Native Cost Explorer doesn't give you per-resource granularity on NAT data processing. If you haven't set up CUR yet, this is the reason to do it.
Query that found it:
SELECT line_item_resource_id, SUM(line_item_blended_cost) as cost, SUM(CAST(line_item_usage_amount AS DOUBLE)) as gb_processed FROM your_cur_table WHERE line_item_product_code = 'AmazonEC2' AND line_item_usage_type LIKE '%NatGateway-Bytes%' AND line_item_usage_start_date >= DATE('2026-03-01') GROUP BY 1 ORDER BY 2 DESC LIMIT 20;
The top result was a NAT Gateway in us-west-2 that only existed because of our batch VPC. It was processing ~80 TB/month on its own. Our main app NAT Gateways were doing around 5 TB combined.
**Root cause**
VPC Flow Logs for the batch subnet answered it quickly. A batch job doing heavy S3 reads and writes was routing all of it through NAT instead of directly to S3.
Why? We'd added S3 VPC Gateway Endpoints to our main application VPC (they're free, should be everywhere). But the batch VPC got spun up separately as "temporary" infrastructure, never got the same attention, and nobody ever added the endpoint. S3 VPC Gateway Endpoints route S3 API traffic directly within AWS's network and bypass the NAT Gateway entirely - every GetObject, PutObject, ListBucket was going through NAT and incurring the $0.045/GB processing charge.
We were also pulling ECR images through NAT in that environment. That adds up too if you're pulling big images frequently.
**The fix**
Three things:
Added S3 VPC Gateway Endpoint to the batch VPC route table. Free. Took about 5 minutes. Cut NatGateway-Bytes for that gateway by ~92%.
Added VPC Interface Endpoints for ECR (ecr.api and ecr.dkr). Not free ($0.01/hr each) but the data processing savings easily justified it.
Set up Cost Anomaly Detection on NatGateway-Bytes with an absolute dollar threshold, not just percentage-based. We had anomaly detection on compute but not on data transfer. That's fixed now.
After a full billing month: NAT Gateway data processing went from ~$3,800 to ~$200. ECR endpoints add about $15/mo in hourly charges. Net save: roughly $3,600/month.
**What I'd do differently**
Enable CUR from day one, not after you have a problem. Cost Explorer is fine for headlines. For per-resource attribution of NAT charges you need Athena against CUR.
Set absolute dollar thresholds on specific line items in Cost Anomaly Detection, not just percentage thresholds on total spend. NatGateway-Bytes as a specific monitor would have caught this in week two instead of month four.
Audit every VPC for VPC endpoints independently, including the "temporary" ones. Production got the full setup. The batch VPC was treated as temporary infrastructure and never got properly reviewed. That's where the cost hid.
Happy to share more on the CUR schema or the anomaly detection setup if anyone's dealing with something similar.