A cautionary tale about AWS VPC networking, NAT gateways, and how a missing VPC endpoint turned our S3 data transfer into an expensive lesson.
I’ve been using AWS since about 2007. At the time, EC2 storage was strictly ephemeral and stopping an instance meant losing all your data. Since then this platform has progressed a lot.
Even after almost two decades with this platform, there is always something new to learn. And sometimes those texts come with a $1,000 price tag.
Net
We have recently started using S3 to mirror some large internal data files for Geocode. We’re talking about geographic datasets (things like address points, boundary data, and census information) that range from a few gigabytes to hundreds of gigabytes. Some of these files are updated with fresh data almost daily, while others are refreshed less frequently. They need to be regularly synced from our ETL platform (which is hosted with Hetzner) to our processing infrastructure on AWS.
AWS’s data transfer costs are extremely high. Cloudflare has written about this extensively, and it’s a common complaint across the industry. Last week’s Corey Quinn at AWS also described AWS managed NAT gateways as particularly expensive. AWS charges $0.09 per GB for data transfer over the Internet in most regions, which adds up quickly when you’re transferring terabytes of data.
So before starting this project, I did my homework. I carefully researched the costs involved and confirmed two important things:
- AWS still allows free migration between EC2 instances and S3 (as long as they are in the same area)
- transfer In S3 are free (This was important because the data comes from our ETL platform hosted with Hetzner)
Great! I had a clear picture of the costs.
…or so I thought.
Wonder
A few days after implementing the new S3 sync process, I received a notification from AWS Cost Anomaly Detection. (Boy, was I glad I enabled it!)
The alert showed something worrying: 20,167.32 GB The data of “NAT Gateway” is transferred in a single day, amounting to $907.53,
As of today, it has already crossed $1,000.
I looked at the dashboard in disbelief. How can this be possible? I specifically confirmed that EC2-to-S3 transfers were free!
But why oh why?
After some frantic investigation (and a little panic), I found the culprit.
S3 transfer when you’re using a VPC with a NAT gateway (which most production AWS setups do) still go through NAT gateway As a default. Even if you are making a request to an AWS service that is in the same region, the traffic is sent out and sent back through your NAT gateway, which incurs a data transfer fee of $0.045 per GB.
Solution? VPC Endpoint for S3Specifically what AWS calls a “Gateway Endpoint”.
A gateway endpoint is a special type of VPC endpoint that allows you to route traffic privately to S3 without going through your NAT gateway or Internet gateway. This is essentially a direct pipe from your VPC to S3.
Gateway endpoints are even better for S3 completely freeNo hourly fees, no data transfer fees, Nothing,
Solution
The solution is to create a VPC gateway endpoint for S3. This is a special type of VPC endpoint that creates a direct route from your VPC to S3, bypassing the NAT gateway completely.
In our case, we manage the infrastructure with Terraform, so it was just a matter of adding the Gateway Endpoint resource and associating it with our routes table. AWS automatically handles routing updates to direct S3 traffic through the endpoint instead of the NAT gateway.
learning
I’ve created countless VPCs, configured security groups, set up load balancers, and optimized costs in dozens of ways over the years. But somehow, VPC endpoints for S3 fell out of my scope of knowledge.
AWS’s networking can be deceptively complex. Even when you think you’ve done your research and confirmed the costs, there are still layers of configuration that can change your bill dramatically.
Don’t make my mistake. Here are some things I would suggest checking out to help you avoid of one’s own Surprise $1,000 bill:
AWS Cost Anomaly Detection is worth installing. It caught the issue within a few days, saving us from an even bigger surprise at the end of the month. If you haven’t enabled it yet, do it Now,
VPC endpoints are your friends. If you’re using S3 or DynamoDB from EC2 instances in a VPC with a NAT gateway, you absolutely need gateway endpoints. There’s really no reason not to use them. They are free and improve performance.
Always validate your assumptions. I thought “EC2 to S3 is free” was enough. I should have tested with a smaller amount of data and monitored the costs before scaling up to terabytes.
The cloud is complex. Even after almost two decades, there’s always more to learn. And that’s okay. It simply means that we need to be careful and alert.
And we are not alone in this. Just last year, Recall.ai discovered they were paying $1M annually in windfall AWS WebSocket data processing fees. Even experienced teams hit these surprises.
what will happen next
We have audited our entire AWS infrastructure to ensure that we have gateway endpoints configured for all VPCs communicating with S3.
If you’re using AWS and haven’t checked your VPC endpoint configuration recently, I’d recommend taking a look. There’s no need to repeat that $1,000 lesson.
TL;DR: NAT gateways charge a fee for all data processing, even for traffic to AWS services like S3, which have no data transfer fees. Use VPC endpoints to bypass this.