At Lob, we currently use Convox as our deployment platform, a “roll your own Platform-as-a-Service” that you can install to handle container orchestration on AWS’s ECS (Elastic Container Service). Convox is showing its age and this year we began the process of replacing Convox with HashiCorp’s Nomad, a flexible workload orchestrator to deploy and manage our containers on AWS.
As we prepared our migration to Nomad, our biggest concern was performance. We started our tests with a barebones proof-of-concept setup of our API in Nomad. We benchmarked and compared the two systems and found Nomad was running slower. This was extremely surprising as Convox (which leverages AWS Elastic Container Services) and Nomad both use Docker containers under the covers. We assumed going from older to newer technology would result in some performance gains.
Due to the stripped-down Nomad cluster, we were initially not set up with Datadog’s agent to collect application traces and profiles.
Our initial performance testing revealed that every request performed poorly with a four-second delay regardless of load, which was unexpected. Even stranger, running a load test showed performance got better and would go up and down over time. Internally, we debated whether poor database performance was the cause. However, with the time-honored process of deleting code and observing the results, we found that the four-second delay occurred on save requests to our AWS S3 bucket.
To determine if the problem was in our code or the infrastructure we needed to isolate the suspect code and configure Datadog to effectively trace interactions between our app and the infrastructure. We did this by extracting the S3 upload method from our production codebase to eliminate noise in our trace, configuring the Datadog APM trace library for this code, and enabling the Datadog agent on our Nomad cluster.
We found 4 sequential calls with exclamation marks in Datadog. Why would an S3 upload cause 4 HTTP PUT requests? And why did each request fail after exactly 1 second?
Looking closer, we found socket hang-ups and an IP address being called.
Let’s zoom out for a minute and discuss how Amazon services share information with each other. All AWS API requests have to be signed with credentials that tie back to an IAM Role with specific permissions, like uploading objects to our AWS S3 bucket for example. Traditionally, this role would be associated with access keys and those credentials would be loaded into your application -- potentially exposing them.
AWS provides a more secure mechanism when the calls originate from an AWS service. For our existing Convox services, this was ECS Container Instance Roles. For services running on ECS with an associated instance role, the AWS SDK would automatically call the metadata API on our ECS cluster to get temporary credentials it can use to access other Amazon services like S3. This means we don’t need to include our AWS credentials in our code and the IAM permissions can be directly tied to the ECS service itself. For EC2, there exist EC2 Instance Profiles, which function almost the same.
The AWS SDKs use IMDSv2 calls by default. If the IMDSv2 call receives no response, the SDK retries the call and, if still unsuccessful, uses IMDSv1. This can result in a delay.
In a container environment, if the hop limit is 1, the IMDSv2 response does not return because going to the container is considered an additional network hop. To avoid the process of falling back to IMDSv1 and the resultant delay, in a container environment we recommend that you set the hop limit to 2. For more information, see Configure the instance metadata options.
Translation: In order for the SDK, running in a Nomad-orchestrated container, to obtain credentials it must pass through the EC2 instance to reach EC2 metadata API. This counts as two “hops” and the default hop limit is one! We changed the hop limit to two and the request time dropped from 4 seconds to 85ms
Here is the Datadog APM trace after increasing the hop limit to two.
To put it bluntly, EC2 instance profiles are not the proper way to provide credentials to applications running in a container orchestrator. They are too coarse-grained. Permissions are granted to all applications running on the EC2 instance, not a specific application or service. After all, that’s why ECS, which runs on EC2 instances and therefore could use the EC2 instance profiles, instead relies on ECS instance profiles.
So why did we? Because we needed a temporary solution in order to trial our proof-of-concept Nomad cluster, and EC2 instance profiles were tantalizingly similar to ECS instance profiles -- our bread-and-butter solution in Convox.
Oh, and they do work. The application was able to access the private S3 bucket. Well, eventually. Mostly.
You might be wondering why isn’t AWS setting the hop limit default higher? We don’t know for certain, but our guess would be for security purposes. A low default hop limit ensures that credential requests originate inside the EC2 instance and forces developers to make a choice to increase the hop limit and be aware of any associated risks.
Several lessons can be drawn from our experience tracking down this performance issue
We hope this article helps other developers who encounter mysterious performance problems and are looking for tips to track them down.
2022 UPDATE: We have completely migrated to Nomad; read more about why we chose Nomad over K8s here.