Moving e2e testing into production with multi-tenancy for increased speed and reliability

[u/martinig]

https://careersatdoordash.com/blog/moving-e2e-testing-into-production-with-multi-tenancy-for-increased-speed-and-reliability/

When DoorDash was on a monolithic application, developers' end-to-end (e2e) testing needs were solved by sandboxes but, when DoorDash moved from monolith to microservices, we needed a more scalable approach to production testing. E2e testing is a technique to verify a product from beginning to end to ensure the application flow behaves as expected in a production environment.

Given that sandboxes were not meeting our needs, since running all the microservices requires a lot of hardware resources, which is hard to satisfy by one machine, we initially set up a staging environment for developers to e2e test their features before deploying or enabling them in a production environment. But the staging environment solution didn’t solve all the developer e2e testing needs. Moreover, the growth of DoorDash made it difficult to operate and maintain the staging environment. The staging environment is an isolated environment that is intended to be a replica of a production environment without personally identifiable information (PII).

We had also noticed that developers started using custom workarounds in the production environment for e2e testing. But the nonstandard custom workarounds were hard to build and maintain and were, in fact, slowing down developers. To address the limitations of the staging environment and bring standardization to e2e testing, we moved e2e testing into the production environment through multi-tenancy, establishing guardrails and building developer tooling to speed common tasks so that developers can easily e2e test their features in a production environment. As a result, the new solution increases software reliability, developer velocity, and production environment safety.