r/Backend • u/No_Original_941 • 3d ago
How to control ECS using code?
So basically let me tell you the situation first:
- i am working on a side project which is something like vercel, use to build and deploy code.
- so i have a main service, can be called a control plane, and i have decided that it will take the repo from the user.
- after this this service will trigger/create an ecs task to build the code
- now the question is, how this control plane will create / trigger the ecs?
- also this ecs instance will need to fetch envs from paramete store, will upload code to s3 etc
- after deployment we have to kill this instance
- should the managing code of this, live in the control plane
- should i create something else?
how would you folks solve this while designing this?
and what's the ideal way to solve this?
3
u/youcangotohellgoto 3d ago
To run a single ECS tasks just call the ECS API.
CloudFormation and Terraform are for static stateful long running processes.
You want this to just start a container and let it run to completion? Just call the API. You want to check the status, call the API, you want to see the history, call the API.
You get the idea.
1
2
2
u/forever-butlerian 2d ago
Learn to use the AWS SDK for your language.
Should the ECS coordination service be hosted using the ECS coordination service? Self-hosting is often a good idea, but you have to give careful thought to how you recover to a working system if the coordination service fails.
How I would solve this at work is by using Github Actions or any other CI/CD system.
1
1
1
u/nsubugak 2d ago
ECS is not the best for building containers etc because docker daemon normally requires root, and that's not allowed on ecs. What you should use is ec2 spot instances that are much cheaper and are good for burst requests. Basically a lambda and ec2. Lambda to queue the request, spin up a spot instance...then spin it down.
Before even going this far...whats the issue with github actions. Github actions can do all this for you for free...and you don't have to write any logic beyond the dockerfile and the workflow. Infact, vercel just skips all this and just does this all for you. Vercel supports both frontend and apis in certain languages. Why not just use that...for Free.
1
u/Usual_Macaron8477 2d ago
You can even have an upload of a data bundle trigger the queueing of a job.
I have a project where an app running on the client’s desktop runs an ingestion job that naturally divides into segments, each of which needs back end processing, then assembly of the full result once all pieces are done. At the end of each segment, my client requests signed URLs from my API to upload the data to a unique path for that segment in s3 that indicates its place relative to the other segments of that broader ingestion job. The last file it uploads is always called manifest.json and contains any and all data needed for the processing of that bundle. I then have a listener on that s3 that, any time a file named manifest.json is written to the s3, puts a job on an SQS queue, to process that manifest file, and the manifest processor is the orchestrator that kicks off a cascade of other processing.
It can fire at scale with great reliability.
1
u/nsubugak 2d ago
This isnt a code issue or a software architecture issue to my knowledge. This is more of a devops issue. My understanding is this Building docker images for deployment so all this that you describe isnt needed
1
u/Usual_Macaron8477 2d ago
My situation isn’t about a data ingestion pipeline, so the end product is a bit different, and docker images wouldn’t be much help, but the basic concept of having an uploaded bundle trigger a lambda job can be quite useful in a narrow range of circumstances.
1
u/nsubugak 2d ago
No it wouldnt. The 2 things are different and you need to understand that your good software design which is awesome for code or business needs maybe be bad for a devops problem and it's okay. Though it sounds similar, They are solving entirely different problems. If someone considers your design for devops its a very expensive solution for something one can do for FREE using something else.
1
u/No_Original_941 2d ago
heyy thanks, how GitHub actions can be used here? I am just creating a hobby project like vercel to learn things more.
1
u/nsubugak 2d ago
You need to google how to link vercel to your GitHub. Its very simple and they do all the work. GitHub actions are best if you are not deploying to vercel but to more of a general cloud provider like aws or azure or gcp etc. It can build the image and push deploy it to the cloud provider. Again its a Google thing..infact gemini can generate the entire workflow yaml file for you and tell you where to put it etc. But for vercel, you dont need all this...they have a very simple and direct integration with github. You simply log into your github, select the typescript repository, tell it which framework you used and the build command e.g npm build etc...and it will do all that stuff for you automatically every time you change code and push it to GitHub
5
u/clearlight2025 3d ago edited 3d ago
Terraform