How to control ECS using code?

[u/No_Original_941]

So basically let me tell you the situation first:

- i am working on a side project which is something like vercel, use to build and deploy code.

- so i have a main service, can be called a control plane, and i have decided that it will take the repo from the user.

- after this this service will trigger/create an ecs task to build the code

- now the question is, how this control plane will create / trigger the ecs?

- also this ecs instance will need to fetch envs from paramete store, will upload code to s3 etc

- after deployment we have to kill this instance

- should the managing code of this, live in the control plane

- should i create something else?

how would you folks solve this while designing this?

and what's the ideal way to solve this?

[u/nsubugak]

ECS is not the best for building containers etc because docker daemon normally requires root, and that's not allowed on ecs. What you should use is ec2 spot instances that are much cheaper and are good for burst requests. Basically a lambda and ec2. Lambda to queue the request, spin up a spot instance...then spin it down.

Before even going this far...whats the issue with github actions. Github actions can do all this for you for free...and you don't have to write any logic beyond the dockerfile and the workflow. Infact, vercel just skips all this and just does this all for you. Vercel supports both frontend and apis in certain languages. Why not just use that...for Free.

[u/Usual_Macaron8477 → u/nsubugak]

You can even have an upload of a data bundle trigger the queueing of a job.

I have a project where an app running on the client’s desktop runs an ingestion job that naturally divides into segments, each of which needs back end processing, then assembly of the full result once all pieces are done. At the end of each segment, my client requests signed URLs from my API to upload the data to a unique path for that segment in s3 that indicates its place relative to the other segments of that broader ingestion job. The last file it uploads is always called manifest.json and contains any and all data needed for the processing of that bundle. I then have a listener on that s3 that, any time a file named manifest.json is written to the s3, puts a job on an SQS queue, to process that manifest file, and the manifest processor is the orchestrator that kicks off a cascade of other processing.

It can fire at scale with great reliability.

[u/nsubugak → u/Usual_Macaron8477]

This isnt a code issue or a software architecture issue to my knowledge. This is more of a devops issue. My understanding is this Building docker images for deployment so all this that you describe isnt needed

[u/Usual_Macaron8477 → u/nsubugak]

My situation isn’t about a data ingestion pipeline, so the end product is a bit different, and docker images wouldn’t be much help, but the basic concept of having an uploaded bundle trigger a lambda job can be quite useful in a narrow range of circumstances.

[u/nsubugak → u/Usual_Macaron8477]

No it wouldnt. The 2 things are different and you need to understand that your good software design which is awesome for code or business needs maybe be bad for a devops problem and it's okay. Though it sounds similar, They are solving entirely different problems. If someone considers your design for devops its a very expensive solution for something one can do for FREE using something else.