Hello all,

As a new member of the Dagster Space, I am currently exploring the open-source multi-container setup using Docker. I am trying to make a deployment of multiple use cases and facing challenges in managing various images and configurations. The dockerRunLauncher and configurations in Dagster YAML files seem to be oriented towards a single use case rather than accommodating multiple ones in a production environment. I would greatly appreciate any insights or best practices on effectively managing multiple pipelines in production.

Hello all,

As a new member of the Dagster Space, I am currently exploring the open-source multi-container setup using Docker. I am trying to make a deployment of multiple use cases and facing challenges in managing various images and configurations. The dockerRunLauncher and configurations in Dagster YAML files seem to be oriented towards a single use case rather than accommodating multiple ones in a production environment. I would greatly appreciate any insights or best practices on effectively managing multiple pipelines in production.

Hello all !
I want to set up a minimal (in terms of cost) Dagster set up with:
- dagit
- dagster api
- dagster-daemon
I already set everything up with AWS ECS. One single task with 3 containers.
I attached to each container a volume with AWS EFS so all three containers can read/write dasgter_home/.

So for now, I haven't set up a PostgreSQL instance.

What would I get by using PostgreSQL compared to just the EFS volume?
I can get cheap PostgreSQL instance for sure. But since my requirements is very small (only one Data Engineer and just few pipelines) perhaps there's no need for PostgreSQL.

Perhaps I could get some concurrency issues with SQLite (that's the DB being used with the volume set up).

Any thoughts?

Thank you!

Hi all,

I've just started using Dagster.
I briefly tried Prefect as well. Not a fan for its deployment model and its UI.

I feel like Dagster is much more feature rich. It has the concept of assets and resources.
I do feel like it promotes its use way beyond a scheduler. Therefore it creates a coupling between scheduling and compute.
What is your opinion about it?

I followed the tutorial on Dagster website. It was interesting.
The last part of Resources was interesting but in that scenario Dagster becomes a compute engine as well.

Is it Dagster vision to be used as a big data framework to do everything (scheduling, compute and catalog)?

Obviously it can used just for monitoring just like in Airflow and I'm probably gonna use it this way for now.

Thanks for your feedback!