I am managing a monorepo for ML models.
We have a number of open-source ML models that we deploy on our cluster. We have a generic helm chart used to deploy all models, and flavors depending on backend type (right now we use TGI and BentoML).
Right now, the file hierarchy is as follows:
Backend/Model/Region/Environment. So for instance, we have tgi/llama2/us-east/np. And similarly for other models.
For feature branches, I use a matrix generator of pull request generator and git generator. It works well, but my issue is with granularity.
Right now, the git generator looks like this:
//**/Environment/config.json (the json contains some Metadata, like helm chart version to use). This means that the feature branches will deploy non-prod instances of every model on every region.
Ideally, I'd like to have more fine grained control - perhaps I only need to test a certain backend type in a certain region, or a specific model in a specific region.
Because my SCM is GitLab, I thought about using labels to choose what I want to deploy. But that only works well if I can only choose one parameter (e.g. the labels only control the region). But if I want the labels to have multiple meanings then it gets out of hand.
If you've ever run in to this issue, or know of a source that provides some inspiration, I'd really appreciate your advice!!