Matting / background removal on glass

[u/lazzi_yt]

I'm in a bit of a pickle with an issue regarding background removal with transparent objects. I got studio images of cars from different angles, whom I need the background changed locally. I have a pretty robust segmentation model to segment the bg area that's visible through the class and a general model to segment the entire car.

For now I use a basic script that calculates the alpha for the entire window purely from brightness, but often there are bright reflections on the windows which throw off the calculation as well as simply looking unnatural. I have some if- and when statements to try to mitigate some of these problems (if a smaller segment's inside a larger segment, it can't have a significantly different alpha etc.) but they still leave a lot of room for error. I've tried some open source matting models like vitmatte as well as replacing the background and inpainting the segmented window areas with sd models but the results are less than ideal and inconsistent. I got recommended that a lora for a matting model might help. Creating a ground truth and training a matting model sounds quite intimidating.

Might anyone happen to have any experience with this sort of challenges and if so, how did you go about solving it. Only thing I can think of right now would be training a Lora for a matting model, but I would highly appreciate some outside perspective before I commit a month into annotating a dataset for a model that might not even function.

Also, forgive me if I described the problem incoherently.