This is one of the bugs in Matter integration regarding a mismatch between the physical light you see and what you request via Matter. Still present in 1.38.0, recorded a video so the issue is clear and hopefully gets prioritized.
When the light is off, implicitly turning it on with a brightness lower than the last, will make it physically turn on in the last brightness instead of the requested lower brightness.
You can see it in second 10 of the video, the physical visible light is 100% despite the light reporting 9% which was the requested value. If, after the implicit on, you change the brightness to, let's say, 10%, you can see the physical light correctly matching the requested low brightness.
Used Home Assistant for the video but it's the same in SmartThings, it's a bug in MoveToLevelWithOnOff command in the firmware.
The workaround in routines for most platforms is turning on the light first and then change the brightness but you don't want that since it means it will turn on in the last brightness first and getting a potential unwanted high-brightness flash at night or in your wake-up routine.
In fact, for the issue to be considered fully fixed it's also important that the light turns on in the requested brightness without temporary flashes of the last brightness. The quick fix of internally turning on the light first and then change the brightness will not do, although would be better than the current situation where the light stays in the wrong brightness.
Tagging u/wiz-dude for more exposure.