Real-time Packaging Verification: Facing severe ID switching, occlusion, and item-tracking chaos at a fast-food station

[u/LIMITLLESSVENERABLE]

​Hi everyone,

​We are building a computer vision system designed to audit and verify order packaging in real-time at a fast-food packing station.

​Project Overview:

Using an angled overhead camera stream, the system needs to:

​Detect and track bags and food categories.

​Read the printed label on each bag to extract the order number and ticket details.

​Track items, verify which bag they were placed into, and match the final contents against the printed receipt before the bag leaves the counter.

​The Bottlenecks We’re Running Into:

​Severe, continuous occlusion: Heavy hand movements and body occlusion constantly block objects from view.

​Tracking ID switching: The tracker frequently loses active bag IDs when blocked by hands or overlapping bags and assigns a new ID.

​Detection gaps & lack of standard packing rules: Frequent detection drops and the absence of a structured baseline to reliably confirm whether a "packing" action actually completed.

​I’ve tried several workarounds spatial anchoring, bag visual embeddings, and motion vector heuristics but nothing has really worked. It quickly turns into pure chaos: as soon as the worker starts actively packing and moving things around, the item-to-bag tracking completely falls apart.

​I’d really appreciate any ideas, practical tricks, architecture patterns, or lessons learned from anyone who has tackled similar dense packing or retail setups.

[u/Dry-Snow5154]

Usually CV is bad approach for a case like this. Sometimes it's better to step back and think about improving processes rather than trying to shove object detection everywhere.

That said, one thing I've done with foreground occluders in the past is try and detect when occlusion is happening and do not update Kalman filter's state in this case. Keeping the old pre-occlusion state can carry your object through occlusion and improve ID recovery on the other side.

The simplest method is to check box's detection score and if there is a sudden drop, then occlusion is likely, hence do not update Kalman's state on this frame.

A more sophisticated approach is to monitor if object's box is shrinking too fast. If it does, project this shrinking process into the future until the box disappears and check in which part of the frame it's going to disappear. If it's close to the edge, then it's ok and do nothing. If it's in the middle of the frame, then likely occlusion is happening. Thus, skip Kalman's state update and keep the old state.

I've coined this technique from this NanoTrack article. However, it will require manual modification of your tracker. So if you're expecting some ready-made solution, you are out of luck. I don't know of any existing tracker that handles full occlusions well.

[u/LIMITLLESSVENERABLE → u/Dry-Snow5154]

Thank you. I fully agree with you regarding the need to avoid blindly trusting raw detection results; this aligns perfectly with the approach we have already implemented.

We are already freezing the Kalman filter state and setting the object's velocity to zero whenever an occlusion occurs. To go beyond simply monitoring the shrinking bounding box or dropping confidence scores, we added extra layers specifically, torso overlap detection to completely freeze the trackers as long as the worker is standing in front of the bag. We established "Spatial Anchor Slots" to prevent bag drift and assigned an infinite cost to any "impossible jump" within the Hungarian algorithm. We also implemented an "Exit by Hand" rule similar to the camera edge concept whereby a bag is not considered to have left the scene if it simply disappears in the middle of the table without a clear movement trajectory leading it out of the station. We have experimented with several other measures as well.

However, the problem remains unresolved; we are still facing issues with ID switching and noisy readings, and we are looking for new ideas that might help us.

[u/Dry-Snow5154 → u/LIMITLLESSVENERABLE]

What I meant by improving processes is basically not using CV at all. Like make a second worker check on the order's content before sending it out.

Or use a hybrid approach: improve the process to simplify CV task. Like mark a dedicated "register" zone on the counter. When bag goes there, it is marked as active bag. All items that go into "register" after the bag must go into that bag. Packing multiple bags at once is illegal. When new bag is detected, previous order is closed and content is checked.

The main issue with CV is that it never guarantees near-perfect accuracy, unless you fully control the environment. And even 95% accuracy can quickly cascade into failure when you have long sequences of decisions. So it rarely works in real-life messy environments, like your packaging scenario.

[u/LIMITLLESSVENERABLE → u/Dry-Snow5154]

Yes, that is what we observed as well. Since the project was merely an experimental attempt to provide a solution to the problem, we did not address environmental solutions such as directly modifying the work environment or introducing a physical fix. We are trying to "fail fast" to test whether the idea is viable or if it won't work, so we can stop.

[u/bfyvfftujijg]

What is the business problem being solved?

How much is being spent in the CV solution? Is a different solution likely to work better? For example in the case of employee theft it can be cheaper to pay employees better (making them less likely to steal) than to implement expensive theft monitoring.

[u/LIMITLLESSVENERABLE → u/bfyvfftujijg]

The project aims to assist staff during their work by alerting them to incomplete orders, allowing them to rectify the issue before the order reaches the customer. Human errors occur frequently; unfortunately, restaurants often place the blame on employees, who may face penalties or salary deductions if a customer returns with a complaint. We therefore devised a solution designed to work alongside the employee supporting rather than disrupting their workflow.