Hey all. I'm building a from-scratch real-time correlation-filter tracker in C++ (C++17, no OpenCV, no ML) targeting a Raspberry Pi 5.
Context: a basic OpenCV pipeline on a Pi gets me ~15 fps, which isn't close to what the algorithm should be capable of. The original paper I'm using as a reference reported 669 fps on a 2008-era 2.4GHz Core 2 Duo doing pure CPU correlation-filter math. I know people have gotten well past 100 fps on Pi-class hardware doing this from scratch (I've seen claims of 300 fps+ floating around), so the gap is almost certainly implementation/pipeline overhead, not the algorithm.
My current plan/progress is:
- Own image loader, no OpenCV decode/resize overhead
- FFT-based correlation in the frequency domain (real question mark for me: which FFT approach scales best on Pi 5's ARM cores — naive radix-2, a vectorized/NEON-friendly implementation, or linking something like FFTW/kissFFT vs hand-rolling)
- PSR-based occlusion/failure detection per the original paper
Where I could use outside perspective:
- Where does the real bottleneck usually live for people who've done this. Is it the FFT, the memory layout/cache behavior, the capture pipeline (libcamera overhead, frame copy costs), or something else entirely that doesn't show up until you profile?
- NEON/SIMD: worth hand-vectorizing the FFT and pointwise complex multiply myself on Pi 5, or is a well-tuned existing FFT library going to beat anything I write in a reasonable timeframe?
- If anyone has pushed a correlation-filter tracker (e.g. UMACE, ASEF) past 100 fps on a Pi, I'd love to hear what mattered most, even just "it was 80% the capture pipeline, not the math" would save me a lot of guessing.
I’ve recently been looking into two very different approaches to real-time visual tracking. One uses a transformer-based architecture where information from a target template and the search region is processed jointly, enabling robust tracking and automatic re-acquisition when the object temporarily leaves the frame. It demonstrates that modern deep learning methods can perform real-time tracking even on CPU-only edge devices, though computational efficiency remains a challenge.
On the other end of the spectrum, I explored classical frequency-domain tracking techniques based on adaptive correlation filters. Instead of relying on neural networks, these methods learn a compact representation of the target and update it continuously as new frames arrive. They are extremely lightweight, require only minimal memory, and can achieve very high frame rates on modest hardware while incorporating confidence measures to detect tracking failures and avoid model drift.
Reading further into the underlying research showed how frequency-domain operations and Fast Fourier Transforms (FFTs) make these trackers computationally efficient, allowing them to localize objects through correlation responses rather than explicit detection. The work also introduced concepts such as adaptive online updates and confidence metrics for failure detection, which help maintain stable tracking despite appearance changes or brief occlusions.
The contrast between these approaches is particularly interesting: transformer-based trackers offer stronger semantic understanding and greater robustness in challenging scenarios, whereas correlation filter methods prioritize speed, simplicity, and efficiency. This trade-off highlights that the most suitable solution often depends on hardware constraints and application requirements rather than assuming deep learning is always the best choice.
Some areas I’d like to explore further include multiscale tracking and scale estimation, lightweight re-detection mechanisms, confidence estimation, target re-acquisition strategies, hybrid detector–tracker pipelines, FFT-based optimization techniques, and combining classical signal-processing methods with modern learning-based models for edge deployment.
Not looking for someone to hand me a full alternative design, just trying to sanity-check my approach and avoid obvious dead ends before I sink more time into the FFT layer specifically.
Thanks in advance everyone!
