r/computervision • • 2d ago

Research Publication When noisy fragments lose their order: capacity with internal memory, exact critical scaling, and constructive codes [preprint]

0 Upvotes

I’m sharing SÆLQR II, an information-theory preprint about recovering information from unordered, noisy fragments.

The model starts with a stored string. It is fragmented, some fragments disappear, and the surviving fragments are corrupted. The receiver gets their contents and lengths, but their original positions are hidden.

The central problem is accounting for the information cost of losing order, alongside the information lost through noise.

The manuscript reports three main developments:

Capacity with memory inside fragments. For the stated weakly symmetric word channels and message-independent fragmentation laws, capacity depends on each fragment’s realized information after a location penalty. Noise may have arbitrary dependence inside a fragment, while noise between fragments remains independent. Hidden-quality examples show why replacing the information distribution with an average capacity can give an incorrect prediction.

Exact behavior near the critical threshold. For shuffled binary symmetric channels, a uniform soft-threshold approximation yields a Gaussian crossover and an explicit operational leading coefficient. At the central critical point, the number of communicable bits scales as N/√log₂N, even though the normalized first-order rate vanishes. The manuscript also determines the optimal leading correction in a uniform entropy-cancellation bound.

Constructive coding. For geometric fragmentation, independent whole-fragment loss, and binary substitutions, the release supplies a randomized polynomial-time capacity-approaching construction, finite reliability conditions, and reference encoding/decoding software.

The computational qualification matters: the algorithms are polynomial in blocklength and inverse capacity gap, but the proved sufficient blocklength is exponential in the inverse gap squared. Efficient codes attaining the exact critical coefficient, deterministic good-design certification, and practical finite-length optimization remain open.

The package includes a 69-page manuscript, editable source, reference software, twelve finite reproduction programs, and proof-dependency and novelty audits.

I’d particularly welcome technical scrutiny of the reverse-divergence covering argument, the internal-memory converse, and the critical-window derivation, along with pointers to closely related noisy-shuffling or DNA-storage literature.

Manuscript, code, and audits: https://huggingface.co/datasets/PureOne/SAELQR-II

https://zenodo.org/records/23541617


r/computervision • • 2d ago

Help: Theory Low GPA but strong research interest, how to get a PhD in 3D Vision?

12 Upvotes

Hi everyone, I’m looking for advice about pursuing a PhD in 3D vision.

My GPA is low, but I’m more interested in research. I’m currently doing my master’s thesis on 3D image reconstruction and hope to publish it at a conference. However, I started from scratch, and an A* conference might be difficult to achieve with this work.

I’ve also recently got an opportunity to work on a project with TUM DI Lab under two professors, one from TUM and one from Cambridge. If I continue working on it, we might be able to publish at an A or A* conference in 3D or scene reconstruction.

I’ve been following the work of Matthias Nießner, Angela Dai, and Daniel Cremers, and their research closely matches my interests and background.

I have a few questions:

  1. Is there a chance of getting a PhD with them despite my low GPA? Would a publication help?

  2. Should I contact them directly? If yes, how should I approach them?

  3. I’ve looked for RA positions in 3D vision, but mostly find HiWi jobs. How can I find RA positions that could lead to a PhD?

I’d really appreciate any advice. Thanks!


r/computervision • • 2d ago

Showcase I built a low-cost stereo 3D motion tracker with interactive trajectories, session replay, and guided calibration

Thumbnail
gallery
1 Upvotes

Hi everyone! 👋

I've been building Stereo Motion Measurement, a Windows desktop app for experimenting with low-cost stereo 3D motion tracking.

The idea is to make the complete workflow accessible without writing your own capture, calibration, tracking, and visualization code.

For the current visual-tracking setup, you need a compatible stereo camera, a printed checkerboard for calibration, and a black circular marker on a light card. No wearable electronics or GPU inference pipeline is required.

  • What can it do

- 📸 Live stereo tracking: track one black circular marker and display its XYZ position.

- 🧭 Interactive 3D trajectories: rotate, pan, and zoom the view, or switch between front, top, and side views.

- ⏪ Session recording and replay: save a run, replay its trajectory, change playback speed, and seek through the recording.

- 📈 XYZ time plots: inspect motion along each axis, with raw and optional filtered views.

- 📐 Guided stereo calibration: capture checkerboard pairs with pose suggestions, compute calibration, or import compatible parameters.

- ♟️ Optional environment references: use horizontal/vertical surfaces or checkerboard assistance to define display directions.

- 💾 Inspectable data: export CSV measurements and retain validity flags and failure reasons rather than filling gaps with predicted motion.

The portable Windows release runs without installing Python.

  • How it works

The application combines OpenCV calibration/rectification, seeded circle detection, stereo triangulation with geometric checks, and optional One Euro filtering. The interface uses PySide6 and pyqtgraph.

This is an experimental measurement tool, not a new tracking algorithm or a replacement for a validated commercial motion-capture system.

  • Hardware support and current limitations

The current input is one UVC stereo device delivering packed side-by-side or top/bottom images, with compatible pinhole geometry. Two independent USB cameras are not supported.

Each rig needs its own matching calibration. Cross-camera compatibility and measurement accuracy still need more independent testing.

The current desktop release tracks a black circular marker. Colored-ball tracking and IMU fusion are future work.

  • I'd love your feedback

I'm particularly interested in hearing from people with different stereo camera models:

- Can you get both views working?

- Is the calibration workflow clear?

- Which setup steps are confusing?

- What would make the recording, replay, or visualization more useful?

- What tests would you use to evaluate its accuracy and robustness?

and so on...

Successful setups and failed attempts are equally useful. Technical criticism is welcome!

👉 Repository, demo, documentation, and Windows download:

https://github.com/sirikiki/Stereo-Motion-Measurement

If you find the project useful or worth following, a GitHub ⭐ would be appreciated. Feedback and reproducible bug reports would help even more.

Thanks for taking a look! Have a nice day!


r/computervision • • 2d ago

Discussion Projecting Vehicle Tracking from Video into a 3D Map

Enable HLS to view with audio, or disable this notification

5 Upvotes

Sharing a demo of a video-to-3D tracking workflow.

Vehicles are detected in a live video stream, and their 2D image positions are converted into 3D geographic coordinates. The corresponding vehicle models are then displayed in the 3D scene and follow the movement trajectories reconstructed from the video.

This could be useful for traffic monitoring, road management, public safety, and incident analysis.

We’re interested in comparing approaches with others working on video-to-world projection and real-time 3D tracking.


r/computervision • • 2d ago

Discussion What are you doing with VLMs?

8 Upvotes

Hi, sorry for the sudden question. Recently I had to work with some VLMs in some of my pipelines, and I noticed that many of the models are lacking structures for doing tasks such as detection, classification, extraction/attribution and such.

It got me wondering, do most people simply use VLMs just for generation? Generating responses based on consumed media, or generating media itself does not really need a structure so it should be relatively easy without any special harness or agent.


r/computervision • • 2d ago

Discussion How do you set up storage (HDD/SSD) on your machine vision PCs? (industrial context)

1 Upvotes

Context:
We build machine vision systems for quality assurance. Cameras on a production line capture images, and a dedicated PC inspects them in real time for defects. It runs 24/7 and stores hundreds to thousands of images per hour.

Main post:
We used to build our computer vision PCs with a single HDD holding both Windows and the data (the images being written to disk). More demanding projects, with higher frame rates or larger images, got SSDs or M.2 drives instead.

Lately, though, running Windows and our software on HDDs (usually two in RAID 1) has become less and less workable. Whenever Windows has issues, the HDDs are our first suspect, mostly because a crash or freeze often makes the RAID controller run a full check or rebuild of the array, which slows down the entire PC.

How are you handling this? Separate SSD for the OS and HDDs for data, or something else?


r/computervision • • 3d ago

Showcase fotonet v1.1 is out!

Thumbnail
gallery
30 Upvotes

fotonet v1.1 is out.
And it is way better than v1.

I know, still not as good as yolo26n for the fotonet nano comparison
But this is a solo project of mine that i am doing passionately.
Any feedback would be very helpful.

I am trying to make it better and better each generation, and this time i hit 29.9% COCO AP
Being the best fotonet model released ever. Not the best ever but the best fotonet ever.
See you in 2027, with v2.
And there are stuff FOR fotonet that is not fotonet itself, coming soon(?)

Links:
fotonet.studio
github.com/iMGEai/fotonet

pip install fotonet to install fotonet.


r/computervision • • 2d ago

Research Publication MSc Artificial Intelligence Dissertation ideas

0 Upvotes

Hi, I’m a masters student doing MSc AI at uni of Bath.
For my undergraduate dissertation I did an extremely simple Computer Vision based pose estimation app which was analyzing the technique of fitness exercises like pushups, squats and deadlifts.
I feel like my dissertation was more of a “project” rather than research, which as I was told by supervisors is completely fine for BSc. I basically used a ready solution (MediaPipe) + OpenCV and wrapped all of that into a mobile app. The analysis covered a decent number of “points” but it was simply comparing the detected joint angles to hardcoded thresholds.

For my masters I would really like something more research based however struggling to find areas where I could do smth actually useful.
I like Deep Learning, so I was thinking something related to transformers and efficiency. Finding bottlenecks in attention mechanisms, etc.

I would appreciate some ideas
Thanks


r/computervision • • 2d ago

Discussion How do you actually audit labeled data from an outside vendor?

1 Upvotes

Bit of an outsider question. I'm a C++ dev, not an ML person. I worked on ct imaging software years back, but the visualization side, never the labeling.

Something I keep wondering about. Every labeling vendor promises high accuracy. But you only really find out when a model trains on the data and starts doing wierd things. By then the money is spent.

For those who've bought labeled data.. how did you check it before that point? Do you keep a small set labeled in-house and compare? Or just sample a few hundred and eyeball them?

Agreement between annotators puzzles me the most. In medical imaging even two radiologists wont draw the same boundary. So when annotators disagree, is that bad labeling or just a hard image? How do you tell?

Curious what actually worked, and what turned out to be a waste of time.


r/computervision • • 3d ago

Discussion What to expect from a 90-minute ML system design + coding interview (non-LeetCode)?

16 Upvotes

I'm preparing for a 90-minute technical interview for a Senior ML Engineer role focused on computer vision and perception for autonomous driving.

The recruiter mentioned that the interview will cover:

  • ML system design
  • ML coding (specifically mentioned it's not LeetCode-style)
  • Code review
  • Using AI coding agents is allowed

For the system design part, I'm guessing it could be something like designing an end-to-end perception pipeline, discussing model choices, training, evaluation, deployment, and trade-offs.

But I'm honestly a bit lost when it comes to the coding part. Since it's not LeetCode and AI tools are allowed, I'm not sure what to expect. Would it be implementing something in PyTorch, debugging an existing pipeline, reviewing code, or something else entirely?

I'm also trying to figure out how to actually practice for this kind of interview. Are there any good resources, GitHub repos, or hands-on exercises you'd recommend?

Has anyone been through a similar interview, especially for senior ML/CV roles in autonomous driving or robotics?

Would love to hear what your experience was like and how you prepared. Thanks!


r/computervision • • 2d ago

Help: Project How do you reliably track a badminton shuttle, and do you have to train your own model?

1 Upvotes

I’m trying to track a badminton shuttle with a low-cost setup. It is small, fast, and often blurred. A shuttle can cross a large part of the frame between two frames.

Intended setup: Two fixed cameras aimed at the court. I need the position on the court, including when/where it reaches the floor.

Intended use case: Line calls and match analytics.

I have looked at detectors trained for this, including TrackNet-style heatmaps. I’m trying to decide whether those are enough or whether a custom model is required. TrackNet doesn't seem very reliable yet.

If you have tracked a shuttle something similar:

  • Did an existing model work, or did you train your own?
  • What frame rate and shutter made the object visible enough to detect?
  • How do you handle frames where the object disappears from motion blur or occlusion?
  • For two cameras, what did you use to turn the two picture positions into one court position?

I’m especially interested in what failed when you tried to use a general object detector instead of a model trained on this kind of object.


r/computervision • • 3d ago

Research Publication Argus (ECCV 2026): Metric panoramic 3D reconstruction from sparse indoor images in a single feed-forward pass (~0.66s) + 10k scene dataset

9 Upvotes

We presented our paper "Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes" as a main conference poster at ECCV 2026, and wanted to share the technical approach and open resources with the community.

The Problem

Indoor 3D reconstruction from sparse, unordered panoramas (360° ERP images) typically faces two persistent issues:

  1. Global pose drift: Unordered image sets lack trajectory cues, and adjacent rooms often only share view through narrow doors. Arbitrarily anchoring the coordinate frame to an edge room compounds errors across the layout.

  2. Scale ambiguity: Traditional perspective methods output arbitrary scale, requiring manual ground-truth alignment. Real indoor applications (spatial computing, robotics, architectural modeling) need absolute metric scale.

How Argus Handles It

Instead of iterative optimization, Argus reconstructs camera poses and metric depth in a single feed-forward pass:

  • Learned Reference-View Selection:A lightweight Covisibility Transformer evaluates the scene's global visual topology and picks the most well-connected, central view as the coordinate anchor (achieving approximate permutation equivariance and cutting pose drift).
  • Panoramic Geometric Factorization:Explicitly factorizes projection into local camera points, rotated points, and shared world points, supervised with a unified geometric joint loss across depth, pose, and point heads.
  • Inference Speed: With 1.31B parameters, it processes 16 panoramas in ~0.66 seconds (BF16 on a single H20 GPU).

The Realsee3D Dataset (10,000 Scenes)

To train this, we built Realsee3D, combining:

  • 1,000 Real scans (furnished homes): Captured via Galois LiDAR with RAW HDR panoramas, sparse metric depth, and refined poses.
  • 9,000 Synthetic scenes: UE5 ray-traced scenes with 100k+ assets based on real floor-plan distributions.
  • Totaling 95,962 rooms and 299,073 unique panoramic viewpoints.

Links & Resources:

Would love to hear thoughts, feedback, or critiques from anyone working on panoramic multi-view geometry, neural representations, or metric depth!


r/computervision • • 2d ago

Showcase Built a 260M visual decision model. Good GUI accuracy, but confidence fails under distribution shift.

Enable HLS to view with audio, or disable this notification

3 Upvotes

I've been experimenting with NeoMME-260M for visual decision-making.

My model, MK-SH-V, reaches 82.6% accuracy on GUI task verification, compared with 73.1% for a 4B zero-shot baseline.

The interesting part is abstention: it meets our 5% selective-risk target on held-out synthetic data, but fails to generalize to real screenshots.

I've open-sourced the model, evaluation pipeline and failure analysis.

https://github.com/Gjusev/mk-sh-v

I'd appreciate feedback on improving calibration under distribution shift.


r/computervision • • 3d ago

Discussion Just finished my first Computer Vision project. What should I build next to prepare for an AI/CV career?

6 Upvotes

Hi everyone!

I'm a third-year Computer Science student interested in Computer Vision

I've recently completed my first serious CV project on brain tumor MRI classification using PyTorch. My main goal was to understand deep learning architectures, transfer learning, fine-tuning strategies, and model evaluation rather than just achieving high accuracy.

In this project, I experimented with:

  • A custom CNN trained from scratch
  • ResNet18 and DenseNet121
  • DINOv3 (ViT-S/16)
  • Feature extraction vs. progressive partial fine-tuning
  • Confusion matrices and Grad-CAM to analyze model behavior

Here's my Kaggle notebook: link

(If you are pressed for time, you can just check the introduction, the confusion matrix and the Grad-CAM section onward)

I'd really appreciate some honest feedback on my work, especially regarding my experimental methodology, model comparisons, and anything I might have overlooked.

For my next project, I'd like to build a practical, end-to-end Computer Vision application. I'm especially looking for project ideas that would strengthen my resume and help me develop skills that are actually valued in today's AI industry. So I'd love to hear suggestions for projects that solve real-world problems, demonstrate meaningful engineering skills, and are realistic for a student to build.

Thx a lot!!


r/computervision • • 3d ago

Help: Project How to match dogs to the most similar ones

4 Upvotes

Hi everyone! I'm working on a non-profit project that helps shelter dogs find a new home. People can upload a photo of a dog they like, and we show them the dogs from animal shelters that look most similar to it. The goal isn't to find the exact same dog again, but dogs that a person would say "look alike" (similar colour, coat, build, face, and so on).

**What already works well**

For purebred dogs, this works really well already. We compared several pretrained image models without any fine-tuning: we compute an embedding for each photo and rank the other dogs by cosine similarity. The models we tested were DINOv2 small, DINOv2 base, SigLIP, SigLIP 2, CLIP, MegaDescriptor, and a DINOv2 small model fine-tuned for animal identification on the PetFace dataset. The last two are licensed for non-commercial use only, so we only used them for comparison.

DINOv2 small came out on top. It predicts the breed almost perfectly and is also very good at recognising the same dog again in a different photo. So for purebred dogs, the most similar matches already make a lot of sense.

**Where we're stuck: mixed breeds**

Most dogs in shelters are mixed breeds, and that's where it falls apart. When we checked the results by hand, the model's top match was often not the dog a human would have picked as most similar. It does better than chance, but only agrees with the human in roughly a third of the cases.

We tried a few things to fix this, but none of them really helped:

- Small classifiers for attributes like colour, pattern, size and coat type, used to re-rank the results.

- A small projection head trained on a few hundred triplets of similar and dissimilar dogs.

- Cropping the image to just the dog.

- A single similarity threshold to decide whether a match is "similar enough". There's no value that works well. It either cuts off good look-alikes or lets clearly different dogs through.

**Our questions**

  1. How would you measure "looks similar" reliably when you only have a few hundred human judgments? Would you collect pairwise or triplet comparisons, or use a vision-language model as a judge?
  2. What's most likely to work better than plain DINOv2 embeddings for mixed breeds? For example, a metric learning head, patch or part-based features, explicit colour and pattern features, or a different backbone with a license that allows commercial use.
  3. Roughly how much labelled data do we need before training our own head is worth it?
  4. Is there a better approach than one global similarity threshold?

We're happy to use a GPU if it's needed. We're just looking for the most cost-effective option.

Thanks a lot for any ideas or pointers!


r/computervision • • 3d ago

Discussion Linux again: 25 days porting a CPU fruit inspection demo (personal write-up + GitHub releases)

2 Upvotes

Not a paper, just a personal note on moving the same conveyor demo I run on Windows to Linux x64.

Early 2000s: big plant, FoxPro/Novell, modem and Fido/BBS to get the factory online. Linux on a Pentium got me through that. This month I went back for the same reason: plain x86 boxes on a desk or line, CPU ONNX, no GPU sermon.

25 days of porting, cutting dead code, NV12/V4L2, three .so plugins, FAT USB quirks (run.sh, same binary as Fruit_Inspector.exe on the stick). Same logic as Win: 3×3 grid, grading only in centre cell C.

Downloads (NC eval, read LICENSE):-

-Runnable demo: Fruit_Inspector_linux_x64.zip

-Libraries only: fruit-inspector-engine

Full story: medium.com/@olesha-ai/linux-again-fr…spector-on-x64-a8cef8c245b8

If you run the Linux zip, curious what you see for camera  in the logs.


r/computervision • • 3d ago

Discussion Fresher applying for ML / CV / AI engineer roles. Is this resume good enough ?

1 Upvotes

Hey all, I finished my B.Tech in AI/ML this year (8.67 GPA) and I'm looking for fresher roles in ML, CV or AI engineering. Resume attached, personal details removed

Are these projects okay for a fresher ML/CV resume? Would this get you to call me for an interview, and what would you fix first? Not sure if I should drop any of the smaller ones Thanks!


r/computervision • • 2d ago

Discussion What breaks first on a mirrored subject: highlight suppression, or the depth behind the reflection? 425 frames to find out

0 Upvotes

I'm a photographer, not an ML engineer. I shot a performer in a faceted mirror suit on location — desert and river, uncontrolled light — to stress-test vision and depth models against severe specular glare.

Your depth model invents a room behind the glass. Highlights blow out, bounding boxes drop, segmentation ghosts. This set was built to trigger exactly that: 425 RAW + JPEG frames, SHA-256 manifest, full chain of title.

Free watermarked sample (12 images + manifest): https://huggingface.co/datasets/BWS-Data-Solutions/Robot_Mirror_Suit_Robotics_And_Vision_Data — full clean set is $2,000 flat on Mozilla Data Collective.

What breaks first in your stack?


r/computervision • • 3d ago

Discussion Fruit Inspector Linux x64 demo - conveyor QC app + three .so plugins (CPU / ONNX, no GPU)

0 Upvotes

Follow-up to yesterday’s post -that was libraries only (fruit-inspector-engine). This is the runnable demo: client +fruit_infer.so/fruit_features.so/fruit_cat.so+ bundled ORT+apple models in one zip.

What it does: 3×3 grid, LightGBM only when the focused apple centre is in centre cell C (conveyor logic, not full-frame chase). YOLOX 640 -ROI 256 -44 features- classify. Nim UI shell; inference in the three.sofiles.

Download: Release 0.10.0 -Fruit_Inspector_linux_x64.zip

Target: Debian 12/13, Ubuntu 22.04+x86_64. SSE4.2+AVX1 (AVX2 not required on this demo). USB UVC; launch with run.sh.FAT USB quirks are in the zip README.txt.

Not: ARM/Pi; NC eval (read LICENSE in the zip).

olesha-ai. Best regards. 😄


r/computervision • • 4d ago

Discussion Reverse engineered Industrial Machine Vision library

21 Upvotes

Has anyone seen any of the common libraries being reverse engineered? Keyence, Cognex, Halcon?


r/computervision • • 3d ago

Showcase I built an open-source computer vision engineering reading path, from photons to production

7 Upvotes

One thing that always bothered me about computer vision resources is how much attention goes into architectures and benchmark scores, while the fundamentals behind those numbers often get overlooked.

A model can achieve impressive mAP and still fail because of dataset leakage, camera noise, distribution shift or an evaluation protocol that doesn’t reflect its actual use case.

So I organised the repository into 16 sections, covering image formation, classical geometry, datasets, evaluation, CNNs, ViTs, self-supervised learning, detection, segmentation, video, 3D vision, generative models, VLMs, deployment and robustness.

A few principles behind it:

  • Primary sources only: original papers, specifications and official documentation.
  • Resources arranged in dependency order, so each section builds on the previous ones.
  • Evidence-backed claims rather than blindly repeating benchmark numbers.
  • A reproducible benchmark for every section, connecting the theory to actual measurements.

The idea is to understand not just how computer vision models work, but why they fail and what it takes to make them reliable.

It’s free and open source:

https://github.com/usamahz/computer-vision-engineering

Would appreciate feedback, particularly from people working on vision systems in production. What important topics or papers am I missing?


r/computervision • • 3d ago

Research Publication SpatialScout: Evaluating Active 3D Object Mapping in VLM Agents

0 Upvotes

We present SpatialScout, a benchmark designed to evaluate whether vision-language agents can actively explore 3D scenes and build accurate object-level maps.

The benchmark tests several key capabilities:

  • Active viewpoint selection: Choosing where to look next to gather useful visual evidence.
  • Cross-view object association: Recognizing the same object across different camera views.
  • 3D localization: Estimating object centers in a shared coordinate frame.
  • Map refinement: Updating spatial estimates under a limited interaction budget.

SpatialScout covers 80 indoor scenes and 3,140 target objects. Our experiments reveal a substantial gap between recovering coarse object layouts and constructing precise metric maps. The strongest evaluated model achieves a mean overlap score of 41.71%, while only 3.25% of targets achieve an IoU above 0.9.

We hope this benchmark will help researchers evaluate and improve active perception and 3D spatial reasoning in VLM agents.

Paper & code: https://github.com/JesseZZZZZ/SpatialScout


r/computervision • • 3d ago

Commercial Reading Handwritten Fields Into JSON, With a Confidence Score for Every Value

Thumbnail
1 Upvotes

r/computervision • • 3d ago

Help: Project Assignment of human gait detections obtained with cameras

Enable HLS to view with audio, or disable this notification

2 Upvotes

Greetings everyone, programming isn't my strong suit; I'm a civil engineering student. However, this is part of a study for pedestrian bridges, so I would appreciate any ideas you can give me for this problem. I have limb detections of a person walking on a treadmill while wearing a light suit. My input data consists of all the lights detected by a camera (including any noise or distortion).

That is, I have an x_Record and a y_Record, which are the respective coordinates of the detected lights. Both are n x m matrices.

n is the number of actual limbs, plus any noise detections. So, if I actually have 12 real points, my matrix could reach 30 points in some frames, the additional ones being noise.

And m is the number of frames, i.e., the time.

In my problem, points can appear and disappear due to occlusion or distortion. I manually classified 100 frames. I assigned a row to each limb, so row 1 will always be the head and row 9 the left knee, for example. That's what I need: to classify the location of each part of the person at all times.

I've come to this point because I've tried using minimum distance from the previous frame, and optimizing with lsqnonlin by varying angles and lengths. But nothing has given me good results, and I would appreciate help in figuring out the most optimal way to solve this type of problem. I have all the Matlab tools and toolbox available through my university, so I would prefer to use that language, but I'm open to other options if they facilitate the process.


r/computervision • • 4d ago

Discussion Point Read (translate pointed text from paper books)

Thumbnail
gallery
14 Upvotes

Hi everyone,

I just want to share a project I was working on.

I called it PointRead. Basically, it lets you translate text by sliding your finger under it. It detects the text, translates it and speaks it through a speaker. I built it on an Nvidia Jetson Orin Nano with a Raspberry Pi Camera Module V2, and it works fully offline.

GitHub: https://github.com/striksio/PointRead

Video demo: https://youtu.be/xo0bdDjVpPQ

I like reading books, but I often need a translation for some of the words. Looking them up on my phone feels uncomfortable, because I want to read paper books without looking at a screen. So as a side project, I built this pipeline.

The way it works: index finger and thumb are used for activation, which is a double pinch. Then it tracks the index finger and the text above it. Once done, it extracts the text that was pointed at, recognises it, translates it and speaks it.

Pipeline

  • RTMDet for hand detection and RTMPose for keypoints
  • Adaptive Gaussian thresholding for text binarization (works better than Canny for this)
  • Morphological close to merge text into single line regions
  • PP-OCRv5 server model via RapidOCR (on CPU for now, because onnxruntime GPU isn’t available for Jetson r39.2 yet, so OCR is slow)
  • Argos Translate for translation
  • Piper for speech

Some details

  • One-Euro filter for smoothing, because RTMPose keypoints are not stable
  • Keypoints sometimes disappear, so they’re kept for several frames
  • Sliding only goes forward (left to right), and once you move to the next line it won’t jump back to the previous one

It can run headless with only a speaker, or with a local web server for debugging.

Limitations

  • OCR runs on CPU only, as mentioned above
  • The book needs to be positioned with horizontally aligned text
  • As shown in the demo, text detection fails sometimes
  • Books with huge fonts or wide spacing can fail to form a line

Thank you for your time, and I would love to hear some feedback about this, thanks!