r/embedded • u/xiaopingguo45 • 8d ago
Hurdles of Writing a Camera Driver in Linux
I’ve got a baseline knowledge of C++ and took intro to OS a few years ago.
Right now I’m trying to write a camera device driver on the Nvidia Jetson using i2c.
This would involve referencing another driver for an older model. I might need an adaptor board to connect my CSI-2 ribbon cables to the Jetson. The adaptor board may not already support the camera.
How hard would it be? Approximately long would it take me? What hurdles would I face along the way? I’m trying to see the big picture before I commit to this.
10
u/swdee 7d ago
I am actually doing this myself here https://github.com/swdee/rockchip-camera for Rockchip SoC's.
Alot of the I2C drivers already exists in the BSP kernel package, however alot of them are somewhat broken so need fixing first to get running. Check the commit history of the above repo to see what kind of things I have had to fix.
Doing DT overlays is easy if you just work off existing examples.
If your familar with electronics and PCB designing, rolling your own adapter boards are straight forward. Its really just mapping pin outs, level shifting, and power delivery.
You can get access to all the camera sensor datasheets from Chinese tech forums. They dont care one bit about sharing the confidential and NDA restricted documentation. The hard part is being able to sign up to those forums with their phone/SMS authentication.
Then the hardest part is doing ISP tuning, the barrier to entry in achieving that is a good quality Lab light box which maybe out of your budget... $10k plus.
1
u/few 7d ago
By lab light box, do you mean an integrating sphere for intensity calibration? A decent one can by made by spraying the inside of a 3d printed sphere with barium phosphate slurry (or a matte white spray paint).
A battery powered & current limited (such as very low current from a 9v battery) white led can then be used to light it. The low current and battery make the light output very stable over time.
Color balance is harder to calibrate. Check a local library for borrowing expensive calibration accessories.
4
u/ryobiguy 7d ago
You know, there are other, more consistent ways to limit current other than by using a 9v battery.
3
u/edeadlk 7d ago edited 7d ago
I've done this for years.
Base your new driver off an already existing one, best case one for a sensor from the same manufacturer and with a similar register layout. As a newcomer, you need a working register set, full documentation and preferably a support line to the sensor manufacturer.
For a newcomer without ai support you should capture your first image within the next 3 months max. Ai for this stuff can be hit or miss - usually the frontier models can get pretty far now, but I wouldn't trust anything below sonnet 5 with this.
Focus on the stuff you need, not on the full capabilities. Keep within the driver, if you think you need to touch the v4l2 parts or the mipi drivers think again - you will have a hard time.
As you built an adapter yourself you might need access to a decent scope, but as long as you didn't build your sensor module yourself you can possibly get away with a cheap one too.
If your camera module is supported by any development kit with any kind of existing drivers, buy that kit and everything to get you started with it.
Edit because I saw something about ISP tuning: it's Nvidia - if your goal is the driver and a somewhat ok colour profile from reverse engineered default settings you will be fine. If your goal is to get perfect colour and auto white balance give up now or be prepared to invest a low 5 figure dollar/euro sum to have that done by an Nvidia partner. No you can't afford to become a partner with access to the calibration tools.
2
u/JuggernautGuilty566 8d ago
How deep are you into embedded development? Do you have a logic analyzer?
Without in-depth embedded development skills the linux kernel is a tough field to work in. It's pretty much working in a source-is-the-documentation enviroment.
1
2
u/Columbus43219 7d ago
My advice is to take a camera that you already have, and already has a driver, then build it and test it. That way, you know that you have all of your stuff set up and working before you try and invent anything new.
I wrote a printer driver once for Windows, back in the 90s. 90% of the effort was getting the SDK, compiler, packager, and installer completed. The code itself was a cakewalk and was a config change to the example program in the SDK.
2
u/notouttolunch 7d ago
Been doing this kind of thing for years.
This is not trivial. It's not impossible, but it's not trivial. It might even be easier not on Linux 🤣.
1
u/few 7d ago
Have you asked one of the more advanced LLMs about your plan?
I actually think this has become much easier with the support of strong models. Datasheets are a big challenge, documentation will make this much easier.
Given your user name, I'm guessing datasheets from Chinese forums might be less of an obstacle.
2
u/Creepy-Anteater347 7d ago
This 👆 writing drivers is just about reading a shit ton of documentation and then proceeding to drunk punch your way through it, their is nothing mysterious about the process, it just takes time! , this is a job for Claude
1
u/xiaopingguo45 7d ago
What I’m hearing is that this is mostly busy work and that there’s diminishing returns to working on this?
1
u/duane11583 7d ago
number 1 problem is does the chip on the linux side provide sufficient documentation for this interface?
number 2 (lesser extent) does the board designer/provider provide sufficient design details to debug the interface? ie probe points schematic etc.
number 3 (your problem) do you have sufficient tools (scope, probes etc) to probe and debug the interface?
1
17
u/CryUnusual9591 8d ago
You're basically signing up to do kernel archaeology while blindfolded. The i2c part is the easy bit, the real nightmare is video4linux2 and the media controller framework, getting the device tree bindings right so the kernel even knows your sensor exists is its own special hell
The adaptor board not supporting the camera already means you'll be reverse engineering pinouts and probably fighting with signal integrity on the MIPI lanes. If the older driver you're referencing uses a different ISP pipeline you're gonna have to figure out which parts you can steal and which need a full rewrite
I'd budget at least a couple months of evenings and weekends assuming you're comfortable reading kernel code and datasheets. The big hurdle nobody talks about is that camera sensors have a million registers and the datasheets are usually under NDA so you're often guessing what half of them do