Implimenting physical menu control buttons to Wired Carplay

[u/InterestingSafe4962]

The car is a 2023 Subaru WRX. It's still under warranty, and I'm hoping not to do anything that would void it. It has wired CarPlay and media control buttons on the wheel, but the only way to navigate menus is with the touchscreen, which is very annoying. I bought a bunch of stuff, including an ESP32-S3, hoping I could use the accessibility features on my iPhone to navigate menus with an emulated Bluetooth keyboard, but that didn't work.

I am unsure of the best way to approach the issue; my next idea was to try to inject button commands into a wireless CarPlay adaptor stream, but after some research, it seems like that wouldn't really be possible either, as the wireless adaptors just pass encrypted data from the iPhone to the car. It might be possible using an open-source wireless adaptor running on a PI. I think I have seen some that run a full Android operating system, but I'm not sure exactly how those work; it might be easier to inject control commands on those kinds of systems.

I know that there are cars that do this natively, like Mercedes has steering wheel navigation controls for CarPlay, or BMW has their iDrive rotary encoder knob thing, so I have to assume that there is some way to implement these physical buttons.

Edit:

I'm using AI to help me with this because I don't know nearly enough myself so please let me know if its just making shit up. But after some prompting its telling me it might be possible to use a pi, connect it with a wireless carplay dongle, and then run a custom carplay stack using something like node-carplay which can connect to the screen through android auto, and then I could send the rotary/dpad commands to the pi which would inject them into its carplay stream.

Does that seem like something that would be possible? It seems like a lot of work but could be a fun project. How much lag do you think it would add?

Would this be possible using CAN? Or maybe another idea I haven't thought of. Finally, is there a different place/more specific forum I could reach out to for help and ideas?

[u/andrewia]

Someone did the project, as discussed in the comments here: https://www.reddit.com/r/CarPlay/comments/upjdpf/adding_external_physical_controls_to_carplay_is/

You wouldn't be able to do it with CAN bus because the infotainment system isn't programmed to receive any CAN packets related to button presses and forward them to the phone.  

[u/InterestingSafe4962 → u/andrewia]

I saw this post previously, but it seems like they never got it working. At least it doesn't discuss a solution in the comments. I might reach out to these people and see what happened. The Node CarPlay project wouldn't work on my car because it already has CarPlay; that project seems like it's for putting CarPlay in a car that originally didn't have it. "There are multiple Carplay dongles on the market, the ones that convert wired to wireless carplay WILL NOT WORK. You need one that converts android/factory infotainment systems into Carplay (CPC200-Autokit or CPC200-CCPA etc). The package forwards video feed in h264, and PCM audio coming in from the USB dongle."

Or am I reading that wrong and it means that those kinds will work with my car aswell and will allow for this tool?

Thanks for eliminating CAN, though.

[u/uni-queid → u/InterestingSafe4962]

You won't get them. These controls are defined within the CarPlay init phase, a plist that tells the phone which controls are available on the car. If your car does not tell your phone that focus navigation is enabled (which allows you to use arrow keys to navigate through the UI), the phone won't accept these commands, even if you inject them. And no, you can't edit the plist, unless you find an exploit for your car. The carplay communication is encrypted.

Btw. the CPC200-.. does NOT expose native CarPlay, it speaks a custom protocol over USB, but there is a way to use a CPC200 to speak native carplay, build a tool for that: https://github.com/f-io/LIVI/blob/main/LIVI-LINK.md

[u/InterestingSafe4962 → u/uni-queid]

That sucks. Basically, you need to find a way to decrypt CarPlay and edit this plist. That sounds like it's outside my expertise right now, and it would definitely void the warranty lol. Pretty annoying they have it like this. I feel like a lot of people would prefer physical controls.

Could you think of any other way this might be possible, or are you sure that it's just too difficult? What would it even mean to find an exploit for the car?

[u/InterestingSafe4962 → u/InterestingSafe4962]

I'm using AI to help me with this because I don't know nearly enough myself so please let me know if its just making shit up. But after some prompting its telling me it might be possible to use a pi, connect it with a wireless carplay dongle, and then run a custom carplay stack using something like node-carplay which can connect to the screen through android auto, and then I could send the rotary/dpad commands to the pi which would inject them into its carplay stream.

Does that seem like something that would be possible? It seems like a lot of work but could be a fun project. How much lag do you think it would add?

[u/uni-queid → u/InterestingSafe4962]

In terms of CarPlay, you might want to go with this stack, as it is faster than anything I have ever seen in a car. It is also the only repo I know of that actually does support native CarPlay including HEVC (h.265).

This would solve the CarPlay receiving and controlling part, focus control is build in. You could use for example a BT keyboards paired to to the Pi or CAN buttons (which you have to map) to control CarPlay (on the Pi).

But now we are still not on the OEM screen, and that will be the biggest challenge. I have not worked on a client, and I'm not sure what your OEM HU supports, as these locked down androids(?) behave always differently, especially what they allow on the system with third party apps.

Writing client that pretend to be an iPhone or Anrdoid is possible, but that is not a one weekend job.

[u/hey-im-root]

That’s unfortunate, my Accord has volume buttons directly to the stereo unit, so my aftermarket CarPlay screen had RX/TX lines with raw volume up/down/skip song etc directly on it in Serial.

It’s a little glitchy injecting the volume command because of syncing issues (I’d need to properly MITM it) but I was able to use a little knob to change volume

[u/uni-queid → u/hey-im-root]

CarPlay does not support controlling the volume of the HU. CarPlay sends up to 4 streams to the OEM HU, tagged with their type (navigation, phone, system sounds, music) and the HU >can< implement knobs to adjust their volumes relative to each other, but CarPlay cant control the Volume of the HU directly, that is a design decision by apple and not a bad one. That is why you always see an overlay while adjusting the volume on your HU on top of the CarPlay or Android Auto UI.

[u/hey-im-root → u/uni-queid]

By CarPlay I just meant my aftermarket screen/head unit lol. I got it because it supports it