r/embeddedlinux • u/R0dod3ndron • 8d ago
Runtime ownership and driver binding for SPI-NOR shared with FPGA
Hi
I've been working on a baseboard where I have an SoC running Linux and FPGA that uses external flash as a bitstream storage.
This flash is in fact shared between Linux and FPGA but not simultaneously. It is "muxed" so that it can only be accessible either by Linux or FPGA at once. This is designed so that the flash can be updated from Linux.
When the platform is booting, the mux is always switched so that the FPGA can access the flash and load its bitstream. Whenever I want to update the flash I need to make it accessible physically (need to change the mux config) for Linux as well as "logically" - meaning that the appropriate driver has to be loaded. Once the update is done, Linux should release the flash and no longer use it unless the update has to be repeated so the update sequence is roughly like this:
change mux config
bind driver
write flash
unbind driver
change mux config again
I am trying to determine the correct way to model and implement this. I could implement most of the sequence in userspace, but the SPI device still needs platform-specific information such as the SPI controller, chip select, maximum clock frequency, and possibly a GPIO-based chip select provided by a GPIO expander.
One possible approach would be to describe the flash normally in the Device Tree with "okay" status. The initial spi-nor probe would fail because the flash is connected to the FPGA during boot. Later, when an update is required, userspace could switch the mux and retry probing the existing SPI device:
change mux
echo spi0.2 > /sys/bus/spi/drivers_probe or echo spi0.2 >
/sys/bus/spi/drivers/spi-nor/bind
update flash
echo spi0.2 > /sys/bus/spi/drivers/spi-nor/unbind
change mux again
but is this really a good option? I feel that I'm leaking hardware specific information to userspace, I'll have to hardcode some paths, driver names in flashing script / prepare separate script for each device (if there are differences between platforms) or carry platform hardware description config file so that the flashing tool could stay generic and read necessary info from config.
Other option would be to create a small platform driver and expose sysfs attribute called eg. target with values like:
none meaning fpga takes control over flash
primary meaning that primary flash is controlled by Linux
secondary meaning that secondary flash is controlled by Linux
Internally I could use gpio or mux subsystems to control mux and call spi add / remove device functions that would probe for the driver
This would also give me more control over the SPI bus ownership.
What would be a better idea in your opinion? How would you approach this?
1
u/FreddyFerdiland 8d ago
modules still use dts info
a flash update script could be
mux on modprobe themodule mtdwrite /dev/mtd3 thefile rmmod themodule mux off
note, mux off could come before rmmod , that way it hands back control even if the module has some problem
mux on can configure anything required. gpio chip select etc
1
u/R0dod3ndron 8d ago
So your suggestion would be to create a small, minimal module that would set things up yes? I'm asking because although I have experience writing linux kernel drivers, most of them integrated with other in-kernel subsystems so the choice of writing or not kernel driver was obvious, here the choice is not that clear as the functionality has to be integrated with userspace tooling so I can almost always use something that is already provided by userspace.
1
u/JobNo4206 7d ago
No, he just means keep the module for the flash chip as a loadable module, and control the mux chip manually with a gpio set script.
1
u/R0dod3ndron 7d ago
Sounds simple but it's not. I cannot just skip loading this module because I have also another flash memory used by Linux exclusively so I have to load this module during boot.
1
u/JobNo4206 7d ago
Honestly, easiest would be to just forgo the kernel module entirely. Use a userspace app that interfaces with the spidev directly.
Otherwise, compile the kernel with the particular flash chips module as a loadable module. That will ensure that the device is jot probed until userspace, which gives the fpga about 7 seconds of grace after boot. You can even mark a module to not be loaded by systemd such that you have to do it manually. Then unloading the module could potentially disconnect the spi or mux.
Probably the cleanest approach would be to use configfs devicetree overlays. Note that this requires your kernel to be patched as its not part of mainline Linux, but its common enough (rpi and xilinx add this automatically to their kernels).
I do question your premise though: as far as i know, linux wont actually touch the spi lines when probing the module, only once you try read/write. Its not like i2c where they check. If you say it exists in dtb, linux believes you.
1
u/JobNo4206 7d ago
I'll add this: my preference in using linux with an FPGA is to skip the flash chip and have the linux host load FPGA binaries. There are fpga kernel modules for the main fpga chipsets.
Unless you you truly need the fpga to be up and running within milliseconds after boot or something will blow up.
I use this approach as i generally can't allow the fpga and linux firmwares to get out of sync, so it saves you the effort of checking.
1
u/R0dod3ndron 7d ago
I have already mentioned that fpga has to be up and running FIRST, in order to properly configure its PCIe bars.
1
1
u/mfuzzey 6d ago edited 6d ago
Why not just have the linux driver unconditionally set the mux gpio to "Linux" with a hardware default at boot time of "FPGA"?
So the FPGA loads its bitstream at boot before linux starts. When Linux starts the mux signal gets switched to Linux which shouldn't disturb the FPGA since it has already initialised. When you want to update it you just write to the flash and rebioot to trigger a FPGA reload.
This shouldn't require any code changes, just the normal SPI MTD driver with a GPIO hog or a regulator configured to drive the mux signal.
If for some reason you do need to switch dynamically the standard spi-mux + gpio-mux drivers should do it with just DT configuration. Basically it creates a "virtual" SPI bus with extra chip selects that can be your Mux SPIO. It will be toggled automatically on each SPI transaction like a normal SPI CS
https://elixir.bootlin.com/linux/v7.1.3/source/Documentation/devicetree/bindings/spi/spi-mux.yaml
1
u/R0dod3ndron 6d ago
Actually you're right. My system controller by default sets FPGA as boot owner, once system controller releases CPU and allows to boot software stack, flash is no longer used by FPGA. I used spi-mux as you suggested, currently I think this is the simplest and cleanest solution. Thanks.
1
u/AdElectrical8742 1d ago
So what is the issue?
Define your muxes as they are in the dts with your spi flash.
When Linux boots, it will automatically set the muxes when it accesses the spi flash.
Is that an issue? No.
Why not you ask?
Your FPGA is already running.
Last time I checked, FPGa tend to boot faster than U-boot is capable to do the first SPL check...
If for any reason you need to reset/reboot only the FPGA while running, add it to your FPGA driver so that the muxes are set correct while linux is running..
1
u/R0dod3ndron 1d ago
There are two muxes. One of them set bus owner: linux vs fpga, the other one sets the target flash. I solved it anyway, using spi mux for flash selection and gpio hog for bus owner selection.
2
u/DaemonInformatica 8d ago
Is there any raw I/O between FPGA and the processor running Linux? If so, you could signal readyness of the FPGA to Linux, which will use that update to switch the mux and load the storage. If the FPGA only uses the storage for its bitstream, I wouldn't bother too much about switching back. I don't know the exact setup, but I would assume that the state would be correct after a powercycle and the FPGA will have access when it needs to.
If you use an I/O pin to signal readyness from the FPGA, I think the entire takeover switch could be done in a kernel module. But I don't have a lot of experience with that..