A universal MIPI camera FMC PoC
Jeff Johnson, Owner and Lead Designer, Opsero
Vision has always been one of the most compelling and natural application areas for FPGAs. Their ability to perform massively parallel computations and interface directly with high-speed image sensors makes them ideal for real-time image processing. Yet despite these strengths, it is still difficult to find modular solutions for connecting FPGAs to image sensors when it comes to rapid prototyping and experimentation.
In an ideal prototyping or Proof-of-Concept (PoC) scenario, a developer would simply pick an FPGA board and an image sensor and connect them through a standard interface. While many FPGA platforms can interface with USB cameras which are often sufficient for low-bandwidth applications, most vision systems demand a higher-performance link. For these, the Mobile Industry Processor Interface Camera Serial Interface 2 (MIPI CSI-2) has become the most widely used interface between image sensors and FPGAs.
MIPI CSI-2 delivers high-bandwidth pixel data over just a few differential pairs, making it ideal for compact, high-speed camera links. Each lane can run at multiple gigabits per second (Gbps), and with up to four lanes per sensor, CSI-2 can easily support multi-megapixel, high-frame-rate video streams. Its low pin count, excellent signal integrity, and broad sensor compatibility have made it the standard for everything from smartphone cameras to industrial vision modules.
I’ve long felt that there should be an FPGA Mezzanine Card (FMC) based solution for connecting MIPI CSI-2 image sensors to FPGA development boards.
A few options do exist including the Opsero RPi Camera FMC but these are essentially custom designs, each tied to one or two specific carrier boards rather than being broadly compatible. This runs counter to the goal of the VITA 57.1 FMC standard which is to modularize I/O and make hardware interchangeable across platforms.
So why isn’t there a universal MIPI CSI-2 FMC on the market? The answer lies in pin restrictions. The AMD MIPI CSI-2 RX and equivalent implementations from vendors such as Altera and Lattice, requires specific FPGA pins that support the electrical and timing characteristics of the MIPI D-PHY physical layer standard interface. The FMC standard, while it defines the locations and electrical specifications for I/Os, clocks, gigabit transceivers and power supplies, does not constrain pins at the level of the byte-group or specialized functions which are important for MIPI implementations (e.g., Dedicated Byte Clocks (DBCs) and Quad Byte Clocks (QBCs) in AMD UltraScale/UltraScale+ devices). As a result, a MIPI CSI-2 FMC can be designed to match one or two FPGA carriers, but it’s not possible to assign the pins to make it broadly compatible.
The solution
A different approach is to break that limitation by adding a small FPGA directly on the FMC card. This way, the MIPI CSI-2 pin assignments are fixed and can be chosen to suit the mezzanine card’s FPGA. On the other side, the interface with the carrier board’s FPGA can be a general-purpose, easily reproducible link such as Aurora, ensuring compatibility with a wide range of carrier boards. The image in Figure 1 illustrates the concept and uses the Artix UltraScale+ for the mezzanine FPGA and two Raspberry Pi cameras for the MIPI CSI-2 image sensors:
Here, the AXI Chip2Chip IP core is used to configure and control the processing blocks. This enables an AXI4-Lite interface to be tunneled between the two FPGAs over a serial Aurora link, effectively extending the control bus across the FMC connection.
At the end of the video pipeline, a Frame Buffer Write IP typically stores pixel data into Double Data Rate (DDR) memory. In our design, the DDR memory resides on the carrier board not the mezzanine.
The AXI Chip2Chip IP bridges this gap again by passing an AXI4 memory-mapped interface across the same Aurora link, allowing the mezzanine’s video pipeline to write directly to the carrier’s memory subsystem.
The PoC
To demonstrate the concept works as expected, we built a demonstration platform (Figure 3) with an AMD ZCU106 Multi-Processor System-on-Chip (MPSoC) serving as the main board and a Tria AUBoard standing in for the mezzanine card. The AUBoard hosts the RPi Camera FMC, allowing two standard Raspberry Pi cameras to be connected. The Aurora serial link is implemented over an SFP+ DAC cable connected between the two boards.
Unifying the software platform
Even with most of the hardware architecture defined for a truly universal MIPI CSI-2 FMC and a prototype to prove the concept, one important factor remains: how to make the main FPGA aware of the hardware that resides in the mezzanine FPGA.
In a typical single-FPGA system, this isn’t an issue. Software tools such as the Vitis IDE already know the details of the hardware design because the information is passed on via a hardware handoff file (XSA). In multi-FPGA designs, however, the situation is more complicated. Each FPGA has its own hardware design and its own XSA file. The software environment usually references only the XSA from the main FPGA, leaving the tools in the dark regarding any IP or peripherals instantiated in the secondary FPGA. To develop a coherent software platform, we need a way to describe the remote hardware like its IP blocks, base addresses, interrupts, and required drivers so that the tools can treat it as part of the overall system.
This is where AMD’s System Device Tree (SDT) flow comes in. Introduced in the 2023.2 release of the Vitis Unified IDE, SDT provides a way to describe externally connected hardware using a device tree. Exactly what is needed to overcome the final hurdle in a universal MIPI camera FMC design.
The basics of the SDT flow
In Vitis Classic, the build system extracts the hardware metadata directly from the XSA file to generate the Board Support Package (BSP) and supporting files such as xparameters.h, the linker script, and the driver config files. In the Vitis Unified IDE, this process changes significantly. The tools now begin by creating an SDT, a structured, open-source hardware description that serves as the foundation for both Linux and bare-metal domains. Hardware metadata is no longer parsed directly from the XSA. Instead, it is extracted from the SDT using a tool called Lopper, which reads device-tree nodes and their properties to populate driver and library configurations.
The advantages of the SDT flow
The SDT-based flow provides several important advantages. Firstly, it’s built on open standards, using Device Tree, CMake, and Lopper to create a more transparent and maintainable build system. With enhanced hardware description, it also allows additional hardware elements such as I²C devices or AXI Chip2Chip links to be represented even if they are not defined in the XSA file. As a result, the build system can become fully board-aware. Finally, the low-level interrupt controller setup is abstracted away, simplifying interrupt and sleep handling. Instead, interrupt IDs and related parameters are automatically made available through SDT-populated config structures.
Key differences for bare metal developers
From a user perspective, several practical differences stand out when working with the new SDT flow compared to the legacy flow:
- The DEVICE_ID macro is no longer generated or used. Devices are now identified and initialized by base address (BASEADDR).
- Interrupt setup is greatly simplified through the helper function XSetupInterruptSystem(), provided in xinterrupt_wrap.h.
- Driver binding uses YAML files instead of MDD files. Drivers are matched to hardware using the compatible strings defined in the SDT, resolved by the Lopper framework.
- Config structures (typically generated in *_g.c files) are populated from SDT node properties like reg, interrupts, and vendor-specific attributes instead of from xparameters.h.
Adding the video pipeline with a User DTS overlay
The SDT flow enables us to add the hardware details of a video pipeline running on the AUBoard by supplying Vitis with a User Device Tree Source (DTS) overlay that contains the nodes for all the elements in our video pipe. To help determine what this overlay should look like, a Vivado project is created for the main board and the video pipe elements are added to the design. After generating an XSA file from this project and importing it into Vitis, the tools automatically create a System Device Tree that includes nodes for each of the IP blocks in the video pipe. Once the SDT is generated, the device-tree nodes of the video pipeline can be copied and pasted into the User DTS overlay. Finally, we need to include each of these devices in our memory map.A simplified example of such an overlay is shown in Figure 4. Note that the first part involves adding the nodes to amba_pl, while the second part involves adding devices to the address-map. Since there is no way to cleanly append to the address map, we first delete the address-map property and redescribe the entire address map with the additional devices included.
At this stage, we can create a Vivado project for the AUBoard, playing the role of the mezzanine card in our PoC. The block design will include the video pipeline IP and the AXI Chip2Chip IP. Note that it does not require a processor since it will be controlled remotely by the main board through the AXI Chip2Chip IP. It is important to ensure that the base addresses assigned to the IP match the addresses specified in the User DTS. These addresses should be chosen based on the main board’s memory map and the available address space in that design. Once the design is built, we can program the AUBoard either via JTAG using Vivado Hardware Manager or by storing the bitstream in flash memory for standalone boot.
To complete the system, we create a new Vivado design for the ZCU106 that includes the processor (PS) and the AXI Chip2Chip IP core. From this design, we generate an XSA file and use it in Vitis to create a new platform component.
In the Advanced Options, Vitis allows us to specify our User DTS overlay. When the platform is built using both the XSA and the User DTS, Vitis automatically pulls in the drivers and generates the configuration data for all the IP blocks defined in our video pipeline.
This means that when we begin developing the bare-metal application for the ZCU106, the build system already recognizes the video pipeline running on the external FPGA (the Artix UltraScale+ of the AUBoard). When the AUBoard is connected to the ZCU106 via the SFP link, the complete system becomes operational. We can now develop and debug it as a single system from Vitis through a JTAG connection to the ZCU106 board.
Conclusion
As FPGA designers, we are all familiar with single-FPGA systems but some problems are now best solved through multi-FPGA architectures, although they bring additional challenges. It’s important to recognize when this approach is appropriate and to understand the tools that can help us to work with these systems. As we’ve seen, the Vitis SDT flow provides a clean and maintainable way to deal with multi-FPGAs, offering a path for simple integration of an FMC with a carrier board. If you would like to reproduce the PoC, the source code is available at fpgadeveloper.com.
