IMX477 Sensor <-> Jetson Orin Nano
ISP issues
This page is dedicated the issue I had regarding the communication between my IMX477 sensor and my Jetson Orin Nano Super. This is an extension of an issue encountered in my UAV project.
Here is the relevant nvidia forum thread if interested: Forum Thread
The intial error was seen qualitatively, where the bottom ~2/3 of the frame was noticeably delayed.
If you gave this frame another second you would see the bottom section slowly render in.
The camera I am using is an Arducam B0274 (IMX477). The issue I was having was not a display artifact, the saved stills also had this visual issue.
I ruled out the first issues which areeasy to test: ribbon orientation, cable (using continuity via multimeter), IR-cut filter, JPEG encoder, both CSI ports on the Jetson, and also looked into a known JetPack 6.0 corruption bug which NVIDIA had.
Everything was checking out.
However (as you can see in the thread) "david.monge" wrote suggesting to capture a scene in 3 ways:
ISP : Image Signal Processor, dedicated silicon on the Jetson that turns raw sensor output into a usable image. It does debayering (Bayer mosaic -> RGB), auto exposure, white balance, noise reduction and sharpening, all in hardware, so the CPU and GPU stay free from it.
Some debugging was made. 2-lane was stock settings. I experimented with switching it over to 4-lane by configuring .dtbo (device tree overlay). Essentially num_lanes tells the Jetson's CSI receiver (where your camera ribbon goes in) how many MIPI (Mobile Industry Processor Interface) data lanes to listen to.
Overlay: overlay in this context is our device tree which is a data file the Jetson's kernel reads at boot to learn what hardware exists, and how it's wired ( which bus the camera sits on, which CSI port, how many MIPI lanes, etc.). An overlay is a small patch applied on top of that base tree at boot, rather than a rewrite of it. Arducam ships a .dtbo that says "an IMX477 is on CSI port 0, num_lanes=2."
MIPI CSI-2 carries pixel data over N differntial pairs. More lanes = more bandwith = higher resolution or framerate.
num_lanes only configures the receiver, the sensor (our IMX477 sitting in our camerea) decides how many lanes it actually transmits on.
So when I set num_lanes="4" when our sensor was actually sending 2, our image was then being processed completely wrong. I did revert it, but I reverted to the wrong overlay, which I only found out much later.
So that was one of the issues.
The ISP file
The main driver of this issue was our ISP file. The camera_overrides.isp I had was built for a Xavier, from a generic "mobile" example. Not an IMX477, not an Orin. Meaning our ISP was never broken, it was just communicating with the wrong settings.
I didn't pick that file. It came bundled with the Arducam driver package (the driver is the code that lets the OS talk to a piece of hardware):
wget https://github.com/ArduCAM/MIPI_Camera/releases/download/v0.0.3/install_full.sh
chmod +x install_full.sh
./install_full.sh -m imx477
We had to explicitly fetch the correct one instead:
wget https://raw.githubusercontent.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3/master/li-camera-calibration-files/camera_overrides.isp
The url says RidgeRun, but li = Leopard Imaging. RidgeRun just hosts it.
That fixed the image.
The second issue
But the sensor modes were still broken. sensor_mode=2 returned EINVAL, asking for 1080p got silently forced to 12MP, and asking for 4K hung on STREAMON and wedged the capture path until reboot.
Remember the 4-lane revert. I put it back to tegra234-p3767-camera-p3768-imx477-dual-4lane.dtbo, which is what it was on originally. But that is the wrong overlay for this camera. The correct one is tegra234-p3767-camera-p3768-imx477-dual.dtbo, no 4lane.
Fix was reinstalling the Arducam package and re-picking the config in jetson-io as "CSI Camera IMX477 Dual".
All three modes register now: 12MP@21, 4K@30, and 1080p@60.
Conclusion
ISP overrides settings were programmed for a different chip. Shouldn't have used the calibrations that came bundled with the Arducam driver, needed to fetch it separately.
Both bugs came from the same place though. There is no Arducam package for the Orin Nano at my L4T version (36.4.3), their Nano builds stop at 36.2.0. So install_full.sh grabs the only build matching my kernel, which is t234-nx, built for the Orin NX. Wrong ISP calibration and wrong default overlay, both downstream of that.