Working Principle and Workflow Summary of RGB-IR Sensors

16 Jul, 2024
- Audio and Video

Generally, the vast majority of the IP Camera (IPC) industry adopts a working mode of full color during the day and black-and-white for night vision. This solution uses two sets of filters (one for day and one for night) and uses the SoC’s GPIO to control the IR-cut switcher (IRCUT) mechanical component to switch between them: in day mode, the ISP is set to full-color mode, the IRCUT switches to the daytime filter, the IR LED is turned off, and a color image is output; in night mode, the ISP is set to black-and-white mode, the IRCUT switches to the nighttime filter, the IR LED is turned on, and a black-and-white image is output. The main difference between the day and night filters is that the daytime filter has a narrower bandpass range to filter out more infrared light for more realistic color reproduction, while the nighttime filter has a wider bandpass range to allow more infrared light in, enabling the infrared light emitted by the IR LED to pass through for illumination.

Of course, there are exceptions, such as the emerging “night-vision full-color” or “starlight full-color” solutions. On the ISP side, AI-ISP can be used to achieve better noise reduction. However, most importantly, the lens and image sensor must also support night-vision full-color combinations. The larger the lens aperture, the better (F1.2 or even F1.0), and the pixel size of the image sensor needs to be 1.8 μm or larger. As a result, the cost of this optical solution is very high—typically $6–$7 for a 2-megapixel sensor, and around $10 for a 2.5K ultra-HD solution.

Compared to the Bayer sensors used in general cameras, the biggest difference of the RGB-IR image sensor is that after light passes through the filter, the image generated by the sensor contains not only the three primary RGB colors but also IR image data. This is equivalent to the sensor simultaneously outputting both a color image and a black-and-white IR image to the backend ISP and encoder, eliminating the dependency on the mechanical IR-cut (IRCUT) switcher.

RGBIR vs Bayer

Currently, the vast majority of image sensors used in various cameras on the market output image data in Bayer format to the ISP for further processing.

In a Bayer image array, every four pixels form a pattern block, containing one blue (B) pixel, one red (R) pixel, and two green (G) pixels.

In contrast, the image array of an RGB-IR image sensor is shown below:

In an RGB-IR image sensor, a single pattern block consists of a 4x4 array (16 pixels). This includes two red (R) pixels, two blue (B) pixels, eight green (G) pixels, and four dedicated IR pixels that are sensitive to the infrared spectrum.

In other words, in an RGB-IR pixel array, the number of red and blue pixels is sacrificed to introduce dedicated IR pixels that sense the infrared spectrum.

Prerequisites for Supporting RGB-IR

Image Sensor

First and foremost, you must select an image sensor that supports RGB-IR. As mentioned above, the vast majority of CMOS image sensors on the market output image data in Bayer format. Therefore, to build an RGB-IR imaging system, you must first choose an image sensor capable of outputting RGB-IR format image data.

Optical Filter

For a standard Bayer-format camera, the filter used during the day only allows light in the visible spectrum to enter, while the filter used at night only allows light in the infrared spectrum to pass. That is to say, both the day and night filters are single-band bandpass filters.
For an RGB-IR image sensor, however, because both visible and infrared light bands must be filtered and captured simultaneously on the sensor, the companion filter must be designed as a dual-band bandpass filter. As shown in the figure below, it simultaneously allows visible light (400–650 nm) and infrared light (800–950 nm) to reach the sensor.

ISP

On the ISP side, the video input (Vin) interface of the ISP must support correctly receiving and parsing the image data transmitted from the RGB-IR image sensor, separating the combined RGB and IR image data into two different image frames. Furthermore, the ISP must be capable of processing the RGB image frames and IR image frames simultaneously. It should also feature an algorithm module to remove IR contamination (crosstalk) superimposed on the RGB channels to ensure that the RGB image frame outputs accurate colors.
The ISP should also have the ability to output both processed RGB and IR frames simultaneously according to system requirements. This means that for every single image frame read from the RGB-IR sensor, the ISP processes it to generate both an RGB image frame and an IR image frame.

Comparison and Typical Applications of RGB-IR Sensors

Compared to mainstream Bayer-format RGB cameras, the drawbacks of RGB-IR are quite obvious:

Adding IR pixels and reducing B and R pixels in the Bayer BGGR array will undoubtedly degrade the color fidelity of the image. Therefore, if high color accuracy is required, RGB-IR is clearly unsuitable.
Although several sensor and ISP manufacturers have launched products supporting RGB-IR, it remains a non-mainstream solution. Consequently, these products inevitably come with higher prices and greater development difficulty, destined for niche markets.

However, the advantages of RGB-IR are also very distinct: it can simultaneously output both color images and black-and-white IR images under any lighting condition. Therefore, the key is to identify application scenarios that demand this capability.

AI Image Detection in Complete Darkness Based on Infrared Images

Most AI algorithms are designed for AI detection on color images under good lighting conditions. Under low-light conditions, the severe loss of edge sharpness and the emergence of various types of noise significantly degrade the accuracy of AI algorithm detection. In such cases, using the IR image output by an RGB-IR sensor for detection yields much better results. Although standard IP cameras using IR-cut switches can also achieve this, the advantage of RGB-IR lies in eliminating the mechanical IR-cut switcher, which leads to more stable product operation and a longer lifespan.

Leveraging the Penetrative Power of IR Light for Differentiated Features

For example, infrared light emitted in the 940nm band is invisible to the human eye, meaning it does not cause any direct distraction or impact on the user experience. However, infrared light in this band has stronger penetrative properties. When combined with a suitable RGB-IR sensor and bandpass filter, it can capture details invisible under visible light, such as penetrating sunglasses or certain mask materials.

Using IR to Reveal Markings Invisible Under Visible Light

For specialized marking or anti-counterfeiting applications, certain coatings exist on the market that are only visible to an IR camera when illuminated by a specific band of IR light. For instance, as shown in the figures below, a QR code can be hidden within a visible traffic sign, or text strings can be concealed on a seatbelt.