Understand the various focusing technologies of Sony CMOS sensors in one article

• 1. Contrast focus

As the earliest autofocus technology, its technical principle is the simplest and most natural.Because the principle based on the focus of the human eye is also contrast-the brain determines the contrast information of the picture to adjust the focus of the lens.

The specific judgment principle of "contrast" is: when the object is not in focus, the picture is blurred into circles of confusion superimposed. At this time, the radius of the circle of confusion is much larger than the point, so the brightness and contrast are low; when the focus is accurate, the focus is one point after another, and the picture is sharp and the contrast is high.

That is to say, when the focus is inaccurate, the entire picture is blurred and there is no obvious contrast information; when the focus is accurate, the image in the focus range is the clearest state, while the area outside the focus is blurred and the contrast is very obvious.

After explaining the judgment principle, the specific working principle is easy to understand. When the lens is aimed at the object being photographed, the motor of the focusing module drives the lens to move from bottom to top. During this process, the sensor performs comprehensive detection.

At the same time, the sensor will continue to record the detected contrast values ​​such as contrast. When the position with the greatest contrast is found, the lens that moves to the top will return to this position to complete the final focus.

In this process, you will see a series of repeated changes in the content of the viewfinder, blurring and clearing - the corresponding term is "bellows".The advantage is that the technology is simple and low-cost, but the disadvantage is also obvious - it takes too long!

• 2. Phase focus/laser focus/TOF focus

Interestingly, Phase Detection Auto Focus (PDAF) technology is actually related to the human eye.But what is relevant is the principle of "human eye distance measurement" - because the two eyes are in different positions, they will receive light and form a parallax to know the distance of the object.

Therefore, the mobile phone CIS will set up some "masked phase focusing pixels", that is, half of the color filters on some pixels will be blocked (part on the left and part on the right), and each two will be paired into a pair to simulate the distance information of the human eye detecting light signals.

The detected information will then be given to the ISP to calculate the phase difference. After the final calculation result is obtained, the lens can be directly guided to move to the appropriate position to complete focus. Since this process is done in one step, the actual focusing speed is obviously faster than contrast focusing.

The specific calculation and judgment process is shown in the figure below. It can be seen from the focus diagram that the distance of the light projected on the photosensitive area is certain.

This given value will correspond to a part of the paired phase focusing pixels——This is a pre-designed focus detection benchmark (this is the core design of phase focusing).

On this basis, the smaller distance projected by the front focus state (representing the close distance of the subject) will show that the wave peak is shifted toward the center, and the offset of this wave peak is the phase difference.

Based on the deflection direction of the wave crest and the phase difference information, the lens will know the direction of movement and the distance of movement. In the same way, the working method of back focus state (representing that the subject is far away) can be reversed in this way.

Although this focusing method does not require back and forth testing and only requires one lens movement to complete focusing, in order to improve focus accuracy it needs to be used in combination with contrast focusing - first use phase focusing to quickly move the lens and then use contrast focus to fine-tune.

Theoretically, the more pixels used for phase focusing, the faster the focusing speed.But since the shielded part does not participate in imaging, so adding too many phase difference detection pixels will make it difficult to interpolate image information in the later stage, resulting in a decrease in image quality.

In addition, the bottleneck of the number of these phase difference detection pixels limits the focusing performance in dark light environments. In order to solve the problem of focusing in dark light, the first thing introduced is the laser focus sensor, which can emit infrared light for physical distance measurement to assist focusing.

However, since this type of laser focusing using STMicroelectronics' solution has a distance limit (generally within 2 meters), Sony's TOF focusing solution later appeared.

There are two types. One is the dTOF solution, which has the same principle as laser focusing, but its power is higher and it is suitable for long-distance focusing. This solution is relatively popular (generally mobile phone manufacturers still call it "laser focusing").

The other is a more expensive iTOF lens solution, which indirectly measures depth information by projecting a surface light source, accumulating reflected light and detecting the phase difference with the light source.

The iTOF solution is suitable for medium and close focusing, but its accuracy is higher.This solution is used in the front-facing 3D TOF face recognition module, and some top flagships will use it in the rear imaging module to participate in hybrid focus.

Small transition:

After talking about the technical principles of phase focusing, you can finally breathe a sigh of relief - because all the following principles of focusing technology are related to phase focusing!

But, remember the two major disadvantages of phase focusing?

That's right, the focusing technologies listed below are designed to solve these two major shortcomings, so the "old antique" masked pixels will be eliminated in these focusing technologies.

• 3. Full-pixel dual-core focusing / “2×1 OCL” dual-core focusing

The first one to hit the historical stage was full-pixel dual-core focusing, which directly added one photodiode (responsible for light sensing, termed "PD") in a unit pixel, thus forming a dual-PD structure that shares OCL (on-chip microlens).

In other words, these two PDs form a new form of "eyes", and the ability to detect phase differences is obtained through the "opening and closing" of the dual PDs.At the same time, since the information of the two PDs can be added, the influence of the phase difference can be eliminated, and lossless image quality can ultimately be obtained.

Obviously, through the full-pixel dual PD phase focusing, its focusing performance has been improved unprecedentedly! The key is that it can also get all the light, so the focusing performance in dark light environments will not be reduced. It can be described as "awesome" in one word!

At that time, the Samsung S7 series relied on the first dual-core focusing IMX260, combined with the advanced DBI technology of the Gen3 architecture, 1.4 micron large pixels and the lens's F/1.7 large aperture, thus gaining an incredible image advantage - the title of "night vision device" can be said to be resounding throughout the industry!

In addition to full-pixel dual-core focusing, there is also a dual-core focusing based on "2×1 OCL", which means that two pixels share a large microlens and form a pair of "eyes" for phase focusing. Obviously, the amount of light entering this phase focusing is directly doubled (no masking loss)!

but,Since these "2×1 OCL" pixels do not participate in imaging, they can only partially exist on the CIS like masked pixels.It seems that it is not as good as the full-pixel dual-core focus, so why did it appear?

It turns out that full-pixel dual-core focusing is good at everything, but it has two shortcomings. The first is that the pixel size cannot be too small, otherwise it will not be able to accommodate two PDs, so the "2×1 OCL" dual-core focus emerged.

In fact, "2×1 OCL" dual-core focusing is not popular in Sony CIS (the first model is the IMX398 customized by Green Factory), because Sony does not like small pixels, and then Samsung stepped forward - since Sony is not interested, I, Samsung, will accept it, but Samsung changed its name to "Super PD".

Later, Samsung developed an improved "Super PD Plus" focusing technology, which changed the large microlens into a shape with better light-gathering performance, which naturally improved the focusing performance.

Knowledge interpolation: Quad Bayer permutation

In order to explain the following focusing technology clearly, this knowledge point needs to be supplemented. As shown in the picture above, each color filter of the Quad Bayer arrangement covers four pixels, which feels like an "enlarged version" of the Bayer arrangement.

This is indeed the case, so why do we need to do something “superfluous”? At the beginning, because the pixel size was too small and the filter technology could not keep up, we could only use this "curve to save the country" approach, and by the way, we could brag about it in the marketing process.

Later, with the popularity of the "roll-up sole" trend, this situation no longer existed, but it has still not been eliminated - which shows that it has "real skills". Let’s talk in detail below about how it works when switching to high-pixel mode.

First, the Quad Bayer arrangement requires PD compensation and bad pixel compensation. Then, due to the different positions of the sub-pixels covered, there is a certain difference in photosensitivity, so it is necessary to introduce the Crosstalk link for calibration compensation.

Specifically, the entire image is divided into multiple ROI squares, and then the energy of each pixel channel is calculated and its compensation data is determined. Finally, these calibration data are used to make the originally uneven energy distribution more balanced.

The effect when implemented is,Crosstalk calibration compensation can remove grid, jagged and other color block interference caused by signal differences - this interference phenomenon is especially obvious when shooting a single color block.

After these compensation operations, the Remosaic link officially begins. This process is also divided into three steps.

first step:Use the interpolation algorithm to convert Quad Bayer Raw data into three sets of RGB data.

Step two:Then use the algorithm for converting RGB Image to Bayer and decompose it into three groups of Bayer Raw Images.

Step three:The three sets of data are combined into a normal Bayer arrangement and sent to the ISP for "demosaic" processing.

Friends who are familiar with the "demosaic" algorithm will probably be familiar with this Remosaic link. After all, some steps are the same. Of course, this is just the most basic set of Remosaic algorithms.

Later, with the development of Quad Bayer technology, Sony also developed other more advanced conversion and arrangement algorithms.In addition, Dafa has always integrated algorithm libraries on CIS to achieve hardware-level Remosaic conversion.(Samsung's GM1 made a joke back then because it did not have this hardware-level function).

As for its inherent shortcomings - the problem of reduced sampling space accuracy, this is the direction that CIS manufacturers need to continue to work hard in the future. In addition, there is a technology called "3-HDR" related to it, which will be discussed in the next issue when we talk about HDR technology.

• 4. Full pixel omnidirectional focusing

Regarding the advantages of Quad Bayer, I just talked about one "3-HDR" technology, but in fact the two super focusing technologies I will talk about next are also related to it.

Full-pixel omnidirectional focusing is also called "2×2 OCL" phase focusing. As the name suggests, a microlens covers four pixels. Combined with the Quad Bayer arrangement, it is equivalent to a microlens covering a color filter.

Since a microlens can cover four pixels, these four pixels can not only be divided into left and right to detect phase differences, but also support all-directional focusing up and down and diagonally. This is the so-called "rice-shaped" focusing.

When talking about the two major shortcomings of dual-core focusing above, we only mentioned one, and its other big shortcoming is that it is not very sensitive to shooting objects that lack pattern changes.

This is like the "one-line focus point" on a traditional SLR, which can only detect vertical textures.

So what should we do to enhance focus performance? The first is to increase the focusing ability of horizontal textures to form a "cross" focusing point array.

The final solution is the "rice-shaped" focusing array to achieve omni-directional stereo focusing. This is the birth background of full-pixel omni-directional focusing.

At the same time, in low-light environments, you can also use the "four-in-one pixel" mode to greatly increase the amount of light entering, so its low-light focusing performance is also worry-free.The first model of this outstanding focusing technology is the IMX689, which is deeply customized by Green Factory.

Since this focusing technology is so good, Samsung will not miss it, but it has changed its name to Super QPD.

However, this structure has a big disadvantage.

Since there are four pixels under one microlens, intermodulation distortion will be caused during the Crosstalk calibration compensation process (a lot of light is wasted at the intersection center of the four pixels).

So Sony showed off its skilled semiconductor process technology.Shift the microlens slightly (OCLs Shift).

In this way, a large amount of incident light will not go to the area without photosensitivity, but will smoothly enter each sub-pixel under the action of the microlens. However, one problem remains unresolved.

Since each color of light has a different wavelength, in the four pixels under each microlens, light crosstalk will occur due to no obstruction, which will also cause trouble in the Crosstalk link.

It doesn't matter, Dafa still has a way to deal with this situation.

It directly isolates four pixels using DTI technology - thus perfectly solving various problems encountered and incidentally enhancing low-light focusing performance (as shown in the picture below).

• 3. Full pixel eight-core focusing

Full-pixel omnidirectional focusing seems to be invincible, but in fact it is not. Because of the phase information, a small part of the image quality will be lost. In other words, this technology is most suitable for medium-sized sensors with high pixel count and small pixel size.

What should we do if we want to maintain a high pixel count and use a large sensor (which means an increase in pixel size) without increasing the loss of image quality? At this time, it’s the turn of the full-pixel eight-core focusing.

Full-pixel eight-core focusing is a technology that combines Quad Bayer arrangement and full-pixel dual-core focusing. Each pixel has 2 PDs, so four pixels have a total of 8 PDs, so its English name is very straightforward "Octa PD".

According to Sony officials, the biggest feature of this focusing technology is that when HDR shooting, all pixels in long exposure, medium exposure and short exposure can obtain phase difference.

This is the effect of 3-HDR technology plus dual-core focus, which is not affected by the brightness of the target. You must know that full-pixel omnidirectional focus cannot do this! In addition, like the full-pixel omnidirectional focusing, they all enjoy the high-sensitivity performance of the Quad Bayer arrangement.

In other words, although the principle of this technology is simple, the effect is very good; the difference from full-pixel omnidirectional focusing is that the applicable sensors are different, and there is no strict distinction between advantages and disadvantages.

In addition, as can be seen from the Quad Bayer arrangement to Bayer arrangement process at the beginning, the performance of full-pixel omnidirectional focusing and full-pixel eight-core focusing will not be affected by the pixel conversion process.

Therefore, during the use of these two focusing technologies, there is no difference in focusing performance between "default synthesis mode" and "high pixel mode". The first model of this equally awesome focusing technology is Huawei’s deeply customized IMX700.

In fact, Samsung's sensor also has full-pixel eight-core focusing, but its green sub-pixel photodiode is divided obliquely, which gives it the ability to focus in all directions and the name "Dual Pixel Pro".

• Summarize:

contrast focus——It can accurately find the focus point with the highest contrast and focus automatically. The technology is simple and low-cost. The disadvantage is that it takes too long.

phase focus——It can calculate the focus point based on the phase difference information to focus quickly. The disadvantage is that the masked pixels damage the image quality. As for the disadvantage of poor low-light focusing performance, it can be compensated for by laser focusing or TOF focusing.

Full pixel dual-core focusing (Dual PD)——The phase difference is obtained independently through dual PDs in the pixel, allowing all pixels to participate in imaging and focusing. The first disadvantage is that the pixel size is required to be larger, and the second disadvantage is the lack of lateral texture detection capabilities.

“2×1 OCL” dual-core focusing——Excellent phase focusing capability is obtained through dual pixels that share microlenses that do not participate in imaging. This technology is designed to address one of the shortcomings of Dual PD, but its own shortcomings are similar to phase focusing.

Full pixel omnidirectional focus (2×2 OCL)——The combination of Quad Bayer arrangement and four pixels of shared microlenses achieves high photosensitivity performance and omnidirectional focusing capabilities, thereby solving the second shortcoming of Dual PD.

Full pixel octa-core focusing (Octa PD)——The combination of Quad Bayer arrangement and Dual PD achieves high photosensitivity performance and super HDR focusing performance, thus avoiding the first shortcoming of Dual PD and weakening the second shortcoming of Dual PD.