An Image Unmixing and Stitching Deep Learning Algorithm for In-Screen Fingerprint Recognition Application

Abstract

The market share of organic light-emitting diode (OLED) screens in consumer electronics has grown rapidly in recent years. In order to increase the screen-to-body ratio of OLED phones, under-screen or in-screen fingerprint recognition is a must-have option. Current commercial hardware schemes include adhesive, ultrasonic, and under-screen optical ones. No mature in-screen solution has been proposed. In this work, we designed and manufactured an OLED panel with an in-screen fingerprint recognition system for the first time, by integrating an active sensor array into the OLED panel. The sensor and display module share the same set of fabrication processes when manufactured. Compared with the current widely commercially available under-screen schemes, the proposed in-screen solution can achieve a much larger functional area, better flexibility, and smaller thickness, while significantly reducing module cost. A point light source scheme, implemented by lighting up a single or several adjacent OLED pixels, instead of a conventional area source scheme as in the CMOS image sensor, or a CIS-based solution, has to be adopted since the optical distance is not long enough due to the integration. We designed a pattern for the point light sources and developed an optical unmixing network model to realize the unmixing and stitching of images obtained by each point light source at the same exposure time. After training, data verification of this network model shows that this deep learning algorithm outputs a stitched image of large area and high quality, where FRR = 0.7% given FAR = 1:50 k. In despite of a poorer quality of raw images and a much more complex algorithm compared with current commercial solutions, the proposed algorithm still obtains results comparable to peer studies, proving the effectiveness of our algorithm. Thus, the time required for fingerprint capture in our in-screen scheme is greatly reduced, by which one of the main obstacles for commercial application is overcome.

1. Introduction

In recent years, many categories of consumer electronics, including smart watches, mobile phones, laptops, monitors, and TVs, have adopted OLED screens instead of conventional liquid crystal display (LCD) screens gradually. Compared with LCD, OLED has many obvious advantages [1], such as higher contrast ratio between light and dark, lower power consumption, a wider color gamut, etc. In particular, for mobile phones, OLED screens provide excellent flexibility, opening up the possibility of more module forms for users in mobile applications. For example, foldable and rollable phones with OLED screens have all been implemented [2]. In addition, due to a narrower bezel, the screen-to-body ratio of OLED screens is also higher than its LCD counterpart. In order to further achieve an extremely high screen-to-body ratio for better custom experience, other mobile phone parts related with screens must also be improved in design and implementation [3,4].

For fingerprint recognition, in terms of hardware, a few manufacturers use ultrasonic solutions [5,6]; some other researchers have also proposed a standalone module, which can be adhered to the screen [7]. The mainstream commercial under-screen fingerprint recognition solution is to stack micro lenses and CIS chips under the OLED screen to act like a camera module to capture fingerprint images [8], as shown in Figure 1a. The obtained images are then compared with the pre-captured “standard” fingerprint pattern and conclude the matching degree [9]. Such a scheme has three disadvantages. First, the screen will become thicker due to the stack of CIS and lenses, and will also add weight to the phone. Moreover, under some extreme conditions such as strong ambient light, the module can be seen, as shown in inset of Figure 1b. Second, CIS is fabricated on silicon, which is a rigid module. Inconvenience in design will be caused if some flexible features such as rollability, are needed. Third, due to the consideration on weight, power consumption, and especially cost, the fingerprint recognition module is limited to a small area of the screen. Large-area or even full-screen fingerprint recognition cannot be achieved. Therefore, many researchers and companies are developing and optimizing solutions with sensors integrated under OLED [10,11,12]. Although the under-screen solution solves the problem of CIS not being able to achieve flexibility, it still causes the screen thickness to increase; in addition, due to a low transmittance of OLED, and obviously due to the sensor performance under the panel process being worse than that of CIS under the standard silicon process, the low light and noise challenges are more serious [13,14]. In contrast, the fingerprint recognition solution integrated in the screen, rather than under the screen, does not increase the module thickness, as can be roughly seen in their cross-sectional schematic views in Figure 1c,d. The in-screen scheme also reduces the transmission requirement, while the cost is lower because the manufacturing process is shared with the backplane of the OLED panel, though crosstalk of display and sensors would be more severe. However, the difficulty of the integration process causes degradation in the sensor performance; thus, the integration time needed is longer. As the optical distance decreases due to the integration, the imaging area becomes smaller, so multi-frame image acquisition is required. Both of these problems require a much longer customer waiting time, which cannot be tolerated. Thus, no mature in-screen scheme has been proposed yet.

In terms of algorithms, in recent years, a lot of progress has been made in fingerprint image enhancement based on deep learning. Examples include the use of adversarial learning and edge loss to solve fingerprint sensor interoperability or cross-sensor matching problems [15,16], and the use of cyclic generation adversarial networks to enhance low-quality fingerprint images [17]. When applied to under-screen fingerprint images, due to restrictions in the acquisition environment (sometimes in low temperature or strong ambient light), image quality (sensor density is generally limited by the display size and resolution), algorithm model size (generally about 20 MB or below since consumer electronics require a rapid unlock), and so on, these algorithms have to be optimized to meet higher requirements. Huang et al. proposed an image preprocessing process for fingerprints under OLED screens [18]. To overcome the image quality problems, researchers from Samsung proposed a matching algorithm using multi-scale texture descriptors, A-KAZE, to improve the accuracy of matching [19]. Wu et al. presents a fingerprint alignment algorithm based on the latest under-screen optical fingerprint image sensors in order to avoid very similar but wrong alignment results [20]. To address blurring issues in fingerprint images due to wet fingers, Zhang proposes an algorithm named EMEE (Ellipse Model Extrapolation Equalization) based on an elliptical model [21]. In spite of these works, there are still many problems to be solved, such as handling strategies for different types of lighting conditions, reducing noise, etc. In addition, the above research results are mostly aimed at the under-screen fingerprint scheme, and the algorithm development for the in-screen fingerprint scheme is still missing.

In this study, in terms of hardware, we introduced an in-screen scheme by integrating the optical sensors and its driving circuits into the OLED panel. The sensors and the OLED pixels are fabricated on the same glass substrate and share the same set of masks when manufactured. The test results show that the integrated hardware shows excellent performance. A point light source scheme is adopted since the optical distance is not long enough due to the integration. We found that such a scheme would lead to insufficiency of the effective fingerprint image area, which makes it difficult to achieve key indicators for matching. Therefore, we designed a lighting pattern of multi-point sources for image capture. In terms of algorithms, in order to solve the key problem of the long unlocking time of the in-screen solution, we developed and trained a deep convolutional network model based on a cross grid structure, to extract, enhance, and stitch fingerprint information in multi-point light source fingerprint images. Additionally, we designed a multi-level preprocessing strategy that separately handles regions of ambient light noise, fingermark reflectance, and point sources in order to reduce the impact of environmental light on the quality of fingerprint images. The results of data verification show that the model can achieve the customers’ target. Our work overcomes an important problem towards the mass production of in-screen fingerprint recognition schemes for OLED.

2. Device and Optical Methods

We developed an OLED panel with sensors integrated to realize the function of in-screen fingerprint recognition. As can be seen in Figure 1e, the sensor array is nested between OLED pixels. Cover glass above the panel is used to support and protect the panel. The sensor, known as PIN, consists of a p-type and n-type silicon, sandwiching a layer of amorphous silicon as the photosensitive material, converting light into electrons [22]. TFT circuits, used as pixel driving scheme for sensors, were fabricated by the low-temperature polycrystalline silicon (LTPS) process. OLED-related designs such as pixel layout, pixel circuits, and gate on array (GOA) circuits in this study generally follow mature commercial schemes, and also share the LTPS process. A specific designed driver IC for PIN sensors was adopted to realize the accumulation of photo-generated charges and the following processes, such as analog-to-digital conversion (ADC).

For CIS or under-screen fingerprint recognition schemes, the sensors are under the display screen, while the thickness of the display module could serve as optical distance of the image system. However, for in-screen scheme, the optical distance will be significantly reduced; thus, the conventional area source scheme is no longer applicable. We developed a point light source scheme for fingerprint image capture to adapt the proposed in-screen solution. The details are as follows.

When a finger presses down on the surface of the cover glass, one “point light source” (an OLED pixel or several adjacent pixels) right under the finger lights up and starts to luminesce towards the interface of finger and cover glass. At the valley of fingerprint, the interface is glass and air. Some of the incident light from the pixels reflects back into panel and the rest is released into the air through the interface. Since the refractive index of glass is greater than that of air, incident light with an angle greater than the critical angle of total reflection (typical value here is 42°) will be entirely reflected at the interface, and then this light will be collected by the sensors. Meanwhile, at the ridge of the fingerprint, the interface is glass and human skin. Due to the small difference in refractive index between skin and glass, reflection here will be largely reduced, and a major part of this incident light escapes through the finger side. Therefore, sensors will capture more light reflected by fingerprint valley than that of ridge. A fingerprint image then can be captured by the sensor array, as depicted in Figure 1f. In Figure 1g, we demonstrate a photo of our fabricated OLED panel, with fingerprint recognition module integrated.

There are some incidental problems with this scheme. At the center of the point light source, in the sensor plane, there will be a non-imaging area because the light intensity reflected back is too strong, which exceeds the full well capacity of the sensors (Area I in Figure 2a). Part of the area around the center of the point light source (Area II in Figure 2a), also could not obtain valid data since the reflected ratio of light versus the incident light is too small; thus, the light intensity difference between fingerprint valley and ridge is beyond the resolution limit of the sensor. Moreover, areas far away from the center of the point light source (the outside areas beyond Area III in Figure 2a) could not image well either, due to the long optical propagation distance. The light intensity difference would thus be too small for sensors to distinguish. Therefore, in the sensor plane, for a single-point light source, the valid imaging area of in this scheme is limited to a specific range (Area III in Figure 2a). Experimental data prove that the valid imaging area is too small to provide enough fingerprint features to realize effective fingerprint matching [23]. This is also one of the main challenges faced by researchers when developing an in-screen scheme. Typically, the valid imaging area by a single-point light source scheme is about 53% of that of current commercial CIS solution (which is around 6.5 mm × 6.5 mm). To solve this problem, naturally, stitching of images from multiple point light sources is considered. There are two ways to achieve this goal, as shown in Figure 2b. One is to light up the multiple point light sources sequentially, obtaining multiple frames of images, and then stitch the images into one. This method could obtain a large image of high quality, but the disadvantage is that it requires multiple exposure times. Based on the characteristic limits of sensor and system, and the principle that the display effect should not be influenced, exposure and processing time of each point light source image needs at least 67 ms. The total image capture time of this sequential scheme would exceed the commercial standard, since a long waiting time (image capture time together with processing time such like fingerprint matching) for fingerprint recognition would not be acceptable by customers. The second method is to light up multiple point light sources at the same time to obtain a large-area image at one-time exposure and capture. However, due to image-to-object optical amplification (roughly ×2 here), the obtained image in the sensor array is enlarged. Therefore, when multiple point light sources are lit up at the same time, the images of each light source will be mixed in the sensor plane, and the correct image cannot be obtained, as depicted in Figure 2c. We cannot avoid this problem by separating the point light sources far apart until they do not mix with each other, because the contact area of the finger and the cover glass is limited.

In order to solve the mentioned problems, we designed a light-up pattern, and developed an optical unmixing network model to realize unmixing and stitching of images obtained by each point light source at the same exposure time. We adopted a 4-point mixed lighting pattern here, as can be seen in Figure 2c. Image quality difference in spatial distribution caused by optical noise was also taken into consideration in the pattern design. The reasons for choosing this pattern will be discussed in Section 4.

3. Data Acquisition and Processing Methods

Data acquisition and processing methods include the three following parts: Section 3.1. fingerprint image acquisition by the 4-point light source scheme; Section 3.2. image preprocessing; and Section 3.3. unmixing and stitching network model. In addition, in order to train the proposed network model, we first lit up the 4-point light sources sequentially and stitched the 4 corresponding images together, and then set it as ground truth.

3.1. Data Acquisition, Ground Truth, and Dataset Preparation

A low-noise data acquisition system based on a customized analog front end was set up for charge accumulation of the PIN sensors. The system works simultaneously with a commercially used driving scheme for OLED panel. For 4-point light source image acquisition, we first lit up the 4-point light sources at the same time, and testees were required to press a finger onto the screen right upon the light sources. After image acquisition, the testees were required to keep their finger still upon the screen, and the 4-point light sources were then lit up sequentially to obtain the fingerprint images of the corresponding single-point light source. Image acquisition for each finger was conducted for several times repeatedly, and statistically abundant quantity of fingerprint images from fingers of different testees were obtained. The image was generated by the system into a 16-bit PNG format.

Fingerprint images of 18 volunteers (12 male and 6 female) in total were collected. Images of 6 fingerprints for each volunteer were obtained, and the numbering sequence is shown in Figure 3a. Due to the limited area of the sensors, multiple entries are required to fully cover the entire fingerprint feature, which is generally set to 20 entries for each finger. Therefore, it is necessary to collect as many fragments of fingerprints as possible in the first 20 times to achieve full coverage. Then, 40 more tests were conducted as validation test sets. Due to the lack of publicly accessible datasets of in-screen fingerprint images, we use these captured data to train and evaluate our algorithm.

Schematic diagram in Figure 3b shows the methods taken in data processing of each single light source image. The problems to be solved in image preprocessing mainly include three aspects: (i) The obtained image by the sensor array itself not being uniform. In the point light source scheme, except for the non-imaging area in the center of light source as depicted in Figure 2c, the luminous intensity of OLED decreases with the increase in the light exit angle, and moreover, the light intensity decreases with the increase in propagation distance when total reflection happens. (ii) Defects such as dead pixels, defective lines, and dynamic patterns. This is due to the process deviations of TFT or metal layers. (iii) Noise in the image. Noise here comes from the sensor and the image acquisition system, including the shot noise of the PIN sensor, electro-magnetic signal noise, reading noise of the system, etc. Among all the processing steps taken, self-adaptive brightness correction is used to solve problem (i). Anomaly detection and correction and image filtering are used to solve the problem (ii). Since valleys and ridges in a fingerprint image are required to be distinguished, self-adaptive contrast correction can significantly enhance the quality of the image. Then, these four images were stitched as one and set as ground truth.

3.2. Image Preprocessing

Similarly, the 4-point mixed light source images obtained by lighting up the light source pattern at the same time not only contain fingerprint information obtained by the sensor through collection of reflected light, but also contain noise information introduced by ambient light and optical signals near the light source points. Before training the proposed convolutional neural networks, it is necessary to preprocess the mixed light image, and strip and segment the primary data and secondary data. As shown in Figure 4a, region A contains major fingerprint information; region B contains a large amount of ambient light noise and a small amount of weak fingerprint data; and region C contains a strong light source signal and a small amount of fingerprint information. Here, we use the threshold segmentation method. The average value range of region A, B, C is calculated through the statistics of a large number of image data. Each pixel and threshold of the mixed light image is compared to segment the image region, as shown in Formula (1):

$${\mathrm{I}}_{(\mathrm{x},\mathrm{y})}^{\mathrm{a}}=\left\{\begin{array}{c}{\mathrm{I}}_{(\mathrm{x},\mathrm{y})} {\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{a}}\le {\mathrm{I}}_{(\mathrm{x},\mathrm{y})}\le {\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{a}}\\ 0 {\mathrm{I}}_{(\mathrm{x},\mathrm{y})}\le {\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{a}} \mathrm{or} {\mathrm{I}}_{(\mathrm{x},\mathrm{y})}\ge {\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{a}}\end{array}\right.$$

(1)

where ${\mathrm{I}}_{(\mathrm{x},\mathrm{y})}^{\mathrm{a}}$ stands for pixel value at position (x, y) in region A; ${\mathrm{I}}_{(\mathrm{x},\mathrm{y})}$ stands for pixel value in the acquired image at position (x, y); ${\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{a}}$ and ${\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{a}}$ are the minimum and maximum pixel values, respectively, in an artificially selected region that is relatively smooth in region A. They are regarded as the lower limit and upper limit of the range of region A. The Laplacian edge detection algorithm is used to detect the edge in the image. The lengths and positions of the detected edges are compared, and the final extracted edge is considered as the boundary between neighboring regions. The image is then segmented. Since fingerprint data are mainly distributed in the small value range of region A, it is necessary to normalize each segmented image. The results of a processed image are shown in Figure 4b–d. Both the mixed light image and the unmixed light image are gray images in a single channel.

3.3. Network Structure of Unmixing and Stitching Algorithm

The structure design of the optical unmixing network model is shown in Figure 5. The network structure is like a cross grid, deepening the fusion between deep features and shallow features, making full use of the limited fingerprint information in the mixed optical image. The network uses spatial separable convolution to realize most of the convolution operations, with the convolution kernel size k = 5. The down-sampling layer is composed of spatially separable max pooling layer, min pooling layer, and concat layer, while the filtering kernel size of pooling layer k = 5. The network takes the above mentioned preprocessed three-layer image as input. Concat processing of the input is performed on the channel dimension. That is, the size of the input image matrix is B × 3 × H × W, where B is the number of mixed light images in a training batch, H is the height of the image, W is the width of the image. The output of the network is a B × 1 × H × W unmixing image matrix.

The function of the unmixing optical network model is to correctly recover the fingerprint image containing clear ridge and valley information from the input multi-point unmixing image. L1 loss is used to calculate the pixel-by-pixel gap between the unmixing image generated by the model and the ground truth image, and the gap can be narrowed by optimizing the network parameters, which can guide the fingerprint image generated by the model to have a more accurate ridge and valley direction. However, models that only use L1 loss training focus more on enhancing the thick and obvious ridges and valleys, while small ridges and valleys in local areas are easy to be blurred. Therefore, this paper adds the edge loss function based on the Sobel operator to improve the model’s attention to small ridges and valleys. The calculation process is shown in Formulas (2)–(4):

$$L_{L1}\left(Y,\widehat{Y}\right)=\frac{1}{W\times H\times C}\sum _{x=1}^W\sum _{y=1}^H\sum _{z=1}^C\left|Y_{\left(x,y,z\right)}-{\widehat{Y}}_{\left(x,y,z\right)}\right|$$

(2)

$$L_{sobel}\left(Y,\widehat{Y}\right)=L_{L1}(S\left(Y\right),S(\widehat{Y}\left)\right)$$

(3)

$$\mathrm{L}\left(\mathrm{Y},\widehat{Y}\right)=L_{L1}\left(Y,\widehat{Y}\right)+\lambda \times L_{sobel}\left(Y,\widehat{Y}\right)$$

(4)

Y stands for the network output of the mixed light images; $\widehat{Y}$ for ground truth; W, H, C, for the width and height of the image and channel number, respectively; S(x) for image edge figure based on Sobel edge detection algorithm; λ value is 1.

AdamW algorithm [24] was used to optimize the model parameters of the network model. The initial learning rate was set to 10⁻⁴ and the batch size of the training data was set to 48.

After the fingerprint image of the point light source is preprocessed and unmixed, we intercept the ring region as the effective fingerprint region, according to the full reflection angle and the requirements of the image signal-to-noise ratio. This is usually realized by using a fixed ring mask. Several independent ring fingerprint regions are obtained. Due to the image amplification oriented by this point light source scheme, the size of the image is larger than the object. The magnification factor is M. In order to stitch, it is necessary to expand the spacing of the center position of each ring region image by M times. Since the position of each point source is set by us, its position on the image can also be calculated by a linear mapping. Therefore, its position on the Mosaic image is also obtained. For the overlapping regions of multiple ring regions, a weighted average can be used to complete the fusion of each ring region, and finally stitched into a complete fingerprint image.

4. Results and Discussion

Figure 6 shows the performance of the in-screen PIN sensor obtained by our test. As can be seen from the figure, although the sensor area of each pixel is small due to the layout limitation of the integration, it still maintains a linear response to light intensity until it reaches full well capacity (about 1800 lx, for the DUT). This shows that the process optimization of the sensor is effective enough to significantly reduce the influence of the edge effect. According to the sensor area, integration time, and IC-related parameters, further calculation can obtain that the optical response degree of our sensor is about 0.0076 fC/(lx·um²). These data are higher than the previous work on using glass-based PIN in an under-screen scheme [4], and are also consistent with the data that we calculated based on the transmittance of the stack of film layers. This proves that the in-screen scheme can effectively increase the semaphore and reduce power consumption compared with the under-screen scheme. However, our data show that the sensor noise of the in-screen scheme is about twice that of the under-screen scheme due to interference from the display signal because of signal coupling. This issue needs to be addressed through further hardware and design optimization.

Figure 7a–d show a single-light point image, single-point stitched images (ground truth), unmixing stitched images by a model trained only using L1 loss, and unmixing stitched images by a model trained using L1 loss and edge loss, respectively. It can be seen that the area of a single-point image is obviously small, and the latter three prove no difference in area size. The terminal manufacturer uses false acceptance rate (FAR) and false rejection rate (FRR) to evaluate the pros and cons of the fingerprint acquisition scheme. False acceptance refers to the event when a “wrong” fingerprint is input, but the matching scores is greater than the given threshold. False rejection refer to the event when a “right” fingerprint is input, but the matching scores fall below a given threshold given by the evaluating system. The mathematical formula is as follows:

$$FRR=\frac{NFR}{NGRA}\times 100\%$$

(5)

$$FAR=\frac{NFA}{NIRA}\times 100\%$$

(6)

where NFR refers to number of false rejections; NGRA refers to number of genuine recognition attempts; NFA refers to number of false acceptances; NIRA refers to number of imposter recognition attempts.

Quantitatively, the general requirements of customer indicators are, in the premise of FAR = 1:50 k, FRR ≤ 1%. We took fingerprint data that had not been used for training or tested in the evaluating system. NGRA = 3850, NIRA = 370 k are conducted in this work. Figure 7e demonstrates the influence of edge loss. Edge loss is helpful to improve the accuracy of the network model to restore the concave and convex curves of ridge and valley edges and to reduce the false adhesion and fusion of multiple bifurcated regions. It can be seen that adding edge loss will lead to better performance on FRR. Figure 7f shows the comparison of FRR of ground truth and the unmixing stitched image with edge loss involved. Also, some intermediate evaluation metrics are listed in

Table 1. Intermediate evaluation metrics to compare the proposed algorithm and ground truth. In the table, SNR stands for signal-to-noise ratio; DYN for dynamic range; NRSS for no-reference structure similarity; SMD for sum of modulus of gray difference; EAV for edge acutance value; AREA for effective area of the fingerprint image; KEYPOINTS for the number of key points available for recognition.

		SNR	DYN	NRSS	SMD	EAV	AREA	KEYPOINTS
Proposed algorithm	Average value	0.0328	198.22	0.7185	0.0105	202.56	16,586	12.000
	Standard error	0.0026	4.2273	0.0407	0.0015	15.677	1251.6	10.779
Ground Truth	Average value	0.0346	198.75	0.7666	0.0123	217.60	16,161	12.254
	Standard error	0.0026	6.5187	0.0375	0.0016	16.072	1259.2	11.040

to quantitatively describe the effect of the proposed algorithm. It can be seen in the table and figures that the single-point stitched image and the unmixing stitched image both meet the requirements of customers, although the latter still cannot match the former (FRR = 0.7% versus 0.48%, given FAR = 1:50 k, respectively). Based on our analysis of the recognition result graph, we found that there is a significant FRR difference between the two methods of obtaining fingerprint images when FAR was less than 1%, and thereafter, the FRR curves obtained by the two methods tended to be basically parallel. We believe that this is mainly due to the insufficient number of samples participating in the training, resulting in overfitting of the model during the training process. Overfitting leads to the model mistakenly restoring some fingerprint details to the features of other fingerprints, and there are also very few cases where fingerprint image details cannot be fully restored. To address this issue, we plan to improve model performance by increasing the number of fingerprint samples in the training set. By introducing more training data, we can more comprehensively cover various details of fingerprints and reduce the occurrence of overfitting, thereby improving the accuracy and robustness of fingerprint recovery models.

The lighting pattern used above is carefully designed. We also compared the image quality generated after the same proposed unmixing and stitching algorithm using different lighting patterns. Specifically, we compared the data of lighting patterns of two points, four points, and six points. From the result of FRR, as can be seen in Figure 8, a four-point pattern is slightly better than six points, which is significantly better than two points. The reasons are as follows: The area of the six-point fingerprint image is larger than that of the four-point fingerprint image, but the most central part of the fingerprint image has more saturated and non-information areas due to the bright spots, as mentioned above in Section 2, which just corresponds to the pressing center of the finger, that is, holes will be generated in the center of the fingerprint image. Although the hole texture can be recovered from the surrounding texture by means of deep learning, for the holes in a large range, there will still be differences between the recovered fingerprint texture and the original fingerprint, and this difference will lead to a difference between the six-point pattern and the four-point one. However, due to the small fingerprint area, the information that can be extracted is less, so the FRR result of the two-point fingerprint image is rather poor.

As far as we know, the in-screen integrated hardware scheme and lighting pattern and the corresponding unmixing and stitching algorithm in this paper are proposed for the first time; thus, there are no identical dataset results to compare with. If we extend the scope to include under-screen schemes, Bae et al. adopt an adhesive hardware scheme, where FRR = 0.73% when FAR= 1:50 k (Figure 6 in ref. [25]). The result of our work is slightly better. In the work of Mathur et al., where a capacitive scheme is adopted, FRR > 0.2% when FAR= 1:100; either the SIFT or VeriFinger algorithm are used for processing, for in-house partial DB (Figure 9 in ref [19]). Our work demonstrates a similar result. Zhang used an under-screen optical hardware scheme, and adopted a fusion feature extraction algorithm to repair the fingerprint. For non-extreme cases, the result obtained is FRR = 1.28% @ FAR = 7.28 × 10⁻⁴ (Table 5-4 in [21]), where our work is apparently superior. Raw images captured from the capacitor scheme or the under-screen scheme used in the above three works are with less noise (especially crosstalk noise), better signal-to-noise ratio, and no need to perform light unmixing and stitching processing. Despite the poor quality of raw images and complexity in the algorithm, we still obtained results comparable to peer studies, proving the effectiveness of our scheme.

Therefore, through the design of a four-point lighting pattern and the development of a deep learning algorithm, we realized large-area and high-quality fingerprint image acquisition, which greatly compressed the acquisition time, and solved the problem of too small fingerprint imaging area caused by the single-point light source amplification imaging of the in-screen fingerprint scheme. Hence, one of the main obstacles for the mass production of the in-screen scheme was cleared.

5. Conclusions

By integrating the PIN sensor array into the OLED screen, for the first time, we designed and prepared an OLED in-screen fingerprint recognition system. Compared to current commercial solutions such as adhesive or under-screen schemes, our solution can achieve larger area, better flexible matching, and a significant reduction in cost, turning out to be a better option. In order to solve the problems of insufficient area of fingerprint imaging at one capture and insufficient time of collection at multiple captures in this hardware scheme, we designed an image stitching scheme through four-point light sources and developed an unmixing optical network model to realize optical unmixing and stitching of the one-time-exposure images. According to the matching scores after a large amount of data trained in this model, it is proved that our model, FRR = 0.7% when FAR = 1:50 k, meets the customers’ target well. Despite the poor quality of raw images and complexity in the algorithm compared with the above-mentioned schemes, we still obtained results comparable to peer studies, proving the effectiveness of our scheme. The results show the proposed lighting pattern and deep learning algorithm we developed can help obtain high-quality and large-area fingerprint images, which can greatly reduce the time required for fingerprint collection, and thus provide a feasible scheme for the mass production of in-screen integration products.

The work ahead needs to be performed are mainly on three aspects. On the hardware side, PIN performance and integration design need to be optimized to reduce noise interference, especially crosstalk between display and sensing, which will radically reduce the size and difficulty of the algorithm. In terms of lighting pattern, although the mode we adopt in this study can meet the requirements, it still needs to be further optimized according to the hardware scheme and algorithm, so as to obtain enough data in a shorter time and reduce the time required to identify, thus providing a better user experience. In terms of algorithms, the accuracy and robustness of our algorithms will be improved if there are more data samples in the future, or if there are datasets of open access available. In addition, there is still a lot of work needed to be performed in the restoration and reconstruction of fingerprint details.