Low-Computational-Cost Algorithm for Inclination Correction of Independent Handwritten Digits on Microcontrollers

Abstract

In recent years, the digitization of documents has progressed, and opportunities for handwritten document creation have decreased. However, handwritten notes are still taken for memorizing data, and automated digitalization is needed in some cases, such as making Excel sheets. When digitizing handwritten notes, manual input is required. Therefore, the automatic recognition and input of characters using a character recognition system is useful. However, if the characters are inclined, the recognition rate will be low. Therefore, we focus on the inclination correction problem of characters. The conventional method corrects the inclination and estimates the character line inclination. However, these methods do not work when characters exist in independent positions. Therefore, in this study, we propose a new method for estimating and correcting the tilt of independent handwritten digits by analyzing a circumscribed rectangle and other digital features. The proposed method is not based on an AI-based learning model or a complicated mathematical model. It is developed following a comparatively simple mathematical calculation that can be implemented on a microcontroller. Based on the results of the experiments using digits written in independent positions, the proposed method can correct the inclination with high accuracy. Furthermore, the proposed algorithm is low-computational cost and can be implemented in real-time on a microcontroller.

1. Introduction

Owing to document digitalization, handwritten document usage is decreasing. However, there are many situations in which handwriting is needed. For example, writing down experimental data in notes during scientific experiments using apparatus is still common. Sometimes, automatically converting memorized data to electronic documents, such as Excel sheets, sheets by OCR platforms (ex: Nanonets-OCR platform), is important because manual insertion is time-consuming. The automatic conversion of written documents to electronic documents does not work well when written data are inclined. This inclination correction problem is solved when the written data exists as character lines. However, they do not work well when characters exist as independent characters.

Many studies have been conducted on document image analysis [1,2,3,4,5,6,7], including mathematical expression recognition [8], character recognition, etc. Character tilt correction is related to document image analysis. In conventional research, some methods are used to correct the inclination of character lines, such as the inclination correction of the character string using a wavelet transform and statistical-based methods [8,9,10,11,12,13,14,15,16,17,18,19,20,21]. When it is a character string (line) with two or more characters, the slope of the character string is estimated, and the slope is corrected. However, these methods cannot be applied to independent character tilt correction because the slope cannot be estimated as a character line. Therefore, in this study, we examined the tilt estimation and correction of handwritten numbers because there are many situations where independent digits are written compared to independent character writing (Figure 1). Furthermore, the conventional methods cannot be implemented on small controllers since the algorithms include comparatively complex computing.

The purpose of this study is to develop a low-computational cost algorithm to estimate the slope of an independent handwritten number. Furthermore, the algorithm should be implemented on a microcontroller. Focusing on the fact that the number is vertically long, we obtained the minimum external rectangle surrounding the digit structure. Accordingly, the tilt was corrected by rotating the image area of the external rectangle. However, we found that tilt correction is not possible with external rectangles in the case of digit “4”. Thus, we perform separated processing to achieve a tilt correction of digit “4” as described below. The number “4” was identified from other numbers, and the tilt was estimated using only the“4” feature without relying on the circumscribed rectangle. Furthermore, to distinguish between the number “4” and other numbers, we pay attention to the unique characteristics of the number “4” and propose a new identification method. In this work, we try to handle a handwritten digit inclination correction problem only using the image data at hand, without using any datasets for training, since the training stage is not included in the proposal. This time the inclination of the digits is measured by calculating its circumscribing rectangle and analyzing other unique features. The overall process comprises trigonometry calculation and the Hough Transform (HT). The HT is comparatively time-consuming. But, in this study, we have reduced the computation cost of HT by reducing the number of pixels for the HT process, and the proposed algorithm can be implemented on microcontrollers.

Based on the experimental results of the proposed method, the inclination of many numbers can be corrected with good accuracy, and the method is very effective for numbers inclined in a certain range. Furthermore, the proposed method has a short processing time, and it can be implemented in real-time with a small computer, such as a Raspberry Pi. Therefore, the proposed method is of an applicable level.

2. Tilt Estimation and Correction

2.1. Tilt Estimation with the Circumscribed Rectangle

A sample input image is displayed on the left side of Figure 2. As shown in the figure, some digits are inclined at a certain level. In this study, we first estimated the tilt angle and then performed tilt correction following the angle. The main processing flow for tilt estimation and correction is shown in Figure 3.

To determine the circumscribed rectangle, the image was first binarized based on discriminant analysis [22]. This time, digits written on a consistent-color paper were used. The binarization following discriminant analysis can separate the digits and background when the digits are written with a colored pen or pencil. Subsequently, the lines related to the digits were further extracted using the Canny edge detector [23,24,25]. For character line components, they were sandwiched between two close contours (edges). Through this feature, the lines that correspond only to the digits were determined, and objects other than the digits were removed. Figure 4 shows an example image obtained following these processes. Subsequently, we calculated the slope of the long side of the circumscribed rectangle. In this work, a vertical circumscribed rectangle was not considered, and the minimum circumscribed rectangle surrounding the digit was estimated, as shown in Figure 5.

Figure 6 shows how to find the slope of the long side of the estimated circumscribed rectangle. Because the coordinates of the four corners of the rectangle can be obtained from the circumscribed rectangle, the length was calculated using those coordinates. As shown in Figure 6, with the corner points of the rectangle (ax, ay), (bx, by), the side length of the axis l can be expressed by Equation (1). Consequently, the lengths of the long and short sides of the rectangle are calculated. Then, the long side length is used. In Figure 6, the rectangle $\theta$, which represents the angle of the long side against the horizontal direction, can be expressed by Equation (2). Conversely, ${\theta }^{\prime }$, which represents the angle of the long side against the vertical direction, can be expressed by Equation (3). ${\theta }^{\prime }$ becomes the tilt and considered the tilt of the digit.

$$l=\sqrt{{\left(ax-bx\right)}^2+{\left(ay-by\right)}^2}$$

(1)

$$\theta ={\cos }^{-1}\left(\frac{bx-ax}{l}\right)$$

(2)

$${\theta }^{\prime }=90-{\cos }^{-1}\left(\frac{bx-ax}{l}\right)$$

(3)

2.2. Separation of “4” and other Digits

Figure 7 shows the minimum circumscribed rectangle surrounding the digit “4”. As shown in Figure 4, the circumscribed rectangle considering the rotation of “4” looks like a rhombus with respect to the number, and the slope of the circumscribed rectangle does not coincide with the slope of the digit. Therefore, we first separated “4” and other digits. In the case of “4”, a special process was performed to estimate the tilt.

To separate “4” from the other digits, we focus on the line crossing part “+” of the “4”. The status of the perpendicular crossing of two lines is not available in the other digits, and it has a unique feature for “4”. Here, we first estimated the circumscribed rectangle and determined its short length ($w$). Then we detect the straight lines within the rectangle. If at least two lines are detected, then we estimated the angles between each two-line pair (${\varnothing }_i$). Here, i = 1,2. In addition, the lengths of the detected lines $l_j$ were determined. The line pair that fulfills the conditions in Equations (4) and (5) were extracted as the crossing line pair of “4”. Through this idea, we were able to separate “4” from the other digits with a high success rate.

$${80}^{°}<{\varnothing }_i<{100}^{°}$$

(4)

$$l_j\ge w\times 0.8$$

(5)

In the above process, line detection was performed following the Hough transform. The Hough transform is used to detect shapes with mathematical forms, such as lines, circles, and ellipses [26,27,28,29]. In this study, we used the Hough transform process for line detection. Generally, a straight line in an (x, y) space can be represented by Equation (6). Here, a and b constants. In the Hough transform, if the (x, y) space line is converted to the ($r,\theta$) space by Equation (7), then the line almost appears as a single dot. As shown in Figure 8, if we calculate all the lines that pass through the single dot in the (x, y) space, they appear as curved lines in the ($r,\theta$) space, as shown in Figure 9. In the case where the dots are on a straight line in (x, y) space, there is a crossing point for curved lines in the ($r,\theta$) space (Figure 9). By following these conditions, line detection can be performed [30,31,32].

$$y=a_{0}x+b_0$$

(6)

$$\rho =x_i\cos \theta +y_i\sin \theta$$

(7)

The Hough Transform (HT) is a comparatively time-consuming method due to its voting process. However, in this study, we apply HT to the image domain within the estimated circumscribed rectangle. Therefore, the HT process is conducted on a comparatively smaller number of pixels. As a result, the computational time for HT could be reduced.

2.3. Tilt Estimation of “4”

The digit “4” structure includes a diagonal line. We used this diagonal line part to estimate the tilt of “4”. As shown in Figure 10, the detected digit “4” is divided into four parts following the perpendicular cross lines. Then, we calculated the angle required for the diagonal line to move to the top-left part (see Figure 10 right). This angle is the tilt of the “4”, considering that “4” is inclined.

In this study, we detected diagonal lines by calculating the white pixels in each of the four parts gained following the cross part of a digit “4” image (Figure 11). The part containing the most white pixels includes the diagonal line. The diagonal line was detected using this feature.

3. Tilt Correction

In the above sections, we mainly discuss the estimation of the tilt of digits that exist in individual locations. After estimating the tilt, correction must be performed. In this study, tilt correction was performed following an affine transformation. Affine transform is a method that can translate images to different sizes and rotate them in different directions [33,34,35].

In this study, we applied an affine transformation to correct the digits by rotating their image parts. Following the above equation, if we rotate an image setting using (0, 0) as the image original, the above equation becomes Equation (8), where $\theta$ is the rotation angle.

$$\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]=\left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

(8)

Figure 12 shows an example of the tilt correction by the affine transformation. Here, the tilt estimation is performed using the following method.

4. Experimental Evaluations

4.1. Experimental Environment

We prepared document images with 300 handwritten digits ranging from 0 to 9. Most of the digits were written in an inclined manner, not vertically. The images were taken while keeping the camera almost 10–15 cm away from the documents. Here, we kept the camera approximately parallel to the documents when the images were captured. The resolution of the image was 480 × 640 pixels. We conducted experiments to confirm the effectiveness of the proposed digit tilt estimation and correction. Considering the application issues, we conducted experiments on a microcontroller (Raspberry Pi 4 hardware). The execution time per image under the above condition is 176 ms. In addition to that, we picked up some relevant image data from famous digit image datasets [36,37,38] and conducted experiments.

4.2. Experimental Results

The input and output images of numbers 0–9 are shown in Figure 13, Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, Figure 21 and Figure 22.

Table 1. Overall success rate.

Number	Success Rate of Inclination Correction (%)
0	96
1	100
2	92
3	91
4	86
5	93
6	91
7	92
8	95
9	90

shows the success rate of the angle corrections for each number.

Table 2. Success rate when the inclined angle is within 30°.

Number	Success Rate of Inclination Correction (%)
0	97
1	99
2	94
3	93
4	90
5	93
6	92
7	93
8	97
9	94

shows the success rate of the angle corrections for each number when the tilt angle is less than 30°.

Table 3. Classification success rate of 「4」 and other numbers.

Number	Success Rate of Classification (%)
0	98
1	100
2	95
3	94
5	97
6	93
7	93
8	90
9	91

shows the success rate of distinguishing between “4” and other digits.

The angles of the digits other than 4 were successfully corrected with a probability of 90% or more. The cause of the failure to correct the angle was that the circumscribed rectangle considering rotations was not generated to surround the entire number. The reason is that during the binarization process, some pixels belonging to the digit line were detected as background pixels. An example of the failure to generate a circumscribed rectangle is shown in Figure 23. As shown in the figure, if one number is split into several parts and recognized, the processing will fail. The number is split and recognized because the lines of the numbers are thin, and the lines may disappear at the binarization stage. However, if the line is clearly written this error will not occur. Solving this issue by proposing the new binarization approaches or applying the latest methods in the literature is an important future work [39]. In Table 1 and Table 2, the slope correction rate is higher regarding many digits when the slope is within approximately 30°. Therefore, this method is considered to be effective when the angle of the number is within 30°.

Table 4. Success rate when the digits and background are clearly separated in the binarized process.

Number	Success Rate of Inclination Correction (%)
0	99
1	100
2	97
3	96
4	92
5	95
6	96
7	96
8	98
9	95

shows the success rate of the tilt correction. Based on the findings, the success rate is very high when the numbers are neatly binarized.

Future prospects include handling the thin lines of numbers, increasing the accuracy rate of circumscribed rectangle acquisition and “4” recognition processing, and handling arbitrary input images. To increase the accuracy of circumscribed rectangle acquisition, other processes can be added, such as increasing the contrast. In this study, we used an input image with less noise and less distortion. Therefore, to support an arbitrary input image, it is necessary to remove noise and correct image distortion.

5. Conclusions

In this study, we propose a method that estimates and corrects the slope of independent numbers using a circumscribed rectangle and other specific digit features. We corrected the slope of the digits when the circumscribed rectangle on the slope could be obtained correctly. In the case of the digit “4”, the tilt angle cannot be easily estimated with the circumscribed rectangle, but it could be corrected using the positional relationship between the cross part of “4” and the diagonal line portion. We conducted experiments to confirm the proposed method by preparing appropriate document images with inclined digits. The results revealed that the proposed method achieved a success rate of approximately 90% or more. The circumscribed rectangle was not estimated correctly, mainly due to the disappearance of the digit line parts at the binarization stage. Improving this issue will be one of the main future stages of this work. In addition, the proposed method has a short processing time, and it can be implemented in real-time with a small computer, such as a Raspberry Pi. Therefore, the proposed method is at an applicable level.