A Survey on Face and Body Based Human Recognition Robust to Image Blurring and Low Illumination

Abstract

Many studies have been actively conducted on human recognition in indoor and outdoor environments. This is because human recognition methods in such environments are closely related to everyday life situations. Besides, these methods can be applied for finding missing children and identifying criminals. Methods for human recognition in indoor and outdoor environments can be classified into three categories: face-, body-, and gait-based methods. There are various factors that hinder indoor and outdoor human recognition, for example, blurring of captured images, cutoff in images due to the camera angle, and poor recognition in images acquired in low-illumination environments. Previous studies conducted to solve these problems focused on facial recognition only. This is because the face is typically assumed to contain more important information for human recognition than the body. However, when a human face captured by a distant camera is small, or even impossible to identify with the naked eye, the body’s information can help with recognition. For this reason, this survey paper reviews both face- and body-based human recognition methods. In previous surveys, recognition on low-resolution images were reviewed. However, survey papers on blurred images are not comprehensive. Therefore, in this paper, we review studies on blurred image restoration in detail by classifying them based on whether deep learning was used and whether the human face and body were combined. Although previous survey papers on recognition covered low-illumination environments as well, they excluded deep learning methods. Therefore, in this survey, we also include details on deep-learning-based low-illumination image recognition methods. We aim to help researchers who will study related fields in the future.

1. Introduction

1.1. Research Motivation

In recent years, studies on human recognition in indoor and outdoor environments have been conducted. For example, there are studies on enhancing blurred human images acquired in indoor environments, human images acquired in low-illumination environments, and low-resolution human face images. The particularities of human recognition in indoor and outdoor environments are as follows: in door environments, more motion-blurred face and body images can be obtained because the distance of the user to the camera is smaller than that in outdoor environments. In addition, more slanted face images can be obtained in indoor environments due to inclined head positioning compared to cameras than in outdoor environments. However, the face and body images having severer, non-uniform illumination and lower image resolution can be obtained in outdoor environments compared to those in indoor environments.

Furthermore, owing to the advances in electronic devices, along with the advances in camera performance, studies on human recognition in indoor environments can be applied to various aspects of people’s lives. For example, they can be used to find missing children and identify criminals. Furthermore, by learning from extensive datasets through artificial intelligence (AI) techniques, real-time detection and recognition based on learned data can be achieved. For commercialization, further research is required. Currently, human recognition studies include those on human face-, body-, and gait-based recognition. Methods based on the combination of facial recognition with body- and gait-based recognition, as well as methods based on face and body recognition, and face and gait recognition, have been also reported. However, the main focus of previous survey papers was on human faces. In some cases, deep-learning-based methods were not covered. The reason for this exclusion is that the human face contains more important features than body and gait features do. Deep-learning-based human recognition is currently a hot research topic because of the recent advancements in graphic processing units (GPUs). In addition, specialized hardware accelerators (for example, field programmable gate arrays (FPGAs)) can be used for human recognition because they are faster than GPUs and are designed for real-time image processing.

All in all, this survey paper covers recognition methods based on the enhancement of blurred images and images captured in low-illumination environments by further expanding the aforementioned human face-, body-, and gait-based recognition methods. In addition, the pros and cons of previous studies, including deep-learning-based studies, are analyzed, and future applications are introduced.

1.2. Research Scope and Human Recognition Methods

1.2.1. Research Scope

In this paper, we summarize previous studies that addressed the problem of human-recognition performance degradation due to image blurring and low illumination. Such previous studies are divided into three categories: face-based recognition, body-based recognition, and face-and-body recognition. An additional classification will be introduced in terms of conventional handcrafted feature-based methods and deep feature-based methods. Studies based on the enhancement of blurred images are explained in Section 2, whereas studies based on low-illumination image enhancement are described in Section 3.

First, we address recognition methods based on blurred-image enhancement. Typical examples of image blurring in indoor environments include optical blur and motion blur. Optical blur refers to a phenomenon that occurs as the object’s position deviates from the camera’s depth of field (DOF) when capturing an image with a camera. Motion blur refers to a problem where the object becomes blurred when capturing an image due to the motion of the camera and the object [1]. Figure 1a,b shows examples of optical and motion blur images, respectively. To solve these problems, previous studies employed either conventional handcrafted feature-based methods or deep feature-based methods. These methods will, in turn, be classified into three recognition categories (face-based, body-based, and face-and-body-based recognition) that will be described in detail in Section 2.

Next, we will describe methods for improving image recognition in indoor and outdoor low-illumination environments. Figure 2a,b shows examples of images captured in an indoor environment with normal and low illuminations, respectively. Many previous studies on human recognition in indoor and outdoor environments were fundamentally based on face detection. Likewise, many studies were primarily based on body-based recognition. In Section 3, we will analyze such studies, which will be divided into face-based, body-based, and face-and-body-based recognition methods. They will be also further classified into conventional handcrafted feature-based and deep feature-based methods.

1.2.2. Human Recognition Methods

Figure 3 shows a flowchart of a human recognition method in common indoor and outdoor environments, and it shows the scope of the studies covered in this survey. After inputting a human image captured in an indoor/outdoor environment, the face and body regions are detected through the face and body detector as shown in steps (1), (6), (11), and (17) in Figure 3. The focal value of the detected face and body is measured using the focus measurement mask. If the focal value is below a certain threshold, it is determined that image blur exists, and a deblurring method is applied, as shown in steps (2), (7), (12), and (18) of Figure 3 [2]. If the brightness of the detected face and body image is below a certain threshold, a low-illumination restoration (illumination compensation) method is applied, as shown in steps (3), (8), (13), and (19) of Figure 3 [3]. In the feature extraction step (i.e., step (2) in Figure 3), features are extracted for recognition from the face and body regions separately, as shown in steps (4), (9), and (15) of Figure 3. As shown in Figure 3, blue, red, and violet dashed boxes represent the modules of face-, body-, and gait-based recognition, respectively. In the step (14), GEI and EGEI represent a gait energy image and an enhanced gait energy image, respectively. As shown in Figure 3, body-based recognition is performed with only one body image. However, gait-based recognition is performed with N successive body images, and one gait image is obtained by GEI, EGEI, etc. from the N successive body images for gait-based recognition, as shown in step (14). In addition, multimodal recognition can be performed based on image-level, feature-level, and score-level fusion methods, respectively, as shown in Figure 3. Image-level fusion means the spatial- or channel-wise concatenation of face and body images. Feature-level fusion represents the concatenation of features from face and body images, and dimension reduction by principal component analysis (PCA), linear discriminant analysis (LDA), etc. Score-level fusion means the fusion of matching scores from face and body images based on weighted SUM, weighted PRODUCT, support vector machine (SVM), etc. Starting in Section 2, we will explain these steps and their related studies in more detail.

2. Face- and Body-Based Human Recognition Robust to Image Blurring in Indoor Environments

2.1. Face Recognition Robust to Image Blurring

Many studies have been conducted for recognition on blurred facial images. Relevant information for human identification is concentrated in the human face, which is studied more actively compared to the body. Various methods are used to detect human faces. They include two types of methods based on detection of regions of interest (ROI). The first type acquires an image that includes both the face and background. The second type focuses on capturing the face region, while minimizing the background. Both methods have their pros and cons. In the former case, a variety of information can be compared when recognizing the face because the image contains the neck. However, because the background is included, it may affect human face-based recognition. In the latter case, the advantage is that there is no effect caused by the background. However, the image is captured with the focus set on the face region. If the face is close to the camera, there is no problem. However, if the person is far from the camera, important features such as the eyes, nose, and mouth are not clearly visible, thereby hindering recognition. Consequently, problems in recognition vary depending on which method is employed to acquire the human face, with studies in the literature addressing such problems.

The flowchart of human face recognition on a blurred image is similar to that shown in Figure 3. The typical steps are (2), (3) preprocessing, (4) feature extraction, and (5) matching. The preprocessing step refers to the task of enhancing and converting the blurred face image into a deblurred face image, and compensation for low illumination on the face image. The feature extraction step refers to the step of extracting the face image features for facial recognition. Steps (2)–(4) can be classified into handcrafted feature-based methods and deep feature-based methods, and the specific methods for steps (2)–(4) are summarized in

Table 1. Studies on face recognition robust to image blurring. All the methods used open databases.

Category	Method	Reference	Used Databases
Handcrafted feature-based method	BIBD	[4]	FERET and CMU-PIE
	Modified FADEIN	[5]	FERET
	Iterative graph-based image restoration method + GLDA + LDRC	[6]	AR
	Coupled learning method + BIQA	[7]	FERET, extended Yale Face B, CMU-PIE, and FRGC 2.0
	bASR	[8]	AR-LQ
	MACE filter, FFT, and PSR	[9]	Self-collected database from 24 subjects at Michigan State University
Deep feature-based method	Multi-scale network	[10]	Helen, CMU-PIE, and CelebA
	TIRFaceNet	[11]	DHUFO and DHU
	UMSN	[12]	Helen, CelebA, and PubFig
	CGAN + HG	[13]	2MF2
	TBE-CNN + MDR-TL	[14]	PaSC, COX face, and Youtube faces
	DGFAN	[15]	LFW
	Transfer learning + FaceNet	[16]	ChokePoint and COX-S2V
	Sensitive CNN	[17]	LDHF
	ATFaceGAN	[18]	LFW, CFP, AgeDB, CALFW, CPLFW, and VGGFace2
	UID-GAN	[19]	CelebA, CFP, and COCO
	CycleGAN + KL divergence	[20]	CelebA and CFP
	Semantic face deblurring network	[21]	Helen and CelebA
	SVDFace	[22]	TINDERS
	DeepDeblur	[23]	CASIA WebFace

. Lastly, in step (5), matching is performed using the features extracted in step (4). Typical matching methods include Euclidean distance and cosine-distance-based matching ones.

Table 1 shows a summary of the methods for enhancing blurred face images. They are classified into handcrafted feature-based and deep feature-based methods. Concerning handcrafted feature-based methods [4,5,6,7,8], a blur-invariant binary descriptor (BIBD) was proposed in [4]. This method consists of three stages. Deblurring is achieved through pixel difference vectors (PDVs) applied to training and testing. The datasets used in the experiments were face recognition technology (FERET) and Carnegie Mellon University pose, illumination, and expression (CMU-PIE). In [5], modified facial deblurring inference (FADEIN) algorithms were proposed and face recognition performance was measured using the FERET dataset. In [6], an iterative graph-based image restoration method was proposed as a deblurring method. Experiments with the Aleix Martinez and Robert Benavente (AR) database were conducted. Furthermore, Gabor linear discriminant analysis (GLDA) was applied to obtain facial features, and a linear discriminant regressing classifier (LDRC) was used to evaluate the recognition performance. In [7], four types of datasets were employed, and the authors proposed a blind image quality assessment (BIQA)-based coupled learning method. In [8], the recognition performance of the blur-adaptive sparse representation (bASR) method was compared with state-of-the-art methods using blurred, low-quality images. A built-in-house dataset called AR low quality (AR-LQ) was used. In [9], they proposed a multimedia analytics system that performed automatic online exam proctoring, which included face verification based on a minimum average correlation energy (MACE) filter, fast Fourier transform (FFT), and peak-to-sidelobe ratio (PSR). The superiority of their system was confirmed by the experiments with self-collected database from 24 subjects at Michigan State University.

Meanwhile, deep feature-based methods [10,11,12,13,14,15,16,17,18,19,20,21,22,23] have also been studied. In [23], DeepDeblur was proposed. It is based on a convolutional neural network (CNN) method to proceed with facial deblurring. As an experiment, face verification (1:1 matching) was performed using The Institute of Automation, Chinese Academy of Sciences (CASIA) WebFace database. In [22], a singular value decomposition (SVD)-based (SVDFace) method was proposed. The recognition performance was evaluated using the Tactical Imager for Night/Day Extended-Range Surveillance (TINDERS) dataset. The main novelties in [22] were that infrared (IR) and short-wave infrared (SWIR) camera images were used instead of visible-light images. The recognition performance was measured as a function of the distance between the camera and the person. In [21], a semantic face deblurring network was proposed, and its performance was evaluated using two datasets: Helen facial feature dataset (Helen) and celebFaces attributes (CelebA). The authors used a method for restoring the part corresponding to each region after performing the semantic segmentation of the face region. In [10], a multi-scale network method was proposed. The authors used three datasets—Helen, CMU-PIE, and CelebA—to measure not only the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM), but also to perform facial recognition through the identity distance. In [11], the TIRFaceNet method was proposed. It is a network for facial recognition on thermal images. The recognition performance was measured using datasets, including that of Donghua University (DHU). In [13], face deblurring was performed through a conditional generative adversarial network (CGAN) with an hourglass network (HG), and an in-house 2MF2 dataset was used. In [12], the authors proposed an uncertainty guided multi-stream semantic network (UMSN) and measured the recognition performance using the Helen, CelebA, and public figure face (PubFig) databases. In [14], the authors proposed a trunk–branch ensemble convolutional neural network (TBE-CNN) and mean distance regularized triplet loss (MDR-TL) and evaluated the performance using the point-and-shoot face recognition challenge (PaSC) of the Institute of Computing Technology, Chinese Academy of Sciences (CAS), under the sponsorship of OMRON Social Solutions Co. Ltd. (OSS) and with the support of the Xinjiang University (COX) face and Youtube face databases. In [15], the authors proposed a deep-gated fusion attention network (DGFAN) and used the labelled faces in the wild (LFW) dataset to evaluate the deblurred face recognition performance. In [16], a method was proposed for solving the blurred face image recognition problem using transfer learning and FaceNet by converting 2D images of the ChokePoint and COX-still-to-video (COX-S2V) datasets into 3D images. In [17], a sensitive CNN research method was proposed, and face deblurring was studied using the Long Distance Heterogeneous Face (LDHF) database obtained from Long Distance. The major novelty in [17] was that the recognition performance was measured using not only visible light, but also near-infrared (NIR) face images. Furthermore, they proposed a sensitive CNN by modifying the cost function of a conventional CNN. In [18], an atmospheric turbulence face GAN (ATFaceGAN) was proposed, and the recognition performance was measured using six face databases: LFW, Celebrities in Frontal-Profile (CFP), AgeDB, cross-age LFW (CALFW), cross-pose LFW (CPLFW), and visual geometry group (VGG) Face2. In [19], the authors proposed an unsupervised image deblurring GAN (UID-GAN) and used the CelebA, CFP, and COCO datasets to measure performance. In [20], a method combining cycle-consistent GAN (CycleGAN) and Kullback–Leibler (KL) divergence was proposed, and the performance was evaluated using the CelebA and CFP datasets. Although it is not related to face recognition, Han et al. newly proposed a crossmodal emotion embedding framework called EmoBed, which aims at leveraging the knowledge from other auxiliary modalities in order to improve the performance of an emotion recognition system at hand [24]. The superiority of their method was confirmed by the experiments with the RECOLA and One-Minute Gradual-Emotional (OMG-Emotion) Behavior datasets.

Regarding conventional handcrafted feature-based methods, we note that they focus on solving the image blurring problem through various filters, in contrast with deep feature-based methods. Furthermore, as shown in Table 1, the handcrafted feature-based methods have a common characteristic: they mainly make use of the FERET and face recognition grand challenge (FRGC) datasets. Concerning deep feature-based methods, their fundamental characteristic is that a variety of databases and conventional datasets are used because a large number of data are required to train CNNs or GAN models. The methods in Table 1 mainly used visible-light images because they are more effective for deblurring.

2.2. Body-Based Human Recognition Robust to Image Blurring

No study has yet been conducted on body-based human recognition robust to image blurring in indoor environments; in this case, only the body region is used, dismissing the face. In other words, neither body-based recognition nor body-based re-identification have been studied yet. This is because, compared to the face region, the body region requires more global features for recognition. This in turn implies that the recognition performance is not significantly affected by image blurring. Although it is not related to human recognition, Hidayati et al. newly proposed a recommender of fashion style based on the user’s body attributes [25]. To obtain body attributes, they introduced features for body shape classification in order to significantly improve the performance over conventional body shape calculators. The superiority of their system was confirmed by the experiments with the Style4BodyShape, DeepFashion, and StyleReference datasets.

2.3. Face- and Body-Based Human Recognition Robust to Image Blurring

Body-based human recognition is classified into gait-based human recognition and stationary body image-based human recognition methods. A representative method for gait-based human recognition was reported in [26]. For gait-based human recognition, most methods create GEIs. In gait-based body and face recognition, if data are obtained at a certain distance, the size of the face image might be relatively small, and accordingly, the major information of the face becomes less important for recognition compared to that of the gait images. Most importantly, it was found that the methods for solving image blurring are different from those for gait-based body recognition and texture- and shape-based body recognition.

We divide face- and body-based recognition robust to image blurring into handcrafted feature-based methods and deep feature-based methods. However, given that there was no previous study related to handcrafted feature-based methods, we only investigated deep feature-based methods. Deep feature-based methods can be divided into gait-based body and face recognition and texture- and shape-based body and face recognition. As a study on gait-based body and face recognition, in [27], the authors proposed a method to be applied on blurred images using a graph neural network (GNN). The performance was evaluated using the complementary metal–oxide–semiconductor (CMOS), University of California San Diego (UCSD), Massachusetts Institute of Technology (MIT) gait, and University of Southampton (Soton) databases. Concerning texture- and shape-based body and face recognition, in [2], the authors conducted a study using the Dongguk face and body dataset version 2 (DFB-DB2) and ChokePoint datasets obtained in indoor environments, and DeblurGAN was employed for deblurring. From the images enhanced using DeblurGAN, image features were obtained through VGG face net-16 for face-based recognition and ResNet-50 for body-based recognition. The final recognition performance was measured through the weighted sum and weighted product using the score-level fusion method.

Table 2. Studies on face and body recognition robust to image blurring. All the methods used open databases.

Category		Method	Reference	Used Databases
Deep feature-based method	Gait-based body and face	GNN	[27]	CMOS, UCSD, MIT gait, and Soton
	Texture- and shape-based body and face	DeblurGAN + CNN	[2]	DFB-DB2 and ChokePoint

lists existing methods for face- and body-based recognition robust to image blurring.

2.4. Analysis and Discussion

Previous research on face and body-based human recognition robust to image blurring in indoor environments is classified into face-based, body-based, and both face- and body-based methods. In general, many studies on blurred human images have been carried out targeting faces [4,5,6,7,8,10,11,12,13,14,15,16,17,18,19,20,21,22,23], whereas there is no body-based research and a few both face- and body-based methods [2,27]. This is because the face has a larger number of relevant features to distinguish the subject than the body does. Moreover, the body could be covered by clothes of seasonal types and colors. The face-based method is further classified into handcrafted feature-based [4,5,6,7,8] and deep feature-based methods [10,11,12,13,14,15,16,17,18,19,20,21,22,23]. Diverse studies have been conducted on face recognition, including on blurred images, leveraging advances on deep learning technology. However, there is only a deep feature-based method for both face- and body-based methods, and they can be further classified into a gait-based body and face recognition method [27] and a texture- and shape-based body and face recognition method [2]. As a whole, both face- and body-based recognition shows a higher accuracy and more processing time than the face-based method. In addition, the deep feature-based method shows a higher accuracy and more generality than the handcrafted feature-based method. Gait-based body and face recognition is usually adopted with the successive images of side views [27], whereas the texture- and shape-based body and face recognition method is used with single and frontal view images, in the case of corridors in indoor environments [2]. This is because side view images can magnify gait features, whereas frontal view image can emphasize the texture- and shape-based features.

The impact of blurred images on body-based human recognition is relatively smaller than that on face-based recognition. For human body recognition, there are many studies related to person re-identification in different environments. These studies were not exclusively focused on the blurring problem. Concerning gait-based body recognition among body recognition methods, progress has taken place in the direction of extracting features robust to the blurred effect by using a GEI, whereby a single image is created from several binary images through different images with respect to the background, as shown in Figure 3. The basic principles and particularities of gait-based human recognition and their relation to body-based human recognition are discussed in Section 1.2.2 and shown in Figure 3. In [2], face and body images were enhanced through the DeblurGAN method. However, the impact of blurring is relatively smaller on the body than on the face. Nevertheless, there are elements to be improved for blurred-image-based face and body recognition methods. Some of them are summarized as follows:

-

Many studies addressed effects such as Gaussian blurring in existing face or body images to obtain blurred images that were then enhanced. Given that many studies focused on image enhancement rather than facial recognition evaluation, further research should be conducted to compare the recognition performance, as well as the quality of enhanced images.

-

Because of the improved camera performance based on technological advances, it may be difficult to obtain blurred images. Nevertheless, studies on recognition, including blurred images, should be conducted. Such images can occur in various environments, for instance, when the distance from the indoor surveillance camera to the subject is short.

-

Based on recent advances in deep learning technology, various studies on CNNs and GANs have been reported. By combining these studies, human recognition research can be advanced. Further, studies considering computing speed should also be conducted.

3. Face- and Body-Based Human Recognition Robust to Low Illumination in Indoor Environments

3.1. Face Recognition Robust to Low Illumination

Facial recognition methods for low-illumination environments belong to an area that has been widely studied. The methods for acquiring face images in low-illumination environments can be divided into methods that convert normal-illumination-based images into low-illumination-based images using an image processing method, and methods that acquire images in actual low-illumination environments. The image enhancement methods for low-illumination-based face images can be, in turn, divided into handcrafted feature-based methods and deep feature-based methods.

Studies on handcrafted feature-based methods include [28,29,30,31,32,33,34,35,36,37,38,39], whereas those on deep feature-based methods include [17,40,41,42,43]. In [28], the authors conducted experiments using a database developed in-house and used MPEf and fMPE, as well as multi-pass enhancement (MPE) methods, to enhance low-illumination images. In [29], the Asian Face and Faces94 databases were used, together with the Gradientfaces method, to study human recognition in low-illumination environments. The most relevant contribution of [30] is that instead of directly enhancing the face’s pose and illumination, the weights calculated by learning from high resolution (HR) and low resolution (LR) images were reflected in the recognition performance and in the process of scale invariant feature transform (SIFT). A total of four datasets were employed in [30]: the Multi-PIE, surveillance camera face (SC face), multiple biometric grand challenge (MBGC), and ChokePoint datasets. In [31], the authors leveraged four datasets (ORL, FERET, FEI, and CMU) to extract facial features using the Gabor filter and Zernike moments. With the obtained features, recognition was performed using an error-correcting output code multi-class model (ECOC) and a support vector machine (SVM) method. In [32], the low-illumination problem for face image recognition was solved through fractional discrete cosine transform (Fr-DCT), a kernel extreme learning machine (KELM), and a genetic algorithm (GA). In [33], the histogram of oriented gradient (HOG) with SVM method exhibited the highest recognition performance. In [34], three datasets were used: the Yale B+, CAS-PEAL-R1, and ORL databases. A method that combines histogram equalization (HE) and the fusion of illumination estimations (FOIE) was used to enhance low-illumination images, and an SVM was used as a recognition method. In [35], the authors suggested a DWT-fuzzy filter based on the combination of an existing fuzzy filter with a discrete wavelet transform (DWT) to solve the low-illumination problem in face images. The recognition performance was also evaluated. In [36], three datasets—KoFace, ORL, and Yale—were used and a method that combines the visual observation confidence (VOC) and a Gaussian mixture model (GMM) was proposed. For VOC, they considered the flatness measure (FM), centrality measure (CM), and illumination normality measure (IM). In [37], the authors improved the edge-oriented histogram scale-invariant feature transform (EOH-SIFT) performance by applying two filtering methods to the input image, rather than using the EOH-SIFT, as originally proposed. The two filtering methods were Medfilt2 and Rangefilt. In [38], a method that combines the correction on large-scale components (CLC) and the logarithmic total variation (LTV) was proposed. Its performance was measured using the CMU-PIE, extended Yale-B, and CAS-PEAL-R1 datasets. In [39], four datasets were used: the CMU-PIE, AR, Yale-B, and extended Yale-B datasets. Moreover, the authors proposed the dynamic morphological quotient image (GDMQI), as well as a method called GDMQI + HE that combines a conventional method and HE.

Studies on deep feature-based methods include [17,40,41,42,43]. In [40], the authors proposed a feature reconstruction network (FRN) and conducted experiments using the spec on faces (SoF) dataset. In [41], the recognition problem in low-illumination face images was solved by combining the soft-margin learning for multiple feature-kernel combinations (SML-MFKC) and domain adaptation (DA). Based on the characteristics of deep neural networks (DNNs), in [42], a deep learning method was proposed that combines the controlled pose feature (CPF) and the multi-depth generic elastic model-strategy 2 (MD-GEM-S2). As shown in

Table 3. Studies on face recognition robust to low illumination. All the methods except for [10,17] used open databases.

Categories	Method	Reference	Used Databases
Handcrafted feature-based method	MPEf and fMPE	[28]	Self-collected
	HOG + SVM	[33]	UTM
	Gradientfaces	[29]	Asian Face and Faces94
	Transform matrix and SIFT	[30]	Multi-PIE, SC face, MBGC, and ChokePoint database
	Gabor filter, Zernike Moments with ECOC and SVM	[31]	ORL, FERET, CMU, and FEI
	Fr-DCT, KELM, and GA	[32]	Extended-Yale B, AR, CMU-PIE, and YALE
	HE_FOIE + SVM	[34]	Yale B+, CAS-PEAL-R1, and ORL
	DWT-fuzzy filter	[35]	Yale B, CMU-PIE, and extended Yale B
	VOC + GMM	[36]	KoFace, ORL, and Yale
	Enhanced EOH-SIFT	[37]	FEI, CALTECH,
	CLC + LTV	[38]	CMU-PIE, extended Yale B, and CAS-PEAL-R1
	GDMQI + HE	[39]	CMU-PIE, AR, Yale-B, and extended Yale-B
Deep feature-based method	FRN	[40]	SoF
	SML-MFKC and DA	[41]	SC face, FR_SURV, and ChokePoint
	Sensitive CNN	[17]	LDHF
	CPF + MD-GEM-S2	[42]	Multi-PIE
	UR2D-R	[43]	FRGC 2.0

, the Multi-PIE dataset was used to evaluate the recognition performance. In [17], the authors proposed not only a method for processing blurred images, included in Table 1, but also a processing method for low-illumination scenarios, as shown in Table 3. In [43], UR2D methods based on a 3D–2D framework were suggested. These recognition methods combined the 3D facial shape with 2D facial feature points. Among the UR2D methods proposed in [43], the method UR2D with light normalization (UR2D-R) showed excellent performance. Table 3 shows a summary of studies on human face recognition in indoor low-illumination environments.

3.2. Body Recognition Robust to Low Illumination

Studies on low-illumination-based body recognition in indoor and outdoor environments can be categorized into gait-based recognition and body-image-based recognition, unlike the aforementioned face-based recognition studies. Concerning gait-based human recognition, the impact of low illumination is small if thermal images are used in addition to visible light images. Furthermore, the impact of low illumination is small because of GEIs created through different images with the background. However, regarding body-image-based recognition, there is a difficulty in recognition because it is affected by the texture and color of the body region due to the low illumination.

Body-based human recognition methods robust to low illumination environments in indoor and outdoor settings require a previous enhancement of the target low-illumination image using an illumination enhancement or deep learning method. Given the significant lack of studies on body recognition, in this review, we additionally summarize body-based person re-identification methods. Existing body-based recognition methods can be categorized into handcrafted feature-based methods [44,45,46] and deep feature-based methods [47,48,49,50]. As a handcrafted feature-based method, in [44], the authors proposed a multi-shot framework. In this architecture, recognition is performed in three stages, i.e., preprocessing, relevant feature selection, and hypergraph learning. In particular, person re-identification is performed in the hypergraph learning stage. Three datasets were employed in [44]: ETHZ, iLIDS-VID, and CAVIAR4REID. In [45], the authors proposed a depth-based person re-identification framework and used RGB-D datasets such as PAVIS, BIWI RGBD-ID, and IAS-Lab RGBD-ID. In [46], variations of the kernel partial least squares (KPLS) method were proposed, among which cross-view KPLS and kernel PLS mode A were used for re-identification. The datasets used in [46] were VIPeR and PRID 450S.

Table 4. Studies on body recognition robust to low illumination. All the methods used open databases.

Categories	Method	Reference	Used Databases
Handcrafted feature-based method	Multi-shot framework	[44]	ETHZ, iLIDS-VID, and CAVIAR4REID
	Depth-based person re-identification framework	[45]	PAVIS, BIWI RGBD-ID, and IAS-Lab RGBD-ID
	Cross-view KPLS and kernel PLS mode A	[46]	VIPeR and PRID 450S
Deep feature-based method	TMD2L	[47]	LIPS, LI-PRID 2011, and LI-iLIDS-VID
	IID network	[49]	Market-1501 and DukeMTMC-reID
	Illumination invariant feature learning framework	[50]	MSMT17, 3DPeS, and simulated low light person datasets based on Market-1501 and DukeMTMC-reID
	HDRNet	[48]	CUHK03, Market-1501, and DukeMTMC-reID

presents a summary of various existing methods for body-based recognition robust to low-illumination environments.

As a deep feature-based method, in [47], the authors proposed triplet-based manifold discriminative distance learning (TMD²L) to implement low-illumination, video-based person re-id (LIVPR), a topic for which no previous study had been conducted. The datasets used in [47] were the low-illumination person sequence (LIPS), LI-PRID 2011, and LI-iLIDS-VID. In [48,49,50], methods were proposed for person re-identification in low-illumination surveillance environments. In [48], the authors proposed a deep learning network called HDRNet and used three datasets: CUHK03, Market-1501, and DukeMTMC-reID. HDRNet has an encoder–decoder structure, and ResNet-50 was used for the encoder. In [49], the authors proposed an illumination identity disentanglement (IID) network which presents a structure that combines an encoder and a generator. Two datasets were employed: Market-1501 and DukeMTMC-reID. In [50], an illumination invariant feature learning framework was proposed. It includes a Retinex decomposition network. This network combines a light estimation network (LE-Net) and a light decomposition network (LD-Net) and performs the role of enhancing low-illumination regions. After enhancement, person re-identification is performed using ResNet-50, which has a two-branch structure. In [50], MSMT17 and 3DPeS, which are real-world person datasets, were used as low-light person datasets based on Market-1501 and DukeMTMC-reID.

3.3. Face- and Body-Based Human Recognition Robust to Low Illumination

Face- and body-based human recognition in indoor and outdoor environments can be mainly classified into gait-based face and body recognition methods, and face and body recognition methods applied on the entire face and body regions in a still image. Furthermore, each method type can be, in turn, divided into handcrafted feature-based methods and deep feature-based methods. Given that gait-based human recognition methods are less affected by low illumination, as mentioned in Section 3.2, there are methods for enhancing low illumination in the human face region only. In the case of recognition based on the entire face and body regions in a still image, the image quality is poor for both the face and the body because of the low illumination. Thus, methods that perform recognition after image enhancement have been proposed.

According to our investigation, there are no handcrafted feature-based studies reported in the literature. This is because although many studies on recognition using both face and body have been conducted in recent years, they are mostly based on deep features owing to the recent advances in deep learning. Among such deep feature-based methods, face- and gait-based recognition using thermal infrared (TIR) images was proposed in [51]. Here, the authors proposed a network that combines an optimized thermal image network (OTI-Net), person detector manner (PDM), and person recognition manner (PRM). The datasets used in [51] were Donghua University (DHU) Night, forward looking infrared autonomous and advanced driver assistance systems (FLIR ADAS), and KAIST. In [3], the authors proposed a method for face- and body-based recognition in extreme low-illumination surveillance environments. They divided the face and body regions, respectively, and after enhancing the low-illumination image through the modified EnlightenGAN, they extracted image features using VGG face net-16 for the face and ResNet-50 for the body. Then, after obtaining two matching scores with the extracted image features, they combined them into one score through score-level fusion to measure the final recognition performance. The datasets include the Dongguk face and body dataset version 3 (DFB-DB3) dataset which was self-collected and an open dataset called ChokePoint.

Table 5. Studies on face and body recognition robust to low illumination. All the methods used open databases.

Category		Method	Reference	Used Databases
Deep feature-based method	Gait-based body and face recognition	OTI-Net, PDM, and PRM	[51]	DHU, FLIR ADAS, and KAIST dataset
	Texture- and shape-based body and face recogntion	Modified EnlightenGAN + score-level fusion of two CNNs	[3]	DFB-DB3 and ChokePoint dataset

presents a summary of human recognition methods including face and body recognition in low-illumination indoor and outdoor environments.

3.4. Analysis and Discussion

As described above, studies on human recognition in low-illumination environments can be generally categorized into three research areas: face-based recognition, body-based recognition, and both face- and body-based recognition. The face-based recognition is further classified into handcrafted feature-based [28,29,30,31,32,33,34,35,36,37,38,39] and deep feature-based methods [17,40,41,42,43]. Like this categorization, body-based recognition can be further classified into handcrafted feature-based [44,45,46] and deep feature-based methods [47,48,49,50]. Lastly, in the both face- and body-based recognition area, there is no previous research of handcrafted feature-based methods, and only deep feature-based methods are proposed [3,51]. That is because both face- and body-based recognition methods were more recently proposed compared to face-based and body-based methods, and most studies were conducted based on deep learning according to the recent advances of deep learning technology. The deep feature-based methods are further classified into gait-based body and face recognition [51] and texture- and shape-based body and face recognition methods [3].

As a whole, both face- and body-based recognition shows a higher accuracy and more processing time than face-based and body-based methods. In addition, deep feature-based methods shows a higher accuracy and more generality than handcrafted feature-based methods. Gait-based body and face recognition is usually adopted with successive images of side views [51], whereas texture- and shape-based body and face recognition methods use single and frontal view images, as in cases of corridor sin indoor environments [3]. This is because side view images can magnify gait features, whereas frontal view images can emphasize the texture- and shape-based features. The basic principles and particularities of gait-based human recognition and their relation to body-based human recognition are discussed in Section 1.2.2 and shown in Figure 3.

Research on low-illumination recognition follows different directions compared to studies on recognition robust to image blurring, as mentioned in Section 2. This is because low-illumination environments facilitate the use of various types of images. For example, instead of just using visible-light images, it is possible to use thermal or near-infrared images, which are relatively robust to low illumination. For this reason, studies on human recognition in low-illumination environments using various types of images have been conducted. However, the following problems still remain:

-

Many studies on human recognition in low-illumination environments focused on the quality improvement of enhanced images rather than on recognition performance, e.g., recognition in blurred environments. Studies should focus on enhancing not only the image quality, but also the recognition performance.

-

Color must be considered as an image enhancement problem in low-illumination environments because it is one of the elements significantly affected by light. Because of this problem, it is difficult to restore the color of clothes or faces perfectly through the task of converting low illumination into normal illumination. Thus, the original colors should be restored, for instance, through deep learning methods.

-

Studies on human recognition in low-illumination environments have been usually conducted in severe conditions, but with facial outlines still visible. However, given that situations with almost no light often exist in real-world environments, research on recognition in ultra-low-illumination surveillance environments should be also conducted.

4. Conclusions

In this review paper, we cover the research areas of human recognition in blurring and low-illumination environments. This paper introduces research methods that perform recognition after enhancing blurred images and proposes future research directions. In addition, we introduce various research directions for recognition in low-illumination environments depending on the impact of light.

As the first step of human recognition on blurred images in indoor and outdoor environments, the degree of blurring is evaluated through an image quality assessment. Concerning human body-based recognition, the color of clothes may affect the recognition. Poor recognition may occur due to color bleeding caused by blur. Lastly, human-gait-based recognition can be used in various environments because of the low impact of blurring.

Human recognition in indoor and outdoor low-illumination environments is a difficult task. As a result, the majority of studies have been conducted for recognition in environments where low illumination is not severe. This is because it is difficult to restore colors perfectly when converting severe low-illumination images into normal-illumination images. It is expected that these problems could be solved through the various deep learning methods. Moreover, some studies on recognition were conducted using not only visible-light images, but also thermal images or NIR images. Concerning human recognition in low-illumination environments, there is a relatively smaller set of studies on body-based recognition than on face-based recognition. Regarding gait-based recognition, which is a widely researched method among those for body-based recognition, the impact of low illumination is relatively small in the different images with a background. However, given that human recognition in blurring and low-illumination environments should be considered, further studies should be conducted to solve these problems.

The results of these studies are increasingly applied to everyday life situations, such as remote driver identification in deluxe cars and the supervision of VIP members in hotels and resorts based on person identification at a distance. In addition, they can be used for the applications of remote gender recognition and age estimation, and in searching for missing children, the aged with dementia, and criminals in intelligent surveillance systems. In order to guarantee the system reliability in these applications, studies should be performed about the algorithms robust to various camera and user’s views, as well as occlusion. In addition, a light-weighted algorithm should be researched in order to be adopted to the embedded system of low processing power in car and surveillance camera environments.