On the Use of Deep Learning for Video Classification

Abstract

The video classification task has gained significant success in the recent years. Specifically, the topic has gained more attention after the emergence of deep learning models as a successful tool for automatically classifying videos. In recognition of the importance of the video classification task and to summarize the success of deep learning models for this task, this paper presents a very comprehensive and concise review on the topic. There are several existing reviews and survey papers related to video classification in the scientific literature. However, the existing review papers do not include the recent state-of-art works, and they also have some limitations. To provide an updated and concise review, this paper highlights the key findings based on the existing deep learning models. The key findings are also discussed in a way to provide future research directions. This review mainly focuses on the type of network architecture used, the evaluation criteria to measure the success, and the datasets used. To make the review self-contained, the emergence of deep learning methods towards automatic video classification and the state-of-art deep learning methods are well explained and summarized. Moreover, a clear insight of the newly developed deep learning architectures and the traditional approaches is provided. The critical challenges based on the benchmarks are highlighted for evaluating the technical progress of these methods. The paper also summarizes the benchmark datasets and the performance evaluation matrices for video classification. Based on the compact, complete, and concise review, the paper proposes new research directions to solve the challenging video classification problem.

1. Introduction

The task of automatically classifying videos has become very successful recently. Particularly, the subject has drawn increased interest since deep learning models became an effective method for automatically classifying videos. The importance of the accurate video classification task can be realized by the large amount of video data available online. People around the world generate and consume a huge amount of video content. Currently, on YouTube only, over 1 billion hours of video are being watched by different people every single day. In recognition to the importance of the video classification task, a combined effort is being made by researchers for proposing an accurate video classification framework. Companies such as Google AI are investing in different competitions to solve the challenging problem under constrained conditions. To further advance the progress of the automatic video classification task, Google AI has released a public dataset called YouTube-8M with millions of video features and more than 3700 labels. All these efforts being made demonstrate the need for a powerful video classification model.

An artificial neural network (ANN) is an algorithm based on interconnected nodes to recognize the relationships in a set of data. Algorithms based on ANNs have shown a great success in modeling both the lineßar and the non-linear relationships in the underlying data. Due to the huge success rate of these algorithms, they are extensively being used for different real-time applications [1,2,3,4]. Moreover, with an increase in the availability of huge datasets, the deep learning models have specifically shown a significant improvement in the classification of videos. This paper reviews studies based on deep learning approaches for video classification.

Contribution

There are several existing reviews and survey papers related to video classification in the scientific literature. Some of the recent works are summarized here in

Table 1. Summary of recent related works.

Reference	Year	Coverage	Highlights	Drawbacks
A. Anusya [5]	2020	2014–2019	Video classification, tagging, and clustering.	Not comprehensive and lacks concise information.
Rani et al. [6]	2020	2001–2016	Text, audio, and visual modalities for video classification.	Missing analysis of recent state-of-art approaches.
Y. Li et al. [7]	2020	2012–2019	Live sport video classification.	More specific to live sport video classification.
Md Islam et al. [8]	2021	2004–2020	Machine learning approaches for video classification.	Focus of review is not on deep learning approaches.
Ullah. H. et al. [9]	2021	2015–2020	Human activity recognition using deep learning.	Focus only on the human activity recognition.
This study	2022	2000–2022	Comprehensive deep learning review for video classification.	-

. However, these review papers do not include the recent state-of-art works, and they have some limitations. In the following text, the limitations and highlights of these works are discussed.

• A more recent review was done by A. Anusya [5]; this review covers very few methods for video classification, clustering, and tagging. However, the review provided is not comprehensive and lacks concise information, coverage of topic, datasets, analysis of state-of-art approaches, and research limitations;
• Rani et al. [6] also conducted a recent review on video classification methods, and their review covered some recent video classification approaches and summary-based description of some recent works. This review also had some limitations including the missing analysis of recent state-of-art approaches and a very limited description of topics covered;
• Y. Li et al. [7] recently conducted a systematic and good review on live sport video classification. This review covers most of the recent works in live sport video classification, including the tools, video interaction features, and feature extraction methods. This is a comprehensive review, but the findings are not summarized in tables for research gaps and advantages and disadvantages of existing methods for a quick review. Moreover, this review is more specific to live sport video classification;
• A recent review was also done by Md Islam et al. [8]; in this review, they included all the methods for video classification, including deep learning. However, as the focus of review is not on deep learning approaches, these methods are therefore not completely covered in this review;
• Ullah. H. et al. [9] also conducted a recent systematic review; however, the focus of their review remained on human activity recognition;
• Z. Wu. [10] presented a concise review on video classification specific to deep learning methods. This review provides a good description on deep learning models, feature extraction tools, benchmark dataset, and comparison of existing methods for video classification. However, this review was conducted in the year 2016, and it does not cover the recent state-of-art deep learning methods;
• Q. Ren [11] conducted a simple review on video classification methods; however, the techniques covered in this review are not well described, and the review also lacks in the description of research gaps, benchmark datasets, limitations of existing methods, and performance metrics.

In contrast to the existing reviews on classification of videos, this paper provides a more comprehensive, concise, and up-to-date review of deep learning approaches for video classification. In this current review, most of the recent state-of-art contributions related to the topic are analyzed and critically summarized. Deep learning is an emerging and vibrant field for the analysis of videos; therefore, we hope this review will help in stimulating future research along the line. The following are the key contributions to this review paper:

• A summary of state-of-art, CNN-based deep learning models for image analysis;
• An in-depth review of deep learning approaches for video classification highlighting the notable findings;
• A summary of breakthroughs in the automatic video classification task;
• Analysis of research trends from past towards future;
• Description of benchmark datasets, evaluations metrics, and comparison of recent state-of-art deep learning approaches in terms of performance.

The rest of the paper is organized as follows: Section 2 reviews some existing CNNs for images; Section 3 provides an in-depth review on deep learning models for video classification; Section 4 provides a summary for benchmark datasets, evaluation metrics, and comparison of existing state-of-art methods for the video classification task; and Section 5 provides conclusion and future research directions.

2. Convolutional Neural Networks (CNN) for Image Analysis

Deep learning models, specifically convolutional neural networks (CNNs), are well known for understanding images. A number of CNN architectures are proposed and developed in the scientific literature for image analysis. Among these, the most popular architectures are LeNet-5 [12], AlexNet [13], VGGNet [14], GoogleNet [15], ResNet [16], and DenseNet [17]. The trend that follows from the formerly proposed architectures towards the recently proposed architectures is to deepen the network. A summary of these popular CNN architectures along with trend of deepening the network is shown in Figure 1, where the depth of network increases from left-most (LeNet-5) to right-most (DenseNet). Deep networks are believed to better approximate the target function and to generate better feature representation with more powerful discriminatory powers [18]. Although deeper networks are better in terms of having more discriminatory powers, the deeper networks require more data for training and more parameters to tune [19]. Finding a professionally labeled, huge dataset is still a big challenge faced by the research community, and therefore, it limits the development of deeper neural networks.

3. Video Classification

In this section, a very comprehensive and concise review for deep learning models employed in the video classification task is provided. This section covers a description on video data modalities, traditional handcrafted approaches, breakthroughs in video classification, and recent state-of-art deep learning models for video classification.

3.1. Video Data Modalities

As compared to images, videos are more challenging to understand and classify due to the complex nature of the temporal content. However, three different modalities, i.e., visual information, audio information, and text information, might be available to classify videos in contrast to image classification, where only a single visual modality can be utilized. Based on the availability of different modalities in videos, the task of classification can be categorized as a uni-modal video classification or a multi-modal video classification, as summarized in Figure 2. The existing literature has utilized both of these models for the video classification task, and it is generally believed that models utilizing multi-modal data perform better than the models based on uni-modal data [20,21]. Moreover, the visual description [22] of a video works better than the text [23] and the audio [24,25] description for the classification purpose of a video.

3.2. Traditional Handcrafted Features

During the earlier developments of the video classification task, the traditional handcrafted features were combined with state-of-art machine learning algorithms to classify the videos. Some of the most popular handcrafted feature representation techniques used in the literature are spatiotemporal interest points (STIPs) [26], improved dense trajectories (iDT) [27], SIFT-3D [28], HOG3D [29], motion boundary histogram [30], action-bank [31], cuboids [32], 3D SURF [33], and dynamic-poselets [34]. These hand-designed representations use different feature encoding schemes such as the ones based on pyramids and histograms. iDT is one of these handcrafted representations that is widely considered the state-of-the-art. Many recent competitive studies demonstrated that handcrafted features [35,36,37,38] and high-level [39,40] and mid-level [41,42] video representations have contributed towards the task of video classification with deep neural networks.

3.3. Deep Learning Frameworks

Along with the development of more powerful deep learning architectures in the recent years, the trend for the video classification task has followed a shift from traditional handcrafted approaches to the fully automated deep learning approaches. Among the very common deep learning architectures used for video classification is a 3D-CNN model. An example of 3D-CNN architecture used for video classification is given in Figure 3 [43]. In this architecture, 3D blocks are utilized to capture the video information necessary to classify the video content. One more very common architecture is a multi-stream architecture, where the spatial and temporal information is separately processed, and the features extracted from different streams are then fused to make a decision. To process the temporal information, different methods are used, and the two most common methods are based on (i) RNN (mainly LSTM) and (ii) optical flow. An example of a multi-stream network model [44], where the temporal stream is processed using optical flow, is shown in Figure 4. A high-level overview of the video classification process is shown in Figure 5, where the stages of feature extraction and prediction are shown with the most common type of strategies used in the literature. In the upcoming sections, the breakthroughs in video classification and studies related to classification of videos, specifically using deep learning frameworks, are summarized, describing the success rate of utilizing deep learning architectures and the associated limitations.

3.4. Breakthroughs

The breakthroughs in recognition of still-images originated with the introduction of a deep learning model called AlexNet [13]. The same concept of still-image recognition using deep learning is also extended for videos, where individual video frames are collectively processed as images by a deep learning model to predict the contents of a video. The features from individual video frames are extracted, and then, temporal integration of such features into a fixed-size descriptor using pooling is performed. The task is either done using high-dimensional feature encoding [45,46] or through the RNN architectures [47,48,49,50]. For un-supervised spatiotemporal feature learning in 3D convolutions, restricted Boltzmann machines [51] and stacked ISA [52] are also studied in parallel. The 3D-CNNs using temporal convolutions to extract temporal features automatically were first proposed by Baccouche et al. [53] and by Ji et al. [54].

3.5. Basic Deep Learning Architectures for Video Classification

The two most widely used deep learning architectures for video classification are convolutional neural network (CNN) and recurrent neural network (RNN). CNNs are mostly used to learn the spatial information from videos, whereas RNNs are used to learn the temporal information from videos, as the main difference between these two architectures is the ability to process temporal information or data that come in sequences. Therefore, both these network architectures are used for completely different purposes in general. However, the nature of video data with the presence of both the spatial and the temporal information demands the use of both these network architectures to accurately process the two-stream information. The architecture of a CNN applies different filters in the convolutional layers to transform the data. RNNs, on the other hand, reuse the activation functions to generate the next output in a series from the other data points in the sequence. However, the use of only 2D-CNNs alone limits the understanding of video to only spatial domain. RNNs, on the other hand, can understand the temporal content of a sequence. Both these basic architectures and their enhanced versions are applied in several studies for the task of video classification.

3.6. Developments in Video Classification over Time

The existing approaches for video classification are categorized based on their working principle in

Table 2. Different categories of approaches of video classification.

Categories	Working Principle	References
Hand-crafted approaches	These representations are handcrafted and employ various feature encoding techniques, such as histograms and pyramids.	Spatiotemporal Interest Points (STIPs) [26], iDT [27], SIFT-3D [28], HOG3D [29], Motion Boundary Histogram [30], Cuboids [32], Action-Bank [31], 3D SURF [33], Dynamic-Poselets [34].
2D- CNNs	These are image based models where frame level feature extraction is performed using CNN architecture and classification is performed using state-of-art classification models, for example SVM.	[55]
3D-CNNs	2D image classification extension to 3D for video (For example the Inception 3D (I3D) architecture).	[56]
Spatiotemporal Convolutional Networks	To aggregate the temporal and the spatial information, these methods primarily depend on convolution and pooling.	[54,57,58]
Recurrent Spatial Networks	To represent temporal information in videos, recurrent neural networks such as LSTM or GRU are used.	[47,53,59,60]
Two/multi Stream Networks	In addition to the context frame visuals, these methods use layered optical flow to identify movements.	[50,61,62,63]
Mixed convolutional models	Models built with the ResNet architecture in mind. They are particularly interested in models that utilize 3D convolution in the bottom or top layers but 2D in the remainder; these are referred to as “mixed convolutional” models. Or the methods based on mixed temporal convolution with different kernel sizes.	[64,65]
Hybrid Approaches	These are models based on integration of CNN and RNN architectures.	[66,67,68]

. The trend observed for the classification of videos from the existing literature is that the recently developed state-of-art deep learning models are outperforming the earlier handcrafted classical approaches. This is mainly due to the availability of large-scale video data for learning deep architectures of neural networks. Besides an improvement in classification performance the recently developed models are mostly self-learned and does not require any manual feature engineering. This added advantage makes them more feasible for use in real applications. However, the better performing recently developed architectures are deeper as compared to the previously developed architectures which brings a compromise on the computational complexity of the deep architectures.

Among the initially developed hand-crafted representations, improved Dense Trajectories (iDT) [27] is widely considered the state-of-the-art. Whereas, many recent competitive studies demonstrated that hand-crafted features [35,36,37,38], high-level [39,40], and mid-level [41,42] video representations have contributed towards the task of video classification with deep neural networks. The hand-crafted models were among the very early developments of video classification problem. Later, 2D-CNNs were proposed for video classification, where image-based CNN models are used to extract frame level features and based on the frame level CNN features, some state-of-art classification models (for example SVM) are learned to classify videos. These 2D-CNN models do not require any manual feature extraction and these models performed better than the competing hand-crafted approaches. After successful development of 2D-CNN models where features are extracted from frame level, the same concept was extended to propose 3D-CNNs to extract features from videos. The proposed 3D-CNNs are computationally more expensive as compared to the 2D-CNN models. However, these models consider the time variations in feature extraction therefore these 3D-CNN models are believed to perform better as compared to 2D-CNN models for video classification [54,58,69].

The development of 3D-CNN models paved the way for fully automatic video classification models using different deep learning architectures. Among the developments using deep learning architectures, spatiotemporal convolutional networks are approaches based on integration of temporal and spatial information using convolutional networks to perform video classification. To collect temporal and spatial information, these methods primarily rely on convolution and pooling layers. Stack optical flow is used in two/multi-stream networks methods to identify movements in addition to context frame visuals. Recurrent spatial networks use recurrent neural networks (RNN) to model temporal information in videos, such as LSTM or GRU. The ResNet architecture is used to build mixed convolutional models. They are particularly interested in models that utilize 3D convolution in the bottom or top layers but 2D in the remainder; these are referred to as “mixed convolutional” models. These also include methods based on mixed temporal convolution with different kernel sizes. Advanced architectures based on DenseNet have also shown promising results for the video classification task. Some of these notable architectures based on DenseNet include region-based CNN (R-CNN) [70,71], faster R-CNN [72,73], and YOLO [74]. Besides these architectures, there are also hybrid approaches based on the integration of CNN and RNN architectures. A summary of these architectures is provided in Figure 6.

The different deep learning architectures described above employ different fusion strategies. These fusion strategies are either for the fusion of different features extracted from the video or for the fusion of different models used in the architecture. The fusion strategies mainly used for the extracted features are (i) concatenation, (ii) product, (iii) summation, (iv) maximum, and (v) weighted, where the concatenation approach simply combines all the features together, and all the features are used for classification. The product/summation approach performs the product/summation between the features extracted using different strategies and uses the result of product/summation to perform classification. The maximum approach takes the maximum value of the features extracted using different strategies and uses that for classification. The weighted approach gives different weights to different features and performs the classification using the weighted features. Different fusion methods are summarized in Figure 7.

3.7. Summary of Some Notable Deep Learning Frameworks Developments

A summary of some deep learnings architectures for video classification is provided in Table 3. These studies are summarized based on the architecture, the datasets, the evaluation metrics, the fusion strategy, and the notable findings. The most common architectures for video classification are fundamentally based on the RNN and CNN architectures; classification accuracy is one of the most common evaluation metrics; UCF-101 and Sports-1M datasets are the choice for validation in most cases, multi-class classification problem is considered in almost all cases, SMART blocks outperform 3D convolutions in terms of spatiotemporal feature learning, and average fusion, kernel average fusion, weighted fusion, logistic regression fusion, and MKL fusion are all proven to be inferior compared to the multi-stream multi-class fusion technique. Moreover, a more applied form of classification in videos is to identify/recommend tags or thumbnails in videos, and this specific task is successfully caried out in [75,76,77,78,79].

3.8. Few-Shot Video Classification

FEW-SHOT learning (FSL) has received a great deal of interest in recent years. FSL tries to identify new classes with one or a few labeled samples [80,81,82,83]. However, due to most recent work in few-shot learning being centered on image classification, FSL in the video domain is still hardly being explored [84,85]. Some of the notable works done in this domain are discussed below.

A multi-saliency embedding technique was developed by Zhu et al. [85] to encode a variable-length video stream into a fixed-size matrix. Graph neural networks (GNN) were developed by Hu et al. [86] to enhance the video classification model’s capacity for discrimination. The local–global link in a distributed representation space was still disregarded nevertheless. To categorize a previously unseen video, Cao et al. [87] introduced a temporal alignment module (TAM) that explicitly took advantage of the temporal ordering information in video data through temporal alignment. To combine the two-stream aspects of videos more effectively, Fu et al. [88] developed a depth-guided adaptive instance-normalization module (DGAdaIN). A C3D encoder was created by Zhang et al. [89] to record close-range action patterns for spatiotemporal video blocks. Few-shot video categorization was addressed by Qi et al. [90] by learning a collection of SlowFast networks enhanced with memory units. To comprehend realistic films of the target classes, Fu et al. [91] presented embodied agent-based one-shot learning, which made use of synthetic videos created in a virtual environment. For the issues of few-shot and zeroshot action recognition, Bishay et al. [92] presented the temporal attentive relation network (TARN), which was trained to compare representations of varying temporal length. By examining local–global linkages and preserving the specifics of properties, Y. Feng et al. [93] recently presented a dual-routing capsule graph neural network (DR-CapsGNN) to address the issue of severely constrained samples in few-shot learning.

Apart from this, contrastive learning has also proved successful in recognizing human actions. Some of the interesting works done in this regard are multi-granularity anchor-contrastive representation learning [94] and X-invariant contrastive augmentation and representation learning [95].

3.9. Geometric Deep Learning

Shape descriptors play a significant role in the description of manifolds for 3D shapes. In general, a global feature descriptor is created by aggregating local descriptors to describe the geometric properties of the entire shape, for example, using the bag-of-features paradigm. A local feature descriptor assigns a vector to each point on the shape in a multi-dimensional descriptor space, representing the local structure of the shape around that point. Most deep learning techniques that deal with 3D shapes essentially use the CNN paradigm. Volumetric 2D multi-view shape representations are applied directly using standard (Euclidean) CNN architectures in neural networks via methods such as [96,97]. These techniques are unsuited for dealing with deformable shapes because the shape descriptors they use are dependent on extrinsic structures that are invariant under Euclidean transformations, as demonstrated in Figure 8a [98], while some other approaches [99,100,101,102,103] create a new framework by adopting the CNN feature extraction pattern to investigate the inherent CNN versions that would enable handling shape deformations by using intrinsic filter structure, as shown in Figure 8b [98]. Geometric deep learning deals with non-Euclidean graph and manifold data. This type of data (irregularly arranged/distributed randomly) is usually used to describe geometric shapes. The purpose of geometric deep learning is to find the underlying patterns in geometric data where the traditional Euclidean distance-based deep learning approaches are not suitable. There are basically two methods available in the literature to apply deep learning on geometric data: (i) extrinsic methods and (ii) intrinsic methods. The filters in extrinsic methods are applied on the 3D surfaces such that it effects the structural deformity due to the extrinsic filter structure. The key weakness of extrinsic approaches [96,97] is that they continue to consider geometric data as Euclidean information. When an object’s position or shape changes, the extrinsic data representation fails. Additionally, for these methods to support the challenging-in-practice task of attaining the invariance of shape deformation, complicated models and extensive training are required. The filters in intrinsic approaches are applied on the 3D surfaces without being affected by the structural deformity. Rather than Euclidean realization, intrinsic methods work on the manifold and are isometry-invariant by construction. Some of the works based on intrinsic deep learning include (i) geodesic CNN [99], (ii) anisotropic CNN [100], (iii) mixture model network [101], (iv) structured prediction model [102], (v) localized spectral CNN [103], (vi) PointNet [104], (vii) PointNet++ [105], and (viii) RGA-MLP [106]. The application of geometric deep learning (mostly intrinsic methods) in analyzing videos can help in better understanding from the machine perspective, but it is still an open research problem and needs further investigation. For further details on geometric deep learning, readers are referred to [98,107].

Table 3. Summary and findings of studies based on deep learning models.

Study	Features	Model	Evaluation	Dataset	Problem	Fusion	Findings
[57]	Automatic spatio-temporal features/self-learning. Temporal features captured both locally and globally.	Multiresolution CNN architecture.	By the fraction of test samples that contained at least one of the ground truth labels in the top k predictions.	Sports-1M, UCF-101.	Multi-class	Single frame, Early Fusion, Late Fusion, Slow Fusion.	When compared to a multilayer neural network with rectified linear units followed by a Softmax classifier built using histogram features, the Softmax classifier performed better (both local features such as texton, HOG, cuboids, etc., and global features such as color moments, and hue–saturation).
[108]	Visual (dense trajectory descriptors): A 30-d trajectory shape descriptor, a 96-d HOG descriptor, a 108-d HOF descriptor, and a 108-d MBH descriptor (local visual descriptors). Audio Features: MFCCs and Spectrogram SIFT.	Deep neural network (DNN).	Mean average precision (mAP).	Hollywood2, Columbia Consumer Videos (CCV), and CCV+.	Multi-class	Regularized fusion of multiple features.	Found better than dense trajectory features and classification utilizing the basic early fusion technique.
[109]	Tensor-Train Factorization (global representation for the whole sequence).	Recurrent neural network (RNN).	Classification accuracy.	UCF11, Hollywood2, YouTube Celebrities Face Data.	Multi-class	-	Tensor-Train layer-based RNN such as LSTM and GRU perform better than the plain RNN architectures for video classification.
[110]	Improved Fisher vector (iFV) and explicit feature maps to represent features of conv and fc layers. Long-term temporal information.	A multilayer and multimodal fusion framework of deep neural networks based on fully connected (FC)-RNN.	Classification accuracy.	UCF101, HMDB51.	Multi-class	Multilayer and multimodal fusion framework.	When compared to enhanced dense trajectories, which require a number of handcrafted procedures such as dense point tracking, camera motion estimation, person detection, and so on, the proposed FC-RNN obtained competitive results.
[50]	Convolutional temporal feature pooling architectures (conv pooling, late pooling, slow pooling, local pooling). Global video-level descriptors.	Two CNN architectures (AlexNet and GoogleNet) and LSTM.	By the fraction of test samples that contained at least one of the ground truth labels in the top k predictions.	UCF101, Sports 1 million.	Multi-class	Late fusion	(i) UCF-101 necessitates the utilization of optical flow. (ii) Optical flow is not always beneficial, especially when the videos are captured in the wild, such as Sports-1M. (iii) To make use of optical flow, a more advanced sequence processing architecture such as LSTM is required. (iv) The maximum documented performance is achieved by using LSTMs on both image frames and optical flow for the Sports-1M benchmark.
[111]	Spatiotemporal feature learning: a SMART block and ARTNet for short-term spatiotemporal feature learning with a possibility to explore long-term learning.	ARTNet by integrating the SMART block into the C3D-ResNet18 architecture, where SMART block architecture is composed of appearance branch and relationship branch.	Top-1 and Top-5 accuracy.	Kinetics, UCF101, and HMDB51.	Multi-class	Concatenation and reduction operation.	(i) In terms of spatiotemporal feature learning, SMART blocks outperform 3D convolutions (3D-CNN). (ii) In the case of ARTNet, supplementing RGB input with optical flow improves performance. (iii) The optical flow modality can give additional information. (iv) Optical flow’s high computing cost prevents it from being used in real-world systems.
[112]	Spatial, short-term motion and audio clues using CNN. Long-term temporal dynamics. (Multimodal features).	CNNs-LSTM model with multi-stream multi-class fusion process to adaptively determine the optimal fusion weights for generating the final scores of each class.	Classification accuracy.	UCF-101, Columbia Consumer Videos.	Multi-class	Multi-Stream Multi-Class Fusion.	Average fusion, kernel average fusion, weighted fusion, logistic regression fusion, and MKL fusion are all proven to be inferior to the proposed multi-stream multi-class fusion technique.
[113]	Two distinct layers: 1 × 1 × 1 conventional convolutions for channel interaction (but no local interaction) and k × k × k depth-wise convolutions for local spatiotemporal interactions (but not channel interaction). Global spatiotemporal average pooling layer.	Channel-separated convolutional network (CSN). Two models: interaction-preserved channel-separated network (ip-CSN) and interaction-reduced channel-separated network (ir-CSN).	Classification accuracy.	Sports1M and Kinetics.	Multi-class	-	(i) In 3D group convolutional networks, the number of channel interactions has a significant impact on accuracy. (ii) Separating channel interactions from spatiotemporal interactions in 3D convolutions improves accuracy and reduces computing cost. (iii) Three-dimensional channel-separated convolutions offer regularization and avoid overfitting.
[114]	The 3D network is optimized with three loss functions: (i) cross-entropy (CE) loss, (ii) pseudo-CE loss, and (iii) soft CE loss. 2D Image and 3D video model capture short and long visual descriptors.	Semi-supervised learning (VideoSSL) with 3D ResNet-18.	Top-1	UCF101, HMDB51, and Kinetics.	Multi-class	-	(i) For 3D video classification, a direct application of current semi-supervised algorithms (which were initially designed for 2D imagery) cannot yield adequate results. (ii) The accuracy of 3D-CNN models is much improved by a calibrated use of object appearance indicators for semi-supervised learning.
[115]	Modal- and channel-wise attentions.	Expansion-squeeze excitation fusion network	Accuracy, confusion matrix	ETRI-ACTIVITY3D, NUT RGB+D	Multi-class	Multi-modal	(i) Modal-fusion nets (M-Nets) and channel-fusion nets (C-Nets) are capable of capturing the modal and channel-wise dependencies between features in order to improve the discriminative power of features via modal and channel-wise ESEs. (ii) By adding the penalty of the difference between the minimum prediction losses on the single modalities and the prediction loss on the fused modality, multi-modal loss (ML) can further enforce the consistency between the single-modal features and the fused multi-modal features.

Table 3. Summary and findings of studies based on deep learning models.

Study	Features	Model	Evaluation	Dataset	Problem	Fusion	Findings
[57]	Automatic spatio-temporal features/self-learning. Temporal features captured both locally and globally.	Multiresolution CNN architecture.	By the fraction of test samples that contained at least one of the ground truth labels in the top k predictions.	Sports-1M, UCF-101.	Multi-class	Single frame, Early Fusion, Late Fusion, Slow Fusion.	When compared to a multilayer neural network with rectified linear units followed by a Softmax classifier built using histogram features, the Softmax classifier performed better (both local features such as texton, HOG, cuboids, etc., and global features such as color moments, and hue–saturation).
[108]	Visual (dense trajectory descriptors): A 30-d trajectory shape descriptor, a 96-d HOG descriptor, a 108-d HOF descriptor, and a 108-d MBH descriptor (local visual descriptors). Audio Features: MFCCs and Spectrogram SIFT.	Deep neural network (DNN).	Mean average precision (mAP).	Hollywood2, Columbia Consumer Videos (CCV), and CCV+.	Multi-class	Regularized fusion of multiple features.	Found better than dense trajectory features and classification utilizing the basic early fusion technique.
[109]	Tensor-Train Factorization (global representation for the whole sequence).	Recurrent neural network (RNN).	Classification accuracy.	UCF11, Hollywood2, YouTube Celebrities Face Data.	Multi-class	-	Tensor-Train layer-based RNN such as LSTM and GRU perform better than the plain RNN architectures for video classification.
[110]	Improved Fisher vector (iFV) and explicit feature maps to represent features of conv and fc layers. Long-term temporal information.	A multilayer and multimodal fusion framework of deep neural networks based on fully connected (FC)-RNN.	Classification accuracy.	UCF101, HMDB51.	Multi-class	Multilayer and multimodal fusion framework.	When compared to enhanced dense trajectories, which require a number of handcrafted procedures such as dense point tracking, camera motion estimation, person detection, and so on, the proposed FC-RNN obtained competitive results.
[50]	Convolutional temporal feature pooling architectures (conv pooling, late pooling, slow pooling, local pooling). Global video-level descriptors.	Two CNN architectures (AlexNet and GoogleNet) and LSTM.	By the fraction of test samples that contained at least one of the ground truth labels in the top k predictions.	UCF101, Sports 1 million.	Multi-class	Late fusion	(i) UCF-101 necessitates the utilization of optical flow. (ii) Optical flow is not always beneficial, especially when the videos are captured in the wild, such as Sports-1M. (iii) To make use of optical flow, a more advanced sequence processing architecture such as LSTM is required. (iv) The maximum documented performance is achieved by using LSTMs on both image frames and optical flow for the Sports-1M benchmark.
[111]	Spatiotemporal feature learning: a SMART block and ARTNet for short-term spatiotemporal feature learning with a possibility to explore long-term learning.	ARTNet by integrating the SMART block into the C3D-ResNet18 architecture, where SMART block architecture is composed of appearance branch and relationship branch.	Top-1 and Top-5 accuracy.	Kinetics, UCF101, and HMDB51.	Multi-class	Concatenation and reduction operation.	(i) In terms of spatiotemporal feature learning, SMART blocks outperform 3D convolutions (3D-CNN). (ii) In the case of ARTNet, supplementing RGB input with optical flow improves performance. (iii) The optical flow modality can give additional information. (iv) Optical flow’s high computing cost prevents it from being used in real-world systems.
[112]	Spatial, short-term motion and audio clues using CNN. Long-term temporal dynamics. (Multimodal features).	CNNs-LSTM model with multi-stream multi-class fusion process to adaptively determine the optimal fusion weights for generating the final scores of each class.	Classification accuracy.	UCF-101, Columbia Consumer Videos.	Multi-class	Multi-Stream Multi-Class Fusion.	Average fusion, kernel average fusion, weighted fusion, logistic regression fusion, and MKL fusion are all proven to be inferior to the proposed multi-stream multi-class fusion technique.
[113]	Two distinct layers: 1 × 1 × 1 conventional convolutions for channel interaction (but no local interaction) and k × k × k depth-wise convolutions for local spatiotemporal interactions (but not channel interaction). Global spatiotemporal average pooling layer.	Channel-separated convolutional network (CSN). Two models: interaction-preserved channel-separated network (ip-CSN) and interaction-reduced channel-separated network (ir-CSN).	Classification accuracy.	Sports1M and Kinetics.	Multi-class	-	(i) In 3D group convolutional networks, the number of channel interactions has a significant impact on accuracy. (ii) Separating channel interactions from spatiotemporal interactions in 3D convolutions improves accuracy and reduces computing cost. (iii) Three-dimensional channel-separated convolutions offer regularization and avoid overfitting.
[114]	The 3D network is optimized with three loss functions: (i) cross-entropy (CE) loss, (ii) pseudo-CE loss, and (iii) soft CE loss. 2D Image and 3D video model capture short and long visual descriptors.	Semi-supervised learning (VideoSSL) with 3D ResNet-18.	Top-1	UCF101, HMDB51, and Kinetics.	Multi-class	-	(i) For 3D video classification, a direct application of current semi-supervised algorithms (which were initially designed for 2D imagery) cannot yield adequate results. (ii) The accuracy of 3D-CNN models is much improved by a calibrated use of object appearance indicators for semi-supervised learning.
[115]	Modal- and channel-wise attentions.	Expansion-squeeze excitation fusion network	Accuracy, confusion matrix	ETRI-ACTIVITY3D, NUT RGB+D	Multi-class	Multi-modal	(i) Modal-fusion nets (M-Nets) and channel-fusion nets (C-Nets) are capable of capturing the modal and channel-wise dependencies between features in order to improve the discriminative power of features via modal and channel-wise ESEs. (ii) By adding the penalty of the difference between the minimum prediction losses on the single modalities and the prediction loss on the fused modality, multi-modal loss (ML) can further enforce the consistency between the single-modal features and the fused multi-modal features.

4. Benchmark Datasets, Evaluation Metrics, and Comparison of Existing State-of-the-Art for Video Classification

4.1. Benchmark Datasets for Video Classification

There are several benchmark datasets being utilized for classification of videos, AND some of these notable datasets are summarized in

Table 4. Benchmark datasets.

Dataset	# of Videos	# of Classes	Year	Background
KTH	600	6	2004	Static
Weizmann	81	9	2005	Static
Kodak	1358	25	2007	Dynamic
Hollywood	430	8	2008	Dynamic
Hollywood2	1787	12	2009	Dynamic
MCG-WEBV	234,414	15	2009	Dynamic
Olympic Sports	800	16	2010	Dynamic
HMDB51	6766	51	2011	Dynamic
CCV	9317	20	2011	Dynamic
UCF-101	13,320	101	2012	Dynamic
THUMOS-2014	18,394	101	2014	Dynamic
MED-2014 (Dev. set)	31,000	20	2014	Dynamic
Sports-1M	1,133,158	487	2014	Dynamic
ActivityNet	27,901	203	2015	Dynamic
EventNet	95,321	500	2015	Dynamic
MPII Human Pose	20,943	410	2014	Dynamic
FCVID	91,223	239	2015	Dynamic
UCF11	1600	11	2009	Dynamic
YouTube Celebrities Face	1910	47	2008	Dynamic
Kinetics	300,000	400	2017	Dynamic
YouTube-8M	6.1 M	3862	2018	Dynamic
JHMDB	928	21	2011	Dynamic
Something-something	110,000	174	2017	Dynamic

. The details related to these datasets, such as total number of videos contained in the dataset, number of classes present in the dataset, the year of publication of dataset, and the background of videos in the dataset, are included in the summary.

4.2. Performance Evaluation Metrics for Video Classification

The evaluation of video classification models is performed using different performance measures. The most common measures utilized to evaluate the models are accuracy, precision, recall, F1 score, micro F1, and K-fold [8]. Some of the recent studies using these measures are listed in

Table 5. Commonly used evaluation metrics for video classification.

Evaluation Metric	Year of Publication	Reference
Accuracy	2020–2021	[116,117,118,119,120]
Precision	2020–2021	[116,118,119]
Recall	2020–2021	[116,118,119]
F1 Score	2020–2021	[116,118,119]
Micro F1	2020	[121,122]
K-Fold	2019	[123]
Top-k	2018,2021	[111,114]

.

4.3. Comparison of Some Existing Approaches on UCF-101 Dataset

UCF-101 is a benchmark action recognition dataset published by the researchers of University of Central Florida in the year 2012 [124], and the videos in the dataset were collected from YouTube. The total videos in the dataset are 13,320, with 101 action categories. The dataset is challenging because of the uncontrolled environment in the captured videos, and it is widely being used by researchers working on the video classification problem. Therefore, it is easy to compare most of the existing literature based on this dataset. The existing works employing UCF-101 are compared in

Table 6. Comparison of video classification method on UCF-101.

Method	Accuracy
LRCN [48]	82.9
DT + MVSV [125]	83.5
LSTM–Composite [49]	84.3
FSTCN [126]	88.1
C3D [127]	85.2
iDT + HSV [128]	87.9
Two-Stream [61]	88.0
RNN-FV [129]	88.0
LSTM [50]	88.6
MultiSource CNN [130]	89.1
Image-Based [55]	89.6
TDD [35]	90.3
Multilayer and Multimodal Fusion [110]	91.6
Transformation CNN [131]	92.4
Multi-Stream [112]	92.6
Key Volume Mining [132]	92.7
Convolutional Two-Stream [62]	93.5
Temporal Segment Networks [39]	94.2

, where the methods are arranged in ascending order based on the performance. The results reported in Table 6 are taken from the existing studies in the literature.

4.4. Comparison of Different Deep Learning Architectures

In

Table 7. Performance comparison of different deep architectures [127].

Architecture Name	Parameters	Error Rate	Depth	Category	Year
LeNet	0.060 M	[dist]MNIST: 0.8 MNIST: 0.95	5	Spatial exploitation	1998
AlexNet	60 M	ImageNet: 16.4	8	Spatial exploitation	2012
ZfNet	60 M	ImageNet: 11.7	8	Spatial exploitation	2014
VGG	138 M	ImageNet: 7.3	19	Spatial exploitation	2014
GoogLeNet	4 M	ImageNet: 6.7	22	Spatial exploitation	2015
Inception-V3	23.6 M	ImageNet: 3.5 multi-crop: 3.58 Single-Crop: 5.6	159	Depth + width	2015
Highway networks	2.3 M	CIFAR-10: 7.76	19	Depth + multi-path	2015
Inception-V4	35 M	ImageNet: 4.01	70	Depth + width	2016
Inception-ResNet	55.8 M	ImageNet: 3.52	572	Depth + width + multi-path	2016
ResNet	25.6 M 1.7 M	ImageNet: 3.6 CIFAR-10: 6.43	152 110	Depth + multi-path	2016
DelugeNet	20.2 M	CIFAR-10: 3.76 CIFAR-100: 19.02	146	Multi-path	2016
FractalNet	38.6 M	CIFAR-10: 7.27 CIFAR-10 +: 4.60 CIFAR-10 ++: 4.59 CIFAR-100: 28.20 CIFAR-100 +: 22.49 CIFAR100 ++: 21.49	20 40	Multi-path	2016
WideResNet	36.5 M	CIFAR-10: 3.89 CIFAR-100: 18.85	28 –	Width	2016
Xception	22.8 M	ImageNet: 0.055	126	Width	2017
Residual attention neural network	8.6 M	CIFAR-10: 3.90 CIFAR-100: 20.4 ImageNet: 4.8	452	Attention	2017
ResNeXt	68.1 M	CIFAR-10: 3.58 CIFAR-100: 17.31 ImageNet: 4.4	29 - 101	Width	2017
Squeeze and excitation networks	27.5 M	ImageNet: 2.3	152	Feature-map exploitation	2017
DenseNet	25.6 M 25.6 M 15.3 M 15.3 M	CIFAR-10 +: 3.46 CIFAR100 +: 17.18 CIFAR-10: 5.19 CIFAR-100: 19.64	190 190 250 250	Multi-path	2017
PolyNet	92 M	ImageNet: Single: 4.25 Multi: 3.45	– –	Width	2017
PyramidalNet	116.4 M 27.0 M 27.0 M	ImageNet: 4.7 CIFAR-10: 3.48 CIFAR-100: 17.01	200 164 164	Width	2017
Convolutional block attention Module (ResNeXt101 (32 × 4d) + CBAM)	48.96 M	ImageNet: 5.59	101	Attention	2018
Concurrent spatial and channel excitation mechanism	–	MALC: 0.12 Visceral: 0.09	–	Attention	2018
Channel boosted CNN	–	–	–	Channel boosting	2018
Competitive squeeze and excitation network CMPE-SE-WRN-28	36.92 M 36.90 M	CIFAR-10: 3.58 CIFAR-100: 18.47	152 152	Feature-map exploitation	2018

, some important deep learning architectures are compared in terms of performance and computational requirement. These architectures are the basis of development of different deep learning models for video classification, and from this comparison, an estimation of the requirement of computational cost for each of these architectures can be drawn.

5. Key Findings

From the analysis of the existing literature, the following key findings are drawn for video classification task: (i) The visual description works better than the text and the audio description, and the combination of all modalities can contribute to better performance with an increase in computational cost. (ii) The architectures employing CNN/RNN for feature extraction have the ability to perform better than handcrafted features provided that enough data are available for training. (iii) Tensor-Train layer-based RNN such as LSTM and GRU perform better than the plain RNN architectures for video classification. (iv) It is sometimes necessary to use optical flow for datasets such as UCF-101. (v) It is not always helpful to use optical flow, especially for the case of videos taken from the wild, e.g., Sports-1 M. (vi) It is important to use a sophisticated sequence processing architecture such as LSTM to take advantage of optical flow. (vii) LSTMs, when applied on both the optical flow and the image frames, yield the highest performance measure for the Sports-1M benchmark dataset. (viii) Augmenting optical flow and RGB input helps in improving the performance. (ix) Optical flow modality provides complementary information. (x) The high computational requirement of optical flow limits its use in real-time systems. (xi) Multi-stream multi-class fusion can perform better than average fusion, weighted fusion, kernel average fusion, MKL fusion, and logistic regression fusion on datasets such as UCF-101 and CCV. (xii) In 3D group convolutional networks, the volume of channel interactions plays a vital role in achieving a high accuracy. (xiii) The factorization of 3D convolutions by separating spatiotemporal interactions and channel interactions can lead to an improvement in accuracy and a decrease in the computational cost. (xiv) Further, 3D channel-separated convolutions results in a kind of regularization and prevents overfitting. (xv) Popular frameworks of conventional semi-supervised algorithms (which were originally developed for 2D images) are unable to obtain good results for 3D video categorization. (xvi) For semi-supervised learning, a calibrated employment of the object appearance cues keenly improves the accuracy of the 3D-CNN models.

6. Conclusions

This article reviews deep learning approaches for the task of video classification. Some of the notable studies are summarized in detail, and the key findings in these studies are highlighted. The key findings are reported as an effort to help the research community in developing new deep learning models for video classification.

The latest developments in deep learning models have demonstrated the potential of these approaches for the video classification task. However, most of the existing deep learning architectures for video classification are basically adopted from the favored deep learning architectures in image/speech domain. Therefore, most of the existing architectures remain insufficient to deal with the more complicated nature of video data that contain rich information in the form of spatial, temporal, and acoustic clues. This calls for attention towards the need for a tailored network capable of effectively modeling the spatial, temporal, and acoustic information. Moreover, training CNN/RNN models requires labeled datasets, and acquiring those datasets is usually time-consuming and expensive, and hence, a promising research direction is to utilize the considerable amount of unlabeled video data to derive better video representations.

Furthermore, the deep learning approaches are outperforming other state-of-the-art approaches for video classification. The deep learning Google trend is still growing, and it is still above the trend for some other very well-known machine learning algorithms, as shown in Figure 9a. However, the recent developments in deep learning approaches are still under-evaluated and require further investigations for the video classification task. One such example is geometric deep learning approaches, and the worldwide research interest in this specific topic is shown in Figure 9b, which describes that this topic is still confined to some states of U.S., Europe, and India. Therefore, it has yet to be developed and investigated further. The use of geometric deep learning in extracting rich spatial information from videos can also be a new research direction as a future work for better accuracy in the video classification task.