Video Summarization Generation Based on Graph Structure Reconstruction

Abstract

Video summarization aims to identify important segments in a video and merge them into a concise representation, enabling users to comprehend the essential information without watching the entire video. Graph structure-based video summarization approaches ignore the issue of redundant adjacency matrix. To address this issue, this paper proposes a video summary generation model based on graph structure reconstruction (VOGNet), in which the model first adopts a variational graph auto-encoders (VGAE) to reconstruct the graph structure to remove redundant information in the graph structure; followed by using the reconstructed graph structure in a graph attention network (GAT), allocating different weights to different shot features in the neighborhood; and lastly, in order to avoid the loss of information during the training of the model, a feature fusion approach is proposed to combine the training obtained shot features with the original shot features as the shot features for generating the summary. We perform extensive experiments on two standard datasets, SumMe and TVSum, and the experimental results demonstrate the effectiveness and robustness of the proposed model.

1. Introduction

With the widespread application of digital camera technology and the Internet, massive amounts of video data have been generated, but their management and utilization are facing challenges in storage, browsing, and searching. Consequently, video summarization techniques have become the focus of research, aiming to achieve compressed storage, fast browsing and efficient searching of video data through automated techniques. Video summarization enables users to grasp the main content of a video rapidly without watching the entire video by extracting key frames or key shots from the original video [1,2,3]. According to the perspective of generating summary content, they are divided into two categories: static video summarization and dynamic video summarization [4]. Static video summarization concludes video content utilizing a set of keyframes, which typically contain only image information, whereas dynamic video summarization consists of video clips, combining a variety of information including sound, images, and text to provide richer content for the user. Under this background, we choose to study dynamic video summarization as it enables a more comprehensive consideration of multiple messages, including sound, images and text, to better meet the needs of information delivery and user experience.

Deep learning methods, especially those based on Recurrent Neural Network (RNN) and Long Short-Term Memory Network (LSTM), have made significant achievements in the field of video summarization [5,6,7,8]. With the introduction of RNN, it is possible to capture the temporal information in the video more effectively, extract the contextual semantic information, and generate more accurate and coherent video summarization [7]. Although designed to capture long-term dependencies, LSTM still faces challenges in dealing with dependencies in very long sequences [9]. Deep learning methods perform excellently in handling Euclidean spatial data, but are ineffective in handling non-Euclidean spatial data. To solve this problem, researchers have introduced Graph Neural Network [10], exhibiting excellent performance. The researchers transform the video into a graph structure and capture the complex dependencies between frames through a GNN, providing a novel approach to video summarization [7,11,12]. Nonetheless, the problem of long-term dependency in the video modeling process remains a notable challenge. Moreover, the graph structures generated based on videos could suffer from large redundancy, hindering the effective learning of structural features of videos. Therefore, addressing these issues remains a key challenge that needs to be addressed in the domain of video summarization.

The GNN is used to model the video in this study for solving the long-term dependency problem of RNN. First, frame features are extracted by GoogleNet [13], and then a representative algorithm is utilized to select representative frame features from them as the features of the current shot. Then, a graph structure is constructed by using these shot features as nodes and using the similarity between them as edges. To further optimize the representation of the video graph, we employ a VGAE to reconstruct the original video graph and remove the redundancy of the adjacency matrix. The graph structure is optimized by introducing a GAT, and aggregation is weighted according to the importance of neighbor information. Besides that, we utilize the raw and structural features of the shots for feature fusion to improve the reasonableness of scoring the importance of the shots. This comprehensive approach effectively combines algorithms such as GNN, VGAE, and GAT to more accurately capture key information in the video, optimize the graph structure, and improve the accurate assessment of shot significance.

The contribution of the method to video summarization is reflected in three aspects:

1.

Transformation of the video into nonlinear data and building of a graph structure in which the shot features are the nodes and the inter-shot similarities are the edges, thus overcoming the long-term dependency problem, effectively extracting the dependencies between shots, and compensating for the limitations of LSTM.

2.

Application of VGAE to reconstruct the graph of the original video for redundant information removal and aggregation of the neighborhood information of the shot nodes by GAT.

3.

Feature fusion of raw and aggregated features of video shots to evaluate the significance of video shots in a more rational way.

The remainder of the paper is organized as follows. Section 2 reviews the related work. Section 3 provides a detailed description of the proposed method. Section 4 provides the results and analysis of the experiments. Finally, conclusions and insights are drawn in Section 5.

2. Related Work

We first present the existing research in the area of video summarization, followed by an introduction to variational graph auto encoders and graph attention network.

2.1. Video Summrization

Existing video summarization is mainly implemented using deep learning methods. In recent years, Recurrent Neural Networks (RNNs) have been widely used to process sequential data. For instance, Zhang et al. [14] propose a video summarization model based on long short-term memory network (LSTM), which utilizes LSTM to model the variable range of temporal dependencies between video frames. LSTM suffers from the problem of long-term dependency when dealing with long sequence data. In order to tackle this problem, Zhao et al. [6] propose a hierarchical recurrent neural network (H-RNN), which is divided into two layers. The first layer of the network employs LSTM to capture the short-term temporal dependencies between frames within a sub-shot, and the second layer of the network uses a bi-directional long short-term memory network (Bi-LSTM) to extract the long-term dependencies between shots. Considering that people focus on key shots, Ji et al. [15] apply the attention mechanism to LSTM. They introduce an encoder–decoder video summarization network based on the attention mechanism, which encodes contextual information between video frames using a bidirectional long short-term memory network (Bi-LSTM), and explore two types of attention-based LSTM networks in terms of the decoder. Although RNN has achieved great success in processing Euclidean spatial data, it is not applicable to processing non-Euclidean spatial data. For the purpose of better mining the hidden structural relationships in videos, video data are constructed into graph structures and graph neural networks, such as Graph Convolutional Neural Network (GCN) [10], are used to learn the topology of the graphs. The idea of GCN is to use a low-dimensional vector to represent the high-dimensional information of the nodes in the graph, which can capture the global information of the graph and better represent the characteristics of the nodes. Park et al. [16] present a recursive graph modeling network (SumGraph) for video summarization, which takes video frames as the nodes of the graph, and the nodes are connected by semantic relationships between frames as the graph edges. Zhao et al. [7] propose a reconstructed sequence-graph network (RSGN) that hierarchically encodes frames and shots as sequences and graphs, where frame-level correlations are encoded by long short-term memory (LSTM) and shot-level correlations are captured by a graph convolutional network (GCN). Li et al. [17] propose that GCAN has two modules, an embedding learning module and a feature fusion module. One branch of its embedding learning module extracts temporal features using DTC and TSA, and the other extracts spatial features using GCN. The fusion module uses the fusion gate mechanism to fuse the original, temporal, and spatial features to generate summary. To highlight critical information and filter irrelevant information, Zhong et al. [7] use the graph attention mechanism GAT to merge the influence of related neighboring nodes, convert the visual features of nodes to higher-level features, and adjust the weights of related nodes.

In conclusion, video summarization is studied, and initially, the sequence information in video summary is successfully extracted by RNN, and then the concept of hierarchical is introduced to solve the long-term dependency problem of RNN. Since LSTM cannot effectively handle non-Euclidean spatial data, graph neural networks are then introduced to learn graph structures.

2.2. Variational Graph Auto Encoders (VGAE)

Graph neural networks are applied with great success to extract features from non-Euclidean spatial data. We classify graph neural network into five main categories, namely graph convolutional networks (GCNs) [10], graph attention networks (GATs) [18], graph autoencoders (GAEs) [19], variational graph autoencoders (VGAEs) [19], and graph sample and aggregates (GraphSAGEs) [20]. This paper focuses on exploring VGAEs and GATs.

Kingma et al. [19] introduce the variational autoencoder (VAE), a generative model that learns latent variables from input data and generates new samples. It is an unsupervised learning algorithm that learns without labeling the data. Kipf et al. [19] migrate VAE to the graph domain and present the variational graph self-encoder. The encoder learns a distribution represented by node vectors using a known graph structure after encoding, after which it samples in the distribution and finally decodes to realize graph structure reconstruction. The method in this paper employs a variational graph auto-encoder to reconstruct the graph structure of the constructed video by feeding the constructed adjacency matrix into the VGAE, allowing the model the learning of a number of distributions using hidden variables, then sampling from these distributions to obtain the latent representation vectors, that is, the encoding part, and then reconstructing the original graph, that is, the decoding part, using the obtained latent representations. Specifically, the encoding part of VGAE uses GCN, while the decoding part is a simple inner product. Finally, the reconstructed graph is used as input for the next part of the model. The purpose of using VGAE in this paper is to learn the potential structural features in the video through VGAE and obtain the graph structure that can better represent video information.

2.3. Graph Attention Network (GAT)

Kipf et al. [21] offered a fresh perspective by proposing graph convolutional networks (GCNs) that fuse graph structural features in a convolutional manner. They motivate the choice of convolutional architecture by a local first-order approximation of the spectral graph convolution. Hidden layer representations of local graph structure and node features are encoded using models. However, when GCN fuses features of graph structures, the weights of the edges of the graph are fixed and not flexible enough. Addressing this problem, Veličković et al. [18] introduced the graph attention network (GAT). This network assigns different weights to different nodes in the neighborhood when dealing with neighborhoods of different sizes and does not rely on knowing the entire graph structure in advance. Specifically, by calculating the attention coefficients, namely the weights, of the current node and its neighbors, and weighting them when aggregating neighbors, the graph neural network is able to pay more attention to important nodes in order to reduce the impact of noise, and also improve the robustness of the model and endow it with a certain degree of interpretability. Zhong et al. [7] applied the graph attention network to video summarization. They utilized a bi-directional long short-term memory network (Bi-LSTM) unsupervised video summarization model based on a graph attention network (GAT) to solve the problem of high redundancy between key frames.

To summarize, GCN can effectively extract the spatial features of the video, but when fusing the features of the graph structure, GAT can assign different weights to different nodes in the neighborhood according to the attention, which can better extract the spatial features of the video.

3. Model

As shown in Figure 1, a Video Summarization Generation Network Based on Graph Structure Reconstruction (VOGNet) is proposed for removing redundancies in graph structures and learning structural features of videos. The major components of this network include three key parts: feature extraction, graph structure reconstruction and feature fusion. For feature extraction, we first extract features from video frames followed by selecting representative frames as shot features. The shot features are used as nodes of the graph, while the original graph is constructed using the similarity between shots as edges. Then, a VGAE is employed to reconstruct the structure of the original graph, learning the structure of potentially information rich graphs. After that, by introducing GAT, we process the reconstructed graph to aggregate the neighbor information of the nodes and make the learned node features more stable and representational through different weights.

3.1. Feature Extraction

First, the original video is defined as $\mathrm{F}={\left\{{\mathrm{f}}_{\mathrm{n}}\right\}}_{\mathrm{n}=1}^{\mathrm{N}}$, $f_n\in {\mathbb{R}}^{w\times h\times 3}$, where ${\mathrm{f}}_{\mathrm{n}}$ denotes the $\mathrm{n}$th video frame, $\mathrm{N}$ denotes the total number of frames of the video, $w$ denotes the width of each image frame, $h$ denotes the length of each image frame, and 3 denotes the number of channels of the image. The specific process of feature extraction is as follows.

3.1.1. Shot Split

First, the KTS algorithm [22] is utilized to segment the original video into a number of shots of varying lengths, and the set of shots of the original video is denoted as ${\mathrm{S}=\left\{{\mathrm{s}}_{\mathrm{m}}\right\}}_{\mathrm{m}=1}^{\mathrm{M}}$, ${\mathrm{s}}_{\mathrm{m}}\in {\mathbb{R}}^{\mathrm{w}\times \mathrm{h}\times 3\times {\mathrm{T}}_{\mathrm{M}}}$, M denotes the total number of shots in which the video is segmented, and ${\mathrm{s}}_{\mathrm{m}}$ denotes that there are ${\mathrm{T}}_{\mathrm{m}}$ video frames in the mth shot and $\mathrm{N}={\sum }_{\mathrm{m}=1}^{\mathrm{M}}{\mathrm{T}}_{\mathrm{m}}$.

3.1.2. Calculation of Shot Features

After the collection of shots is obtained, the depth features of each frame are extracted using GoogLeNet. Since shots are short sequences of consecutive frames that are captured in a single shot and characterize the same semantic action, the visual features extracted by GoogLeNet for each frame in the shot are relatively similar. And since the number of video frames in each shot is different, the dimensions of the shot features are not the same. In order to construct the video summary graph structure, in this paper, the features of representative frames in each shot are selected as the features of a shot, and the shot features are denoted as $\mathrm{G}={\left\{{\mathrm{s}}_{\mathrm{m}}^{\prime }\right\}}_{\mathrm{m}=1}^{\mathrm{M}}$, $s_m^{\prime }\in {\mathbb{R}}^{F\times T_m}$, where $s_m^{\prime }$ is the depth video feature of shot ${\mathrm{s}}_{\mathrm{m}}^{\prime }$, $F$ is the feature dimension of frames, and $T_m$ is the number of frames in shot ${\mathrm{s}}_{\mathrm{m}}$. The average of the 2 parameters between the frames in a shot and the rest of the frames is computed, with the representative frame being the one with the lowest average.

$$\mathrm{represent}\mathrm{frame}=exp(-\frac{1}{{\mathrm{T}}_{\mathrm{m}}}\sum _{\mathrm{t}=1}^{{\mathrm{T}}_{\mathrm{m}}}\underset{{\mathrm{t}}^{\prime }\in \mathrm{y}}{min}\left|\right|{\mathrm{x}}_{\mathrm{t}}-{\mathrm{x}}_{{\mathrm{t}}^{\prime }}{\left|\right|}_2).$$

(1)

We denote by $\mathrm{X}$ the nodes (shot features) of the graph.

$$\mathrm{X}={\mathrm{s}}_{\mathrm{represent}\mathrm{frame}}$$

(2)

where ${\mathrm{s}}_{\mathrm{represent}\mathrm{frame}}\in {\mathbb{R}}^{\mathrm{F}\times \mathrm{M}}$ denotes the features of the representative frame.

3.1.3. Adjacency Matrix Construction

We construct adjacency matrix A using the extracted node features X. There are four general ways to construct the adjacency matrix:

Dot Product [11]:

$${\mathrm{e}}_{\mathrm{ij}}=\mathrm{f}({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}})=-\varphi {\left({\mathrm{x}}_{\mathrm{i}}\right)}^{\mathrm{T}}\phi \left({\mathrm{x}}_{\mathrm{j}}\right);$$

(3)

Gaussian [11]:

$${\mathrm{e}}_{\mathrm{ij}}=\mathrm{f}({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}})=exp\{-\varphi {\left({\mathrm{x}}_{\mathrm{i}}\right)}^{\mathrm{T}}\phi \left({\mathrm{x}}_{\mathrm{j}}\right)\};$$

(4)

Concatenation [11]:

$${\mathrm{e}}_{\mathrm{ij}}=\mathrm{f}({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}})={\mathrm{W}}_{\mathrm{e}}^{\mathrm{T}}[\varphi \left({\mathrm{x}}_{\mathrm{i}}\right)\left|\right|\phi \left({\mathrm{x}}_{\mathrm{j}}\right)];$$

(5)

Cosine similarity [23]:

$${\mathrm{e}}_{\mathrm{ij}}=\mathrm{f}({\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}})=\frac{{\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}{\mathrm{x}}_{\mathrm{j}}}{\left|\right|{\mathrm{x}}_{\mathrm{i}}{\left|\right|}_2·\left|\right|{\mathrm{x}}_{\mathrm{j}}{\left|\right|}_2},$$

(6)

where x is each node, $\left|\right|$ denotes the connection operation. $\varphi (·)$ and $\phi (·)$ are linear transformations based on $W_{\varphi }$, $W_{\phi }$. $W_{\varphi }$, $W_{\phi }$, $W_e$ are all learnable parameters. When it comes to video summarization, there is not a defined graph structure. To create one, we consider the shot features as the graph’s nodes, while the edges are determined by comparing the similarity of the two nodes. According to literature [11], we can obtain the edge weight calculated using the dot product, and the F1 value obtained by this training is slightly higher than the values obtained by the two methods of Gaussian and concatenation. According to literature [24], it is proven that cosine similarity is more suitable for calculating the similarity between two nodes, so this paper uses cosine similarity to calculate the similarity of two nodes.

$${\mathrm{e}}_{\mathrm{ij}}={\frac{{\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}{\mathrm{x}}_{\mathrm{j}}}{\left|\right|{\mathrm{x}}_{\mathrm{i}}{\left|\right|}_2·\left|\right|{\mathrm{x}}_{\mathrm{j}}\left|\right|}}_2,$$

(7)

where $\mathrm{i},\mathrm{j}=1,2,\dots ,\mathrm{M}$ and ${\mathrm{x}}_{\mathrm{i}}$ denotes the feature of the $\mathrm{i}$th shot. $\left|\right|{\mathrm{x}}_{\mathrm{i}}{\left|\right|}_2$ denotes the L2 norm for ${\mathrm{x}}_{\mathrm{i}}$.

At this point, we have a complete graph $\mathrm{G}=(\mathrm{X},\mathrm{E})$. $\mathrm{X}$ is the set of nodes of the graph, specifically consisting of the video shot features $\{{\mathrm{x}}_1,{\mathrm{x}}_2\dots {\mathrm{x}}_{\mathrm{M}}\}$. E is the set of edges of the graph, consisting of $\{{\mathrm{e}}_{11},\dots ,{\mathrm{e}}_{\mathrm{ij}},\dots {\mathrm{e}}_{\mathrm{MM}}\}$. ${\mathrm{e}}_{\mathrm{ij}}$ is the similarity of two nodes, i.e., the adjacency matrix $\mathrm{A}\in {\mathbb{R}}^{\mathrm{M}\times \mathrm{M}}$ is obtained.

3.2. Reconstruction the Adjacency Matrix

VGAE consists of an encoder and a decoder. The encoder learns the mean $\mu$ and variance $\log \sigma$ of the node’s low-dimensional vector representation, uses the posterior probability $\mathrm{q}\left(\mathrm{Z}\right|\mathrm{A},\mathrm{X})$ to obtain the hidden variable $\mathrm{Z}$, and then reconstructs the adjacency matrix A with the hidden variable $\mathrm{Z}$.

The encoder is implemented by a two-layer graph convolutional network:

$$\mathrm{GCN}(X,A)=\tilde{\mathrm{A}}\mathrm{ReLU}\left(\tilde{\mathrm{A}}\mathrm{X}{\mathrm{W}}_0\right){\mathrm{W}}_1,$$

(8)

where $W_0$ is the weight matrix of the first layer, $\tilde{\mathrm{A}}={\mathrm{D}}^{-\frac{1}{2}}{\mathrm{AD}}^{-\frac{1}{2}}$ is the symmetric normalized adjacency matrix, and $\mathrm{D}$ is the degree matrix.

Here are the specific steps:

The node features X and the adjacency matrix A are fed into the encoder of VGAE to obtain the mean $\mu$ and variance $\sigma$ of the low-dimensional vector representation of the node. Expressions for $\mu$ and $\sigma$ are computed, respectively:

$$\begin{array}{cc}& \mu ={\mathrm{GCN}}_{\mu }(X,A),\end{array}$$

(9)

$$\begin{array}{cc}& \log \sigma ={\mathrm{GCN}}_{\sigma }(X,A).\end{array}$$

(10)

In normal distribution $\mathcal{N}(\mu ,{\sigma }^2)$ by the nodes’ low-dimensional vectors, we sample hidden variables Z. The posterior probability distribution of the hidden variable Z is

$$\begin{array}{cc}& q\left(Z|X,A\right)={\prod }_{i=1}^{N}q\left(z_i\right|X,A),\end{array}$$

(11)

$$\begin{array}{cc}& q\left(z_i|X,A\right)=\mathcal{N}\left(z_i\right|{\mu }_i,diag\left({\sigma }_i^2\right)).\end{array}$$

(12)

It is required that each sample of VGAE has a corresponding normal distribution, which can make a one-to-one correspondence between $z_i$ obtained from sampling and the real samples, which can ensure the learning effect of the model.

The sampling process cannot be derived, which means that the model cannot perform backpropagation and gradient descent. In order to resolve this problem, reparameterization is introduced in order to transform the sampling relationship into a deterministic one.

$$z=\mu +\sigma \odot ϵ,ϵ\sim \mathcal{N}(0,I).$$

(13)

To reconstruct the graph structure, the decoder uses an inner product calculation between two hidden variables.

$$\begin{array}{cc}& p\left(A|Z\right)={\prod }_{i=1}^N{\prod }_{j=1}^{N}p\left(A_{ij}\right|z_i,z_j),\end{array}$$

(14)

$$\begin{array}{cc}& p\left(A_{ij}=1|z_i,z_j\right)=\sigma \left(z_i^T{z}_j\right),\end{array}$$

(15)

where $\sigma (·)$ is sigmoid. We reconstruct adjacency matrix $\widehat{\mathrm{A}}$ using the inner product of hidden variables $\mathrm{Z}$.

$$\widehat{A}=\mathrm{logistic}\mathrm{sigmoid}\left({\mathrm{ZZ}}^{\mathrm{T}}\right).$$

(16)

3.3. Optimizing the Adjacency Matrix

In our model, we utilize the GAT to consider the weight information of various neighbors. This helps us optimize the adjacency matrix and minimize its redundancy.

We calculate the attention scores of the central node and the neighboring nodes.

$$e_{ij}=\varphi (W_{X_i},W_{X_j}),$$

(17)

where $e_{ij}$ denotes the importance of the $\mathrm{j}$th node to the $\mathrm{i}$th node, i.e., the attention score. $X_i$ and $X_j$ represent the features of the $\mathrm{i}$th node and the $\mathrm{j}$th node. $\varphi$ denotes the method of computing the attention score, where the inner product operation is commonly used. $W$ is a learnable linear transformation $\mathrm{W}\in {\mathbb{R}}^{\mathrm{F}\times {\mathrm{F}}^{\prime }}$ of the graph attention GAT setup that transforms the input features into higher-level features.

Our model makes use of MLP, where we define the a-vector dimension as twice the dimension of the mapped node features. We splice and compute the inner product with the a-vector by columns.

Attention scores of nodes and neighboring nodes are

$${\mathrm{e}}_{\mathrm{ij}}=\mathrm{a}({\mathrm{W}}_{{\mathrm{X}}_{\mathrm{i}}},{\mathrm{W}}_{{\mathrm{X}}_{\mathrm{j}}}).$$

(18)

Activating Attention Score $e_{ij-GAT}$,

$$e_{ij-GAT}=\mathrm{LeakyRe}\mathrm{L}\mathrm{U}({\mathrm{a}}^{\mathrm{T}}[{\mathrm{W}}_{{\mathrm{X}}_{\mathrm{i}}}\left|\right|{\mathrm{W}}_{{\mathrm{X}}_{\mathrm{j}}}\left]\right),$$

(19)

where $\mathrm{T}$ denotes the transpose and $[·||·]$ denotes the concatenation operation.

We updated attention scores and optimized the graph structure. In order to highlight the key information and optimize the structure of the graph, we use the reconstructed adjacency matrix $\widehat{\mathrm{A}}$ and the activated attention scores ${\mathrm{e}}_{\mathrm{ij}-\mathrm{GAT}}$ to perform matrix multiplication to obtain updated attention scores ${\tilde{\mathrm{e}}}_{\mathrm{ij}}$.

$${\tilde{\mathrm{e}}}_{\mathrm{ij}}=\widehat{\mathrm{A}}\otimes {\mathrm{e}}_{\mathrm{ij}-\mathrm{GAT}}.$$

(20)

We performed attention score normalization. We normalized all neighboring nodes of node i using softmax function

$${\tilde{\mathrm{e}}}_{\mathrm{ij}}=\mathrm{softmax}\left({\tilde{\mathrm{e}}}_{\mathrm{ij}}\right)=\frac{exp\left({\tilde{\mathrm{e}}}_{\mathrm{ij}}\right)}{{\sum }_{\mathrm{k}\in {\mathrm{N}}_{\mathrm{i}}}exp\left({\tilde{\mathrm{e}}}_{\mathrm{ik}}\right)},$$

(21)

where $\mathrm{k}\in {\mathrm{N}}_{\mathrm{i}}$ denotes the neighbor information of node $\mathrm{i}$.

3.4. Feature Fusion

Feature fusion can improve the quality of key shot selection. To achieve this, the fusion feature F is obtained by combining shot structural features ${\tilde{\mathrm{e}}}_{\mathrm{ij}}$ that were optimized by VGAE and GAT, along with shot depth features $\mathrm{G}$ extracted by GoogleNet. The resulting $\mathrm{F}$ is then inputted into the MLP to obtain the shot prediction score, ${\mathrm{S}}_{\mathrm{pred}}$, for the network. By using feature fusion, it becomes possible to evaluate the importance of video shots in a more rational way.

$${\mathrm{S}}_{\mathrm{pred}}=\mathrm{MLP}\left(\mathrm{F}\right)=\mathrm{MLP}\left(\mathrm{cat}({\tilde{\mathrm{e}}}_{\mathrm{ij}},\mathrm{G})\right).$$

(22)

3.5. Loss Function

The loss function for VGAE, called ${\mathrm{L}}_{\mathrm{VGAE}}$, includes two components: reconstruction loss and $\mathrm{KL}$ divergence. To determine the distance between generated and original graphs, the reconstruction loss uses cross-entropy loss. Meanwhile, $\mathrm{KL}$ divergence calculates the distance between the node representation vector distribution and the normal distribution.

$$\begin{array}{c}\hfill {\mathrm{L}}_{\mathrm{VGAE}}={\mathrm{E}}_{\mathrm{q}}\left(\mathrm{Z}|\mathrm{X},\mathrm{A}\right)\left[{\log }_{\mathrm{p}}\left(\mathrm{A}|\mathrm{Z}\right)\right]\\ \hfill +KL\left[q\right(Z|X,A)\left|\right|p\left(Z\right)].\end{array}$$

(23)

VOGNet’s loss function L uses the mean square loss function MSELoss to represent the gap between predicted values and user ratings.

$$\mathrm{L}={({\mathrm{S}}_{\mathrm{real}}-{\mathrm{S}}_{\mathrm{pred}})}^2,$$

(24)

where ${\mathrm{S}}_{\mathrm{real}}$ denotes the real score of the shots level and ${\mathrm{S}}_{\mathrm{pred}}$ denotes the score predicted by the model.

4. Experiments

For this section, we describe two table-aligned datasets and evaluation metrics for video summarization.

4.1. Datasets

Two standard datasets, the SumMe [25] dataset and the TVSum [26] dataset, are employed in this experiment. The SumMe dataset consists of 25 videos recorded by users. The duration of each video ranged from 1.5 to 6.5 min, with a minimum duration of 32 s and a maximum duration of up to 324 s, and an average duration of about 146 s. There are 15–18 manual annotations for each video, and these annotations are categorized as important (0) and unimportant (1). The TVSum dataset includes a total of 50 videos covering a variety of topics including news stories, recorded articles, operating tips, and video blogs. These videos are between 2 and 10 min long, with a minimum of 83 s and a maximum of 647 s, for an average duration of about 235 s. One to five different scores are assigned to each video after it is viewed by 20 users, and the average of these scores is used as the final rating result. Both SumMe and TVSum offer us frame-level scores. Shots are segments made up of sequences of frames, and within the same shot, the frames typically vary relatively steadily. Since our experiments require shot-level scores, we utilize the KTS algorithm to divide the video into video segments of varying lengths. Following modeling, we obtain shot-level scores and ultimately generate a summary using a 0/1 backpacking algorithm.

During the experimental stage, we split the dataset in the ratio of 80% for the training set and 20% for the test set. In contrast to the previous segmentation of the overall dataset, we perform the segmentation of the training and test sets after randomly disrupting the video numbers. To ensure that the experimental results are broadly applicable, we use Python’s random library to randomly divide the data into K parts, and then perform a K-fold cross-validation, taking the average of the K values as the final result.

4.2. Evaluation of Indicators

When it comes to assessing video summarization studies, the $\mathrm{F}1$ score is the most frequently utilized evaluation indicator. In order to ensure that the $\mathrm{F}1$ score is precise, it must be determined via $\mathrm{precision}$ and $\mathrm{recall}$. Precision denotes the ratio between the accurate prediction and the model’s prediction, while recall measures the ratio of correctly predicted outcomes to manually predicted ones.

$$\mathrm{Precision}=\frac{\mathrm{N}\_\mathrm{match}}{\mathrm{N}\_\mathrm{extract}},$$

(25)

where $N\_match$: is the total number of frames for which the manual and the model can make a match; $N\_extract$ is the number of frames generated by the model.

$$\mathrm{Recall}=\frac{\mathrm{N}\_\mathrm{match}}{\mathrm{N}\_\mathrm{US}},$$

(26)

where $N\_US$ is the number of manually labeled video summarization frames.

$$\mathrm{F}1=\frac{2\ast \mathrm{Precision}\ast \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.$$

(27)

4.3. Results

In this section, we first compare the proposed model with RNN-based methods, GNN-based methods and Reconstruction-based methods as well as state-of-the-art video summarization methods. The model is then subjected to ablation experiments to verify the validity of each module. Lastly, the results of the experiments are visualized.

4.3.1. Contrast Experiment

Table 1. Comparison of F1 scores with existing advanced methods.

Method	SumMe	TVSum
vsLSTM [14]	37.6	54.2
DPP-LSTM [14]	38.6	54.7
SUM-GAN [8]	41.7	56.3
DR-DSN [27]	42.1	58.1
CSNetsup [28]	48.6	58.5
HAS-RNN [5]	44.1	59.8
M-AVS [29]	44.4	61.0
FCSN [30]	48.8	58.4
RSGN [11]	45.0	60.1
GATVSunsup [7]	51.5	59.1
GCANsup [17]	53.0	60.7
VOGNet (ours)	49.8	60.8

shows the results of comparing the F1 scores of this paper’s method to those of other state-of-the-art methods on the two standards datasets. In particular, vsLSTM and dppLSTM introduce the determinatal point process (DPP) to obtain representative and concise video summary of the sequence. SUM-GAN is an unsupervised video summarization model based on variational cyclic auto-encoder and generative adversarial network which consists of a summarizer and a discriminator. The purpose of the summarizer is to generate a summary to fool the discriminator, and the purpose of the discriminator is to identify the summarized video from the original video. DR-DSN is trained unsupervised by reinforcement learning by designing different reward functions to obtain a more representative and diverse video summary. CSNetsup designs a simple but effective regularized loss term, called variance loss and also designs a chunk and stride network (CSNet) that utilizes both local (chunk) and global (stride) temporal views of video features. HAS-RNN is a hierarchical structured adaptive network used in order to preserve the hierarchical structure of video data and improve the quality of the generated summary. M-AVS is divided into the encoding part and the decoding part, and introduces an attention mechanism in the decoder part. Concretely, the encoding part is Bi-LSTM and the decoding part is Bi-LSTM with the introduction of attention. FCSN describes video summarization as a sequential problem, establishes a new link between semantic segmentation and video summarization, and applies semantic segmentation network to video summarization. RSGN divides the video into a frame level and a shot level, and the frame-level dependencies are represented by the long short-term memory network (LSTM) to encode them as sequences, and shot-level dependencies are encoded as graphs, which are implemented by GCN. GATVSunsup uses GAT to aggregate the frame features into depth features, and after that proposes a new spatial attention model to extract the visual features of the frames, and finally combines the high-level visual features with the sequence features extracted by BiLSTM. GCANsup consists of Embedding Learning and Context Fusion; embedding learning is subdivided into temporal branching and graph branching. It learns the embedding representation of the graph through a multilayer graph convolutional network.

The data in Table 1 show that the proposed method achieves high results on both datasets, which proves the effectiveness and robustness of the method. Although the results of this paper’s method on the SumMe dataset are 3.2% lower than those of GCANsup, the F1 scores on TVSum are higher than those of GCANsup. The results of M-AVS on the TVSum dataset are 0.2% higher than the results of the proposed method, but on the SumMe dataset, the results are 5.4% lower than those of the proposed method. Although the method does not reach the highest level on both datasets individually, it is comparable to the highest level on both datasets compared to other state-of-the-art methods, proving the generalization of our method.

4.3.2. Ablation Experiment

To further validate the effects of VGAE and GAT on the video summary generation model, we design three ablation experiments on two datasets, SumMe and TVSum, namely M_model, G_model, and VG_model, where M_model indicates that the model of the experiment contains only multilayer perceptron, G_model indicates that the model contains GAT and does not contain VGAE, and VG_model indicates that the model includes VGAE and GAT. The experimental results are shown in

Table 2. F1 score results of ablation experiments on SumMe and TVSum datasets.

Model	SumMe	TVSum
M_model	48.9	60.8
G_model	45.3	58.9
VG_model	49.8	60.8

.

From Table 2, it can be observed that the F1 score of G_model is lower than that of M_model on both datasets, which proves that the features learned by multilayer perceptron are superior to those learned by GAT, and the probable explanation for this is that there is a large amount of redundant information in the adjacency matrix of the GAT inputs. Comparison of VG_model and M_model shows that on the SumMe dataset, the F1 score of VG_model is 0.9% higher than that of M_model, and on the TVSum dataset, the results of the two models are comparable. Further comparing VG_model and G_model, the F1 score of VG_model is 4.5% higher than that of G_model on the SumMe dataset and 1.9% higher on the TVSum dataset. From the above analysis, it can be concluded that the G_model using GAT has lower performance due to the redundant information in the graph, while the VG_model consisting of VGAE and GAT substantially improves the performance of the model because of the reconstruction of the graph structure by VGAE, which removes the redundant information in the graph.

4.3.3. Visualization Result

In our experiments on video summarization visualization, we utilized our VOGNet model and compared it with ground truth to evaluate the quality of the summarization generated. The video examples shown in Figure 2 are from the 15th and 25th videos in SumMe, and the 37th and 39th videos in TVSum. A comparison of the graphs reveals that VOGNet generated a summary that closely matches the ground truth and accurately selects the keyframes.

Figure 3 shows the hot map of the shot adjacency matrix, where darker colors in the hot map indicate higher similarity. From Figure 3, it can be seen that the highest similarity is found at the diagonal line, which is due to the cosine similarity value of one between itself and itself. Additionally, we can observe that the hot map is roughly symmetric because the adjacency matrix of the undirected graph generated by the model is symmetric.

5. Conclusions

A video summarization generation model based on graph structure reconstruction is proposed in this paper. It is designed to remove the redundant information in the adjacency matrix through graph structure reconstruction and effectively extract the structural features of the video. Specifically, the video is first shot-sliced and frame features are extracted, and then the features of representative frames are selected from them as the features of the current shot. Next, with the shot features as nodes, and using the similarity between the shots as the weights of the edges, the graph is constructed to obtain the adjacency matrix. The original graph is input into VGAE to obtain the reconstructed graph structure, and then the neighbor information in the graph structure is learned through the processing of GAT. Finally, to avoid losing critical information during model training, we perform feature fusion of the GAT output with the original shot features to obtain the importance scores of the shots and generate a summary. In the coming research, we intend to extend the study of video summarization to the multimodal domain to fuse video, audio, and textual information more comprehensively. It will contribute to the generation of richer, more informative summary that will enable users to better understand video content. Given the limited number of labeled datasets and the huge human and material investment required for the labeling process, we will consider exploring unsupervised methods to reduce the reliance on large amounts of labeled data. Aiming to further improve the quality of generated summary, we intend to design new evaluation metrics to more accurately measure model performance. Furthermore, taking into account the importance of user privacy protection, we will work on ways to protect private information in videos in video summarization generation.