Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models

Abstract

In the quest for industrial efficiency, human performance within manufacturing systems remains pivotal. Traditional time study methods, reliant on direct observation and manual video analysis, are increasingly inadequate, given technological advancements. This research explores the automation of time study methods by deploying deep learning models for action segmentation, scrutinizing the efficacy of various architectural strategies. A dataset, featuring nine work activities performed by four subjects on three product types, was collected from a real manufacturing assembly process. Our methodology hinged on a two-step video processing framework, capturing activities from two perspectives: overhead and hand-focused. Through experimentation with 27 distinctive models varying in viewpoint, feature extraction method, and the architecture of the segmentation model, we identified improvements in temporal segmentation precision measured with the F1@IoU metric. Our findings highlight the limitations of basic Transformer models in action segmentation tasks, due to their lack of inductive bias and the limitations of a smaller dataset scale. Conversely, the 1D CNN and biLSTM architectures demonstrated proficiency in temporal data modeling, advocating for architectural adaptability over mere scale. The results contribute to the field by underscoring the interplay between model architecture, feature extraction method, and viewpoint integration in refining time study methodologies.

1. Introduction

Industry 4.0 marks a transformative shift in manufacturing, characterized by the integration of machines and humans within complex Cyber-Physical Systems (CPS) driven by extensive data collection through sensor networks [1,2]. In this advanced technological landscape, the agility of the system increasingly depends on skilled and trained employees [3], necessitating a re-evaluation of the interaction dynamics between machines and humans [1,4]. A pivotal aspect of this evolution is the efficiency of the human factor, which continues to significantly impact the overall productivity of manufacturing systems. This necessitates the monitoring and quantification of human efficiency, for which time-based metrics have emerged as a good indicator [5]. Traditional time study methodologies, including the use of stopwatches or Predetermined Time Standards (PTS) [5], alongside tools like worksheets or video recordings [6], have been the norm. However, these methods exhibit inefficiencies, such as the inability for real-time analysis in manual observation processes, limiting the scope and frequency of analyses [7]. Furthermore, manual video analysis is time-consuming, often requiring two to five hours for one hour of footage, and it is prone to subjectivity and inconsistency [6].

The field of computer vision, particularly action segmentation or detection from video input, offers a promising solution to these challenges [8]. This approach entails the automated recognition and duration estimation of human activities, with a potential to upgrade work productivity analysis and aid in production planning and load balancing. Traditional machine learning algorithms, that were dominant in human activity recognition until a few years ago [8,9], required extensive domain knowledge and time investment for feature engineering. In contrast, deep learning methods circumvent manual feature extraction through the hierarchical learning of complex data representations, although they demand significant data and computational resources [10].

The deficiencies of existing time study methods and technological trends are the motivation for researching artificial intelligence techniques aimed at automating the recognition and estimation of the duration of human activities in manufacturing processes. This study aims to explore the application of deep learning models to enhance time study methodologies, by developing multiple action segmentation models and evaluating their efficiency on real manufacturing data. To achieve this goal, a sample was collected from a real manufacturing process, which consists of nine work activities. During the video recording of the process, the work activities were performed by four subjects on three different types of products, while the recording itself was performed from two different viewpoints. A total of 27 different models have been developed, which differ with respect to recording viewpoint, feature extraction method, and model architecture responsible for action segmentation.

2. Related Work

The problem of the simultaneous recognition and time segmentation of activities from video input, known as action segmentation or action detection, remains a critical area in computer vision research. The objective is to construct models capable of temporally segmenting and recognizing activities in untrimmed videos containing multiple temporal segments. “Action detection” typically refers to processing videos consisting of a predominantly background class (unlabeled activities) to detect sparse segments with labeled activities [11]. In contrast, “action segmentation” involves processing densely labeled videos and predicting action at every frame [12], a common scenario in manufacturing processes. This review focuses on deep learning-based action segmentation models and approaches used in a manufacturing context.

Early attempts to diminish the dimensionality of input data and define descriptive features commonly employed 2D convolutional neural networks (CNNs) such as VGG16 [12,13,14,15,16], VGG19 [17], AlexNet [18], and ResNet50 [19,20]. These models, pretrained on datasets like ImageNet, were fine-tuned for specific tasks. Since 2D CNNs can only capture spatial features from individual images, features derived from optical flow computations [14,15,16,20] or similar, less computationally demanding algorithms [12,13,21] were often used to incorporate a limited temporal context. Some researchers have employed 3D CNNs like C3D [22,23], which consider both spatial and short-term temporal components. Papers [11,24] combined 3D CNNs and optical flow for enhanced temporal feature representation. However, the main drawback of 3D CNNs is their computational cost compared to 2D CNNs.

Long Short-Term Memory (LSTM) and 1D CNN models have been popular for classification in action segmentation. For example, paper [13] utilized bidirectional LSTMs (biLSTM) to leverage information from both ends of a time series. This study introduced an approach using a combination of four distinct 2D CNN models, two focusing on full-image dimension data and the other two trained on person-centric data, to capture both location-dependent and -independent activity information fed into the biLSTM model. Paper [12] proposed an encoder–decoder architecture based on 1D convolutional layers hierarchy, designed to capture long-term temporal regularities. This model, compared to [13], showed much faster learning (30 times faster) on the same dataset, with better or comparable efficiency. Research study [14] combined ideas from [12,13], employing the architecture from [12] but replacing the encoder’s 1D CNN with biLSTM. These modifications resulted in slightly higher accuracy but slower learning. Study [19] suggested the simultaneous training of feature extraction and classification models, with ResNet50 for feature extraction and LSTM for classification, albeit requiring the processing of 2 s video clips. The results were marginally better than training the models separately but with a longer training time for the integrated model. In paper [16], improvements were proposed over similar research like [12,14] by designing a model with two parallel temporal branches. The branch responsible for analyzing videos at full temporal resolution for fine-grained action segmentation was named the residual branch. The other branch used downsampling and upsampling to process videos at various temporal resolutions, improving frame-by-frame classification accuracy. These branches were linked using a deformable temporal residual module employing deformable convolutional filters. However, this model struggled with accurately segmenting shorter actions between longer segments.

Ishikawa et al. [25] introduced an ASRF framework to address excessive segmentation in existing models like MS-TCN [24]. This framework had three phases, using features extracted via models like I3D [26] and refined via MS-TCN. The HASR framework was developed in [27] to refine predictions from models such as MS-TCN, establishing a hierarchy of features at both segment and video levels and refining the output with additional models like the Gated Recurrent Unit (GRU). Kaku et al. [28] addressed the limitations of existing approaches in processing very short activities and proposed an encoder–decoder architecture, with MS-TCN as the encoder and LSTM or biLSTM as the decoder.

Transformers have received much attention regarding their capabilities in solving sequence-to-sequence tasks related to natural language processing. This motivated research into their application in action segmentation because of the apparent similarity between these two problems. Yi et al. [29] proposed a Transformer model for action segmentation, adapting the original architecture to video data. They restricted the attention mechanism to local neighborhoods and re-engineered the decoder to learn temporal relations between segments. Paper [30] introduced a similar idea, combining a 1D convolution and attention mechanism in an encoder–decoder architecture. The authors of paper [31] suggested an alternative approach to action segmentation, advocating for direct segment prediction using an autoregressive method akin to natural language translation tasks. Their model was grounded in a modified Transformer architecture. A model entirely based on the Transformer architecture was proposed in paper [32], and its structure resembled a U-net network used in image segmentation, designed to build a feature hierarchy in the encoder and decoder. This study utilized local attention limited to the neighborhood of each frame.

In the context of manufacturing, a limited number of papers were found, with notable works being [6,7,33,34,35,36,37] The application of classic machine learning models was predominant in these papers, where models like Hidden Markov Models [6,34,35] or Support Vector Machines [7] were employed following the manual extraction of features. Makantasis et al. [33] applied a deep learning model based on a 2D convolutional neural network and multi-layer perceptron, using manually created features with the Motion History Image algorithm. Zhang et al. [36] proposed an encoder–decoder framework for workflow recognition using a 3D CNN, transforming the activations of the last convolutional layer into clip-level representations, which were then fed into an LSTM network with an attention mechanism for enhanced recognition. In paper [37], a system comprising three stages was developed: spatial feature extraction using a Vectors Assembly Graph (VAG) and graph networks from RGB-D video frames; contact force feature extraction via a sliding window technique; and action segmentation through a multi-stage temporal convolution network (MS-TCN) that combines these features. Jiang et al. [7] collected data in laboratory conditions, while studies [33,34,35] were conducted using a dataset described in [38], which is no longer publicly available. Rude et al. [6] used a dataset from [39], collected with a depth sensor for painting manufactured parts, recording data over a single workday with two different workers. The XIOLIFT dataset was utilized in [36], capturing data from six production lines in an elevator factory over one working month. A newly created dataset of 11 assembly actions performed in laboratory conditions, featuring RGB-D videos and contact force data, was used in paper [37].

The need for further research in this domain is evident, particularly in applying deep learning algorithms to real manufacturing data without manual feature engineering. This gap in research, especially in manufacturing contexts, highlights the potential of deep learning models for the simultaneous recognition and temporal segmentation of human activities, a task still underexplored in various human endeavors, including manufacturing.

3. Problem Formulation

Our goal was the development of a deep learning model with the capability of recognizing and temporally segmenting a series of human activities from videos collected in manufacturing processes. The intention was to investigate the effectiveness of different model architectures for action segmentation. We also wanted to explore the effects of different feature extraction methods, such as fine-tuning 2D CNN models on this dataset and training 2D CNN models exclusively on this dataset. We were interested in the effects of different viewpoints on the models’ results. Our research task involved processing video frames ${\mathit{X}}_1^{\mathrm{T}}=\left({\mathit{x}}^{\left(1\right)},{\mathit{x}}^{\left(2\right)},\dots ,{\mathit{x}}^{\left(\mathrm{T}\right)}\right)$, with the goal of inferring the class label of each frame ${\mathit{c}}_1^{\mathrm{T}}=\left(c^{\left(1\right)},c^{\left(2\right)},\dots ,c^{\left(\mathrm{T}\right)}\right)$, where T is the video length. This task required identifying specific activities within the video and determining their durations.

4. Dataset

We collected a dataset from the manual assembly process of metal grills in a heating, ventilation, and air conditioning system (HVAC). Data collection spanned seven working days, with recordings conducted in manufacturing conditions (see Figure 1). This setting differentiates our dataset from those commonly found in action segmentation literature, which are usually collected in controlled environments or include videos of everyday human activities. The collected videos contain interactions between the worker, product parts, and tools. With minimal differences between individual activity classes, the identification of certain activities relies on models capable of capturing temporal context. Additionally, the background’s appearance remains constant throughout the videos, offering no information about the activity class.

The assembly activities were conducted on three different product types of varying dimensions by four workers, each with a unique ID (O1, O2, O3, O4), as illustrated in Figure 2. This variability captures the human factor in our dataset. The products within the dataset are labeled as T1, T2, and T3.

The videos were recorded from two viewpoints: an overhead (OH) viewpoint and a hand-focused (HF) viewpoint, as shown in Figure 3. This multi-viewpoint approach was driven by the objective of assessing the impact of viewpoint on model performance. The use of OH and HF viewpoints was inspired by research in [40], which suggested that using lower resolution images could accelerate the learning process but at the cost of losing vital visual details. Therefore, we opted for different viewpoints with lower resolutions focused on the work object.

The recordings were made using two cameras at a resolution of 1920 × 1080 and 30 frames per second in MP4 format. We performed subsampling of the video frames at a rate of 5 frames per second. This formatting aimed to expedite model training, enable learning with the existing infrastructure, and address the redundancy in adjacent video frames. Subsequently, videos were segmented into shorter clips based on specific criteria: each clip had to encompass all assembly process activities, including background activities, without exceeding two minutes. This process yielded 620 clips with a total duration of 10 h.

Activity labeling and duration in the video clips were crucial for employing supervised machine learning algorithms. The assembly process included nine work activities for assembling the product and placing it on the shelf, as demonstrated in Figure 4.

We developed precise labeling rules to define the start and end moments of each activity, ensuring consistent data labeling, as shown in Figure 5, for activity 1. Each frame was labeled with the corresponding activity, including a background activity label (label 0), with a time resolution of 0.2 s between frames implicitly indicating activity duration.

For data preparation, we split the dataset into training, validation, and test sets: 480 videos for training and 70 videos each for validation and testing. The split was conducted through random sampling with stratification based on worker ID and product type to ensure that the dataset’s population strata were reflected in all subsets, aiming for better model generalization.

The dataset was designed to have a uniform number of samples per worker and product type, which can be seen from Figure 6.

The typical video duration (see Figure 7), with a median value of 52.8 s, varied depending on the worker and product type, ranging from 36.4 to 90.8 s. Yellow highlight indicates position of a median value.

However, total assembly time, measured from the start of the first to the end of the last activity, often differed from video duration due to the presence of background activities. These activities are a consequence of variations in the assembly process. A comparison of total assembly cycle time per worker and product type, using the median and interquartile range, is provided in Figure 8.

Finally, the distribution of individual activity durations varies among different workers and product types, as detailed in Figure 9, where “Me” stands for median value.

5. Methodology

The procedure used in the development of models in this paper is shown in Figure 10, which highlights the fact that experiments were conducted with three variable factors: recording viewpoint, feature extraction method from video frames, and model architecture for action segmentation.

As mentioned in the previous section, data from two recording viewpoints were used: data collected by a camera placed above the head of the activity performer (OH) and data from a frame focused on the performer’s hands (HF). The third state of the recording viewpoint factor refers to the fusion of data from both viewpoints, denoted as OH + HF.

The factor of the feature extraction approach and the factor of the final model architecture are a direct consequence of the two-step video processing approach commonly used in the field of action segmentation. Although the literature review lists the advantages of end-to-end learning in deep learning models, due to the complexity of video data, the practice is to process through separate steps of feature extraction, and final classification and segmentation.

5.1. Feature Extraction Methods

The purpose of using feature extraction models is to reduce the dimensionality of input data and create descriptive features. The practice of using transfer learning for feature extraction is very common in the context of action segmentation, where all studies from the literature review use some form of transfer learning. Three different approaches were used for feature extraction:

• Feat. extraction from a pretrained model without fine-tuning (FE);
• Feat. extraction from a pretrained model with fine-tuning on our dataset (FT);
• Feat. extraction from a new 2D CNN model trained only on our dataset (TN).

The initial assumption was that transfer learning without fine-tuning (the FE approach) allows for quick extraction of lower-quality features, in the sense that the final models learned on them will be less effective compared to models that will use features obtained by some of the other two approaches.

In all experiments related to feature extraction methods, the input data was preprocessed in such a way that the dimensions of the input video frames were ${\mathit{X}}^{\left(i\right)}\in {\mathbb{R}}^{224\times 224\times 3}$. Also, for experiments related to the FE and FT methods, pixel values in individual channels of images were centered by subtracting the arithmetic means calculated on the ImageNet dataset. The result of feature extraction from image ${\mathit{X}}^{\left(i\right)}$ in each approach was a feature vector ${\mathit{x}}^{\left(i\right)}\in {\mathbb{R}}^{2048}$.

We conducted the experiment with the FE method using the original ResNet50 model described in [41], which was trained on the ImageNet dataset. Feature extraction was performed by removing the classification layer from the ResNet50 model and making a forward pass through the model for each frame from both viewpoints.

Feature extraction from the pretrained model with fine-tuning on our dataset was carried out by removing the classification layer from the ResNet50 model. We added three new fully connected layers with 2048, 512, and 10 neurons to the ResNet50 base model, with dropout regularization applied to the output vectors of the first and second fully connected layers, with a neuron dropout rate of 50%. After that, models were fine-tuned on our dataset. We trained two models, one on images from the videos recorded from the OH viewpoint and the other on images from the HF viewpoint. The number of parameters in these models was 28,785,162, of which 5,250,570 belonged to the newly added layers. Models were trained using the cross-entropy loss function, and ADAM [42] was chosen as an optimization method, with a learning rate of $1\times {10}^{-4}$ and a batch size of 64 images. Only newly added layers were trained. Both models were trained until convergence, which was determined by the criteria of early stopping in the case of an increase in the ratio of loss on the validation set and loss on the training set above the defined threshold, or stagnation in a decrease of loss on the validation set in three consecutive epochs. After that, we removed the last two layers and extracted features in a similar way as in the FE method.

The last approach to feature extraction (TN) was based on a model that was trained only on images from our dataset. The developed model (see Figure 11) uses a type of residual block in its architecture that was introduced in study [43]. In developing the model, good practices observed in the ResNet50 architecture were used.

The model contains 8,040,662 parameters, which is about three times less than ResNet50. This makes sense, considering that this model was trained on hundreds of thousands of images compared to ResNet50, which was trained on the ImageNet dataset. Again, we trained two models: one for OH images and the other for HF images. Models were trained using cross-entropy as the loss function and stochastic gradient descent, with a momentum factor of 0.9 and a batch size of 64 images. Due to the use of residual blocks with skip connections, a learning rate of $1\times {10}^{-1}$ was chosen to accelerate model convergence. Feature extraction from the developed models followed a similar pattern as the previous two approaches, the only difference being in the preparation of the input images, for which pixel scaling was performed in the range [0, 1]. After removing the last fully connected layer of the model, feature vectors of dimension 2048 were calculated for each image.

Feature vectors for the OH + HF viewpoint were obtained by concatenating the calculated feature vectors of both viewpoints for the same image into a combined vector of dimension 4096. These calculated vectors were used as input for action segmentation models.

5.2. Action Segmentation Models

The literature review pointed to three types of architectures that dominate in the action segmentation research. These are architectures based on LSTM/biLSTM, 1D CNN, and Transformers (see Figure 12).

The developed models received input data in the form of a sequence ${\mathit{X}}_1^{\mathrm{T}}=\left({\mathit{x}}^{\left(1\right)},{\mathit{x}}^{\left(2\right)},\dots ,{\mathit{x}}^{\left(\mathrm{T}\right)}\right)$ of total length T. The task of the models was to make a prediction of the activity class occurring at each timestep in the output sequence ${\widehat{\mathit{Y}}}_1^{\mathrm{T}}=\left({\widehat{\mathit{y}}}^{\left(1\right)},{\widehat{\mathit{y}}}^{\left(2\right)},\dots ,{\widehat{\mathit{y}}}^{\left(\mathrm{T}\right)}\right)$, where the member of the prediction in the output sequence at timestep t was a vector ${\widehat{\mathit{y}}}^{\left(t\right)}\in {\mathbb{R}}^{10}$, for which the sum $\sum _{i=1}^{10}{y}_i^{\left(t\right)}=1$.

We used the random search procedure [44] for the selection of optimal hyperparameters for all the models. The cross-entropy loss function was used for model training. The stochastic gradient descent optimization method with a momentum of 0.9 and a learning rate of $1\times {10}^{-2}$ was used in the case of the biLSTM and 1D CNN models, while ADAM, with a learning rate of $1\times {10}^{-3}$, was used in the case of the Transformer model. The learning rate was selected using the procedure described in paper [45]. The mini-batch size in the experiments was 32.

biLSTM models: Bidirectional Long Short-Term Memory networks are an advanced variant of recurrent neural networks (RNN), adept at capturing long-range dependencies in sequential data. Their design addresses the limitations of traditional RNNs, particularly the vanishing gradient problem, by introducing a gated mechanism with additional paths for gradient flow and learning. Another limitation of recurrent neural networks is that they make predictions at the current timestep based on the current element of the input sequence and all previous hidden states. biLSTM allows for the use of information from the entire sequence at each timestep, a useful property if the application of the model has no limitations in terms of the availability of future timesteps. biLSTM can be implemented using two parallel LSTM layers, each processing the sequence in a different direction. The operations within the biLSTM unit can be mathematically described as follows:

$${\mathit{f}}^{\left(t\right)}=\sigma \left({\mathit{W}}_f{\mathit{x}}^{\left(t\right)}+{\mathit{U}}_f{\mathit{h}}^{\left(t-1\right)}+{\mathit{b}}_f\right)$$

(1)

$${\mathit{i}}^{\left(t\right)}=\sigma \left({\mathit{W}}_i{\mathit{x}}^{\left(t\right)}+{\mathit{U}}_i{\mathit{h}}^{\left(t-1\right)}+{\mathit{b}}_i\right)$$

(2)

$${\tilde{\mathit{c}}}^{\left(t\right)}=\mathit{tanh}\left({\mathit{W}}_{\tilde{c}}{\mathit{x}}^{\left(t\right)}+{\mathit{U}}_{\tilde{c}}{\mathit{h}}^{\left(t-1\right)}+{\mathit{b}}_{\tilde{c}}\right)$$

(3)

$${\mathit{o}}^{\left(t\right)}=\sigma \left({\mathit{W}}_o{\mathit{x}}^{\left(t\right)}+{\mathit{U}}_o{\mathit{h}}^{\left(t-1\right)}+{\mathit{b}}_o\right)$$

(4)

$${\mathit{c}}^{\left(t\right)}={\mathit{f}}^{\left(t\right)}⨀{\mathit{c}}^{\left(t-1\right)}+{\mathit{i}}^{\left(t\right)}⨀{\tilde{\mathit{c}}}^{\left(t\right)}$$

(5)

$${\mathit{h}}^{\left(t\right)}={\mathit{o}}^{\left(t\right)}⨀\mathit{tanh}\left({\mathit{c}}^{\left(t\right)}\right)$$

(6)

In Equations (1)–(6), $\sigma \left(x\right)=\frac{1}{1+e^{-x}}$ is a sigmoid function that maps inputs to the range [0, 1], and $tanh\left(x\right)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$ is the hyperbolic tangent function that outputs values in the range [−1, 1]. ${\mathit{W}}_{(·)}\in {\mathbb{R}}^{H\times D},{\mathit{U}}_{\left(·\right)}\in {\mathbb{R}}^{H\times D}$, and ${\mathit{b}}_{\left(·\right)}\in {\mathbb{R}}^H$ are model parameters and $⨀$ is an element-wise multiplication operator. ${\mathit{f}}^{\left(t\right)}\in {\mathbb{R}}^H$ is a forget gate that decides which information to discard from the cell state ${\mathit{c}}^{\left(t\right)}\in {\mathbb{R}}^H$, guided by the sigmoid function. The input gate ${\mathit{i}}^{\left(t\right)}\in {\mathbb{R}}^H$ and the modulation gate ${\tilde{\mathit{c}}}^{\left(t\right)}\in {\mathbb{R}}^H$ collectively update the cell state with new information. The cell state update, as defined in Equation (5), combines past information with new insights, ensuring relevant information is retained. The output gate ${\mathit{o}}^{\left(t\right)}\in {\mathbb{R}}^H$ decides which parts of the cell state contribute to the output at the current timestep $t$. The hidden state ${\mathit{h}}^{\left(t\right)}\in {\mathbb{R}}^H$, is then updated to reflect this output.

The hyperparameters of biLSTM models were adjusted as shown in

Table 1. Hyperparameters tuned in experiments with biLSTM architecture.

Label 1	Hyperparameters	Set of Values Used
M	Number of biLSTM layers	$\left\{1,2,3\right\}$
BiN	Number of neurons per biLSTM gate	$\{4,8,16,32,64,128\left.,256,512,1024\right\}$
N	Number of additional fully connected layers	$\left\{0,1\right\}$
FN	Number of neurons in additional fully connected layers	$\left\{128,256,512\right\}$
DR	Dropout rate	$\left\{0,0.1,0.2,0.5\right\}$

¹ Labels of hyperparameters used in Figure 12a.

, with their ranges determined based on the literature and preliminary experiments.

1D CNN models: The convolutional models used in our experiments were based on two types of dilated convolutional blocks, as identified in the research [11,27]. The use of dilated 1D convolutional layers in the architecture allows for efficient modeling of temporal context. Dilation works on the principle of expanding the kernel by adding gaps between individual elements, thus capturing a longer temporal context without increasing the number of parameters, due to the expansion of the receptive field. The first type of block is the dilated residual block, which begins with a dilated convolutional layer with a kernel size of 3 (${\tilde{\mathbf{W}}}_l\in {\mathbb{R}}^{3\times H\times H}$). The dilation factor for the layer $l$ is set to $2^l$, with the number of kernels $H$ being a hyperparameter adjusted based on experimental results. This approach allows for the capturing of increasingly larger temporal contexts as the model deepens. This block also includes a non-linear transformation using the ReLU activation function, followed by a convolution with a kernel size of 1 (${\mathbf{W}}_l\in {\mathbb{R}}^{1\times H\times H}$) for controlling the model’s representational capacity. A dropout layer is added to mitigate overfitting, along with a skip connection to facilitate the learning process. The operations within this block can be mathematically described as follows:

$${\tilde{\mathit{H}}}_l=ReLU\left({\tilde{\mathbf{W}}}_l\ast {\mathit{H}}_{l-1}+{\mathbf{1}\tilde{\mathit{b}}}_l^T\right)$$

(7)

$${\mathit{H}}_l={\mathit{H}}_{l-1}+{\mathbf{W}}_l\ast {\tilde{\mathit{H}}}_l+\mathbf{1}{\mathit{b}}_l^T$$

(8)

In Equations (7) and (8) ${\mathit{H}}_{l-1}\in {\mathbb{R}}^{T\times H}$ and ${\mathit{H}}_l\in {\mathbb{R}}^{T\times H}$ are the input and output of dilated residual block $l$; ${\tilde{\mathit{b}}}_l\in {\mathbb{R}}^H$ and ${\mathit{b}}_l\in {\mathbb{R}}^H$ are bias terms; $\ast$ represents the convolution operator; and $\mathbf{1}$ is a vector of ones of dimension T. The second type of block is the dual dilated block, consisting of two convolutional branches. In one branch, the dilation factor of the convolutional layer increases with the depth of the model, similar to the dilated residual block. In the other branch, it decreases from the largest value to the smallest, according to the expression $2^{L-l}$, where $L$ is the depth of the last layer and $l$ is the depth of the current layer. The goal is to simultaneously capture both short-term and long-term temporal contexts present in the data. Outputs from these two branches are merged and passed through ReLU activation, with a convolution of dimension 1 used to adjust the dimensionality. Like the dilated residual block, it also includes a dropout layer and a skip connection. The operations are summarized as follows:

$${\tilde{\mathit{H}}}_{l,1}={\tilde{\mathbf{W}}}_{l,1}\ast {\mathit{H}}_{l-1}+{\mathbf{1}\tilde{\mathit{b}}}_{l,1}^T$$

(9)

$${\tilde{\mathit{H}}}_{l,2}={\tilde{\mathbf{W}}}_{l,2}\ast {\mathit{H}}_{l-1}+{\mathbf{1}\tilde{\mathit{b}}}_{l,2}^T$$

(10)

$${\tilde{\mathit{H}}}_l=ReLU\left(\left[{\tilde{\mathit{H}}}_{l,1},{\tilde{\mathit{H}}}_{l,2}\right]\right)$$

(11)

$${\mathit{H}}_l={\mathit{H}}_{l-1}+{\mathbf{W}}_l\ast {\tilde{\mathit{H}}}_l+\mathbf{1}{\mathit{b}}_l^T$$

(12)

In Equations (9)–(12) ${\tilde{\mathbf{W}}}_{l,1}$ ${\in \mathbb{R}}^{3\times H\times H}$ and ${\tilde{\mathbf{W}}}_{l,2}$ ${\in \mathbb{R}}^{3\times H\times H}$ are the parameters of the dilated convolution kernels in the branches with increasing and decreasing dilation factors. ${\mathbf{W}}_l\in {\mathbb{R}}^{1\times 2H\times H}$ is the parameter of the convolution with a kernel size of 1. The architecture of the convolutional model containing these two types of blocks (see Figure 12b) can be divided into two parts, which in the literature are called the prediction generation stage and the prediction refinement stage. Preliminary experiments indicated that only one stage of prediction generation was sufficient for models developed on the collected sample. The number of refinement stages was one of the hyperparameters adjusted in the experiments (see

Table 2. Hyperparameters tuned in experiments with 1D CNN architecture.

Label 1	Hyperparameters	Set of Values Used
D	Number of dilated residual layers	$\left\{1,5,10\right\}$
DD	Number of dual dilated layers	$\{1,5,10\}$
R	Number of refinement stages	$\left\{1,2,3,4,5\right\}$
CK	Number of kernels in each CNN layer	$\left\{64,128\right\}$
DR	Dropout rate	$\left\{0,0.1,0.2,0.5\right\}$

¹ Labels of hyperparameters used in Figure 12b.

).

Transformer models: The Transformer architecture offers a distinct approach to handling sequential data. Unlike RNN-based models, the Transformer operates without recurrence, relying on self-attention mechanisms to process entire sequences simultaneously. This design allows for the efficient modeling of long-range dependencies and parallel processing, especially in cases where temporal dynamics span over various lengths. Convolutional architectures, despite being parallelizable, struggle to model long-term contexts without the stacking of multiple layers, which can increase complexity and computational load. As the Transformer lacks recurrent mechanisms, it employs positional encodings to incorporate sequence order. Positional encodings $\mathit{P}\in {\mathbb{R}}^{T\times D}$ are added to the input sequence to provide the model with information about the relative or absolute position of the elements in the sequence, and we us those proposed in the original Transformer paper [46] based on sine and cosine functions. The core of the Transformer is the multihead attention mechanism, which allows the model to focus on different parts of the sequence simultaneously. The attention mechanism is applied independently in parallel heads, allowing the model to capture different aspects of the input sequence. In a single head attention, the input $\mathit{X}\in {\mathbb{R}}^{T\times D}$ is transformed into three matrices called queries ($\mathit{Q}\in {\mathbb{R}}^{T\times D_q}$), keys ($\mathit{K}\in {\mathbb{R}}^{T\times D_k}$), and values ($\mathit{V}\in {\mathbb{R}}^{T\times D_v}$), and the attention mechanism can be mathematically described as:

$$\mathit{A}\mathit{t}\mathit{t}\mathit{e}\mathit{n}\mathit{t}\mathit{i}\mathit{o}\mathit{n}\left(\mathit{Q},\mathit{K},\mathit{V}\right)=\mathit{s}\mathit{o}\mathit{f}\mathit{t}\mathit{m}\mathit{a}\mathit{x}\left(\frac{\mathit{Q}{\mathit{K}}^{\mathit{T}}}{\sqrt{D_k}}\right)\mathit{V}$$

(13)

where $\mathit{Q}{\mathit{K}}^{\mathit{T}}$ is the dot product similarity score for each query–key pair, which is calculated to determine how closely related each query (representing a specific segment or focus in the manufacturing assembly sequence) is to all possible keys (representing different parts or aspects of the video sequences). These scores are then scaled down by dividing them by the square root of the dimension of the keys, which helps in stabilizing the gradients during training. Applying softmax normalizes these scores, so they all add up to one, forming a probability distribution. This normalization allows the model to focus more on the keys that are more relevant to each query. Once we obtain the attention weights (the output of the softmax function), they are used to appropriately weight the values ($\mathit{V}$). In the context of action segmentation, these values represent the detailed content or features of the actions in the video sequences. By weighting these values according to the attention scores, the model selectively focuses on the most informative parts of the video for understanding and classifying the actions in each segment.

In our study, we aim to explore the effectiveness of a Transformer encoder-only architecture in the task of action segmentation, particularly focusing on its capability without integrating dilated temporal convolutional layers. This choice is motivated by the characteristics of our dataset, which comprises videos with shorter time spans. In such scenarios, the global attention mechanism of the Transformer, unrestricted to local neighborhoods, could potentially capture the temporal dynamics more effectively than when limited to local connectivity.

We are deliberately not including dilated temporal convolutional layers in our architecture. The rationale behind this is to assess the innate ability of the Transformer encoder to handle the action segmentation task in its more original form. This approach will allow us to understand the fundamental strengths and limitations of the Transformer encoder in processing sequential video data for action segmentation. Previous studies in this domain have identified that the original decoder architecture of the Transformer is not particularly well-suited for the nuances of the action segmentation task [29]. By focusing solely on the encoder architecture, we aim to investigate how well the Transformer can model the sequential relationships in action data without the influence of a decoder. Hyperparameters used in experiments are given in

Table 3. Hyperparameters tuned in experiments with Transformer encoder architecture.

Label 1	Hyperparameters	Set of Values Used
T	Number of Transformer encoder layers	$\left\{1,2,8,16,32\right\}$
NH	Number of heads in multihead attention	$\{2,4,8\}$
TN	Number of neurons in Transformer hidden layer	$\{128,256\}$
FN	Number of neurons in two-layer feedforward network	$\left\{128,256,512\right\}$
DR	Dropout rate	$\left\{0,0.25,0.5\right\}$

¹ Labels of hyperparameters used in Figure 12c.

.

6. Evaluation

Two metrics were selected for model evaluation on our dataset: frame-wise accuracy and a segmental F1@IoU [12] score with a defined minimum overlap threshold of 50%, in which the overlap is determined by calculating the ratio between intersection and union (IoU) of a true and predicted segment. Accuracy measures the proportion of accurately classified frames in a video relative to the total number of frames. The drawback of this metric is that models of similar accuracy can have large qualitative differences in terms of recognized segments in the video (a good example of this is given by Figure 5 in [13]). F1@IoU has three properties that make it suitable for evaluating models in the domain of action segmentation. The first property is that it penalizes over-segmentation errors, and the second property is that it does not penalize minor time shifts that may be due to variability and subjectivity in labeling, while the final property is that results depend on the number of activities rather than their duration. The accuracy metric and F1 metric are complementary because they penalize different model errors, which is why we selected both metrics as part of the procedure for the evaluation of models.

7. Discussion

Viewpoints Impact: Utilizing different viewpoints, specifically Overhead (OH) and Hand-Focused (HF), has been shown to influence model performance. The OH viewpoint provides a comprehensive perspective, capturing large-scale activities, but it may miss finer details critical for complex tasks. Conversely, the HF viewpoint excels in capturing detailed hand movements but may lose sight of the broader context. Combining OH and HF (OH + HF) aims to merge these strengths, leading to a more balanced and comprehensive understanding of activities. This integrated approach improves the F1@IoU scores, as demonstrated by the results in

Table 4. Evaluation of the best models from each of the 27 approaches ranked by F1@50 metric.

Model Type	Feature Type	Viewpoint	Number of Parameters	Accuracy	F1@50
CONV *	TN	OH + HF	511,572	97.84	93.64
CONV	FT	OH + HF	511,572	97.74	93.46
biLSTM *	FT	OH + HF	262,890	97.78	93.16
CONV	FT	HF	380,500	97.50	92.63
CONV	TN	OH	380,500	97.60	92.48
biLSTM	FT	HF	131,818	97.52	92.45
CONV	TN	HF	380,500	97.41	92.33
biLSTM	TN	OH	135,018	97.37	92.10
biLSTM	TN	OH + HF	262,890	97.66	91.80
CONV	FE	HF	380,500	96.97	90.49
biLSTM	TN	HF	131,818	97.24	90.39
CONV	FE	OH + HF	511,572	97.00	90.34
CONV	FT	OH	380,500	96.75	89.85
biLSTM	FE	HF	264,650	96.33	88.30
CONV	FE	OH	627,860	95.66	86.60
biLSTM	FE	OH + HF	1,057,674	96.18	85.59
biLSTM	FT	OH	131,818	95.53	84.48
TFORM *	TN	OH	8,960,778	96.14	83.50
TFORM	FT	OH + HF	5,268,234	96.45	83.17
TFORM	TN	OH + HF	17,918,730	94.41	82.83
TFORM	FT	OH	17,394,442	95.95	82.37
TFORM	TN	HF	8,960,778	96.06	81.62
TFORM	FT	HF	4,743,946	95.70	80.47
TFORM	FE	OH + HF	2,105,610	95.17	78.03
TFORM	FE	HF	4,743,946	95.42	78.02
biLSTM	FE	OH	25,194,506	90.38	76.14
TFORM	FE	OH	4,743,946	94.08	74.25

* Best models from each architecture type.

, where the three best models use this approach, indicating enhanced segmentation precision by incorporating a variety of visual cues.

Feature Extraction Methods: The results outlined in Table 4 reveal that models using the FE method for feature extraction displayed higher frame-wise accuracy but lower F1@IoU scores. This suggests that, while these models accurately classify individual frames, they struggle with precise action boundary delineation. The general features learned from extensive datasets like ImageNet may not provide the specificity needed for accurate temporal segmentation in our dataset. In contrast, the FT method showed significant improvements in F1@IoU scores, reflecting the success of adapting pretrained features to our dataset’s unique characteristics. However, the benefits of the TN method, which involves training a model from scratch on our dataset, are not as clearly defined. The impact on the F1@IoU metric is not a marked improvement over the FT method, especially considering the resource-intensive nature of training a new model, suggesting a delicate balance between dataset specificity and pretrained features.

Action segmentation model type: The Transformer (TFORM) models, designed with an emphasis on the encoder architecture and devoid of dilated temporal convolutional layers, were expected to leverage the global attention mechanism for capturing broad temporal relationships. While their global attention mechanism effectively captured broader temporal relationships, as indicated by their accuracy scores, they fell short in F1@IoU performance. This suggests that the global attention mechanism, while powerful, may not be sufficient for the fine-grained segmentation required in complex action segmentation tasks. The number of parameters in these models, while substantial, did not directly correlate with superior F1@IoU performance, suggesting that architectural choices play a more significant role than model size in this context. A potential reason for this could be the inherent lack of inductive bias in the Transformer architecture, which, when combined with a relatively smaller dataset, could impede the model’s ability to learn fine-grained temporal dynamics essential for high F1@IoU scores. Furthermore, the 1D CNN and biLSTM models demonstrated a commendable balance between accuracy and F1@IoU scores. These architectures’ ability to efficiently handle temporal data, despite having fewer parameters than some Transformer models, underscores the importance of architectural suitability over sheer model size for specific tasks like action segmentation.

Segmentation challenges: By observing the qualitative results in Figure 13, specifically in the column “Worst segmentation results”, it is evident that all model types face challenges in distinguishing background activities (labeled as light yellow) from other actions. The difficulty with background activities could stem from their diverse nature and under-representation in the dataset. The inconsistency in appearance makes it challenging for models to establish stable prediction rules, leading to lower F1@IoU scores in the worst-case scenarios. Conversely, the best-case scenarios showcase instances where background activities are minimal, or the models have learned to segment actions with high precision due to specific contextual features present in those examples.

8. Conclusions

This study successfully demonstrates the application of deep learning action segmentation models to improve time study methods in manufacturing, aligning with the operational demands of Industry 4.0. By combining different viewpoints and utilizing advanced feature extraction methods, we have advanced the precision of action segmentation models, as evidenced by the enhanced F1@IoU scores. This research contributes to the broader context by addressing the pressing need for automated and precise human activity analysis in complex manufacturing environments.

The evaluation of our models reveals a nuanced understanding of the interconnection between model architecture, feature extraction methods, and viewpoints. The fine-tuning method, which adapts pretrained features to a specific dataset, proved to be particularly effective, suggesting that contextualized learning is crucial for temporal segmentation. Our analysis also highlights the limitations inherent in basic Transformer models for action segmentation tasks, as their global attention mechanism does not compensate for the absence of inductive bias, especially when working with limited datasets.

A key limitation of our research lies in the dataset’s scope, which, while diverse, may not capture the full complexity of industrial activities. Furthermore, the under-representation of background activities points to the need for datasets that mirror the variability and unpredictability of real-world manufacturing scenarios more closely.

Moving forward, several avenues for future research emerge. Investigating the scalability of models to larger and more varied datasets, the integration of multi-modal data sources, and the real-time application of these models in manufacturing settings are pivotal. Additionally, exploring the augmentation of basic Transformer models with structural inductive biases could bridge their current performance gap in the context of manufacturing video datasets.

This research contributes to the domain of automated time studies by providing a comprehensive analysis of deep learning-based action segmentation models. The insights garnered here will serve as a motivation for further advancements, ensuring that human efficiency continues to be a cornerstone of productivity in the era of smart industry.