Path-Wise Attention Memory Network for Visual Question Answering

Abstract

Visual question answering (VQA) is regarded as a multi-modal fine-grained feature fusion task, which requires the construction of multi-level and omnidirectional relations between nodes. One main solution is the composite attention model which is composed of co-attention (CA) and self-attention (SA). However, the existing composite models only consider the stack of single attention blocks, lack of path-wise historical memory, and overall adjustments. We propose a path attention memory network (PAM) to construct a more robust composite attention model. After each single-hop attention block (SA or CA), the importance of the cumulative nodes is used to calibrate the signal strength of nodes’ features. Four memoried single-hop attention matrices are used to obtain the path-wise co-attention matrix of path-wise attention (PA); therefore, the PA block is capable of synthesizing and strengthening the learning effect on the whole path. Moreover, we use guard gates of the target modal to check the source modal values in CA and conditioning gates of another modal to guide the query and key of the current modal in SA. The proposed PAM is beneficial to construct a robust multi-hop neighborhood relationship between visual and language and achieves excellent performance on both VQA2.0 and VQA-CP V2 datasets.

1. Introduction

Traditionally, computer vision and natural language processing are two important but mutually independent research fields of Artificial Intelligence. Both fields have made significant progress toward their goals and have gradually been driven to mutual convergence by the explosion of visual and textual data and the requirements of complex real-world tasks. At present, multi-modal learning has bridged the gap between visual and language and has been widely concerned [1,2,3,4,5]. Remarkable progress has been made in many multi-modal learning tasks, e.g., image captioning [6,7,8,9], video captioning [10,11,12], cross-modal retrieval [13,14,15,16,17,18,19,20,21,22], and visual question answering(VQA) [7,23,24,25,26,27,28,29,30,31].

The VQA task is more challenging than other multi-modal learning tasks because it requires a full understanding of textual information in the question and visual information in the image and finding out the key information to solve the problem for comprehensive reasoning.

Inchoate methods coarsely learn joint embedding representation with global features [32,33,34], which contains more noise and has difficulty answering the fine-grained questions. To address the problem, two lines of work have made major contributions and shown effective improvement in accuracy. Firstly, on data representations, global features are replaced by image regions features [27,29,35,36] and words features [29,37], which enable the model to fuse modal information at a finer level. Secondly, different variants of attention mechanisms are applied to enhance the interaction between fine-grained visual features and language features. The underlying motivation is to selectively focus on important parts and ignore irrelevant information. We combine fine-grained representation and attention mechanism and further improve the multi-step joint reasoning ability of the model; then we propose a Path Attention Memory network (PAM). By recording and tracking the attention distribution pattern, the path-wise memory attention block is constructed to strengthen the effective signal and weaken the invalid noise.

The application of the attention mechanism in VQA is very extensive, and various types of attention have been proposed. REGAT [27] uses question self-attention to aggregate words embedding into a global sentence feature. MCAN [36] uses self-attention on attended word features and attended regions features in the fusion stage so that an answer vector is obtained by adding the learned global question vector and global visual vector. The attention mechanism mentioned above fuses information from a sequence of features into a single feature vector, while graph-based attention similar to graph attention network (GAT) [38] does not change the length of the input feature sequence. Graph-based self-attention can be used to establish region-to-region relations on the visual channels and word-to-word relations on the textual channels. Co-attention is able to learn fine-grained correlations between two feature sequences [25,37], so region-to-word and word-to-region relations can be acquired when applied to the VQA task. Compared with single-stream early fusion and dual-stream late fusion, the introduction of co-attention can flexibly adjust the hidden values according to the context of the other modal and learn the inter-modal relations.

Many existing works use self-attention [23,27,35]; only a few works use co-attention and self-attention simultaneously [29,36]. Existing models aggregate important information in multi-hop neighborhoods by stacking several layers of attention modules. An encoder–decoder architecture similar to Transformer is also proposed in MCAN [36] to stack self-attention and co-attention. However, visual features and linguistic features are in different feature spaces, and some deviations may exist even if transformation matrices are used for feature projection. Inspired by the channel-wise conditioning gates used in DFAF [29], we used channel-wise guard gates to check and reshape the projected values from another modal in co-attention.

In addition, we believe that the multi-hop neighborhood relationship constructed by stacking attention modules needs a path-wise module to summarize the previous learning effects and correct some learning biases. This path-wise module needs to be directly connected to the previous attention modules and play a role of centralized control to instruct and constrain the learning of single-hop modules. We refer to the idea of a memory network to memorize four attention matrices that represent pairs’ relationships in a set of regions and words. These attention matrices are regarded as the direct link between the single-hop module and the path-wise module. In short, we propose the path-wise attention block(PA) which is designed as a special co-attention block. The PA takes four attention matrices and two modality features as inputs to calculate a path-wise attention matrix with the adjusted features as output, so it can directly guide the attention matrix learning of the single-hop attention module along the entire path in the backpropagation. The attention matrix of the general attention module is calculated by query and key and is directly related to the current input features, while the attention matrix of PA is not directly related to the input features but to the updated history of features.

The contributions of our work are four-fold:

(1) We propose a novel framework Path-wise Attention Memory (PAM) Network to construct a robust multi-hop neighborhood relationship between visual and language.

(2) We design a central governor for attention-based models, namely Path-wise Attention, to instruct and constrain the learning of single-hop attention blocks in the path.

(3) We use the cumulative nodes’ importance to calibrate the signal strength of regions and words after each single-hop attention block. This strategy is simple, inexpensive but effective.

(4) We adaptively adopt a new gate mechanism in self-attention and co-attention, to make the information interaction between modalities tight and useful.

Roadmap. The rest of this paper is organized as follows. The related work is summarized in Section 2. The details of the proposed method PAM are presented in Section 3. The experimental results and evaluations are reported in Section 4. Finally, we conclude this paper in Section 5.

2. Related Work

2.1. Visual Question Answering

Visual question answering requires a comprehensive and fine-grained understanding of both visual information and text information. One line of work focuses on designing vector fusion functions to capture a high-level correlation between the global visual vector space and the global text vector space [32,33,39]. Some approaches on this line achieve excellent results. However, the direct use of coarse-grained global features in this way may lead to the loss of local details, and inter-modal relations are highly abstract and poorly interpretable. Therefore, research began to shift from coarse-grained to fine-grained, and many methods gradually use image regions features to replace global image vectors and words features to replace global sentence vectors [25,40]. Moreover, the attention mechanism was also introduced to aggregate fine-grained node information and has become the most mainstream inference framework. Ref. [27] takes the question as a global sentence vector to be concrete after each visual region feature. Three types of visual relationships are considered, and their relations matrices are pre-trained and applied to mask the soft attention of a bidirectional graph attention network. Fine-grained representations of both visual and language features are adopted by [29]. They use intra-modal apply self-attention, and inter-modal apply cross-modal feature gate vector to guide self-attention; that is, the query and key in inter-modal self-attention need an element-wise product with the average pooling aggregate cross-modal features [36]. The stack question self-attention module and question guided visual self-attention module are instituted through the proposed encoder–decoder stacking form.

2.2. Attention Mechanisms

By selectively focusing on important parts and ignoring irrelevant information, an attention mechanism can effectively solve the problem of long-term dependencies in model training and improve the interpretability of neural networks and has been widely applied to many unimodal tasks(e.g., machine translation [41,42], visual detection [43], image classification) and multimodal tasks(e.g., multimedia retrieval, visual question answering [23]).

A typical attention module uses a query and key to calculate the weight of each item in the input sequence then sums the weighted values to obtain the output sequence. The attention mechanism is not restricted to the length of the input sequence suitable for an inductive learning problem. Different varieties of attention modules are produced in different application scenarios.

The attention mechanism is usually combined with an RNN for machine translation tasks in the early stage. The attention model was first introduced for Machine Translation by Bahdanau et al. [41] with sequence input but non-sequence output. Similarly, Yang et al. use an RNN and attention mechanism to capture document information gradually at word level and statement level and finally construct a document level feature vector. In such a scenario, attention is used to weight the tokens in the input sequence and aggregate the representation of those tokens to form a holistic vector. An RNN’s recurrent architectures are nonparallel which result in computational inefficiency. To address this, Vaswani et al. proposed eminent Transformer architecture which only relies on a self-attention mechanism to calculate the relationship between each word and all the other words [44].

In the VQA task, self-attention is extensively used to model word-to-word relationships for questions and region-to-region relationships for images in the VQA task. Question-guided attention on image regions or video frames is generally explored for visual question answering [27,36,45,46], video question answering [47,48,49], image captioning [6], etc. In order to capture more intensive correlation between cross-modal nodes, co-attention-based approaches [25,29,36,37] use bi-direction attention to learn the relationships between word–region pairs.

2.3. Graph Attention Network

When processing graph structure data, graph convolution networks (GCNs) use the Laplacian matrix to aggregate node information leading to fixed-size neighborhoods and allocate non-parametric node weight according to node degree. However, graph attention networks(GATs) use the attention mechanism to aggregate node information, which guarantees that more important nodes receive higher weight. The parallelism, flexibility, and interpretability of the attention mechanism make it overcome such problems and perform well. Presented by Smith et al. [38], graph attention networks aggregate multi-hop neighborhood information as the number of layers deepens. Meanwhile, each layer applies the multi-head attention to stabilize the learning process, K hidden states are computed by K independent mechanisms and can be interpreted as observing features from different K perspectives.

When GAT applies to the VQA task, visual regions nodes and language words nodes jointly compose a fully connected graph, whose edges weights can be learned from associative node features by attention alone or constrained by calculating relative spatial positions between nodes or additional relational label data [27,50]. Our model represents both visual features and language features as graph structure, simultaneously learning the inter-relation and intra-relation of two types of nodes by attention mechanism. The fine-grained aggregation at the node level makes the reasoning process of the model more meticulous and precise. For the sequence of the nodes processed by multiple graph attention, we retrace its importance on the meta-path to construct meta-path-based cross-modal attention, thereby reinforcing and verifying the previous reasoning.

3. The Proposed Approach

According to the input visual information and question information, a solver of VQA extracts features from a singular modality, fusions feature between the visual and language modalities, finally predicting the answer based on the learned joint representation. The VQA model can be divided into three steps: feature extraction, feature fusion, and answer prediction.

Our proposed Path-wise Attention Memory Network (PAM) also follows this structure, as shown in Figure 1. In feature fusion, we use a composite attention network composed of co-attention, self-attention, and path-wise attention. The node impact factor is used to self-reinforce the signal. In the following paragraphs, we introduce the problem definition first and then discuss PAM in that order.

3.1. Problem Definition

As most existing approaches do, we regard the visual question answering task as a multi-class problem rather than a generation problem. Let (X,Y,Z) denote the training set, where X is the space of grounded images, Y is the space of questions, and Z is the space of labels. Following previous VQA methods, we consider the multi-class classification problem with binary cross-entropy (BCE) loss.

$$\begin{array}{cc}\hfill & \mathrm{P}=\mathrm{Sigmoid}\left(\mathrm{f}\right(\mathrm{X},\mathrm{Y};\mathsf{\theta }\left)\right)\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & \mathrm{L}(\mathrm{P},\mathrm{Z})=-\sum _{\mathrm{i}=1}^{\mathrm{C}}{\mathrm{z}}_{\mathrm{i}}\log \left({\mathrm{p}}_{\mathrm{i}}\right)+(1-{\mathrm{z}}_{\mathrm{i}})\log (1-{\mathrm{p}}_{\mathrm{i}})\hfill \end{array}$$

(2)

C denotes the number of categories, and f(.) represents the VQA model that fuses the visual and language information to a C dim predictive vector.

Our model takes fine-grained visual features and question features as input to divide the image into multiple regions according to the different objects identified in the image and divide the text of the question into consecutive words. The image regions and question words are collectively called nodes.

3.2. Build Fine-Grained Feature Vectors

The visual information used in our model is fine-grained image region information. A Faster-RCNN [51] model is pre-trained on a visual genome dataset to detect objects in origin images. Then the image regions feature is extracted. Each image is represented as a feature vector $\mathrm{R}\in {\Re }^{\mathrm{m}\times 2048}$ with at most m = 100 regions.

The input question is first tokenized into a word sequence and padded or truncated to the maximum length of $n=14$; each word has 600 dimensions. Then the words sequence is further transformed into feature vector $\mathrm{W}\in {\Re }^{\mathrm{n}\times 1024}$ by a bidirectional RNN with a gated recurrent unit (GRU) [52]. In order to better capture the overall information in the language modality, we use the simple self-attention mechanism [41] to fuse the information of the entire words sequence feature vector to obtain a global information node of the question with the same dimension of 1024, which is concatenated after the words feature vector, and the final language feature vector is 15 × 1024.

To facilitate the subsequent processing of cross-modal and self-modal graph attention information, two linear layers with an ReLU activation function and 0.1 dropout is used to map the visual feature vector and language feature vector to the same dimension D. That is, the visual feature vector is represented as $\mathrm{R}\in {\Re }^{\mathrm{N}1\times \mathrm{D}}$ and the language feature vector is represented as $\mathrm{W}\in {\Re }^{\mathrm{N}2\times \mathrm{D}}$. For the feature with less than N1 or N2 nodes, zero-padding is used to add a fake empty node, and the attention logits of an empty node are filled with negative infinity by using nodes masks in the following several attention modules.

3.3. Attention Blocks with Memory

Taking the input sequences of regions R and words W, we use a composite attention network composed of three types of attention blocks to fuse the dual-modal information.

In this subsection, we first introduce the standard attention block and then introduce our memory improved attention blocks, that is co-attention with guard gates, self-attention with conditioning gates, and path-wise attention block. Then we introduce the details of the node impact factor used in co-attention and self-attention.

3.3.1. Standard Attention Block

For the standard Transformer attention [44], given the input features $\mathrm{I}\in {\Re }^{\mathrm{N}\times \mathrm{D}}$, three linear layers project it to the query, key, and value matrices and divide it into multiple heads, $\mathrm{Q}\in {\Re }^{\mathrm{H}\times \mathrm{N}\times {\mathrm{D}}_{\mathrm{h}}}$, $\mathrm{K}\in {\Re }^{\mathrm{H}\times \mathrm{N}\times {\mathrm{D}}_{\mathrm{h}}}$, $\mathrm{V}\in {\Re }^{\mathrm{H}\times \mathrm{N}\times {\mathrm{D}}_{\mathrm{h}}}$,

$$\begin{array}{cc}\hfill & \mathrm{Q},\mathrm{K},\mathrm{V}=\mathrm{Linear}\left(\mathrm{I}\right),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & \mathrm{Att}=\mathrm{Softmax}(\frac{{\mathrm{QK}}^{\mathrm{T}}}{\sqrt{{\mathrm{D}}_{\mathrm{h}}}}),\hfill \end{array}$$

(4)

where “Linear” denotes a fully-connected layer, ${\mathrm{D}}_{\mathrm{h}}$ is the dimension of heads, and ${\mathrm{K}}^{\mathrm{T}}$ denotes the transpose of K. Query feature $\mathrm{Q}$ and key feature $\mathrm{K}$ are used to calculate the attention matrix $\mathrm{Att}$ which is regarded as the weight of value feature $\mathrm{V}$, so then the sum of the weighted value feature is fed to the Feed Forward Network(FFN) to obtain the updated feature ${\mathrm{I}}^{*}$,

$$\begin{array}{cc}\hfill & {\mathrm{I}}^{*}=\mathrm{FFN}\left(\mathrm{Att}\mathrm{V}\right),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & \mathrm{FFN}\left(\mathrm{x}\right)=\mathrm{Linear}\left(\mathrm{ReLU}\right(\mathrm{Linear}\left(\mathrm{x}\right)\left)\right).\hfill \end{array}$$

(6)

“ReLU” is used to filter positive values; that is, $\mathrm{ReLU}\left(\mathrm{x}\right)=\max (0,\mathrm{x})$.

Nowadays, many works follow the attention mechanism in Transformer which brings huge computing and memory overhead. Our work introduce gates to it and pruned it. To simultaneously gain a simpler network and good performance, we trimmed the single attention module of Transformer appropriately to delete the Feed Forward Network and some projection linear layers. Details are covered separately in specific attention blocks.

3.3.2. Co-Attention with Guard Gates

The co-attention block as shown in Figure 2 learns to capture the attention score between each pair of regions and word feature. This information flow structure is able to learn cross-modal relations and facilitate cross-modal information interaction.

Given the visual regions feature and words feature, we first transformed the input feature into query, key, and value features.

$$\begin{array}{cc}\hfill & \mathrm{Q}=\mathrm{Linear}\left(\mathrm{R}\right),\mathrm{K}=\mathrm{W}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & {\mathrm{V}}_{\mathrm{r}}=\mathrm{Q},{\mathrm{V}}_{\mathrm{w}}=\mathrm{W}\hfill \end{array}$$

(8)

For the co-attention block, query and value of visual are the heads divided visual regions feature with linear projection; key and value of language are the heads divided words feature without linear projection.

By calculating the matrix multiplication between query and key, then applying the Softmax function on different dimensions, we obtain two inter-modality attention matrices.

$$\begin{array}{cc}\hfill & {\mathrm{Att}}_{\mathrm{r}2\mathrm{w}}=\mathrm{Softmax}(\frac{{\mathrm{QK}}^{\mathrm{T}}}{\sqrt{{\mathrm{D}}_{\mathrm{h}}}},-1)\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & {\mathrm{Att}}_{\mathrm{w}2\mathrm{r}}=\mathrm{Softmax}{(\frac{{\mathrm{QK}}^{\mathrm{T}}}{\sqrt{{\mathrm{D}}_{\mathrm{h}}}},-2)}^{\mathrm{T}}\hfill \end{array}$$

(10)

where ${\mathrm{Att}}_{\mathrm{r}2\mathrm{w}}\in {\Re }^{\mathrm{H}\times \mathrm{N}1\times \mathrm{N}2}$ represent regions’ attention to words, and ${\mathrm{Att}}_{\mathrm{w}2\mathrm{r}}\in {\Re }^{\mathrm{H}\times \mathrm{N}2\times \mathrm{N}1}$ represent words’ attention to regions.

Then, the two attention matrices are used to gather the cross-modal information from the value of regions or words. To reduce the noise caused by the difference between two modal feature spaces, we use the guard gates to check and filter the noise info in cross-modal values. The guard gates are computed by average pooling of target modal values and used in the element-wise product with the source modal values to reduce the noise of cross-modal message passing, defined as ${\mathrm{g}}_{\mathrm{r}}$ and ${\mathrm{g}}_{\mathrm{w}}$.

$$\begin{array}{cc}\hfill & {\mathrm{g}}_{\mathrm{r}}=\mathrm{Avg}\left({\mathrm{V}}_{\mathrm{r}}\right),{\mathrm{g}}_{\mathrm{r}}\in {\Re }^{\mathrm{H}\times {\mathrm{D}}_{\mathrm{h}}}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & {\mathrm{g}}_{\mathrm{w}}=\mathrm{Avg}\left({\mathrm{V}}_{\mathrm{w}}\right),{\mathrm{g}}_{\mathrm{w}}\in {\Re }^{\mathrm{H}\times {\mathrm{D}}_{\mathrm{h}}}\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & \mathrm{Avg}\left(\mathrm{Y}\right)=\frac{1}{\mathrm{N}}\sum _{\mathrm{i}}^{\mathrm{N}}{\mathrm{y}}_{\mathrm{i}},\mathrm{Y}\in {\Re }^{\mathrm{H}\times \mathrm{N}\times {\mathrm{D}}_{\mathrm{h}}}\hfill \end{array}$$

(13)

We denote the updated information flows as updated regions feature ${\mathrm{R}}^{*}$ and updated words feature ${\mathrm{W}}^{*}$, respectively,

$$\begin{array}{cc}\hfill & {\mathrm{R}}^{*}=\mathrm{Linear}\left({\mathrm{Att}}_{\mathrm{r}2\mathrm{w}}({\mathrm{V}}_{\mathrm{w}}\times {\mathrm{g}}_{\mathrm{r}})\right)\times {\mathrm{r}}_{\mathrm{imp}}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & {\mathrm{W}}^{*}={\mathrm{Att}}_{\mathrm{w}2\mathrm{r}}({\mathrm{V}}_{\mathrm{r}}\times {\mathrm{g}}_{\mathrm{w}})\times {\mathrm{w}}_{\mathrm{imp}}\hfill \end{array}$$

(15)

Here ${\mathrm{r}}_{\mathrm{imp}}$ is the accumulated regions importance, and ${\mathrm{w}}_{\mathrm{imp}}$ is the accumulated words importance. See details in Section 3.3.5.

The updated regions feature and words feature would then be fed into the following self-attention block to learn the mutual relations between the same modal nodes.

3.3.3. Self-Attention with Conditioning Gates

Co-attention promotes the fusion of relevant information between visuals and language, while self-attention facilitates the accurate location of important nodes through multi-hop relations. For example, for the question, “What is the man doing under the tree?”, the self-attention block helps the model focus on the right man by recognizing the relation between the man and the tree.

Two symmetrical self-attention (SA) blocks are used respectively for regions and words; the SA for regions is shown in Figure 3. We pruned the network and introduce conditioning gates to standard transformer attention which is the base of our proposed self-attention.

The conditioning gate from the other modal is an element-wise product with the query and key to building the context-appropriate relations between nodes. Then, we obtain a pair of different attention matrices by the gated query and key. ${\mathrm{Att}}_{\mathrm{r}2\mathrm{r}}\in {\Re }^{\mathrm{H}\times \mathrm{N}1\times \mathrm{N}1}$ represent regions’ attention to regions, ${\mathrm{Att}}_{\mathrm{w}2\mathrm{w}}\in {\Re }^{\mathrm{H}\times \mathrm{N}2\times \mathrm{N}2}$ represent words’ attention to words.

$$\begin{array}{cc}\hfill & {\mathrm{Att}}_{\mathrm{r}2\mathrm{r}}=\mathrm{Softmax}(\frac{(\mathrm{Linear}\left(\mathrm{R}\right)\times \mathrm{Avg}\left(\mathrm{W}\right)){(\mathrm{R}\times \mathrm{Avg}\left(\mathrm{W}\right))}^{\mathrm{T}}}{\sqrt{{\mathrm{D}}_{\mathrm{h}}}})\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & {\mathrm{Att}}_{\mathrm{w}2\mathrm{w}}=\mathrm{Softmax}(\frac{(\mathrm{Linear}\left(\mathrm{W}\right)\times \mathrm{Avg}\left(\mathrm{R}\right)){(\mathrm{W}\times \mathrm{Avg}\left(\mathrm{R}\right))}^{\mathrm{T}}}{\sqrt{{\mathrm{D}}_{\mathrm{h}}}})\hfill \end{array}$$

(17)

In the self-attention blocks, the image regions feature and words feature are finally updated as follows,

$$\begin{array}{cc}\hfill & {\mathrm{R}}^{*}=\left({\mathrm{Att}}_{\mathrm{r}2\mathrm{r}}\mathrm{R}\right)\times {\mathrm{r}}_{\mathrm{imp}}\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & {\mathrm{W}}^{*}=\left({\mathrm{Att}}_{\mathrm{w}2\mathrm{w}}\mathrm{W}\right)\times {\mathrm{w}}_{\mathrm{imp}}\hfill \end{array}$$

(19)

The details of accumulated importance are described in Section 3.3.5.

3.3.4. Path-Wise Attention Block

A co-attention block and two self-attention blocks have been used previously; hence four attention matrices have been memorized. These memorized attention matrices are good for exactly two things. One is used to calculate the importance of global nodes which will be declared in Section 3.3.5, and the other important usage is to calculate a path-wise attention matrix.

Path-wise attention block is a special co-attention block without the linear projection from features to query and key. Instead of that, the memoried four attention matrices are used to calculate its attention matrix which could influence all the single-hop attention blocks along the entire path in the backpropagation. Adding a PAM after the stack of single-hop attention modules enables the model not only to gradually transmit information in the single-hop neighborhood, but also to adjust the signal strength along the multi-hop constructed path.

To be clear, four memorized matrices from previous attention blocks are region-to-region attention ${\mathrm{Att}}_{\mathrm{r}2\mathrm{r}}$, word-to-word attention ${\mathrm{Att}}_{\mathrm{w}2\mathrm{w}}$, region-to-word attention ${\mathrm{Att}}_{\mathrm{r}2\mathrm{w}}$, and word-to-region attention ${\mathrm{Att}}_{\mathrm{w}2\mathrm{r}}$. To calculate the path-wise attention matrix, we first use the sum of the attention matrix to calculate the four importance scores with the multi-head structure remaining. For example,

$$\begin{array}{cc}\hfill & {\mathrm{S}}_{\mathrm{r}2\mathrm{w}}=\sum _{\mathrm{i}}^{\mathrm{N}1}{\mathrm{Att}}_{\mathrm{r}2\mathrm{w}}^{\mathrm{i}},{\mathrm{S}}_{\mathrm{r}2\mathrm{w}}\in {\Re }^{\mathrm{H}\times \mathrm{N}2\times 1}\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & {\mathrm{S}}_{\mathrm{w}2\mathrm{w}}=\sum _{\mathrm{i}}^{\mathrm{N}1}{\mathrm{Att}}_{\mathrm{w}2\mathrm{w}}^{\mathrm{i}},{\mathrm{S}}_{\mathrm{w}2\mathrm{w}}\in {\Re }^{\mathrm{H}\times \mathrm{N}2\times 1}\hfill \end{array}$$

(21)

${\mathrm{S}}_{\mathrm{r}2\mathrm{w}}$ represents the word’s importance scores which are rated by visual regions, and ${\mathrm{S}}_{\mathrm{w}2\mathrm{w}}$ represents the word’s importance scores which are rated by language words. The importance score rated by two different modalities can be combined into a unified rating score; we perform the matrix multiplication with a tanh activation function to obtain both regions unified rating score and words unified rating score. Then we can calculate the PAM attention matrix score,

$$\begin{array}{cc}\hfill & \mathrm{S}=\tanh \left({\mathrm{S}}_{\mathrm{w}2\mathrm{r}}{\mathrm{S}}_{\mathrm{r}2\mathrm{r}}^{\mathrm{T}}\right){\mathrm{Att}}_{\mathrm{r}2\mathrm{w}}\times {\left(\tanh \left({\mathrm{S}}_{\mathrm{r}2\mathrm{w}}{\mathrm{S}}_{\mathrm{w}2\mathrm{w}}^{\mathrm{T}}\right){\mathrm{Att}}_{\mathrm{w}2\mathrm{r}}\right)}^{\mathrm{T}},\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill & \mathrm{scores}=\mathrm{Conv}1\mathrm{d}\left(\mathrm{S}\right),\hfill \end{array}$$

(23)

where S comprehensively considers the importance of the previous attention module to each node, and final PAM attention matrix $\mathrm{scores}$ are further fused by one-dimensional convolution, $\mathrm{scores}\in {\Re }^{\mathrm{H}\times \mathrm{N}1\times \mathrm{N}2}$. Such a calculation helps each attention block of the model construct the same-level interdependence in the backpropagation. The interactivity between attention blocks is stronger, so that the PAM has the ability to directly regulate the blocks along the entire attention path.

As shown in Figure 4, the rest part of PA block is similar to CA block. we calculate the cross-modal path-wise attention matrix by Softmax on different dimensions of the same attention scores,

$$\begin{array}{cc}\hfill {\mathrm{Att}}_{\mathrm{r}2\mathrm{w}}^{\mathrm{p}}& =\mathrm{Softmax}(\mathrm{scores},-1),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathrm{Att}}_{\mathrm{w}2\mathrm{r}}^{\mathrm{p}}& =\mathrm{Softmax}{(\mathrm{scores},-2)}^{\mathrm{T}}.\hfill \end{array}$$

(25)

Then the two path-wise attention matrices are used to update values and output the updated ${\mathrm{R}}^{*}$ and ${\mathrm{W}}^{*}$. The guard gates of values mentioned in Equations (11) and (12) are also adopted in PA.

3.3.5. Node Impact Factor

The node impact factor measures the relative importance of a node according to how much attention has been received in the attention network. It is declared as global node importance in our paper. This concept is introduced to amplify the signal of important nodes and reduce the signal of irrelevant nodes after each attention operation.

$$\begin{array}{cc}\hfill & \mathrm{imp}*=\mathrm{Sigmoid}((1-\mathsf{\alpha })\mathrm{imp}+\mathsf{\alpha }\sum _{\mathrm{i}}^{\mathrm{H}}\sum _{\mathrm{j}}^{\mathrm{N}}\mathrm{Att}).\hfill \end{array}$$

(26)

Global nodes importance updates after each attention block and are obtained by the sum in another dimension and activated with a sigmoid function, $\mathrm{imp}\in {\Re }^{\mathrm{N}}$. In order to remember the importance of nodes along the whole attention path, a cumulative strategy is adopted to update the importance of nodes. H is the number of heads, and $\mathsf{\alpha }$ is the cumulative coefficient, ranging from 0 to 1. Experiments show that the value of $\mathsf{\alpha }$ between 0.9 and 0.96 will have better performance. Each region or word feature multiplies by its global node importance to adjust the signal strength of node.

3.4. Answer Prediction

After several blocks of feature updating by CA, SA, and PA, we obtain the final visual regions and language words features by gathering nodes information via robust multi-hop bi-modality neighborhood relationships. As described in Figure 1, two attention blocks are used to turn regions and words features into global visual feature $\mathrm{r}\in {\Re }^{\mathrm{D}}$ and global language feature $\mathrm{w}\in {\Re }^{\mathrm{D}}$. The attention block uses multi-layer perceptron to compress the input’s dimension from D to 1, then uses the Softmax function to obtain each item’s contribution to global representation, $\mathrm{a}\in {\Re }^{\mathrm{N}1\times 1}$, and finally, computes a matrix multiplication between input and a.

$$\begin{array}{cc}\hfill & {\mathrm{a}}_{\mathrm{r}}=\mathrm{Softmax}(\mathrm{MLP}\left(\mathrm{R}\right),-2),{\mathrm{a}}_{\mathrm{r}}\in {\Re }^{\mathrm{N}1\times 1},\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & \mathrm{r}={\mathrm{R}}^{\mathrm{T}}{\mathrm{a}}_{\mathrm{r}},\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & {\mathrm{a}}_{\mathrm{w}}=\mathrm{Softmax}(\mathrm{MLP}\left(\mathrm{W}\right),-2),{\mathrm{a}}_{\mathrm{w}}\in {\Re }^{\mathrm{N}2\times 1},\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill & \mathrm{w}={\mathrm{W}}^{\mathrm{T}}{\mathrm{a}}_{\mathrm{w}}.\hfill \end{array}$$

(30)

The global features $\mathrm{r}$ and $\mathrm{w}$ are eventually fused to generate the last answer prediction,

$$\begin{array}{cc}\hfill & \mathrm{answer}=\mathrm{MLP}\left(\mathrm{FC}\right([\mathrm{r},\mathrm{w}])+\mathrm{r}+\mathrm{w}),\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill & \mathrm{FC}\left(\mathrm{x}\right)=\mathrm{ReLU}({\mathrm{xW}}_{\mathrm{x}}+\mathrm{b}),\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill & \mathrm{MLP}\left(\mathrm{x}\right)=\mathrm{ReLU}({\mathrm{xW}}_1+{\mathrm{b}}_1){\mathrm{W}}_2+{\mathrm{b}}_2.\hfill \end{array}$$

(33)

FC is a linear layer with ReLU and 0.1 dropout. Its MLP maps the hidden dimension H of the fused feature vector to C, where C is the number of the most frequent answers in the training set. So the binary cross-entropy (BCE) loss is calculated with the prediction and data labels by Equation (2).

4. Experiment

4.1. Datasets

We conduct our experiments mainly on VQA2.0 [53], but also test out-of-distribution generalization on VQA-CP [54].

VQA2.0 [53] is the most commonly used VQA benchmark dataset. It is composed of human-annotated question–answer pairs for the real images from the Microsoft COCO dataset [55]. Compared with previous VQA 1.0 [56], VQA 2.0 has many more annotations and less dataset bias. The entire dataset is split into training (82,783 images and 443,757 QA pairs), validation (40,504 images and 214,354 QA pairs) and test-standard sets (81,434 images and 447,793 QA pairs). Additionally, the test-dev set is a 25% subset of the size of the test-standard set. An average of 3 questions are generated for each image; 10 answers are collected for each image–question pair by human annotations, and the most frequent answer is treated as the ground truth. The results are divided into three per-type accuracies(yes/no, number, and other) and an overall accuracy according to question type.

VQA-CP v2 [54] dataset was constructed by reorganizing VQA2.0, which is called visual question answering under changing priors, such that the distribution of answers for each question type differs in the training set and the testing set to avoid the model’s over-reliance on the prior knowledge of potential relationships between questions and answers.

4.2. Implementation Details

Our model is implemented with PyTorch. All initialization is the Pytorch’s default initialization; that is, the initial weights of linear layers and conv1d layers are sampled from the uniform distribution. We use cross-validation to evaluate the accuracy under different hyperparameters and select the group with the best performance. We declare the final hyperparameters setting in

Table 1. Setting of hyperparameters.

Hyperparameters	Value
Hidden dimension	2048
Number of attention heads	8
Learning rate (lr)	0.001
Decay start epoch of lr	10
Decay interval of lr	2
Decay factor of lr	0.5
Dropout factor	0.1
Batch size	64
Epochs	15

. The batch size is set to 64. We adopted 8 heads (H = 8) of attention in CA/SA/PA, hidden dimension is 2048 (D = 2048), and the dimension of each head is 256 (HD = 2048/8).

The dropout rate of fully connected layers is set as $0.1$. We use binary cross entropy as the loss function with the Adamax optimizer. The basic learning rate is set as 0.001, and the hot start strategy is adopted. The learning rates of the first three epochs are 0.0005, 0.001, and 0.0015, and then the learning rate remains unchanged at 0.002 until the 10th epoch begins to decay with a step length of 2 epochs and a decay rate of 0.5. The cumulative coefficient of global nodes importance is set as 0.95.

4.3. Baselines

In this paper, the proposed PAM method is compared with several baselines which can be divided into two categories based on the dataset they are focused on.

One is mainly concerned with the performance on VQA2.0, including Teney et al., 2018 [57], DCN [25], DRAU [58], BLOCK [26], Zhang et al., 2020 [59], MuRel [28], RAMEN [60], and ReGAT-implicit [27].

The other is mainly concerned with VQA-CP to verify its generalization performance, including AReg [24], Grand et al., 2019 [61], CSS-UpDn [30], Teney et al., 2021 [31], and Whitehead et al., 2021 [62].

To reflect the improvements in the model structure, we select their models with similar feature extraction methods and without extra annotations, without ensembling or tuning. Here is a brief introduction to these baselines.

• Teney et al., 2018 [57] adopts the question/image early joint embedding and the single-layer question-guided image self-attention mechanism.
• DCN [25] designs a bi-directional interactions hierarchy network stacked by several dense co-attention maps to fuse visual and language features.
• DRAU [58] combines convolutional attention and recurrent attention to promote bi-modal fusion reasoning.
• BLOCK [26] is a multi-modal fusion method based on the tensor composition which can perform fine-grained inter-modal representation while maintaining strong single-modal representation ability.
• Zhang et al., 2020 [59] designs a question-guided top-down visual attention block and a question-guided convolutional relational reasoning block.
• MuRel [28] adopts a method similar to the graph attention mechanism to learn the inter-regional relations of the image and updates the regions hidden representation under the guidance of questions and image geometric information.
• RAMEN [60] performs an early fusion of fine-grained image regions’ feature and global question feature and then processes the fused feature by bi-directional gated recurrent unit(bi-GRU) but without an attention or a bi-linear pooling mechanism [63].
• ReGAT-implicit [27] models implicit relations between visual regions via graph attention network.
• AReg [24] prevents the VQA model from capturing language bias by introducing the question-only adversary to encourage visual grounding.
• Grand et al., 2019 [61] research how to alleviate linguistically biased by introducing adversarial regularization.
• CSS-UpDn [30] forces the model to reason correctly by adding custom counterfactual samples to the training data; then the model could achieve better performance on VQA-CP.
• Teney et al., 2021 [31] adopts a new multiple environment training scheme to improve the out-of-distributed generalization. We choose its best non-ensemble model to compare with our PAM in

  Table 2. Model accuracy on the VQA 2.0 and VQA-CP benchmark.

  Model		VQA2.0		VQA-CP
  	Test-Dev	Test-Std	val	Test
  Teney et al., 2018 [57]	65.32	65.67	63.15	-
  DCN [25]	66.60	67.00	-	-
  DRAU [58]	66.45	66.85	-	-
  BLOCK [26]	66.41	67.92	-	-
  Zhang et al., 2020 [59]	67.20	67.34	-	-
  MuRel [28]	68.03	68.41	65.14	39.54
  RAMEN [60]	65.96	-	-	39.21
  ReGAT-implicit [27]	67.6	67.81	65.93	40.13
  Our PAM	69.01	69.24	66.20	41.60
  AReg [24]	-	-	62.75	41.17
  Grand et al., 2019 [61]	-	-	51.92	42.33
  CSS-UpDn [30]	-	-	59.21	41.16
  Teney et al., 2021 [31]	-	-	61.08	42.39
  whitehead et al., 2021 [62]	-	-	61.08	41.71

  .
• Whitehead et al., 2021 [62] utilize both labeled and unlabeled image–question pairs and try to separate skills and concepts for the model to improve its generalization.

4.4. Results and Analysis

The PAM is trained on the training set of VQA2.0 [53]. Table 2 shows its overall performance on VQA2.0 test-dev, test-std, and val splits using the accuracy metric [56] in comparison to the baselines and evaluated on the VQA-CP v2 [54] dataset to demonstrate its generalizability. Our PAM has already achieved the state-of-the-art overall accuracy on VQA2.0 ($69.24\%$ vs. $68.41\%$ in test-std split, $66.20\%$ vs. $65.93\%$ in val split), and $41.60\%$ overall accuracy on the test set of VQA-CP. It states that the PAM is not only the best performing among the baselines focused on VQA2.0, but also very competitive in VQA-CP focused baselines, which means our PAM can rely less on prior knowledge and more on correct inference understanding and can be generalized to the VQA-CP dataset as well.

To be more specific about PAM’s generalizability comparison, the bottom five other models focus on VQA out-of-distribution generalization which leads them to achieve higher accuracy on the VQA-CP dataset, but obviously at the cost of huge performance degradation on the VQA2.0 dataset. Take Grand et al., 2019 [61] for example, compared with ReGAT-implicit it gained a $2.2\%$ performance improvement on the VQA-CP test, but a $14.01\%$ performance degradation on the VQA2.0 val.

Our PAM did not specifically optimize performance on VQA-CP v2 like the models in the bottom five rows in Table 2; however, without adversarial regularization or optimized training data, our PAM can achieve competitive performance on the VQA-CP dataset and avoid the obvious degradation on VQA2.0. PAM outperforms other popular models by $\pm 0.8\%$ on the VQA-CP test set, and obtains an improvement of at least $3.45\%$ on the VQA2.0 val set. This shows that our model achieves a more suitable balance between generalization ability and fitting ability.

4.5. Ablation Study

Ablation studies are performed on the VQA2.0 [53] validation dataset to evaluate the effectiveness of our proposed actions. The results are shown in

Table 3. Ablation studies of our proposed actions on the VQA2.0 validation dataset. “-” is the operation to delete trick based on complete model PAM.

Model	VQA v2 val
PAM	66.20
-PA	65.69
-PA+CA	65.32
-PA-imp	65.01
-gate in SA	66.11
-gates in CA/SA/PA	65.85
-PA-imp-gates in CA/PA	64.66
-PA-imp-gates in SA	65.58
-PA-imp-gates in CA/SA/PA	65.34

.

The “PAM” model in the first row is our complete model involving all actions. The next few rows progressively remove blocks based on PAM. “-PA” indicates that path attention is removed, which is the block marked in purple in Figure 1, while “-PA+CA” means to replace PA with CA. The “-imp” indicates the importance of the global nodes after each SA and CA block is removed; that is, the operations marked in red at the end of the SA (Figure 3) and CA (Figure 2) are deleted. We remove guard gates marked in orange in CA (Figure 2); it shows as “-gates in CA” in Table 3 and “-gate in CA/SA” if gates in SA are also removed.

Comparing “PAM” with “-PA”, we achieved a 0.51% performance improvement by adding the path-wise attention. To eliminate the influence of the change of parameters number, we also experimented with replacing a PA with CA in “-PA+CA”, as PA is designed as a special CA, and their parameters are mainly derived from two linear layers. Results in Table 3 show that “-PA+CA” does not achieve a similar performance as PAM and even has a performance decrease compared to “-PA”. We believe that the reason is that CA is only a single-hop attention module and cannot summarize and strengthen the multi-hop neighborhood formed by multiple attention modules along the whole path like PA.

Comparing “-PA” with “-PA-imp”, we achieved a 0.68% performance improvement by applying the importance of the global nodes. The “imp” trick is universal and easy to operate for all the popular attention blocks with almost no extra overhead. Moreover, the importance of accumulation is very suitable for concatenating multiple models in boosting-based ensemble learning. Due to the limitation of computing resources, we only tried concatenating two PAM on small dimensions, and it shows performance improvement compared with one PAM. The gates vector is inspired by the conditioning gating vector mentioned in DFAF. Some adaptive changes have been made in the calculation method and applied to both SA and CA for different purposes, while the gating vector in DFAF is only used to guide SA learning. The improved guard gates in CA are effective, as removing them from “-PA-imp” decreases accuracy from 65.01 to 64.66.

From Table 3, removing gates in SA from “-PA-imp” brings performance improvement, while removing gates from PAM brings performance decrement. We believe the reason is that SA can perfect and supplement other proposed strategies in PAM, which is beneficial to the overall performance.

Figure 5 shows the validation accuracy versus epochs of different ablations of our model, and our PAM model keeps the advantage of accuracy in the whole training process.

4.6. Distribution of Attention and Prediction

In Figure 6, we analyzed the distribution of eight attention heads. Each colored line represents a head in the attention block. Its calculation formula is similar to Equation (20). The PAM successfully focuses on the important words and regions compared with the model without path-wise attention block.

In Figure 7 and Figure 8, our PAM gave a higher degree of confidence in correct answers compared to the model without path-wise attention block.

5. Conclusions

In this paper, we proposed a novel path-wise attention memory(PAM) network framework for visual question answering. Four attention matrices of self-attention and co-attention are memorized by the network. They benefit to calculate global nodes importance and a new attention matrix of path-wise attention. The cumulative nodes importance could calibrate the signal strength of regions and words after each single-hop attention block. The path-wise attention could directly guide the attention matrix learning of a single-hop attention module as a central governor. The gates vectors in SA/CA/PA are used to enhance the interaction between the visual and the language modal. These actions bring model effectiveness improvement, and PAM shows good accuracy and generalization performance in experiments.