Region Collaborative Network for Detection-Based Vision-Language Understanding

Abstract

Given a query language, a Detection-based Vision-Language Understanding (DVLU) system needs to respond based on the detected regions (i.e.,bounding boxes). With the significant advancement in object detection, DVLU has witnessed great improvements in recent years, such as Visual Question Answering (VQA) and Visual Grounding (VG). However, existing DVLU methods always process each detected image region separately but ignore that they were an integral whole. Without the full consideration of each region’s context, the image’s understanding may contain more bias. In this paper, to solve the problem, a simple yet effective Region Collaborative Network (RCN) block is proposed to bridge the gap between independent regions and the integrative DVLU task. Specifically, the Intra-Region Relations (IntraRR) inside each detected region are computed by a position-wise and channel-wise joint non-local model. Then, the Inter-Region Relations (InterRR) across all the detected regions are computed by pooling and sharing parameters with IntraRR. The proposed RCN can enhance the features of each region by using information from all other regions and guarantees the dimension consistency between input and output. The RCN is evaluated on VQA and VG, and the experimental results show that our method can significantly improve the performance of existing DVLU models.

1. Introduction

Visual object detection has been studied for years and has been applied to many applications such as smart city, robot vision, consumer electronics, security, autonomous driving, human–computer interaction, content-based image retrieval, intelligent video surveillance, and augmented reality [1,2,3]. Recently, many researchers found that vision-language understanding tasks based on detected regions or called Detection-based Vision-Language Understanding (DVLU) can achieve better performance than many traditional vision-language understanding methods that directly use the pixel-based image information [4,5,6,7]. For instance, the key to the success of the winning entries to the Visual Question Answering (VQA) challenge [8,9,10,11,12] is the use of detected regions. Moreover, the success of visual object detection leads to many new DVLU tasks, such as Visual Grounding [13,14,15,16,17], Visual Relationship Detection [18,19] and Scene Graph Generation [20,21,22,23,24]. With significant advancements made in the area of object detection, DVLU and its related tasks have attracted much attention in recent years. As shown in Figure 1, in a smart city, a monitor may shoot a picture from a crowded street, and a DVLU system can answer the human question and give the following response. For example, when given a query question “How many people are in the picture?”, a DVLU system can provide the correct answer via the detected regions.

A DVLU task is defined as that given several detected regions (hereinafter, all regions refer to detected regions or bounding boxes for simplicity) from a nature image, a model automatically recognizes the semantic patterns or concepts inside the image. Specifically, a DVLU model first accepts some region inputs from an off-the-shelf object detection model (detector). By applying any well-designed model (i.e., a deep neural network or DNN), these detected regions can be treated as finer features or bottom-up attentions [6] of the original image. Then, each region is fed to the DVLU model as a kind of meaningful information to help understand images at a high level. Unlike usual image understanding, DVLU is based on multiple detected or labeled regions, which makes DVLU receive more fine-grained information than non-region image understanding.

Nevertheless, when facing numerous independent detected regions in an image, existing DVLU methods always ignore their distributions and relationships, which is inadequate for different regions consisting of a real-world scene. Furthermore, it is insufficient to separately reprocess each region for features after detecting them while ignoring that they were an integral whole and discarding their original cooperation. This motivates our new design of collaboratively processing all the regions for DVLU, which will help a DVLU task make full use of regions to collaborate and enhance each other.

In this paper, we propose a simple yet effective add-in block of Region Collaborative Network (RCN) that makes every region collaborate with others in DVLU. With RCN, the feature of a region is enhanced by all other regions in the image. As shown in Figure 2, in a DVLU pipeline, the RCN block can be easily inserted by keeping the dimension consistent. First, the RCN block receives the feature of regions as input. For each region feature, we compute Intra-Region Relation (IntraRR) across positions and channels to achieve self-enhancement. Then, the Inter-Region Relation (InterRR) is applied to all regions across positions and channels. To reduce redundant connections, we apply a pooling and sharing strategy relative to IntraRR. Finally, a series of the refined feature of regions are obtained by linear embedding. In general, we can easily add an RCN block to any DVLU model to process all the regions collaboratively. The proposed RCN is put into two DVLU tasks in the experiment: Visual Question Answering and Visual Grounding. The experimental results show that the proposed RCN block can significantly improve the performance against the baseline models.

Our contributions are three-fold:

• In RCN, we propose to build Intra-Region Relation (IntraRR) across positions and channels to achieve self-enhancement.
• We propose to build Inter-Region Relation (InterRR) to all regions across positions and channels for relationships among regions.
• We evaluate the effectiveness of RCN on VQA and VG, and the results show the proposed method can improve the performance of existing DVLU methods.

2. Related Work

2.1. Detection-Based Vision-Language Understanding

Detection-based Vision-Language Understanding (DVLU) attracted much attention in recent years due to the significant technological advancement of visual object detection [1,25,26,27,28]. With semantically labeled regions, DVLU methods can mine deeper-level information more effectively than those based only on pixel-based images. DVLU methods have been applied to many high-level vision-language tasks, such as VQA [6,7,13,29,30] and VG [13,31,32]. In VQA, Anderson et al. [6] build an attention model on detected bounding boxes for image captioning and visual question answering. Hinami et al. [33] incorporate the semantic specification of objects and intuitive specification of spatial relationships for detection-based image retrieval. The up–down [8] model is based on the principle of a joint embedding of the input question and image, followed by a multi-label classifier over a set of candidate answers. In pythia [9], they demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, fine-tuning image features, and adding data augmentation, pythia can significantly improve the performance of the up–down model. In VG, Deng et al. [13] propose to build the relations among regions, language, and pixels using an accumulated attention mechanism. Lyu et al. [32] propose constructing the graph relationships among all regions and using a graph neural network to enhance the region feature. In [34], the model exploits the reciprocal relation between the referent and context, either of them influences the estimation of the posterior distribution of the other, and thereby the search space of context can be significantly reduced. However, most of these methods do not fully use the relationship among regions, and some only consider the relative spatial relation of regions. In general, these methods ignore the cross-region problem. This may isolate each region from the whole scene and negatively affect performance.

2.2. Region Relation Network

Relationships among objects or regions have been studied for years. Some work is based on hyperspectral imaging [35,36,37,38] and some on deep learning [39,40,41,42,43]. One way is to study the explicit relationship between objects. It usually starts with detecting every object pair with the relationship in a given image [18,19]. After that, by combining detected relationship triplets (〈subject, relationship, object〉), an image can be transformed into a scene graph, where each node represents the region and each directed edge denotes the relationship between two regions [4,20,21,22]. Unfortunately, it is still unclear how to effectively use this structural topology for many end-user tasks, although it is based upon a large number of labeled relationships. Another way is to focus on building a latent relationship between objects. Zagorukyo et al. [44] explore and study several neural network architectures that compare image patches. Tseng et al. [45] was inspired by the non-local network [46] and proposed to build non-local ROI for cross-object perception. This paper is different from [45] in that the proposed RCN: (1) can be inserted into any DVLU model while keeping dimension consistency without modifying the original model; (2) computes intra-region and Inter-Region Relations in terms of both spatial positions and channels; and (3) applies to fully connected region features.

3. Method

3.1. Review of Non-Local Network

Recently, the non-local network has been verified as effective to improve recognition by modeling the long-range region relation in a network [45,46,47,48] in which a self-attention mechanism is used to form an attention map. For each position in the feature (say, query point), the non-local network first computes its pairwise relations to all other positions and then aggregates the features of all positions by weighted sum. The non-local network inspired us to construct the region relation in our RCN. The RCN builds not only Intra-Region Relation, like [45,46,47,48] but also Inter-Region Relation. Following [46,47], a generic non-local operation in DNNs can be defined as

$${\mathbf{y}}_i=\frac{1}{\mathcal{C}\left(\mathbf{x}\right)}\sum _{j=1}^{D}f({\mathbf{x}}_i,{\mathbf{x}}_j)g\left({\mathbf{x}}_j\right),$$

(1)

where the pairwise function $f(·)$ computes a scalar that reflects the correlation between the target position and every other position in the feature map $\mathbf{x}\in {\mathbb{R}}^{H\times W\times d}$, and $g(·)$ is a mapping function. $D=HW$ denotes all positions in a feature. This representation is normalized by a factor $\mathcal{C}\left(\mathbf{x}\right)={\sum }_{j=1}^{D}f({\mathbf{x}}_i,{\mathbf{x}}_j)$. Here, we name ${\mathbf{x}}_i$ a query point, while calling each ${\mathbf{x}}_j$ a connection point. The non-local network is a simple but effective solution to extend the local processing of DNN’s convolutional layer for any position to a non-local pattern. It is even capable of capturing long-range dependencies between positions.

3.2. Region Collaborative Network

Given an image I, a set of detected regions $\mathcal{O}=\{o^1,\cdots ,o^N\}$ is obtained by leveraging an object detector such as Faster R-CNN [25] and YOLO [26], or even from manual labeling. By using a feature extractor (for example RoI-pooling/aligning or linear embedding), regions are represented by fixed-size features $\mathcal{X}=\{{\mathbf{x}}^1,\cdots ,{\mathbf{x}}^N\}$. Here, the Region Collaborative Network block (RCN block) is added in any DVLU task denoted as

$${\widehat{\mathbf{y}}}_i=\frac{1}{\mathcal{C}\left(\mathbf{x}\right)}\sum _{n=1}^N\sum _{j=1}^{D}f({\mathbf{x}}_i,{\mathbf{x}}_j^n)g\left({\mathbf{x}}_j^n\right).$$

(2)

The differences between RCN and the original non-local network are: (1) The input to our model is the high-level feature of all detected regions of an image; (2) The pairwise function $f(·)$ computes correlations not only inside one region but across multiple regions; (3) The normalization factor is distributed through all positions of all regions $\mathcal{C}\left(\mathbf{x}\right)={\sum }_{n=1}^N{\sum }_{j=1}^{D}f({\mathbf{x}}_i,{\mathbf{x}}_j^n)$. Specifically, as shown in Figure 3, RCN holds two main operations, i.e.,Intra-Region Relation (IntraRR) and Inter-Region Relation (InterRR). The IntraRR is particularly designed for one region’s feature, which enhances each position and channel by other positions and channels. The InterRR is adapted to the features from multiple regions, in which other regions enhance each position and channel in one region.

3.2.1. Intra-Region Relation

First, we argue that each region that is framed represents a conception or a goal. Thus, the feature directly extracted from a region by a DNN can be seen as a reasonable representation. To further capture the long dependencies inside the region, we construct the Intra-Region Relation (IntraRR). A simple and straightforward way to grab IntraRR is by treating each region as an independent patch and applying non-local operations directly. However, the operation computes relationship only over spatial distribution (or position-wise), while CNN features are naturally spatial and channel-wise, which may result in losing channel-wise dependencies. Hence, as shown in Figure 4, a position-wise and a channel-wise non-local operation are built, respectively, on the high-level feature tensor $\mathbf{x}\in {\mathbb{R}}^{H\times W\times d}$ of every region

$$\mathbf{y}={\mathbf{y}}^{\mathrm{position}}+{\mathbf{y}}^{\mathrm{channel}}+\mathbf{x},$$

(3)

where for ${\mathbf{y}}^{\mathrm{position}}$ and ${\mathbf{y}}^{\mathrm{channel}}$, D is equal to $HW$ and d, respectively, in Equation (1), which is the same with the normalization. This operation can be regarded as a feature refinement. In other words, by considering dependencies across positions and channels, a region feature guarantees a dense self-correlation. Note, if the region feature is from a fully connected layer, we do not compute ItraRR, since the fully connected layer is already a suitable embedding method that adds the linear combination for each value.

3.2.2. Inter-Region Relation

In the IntraRR, each region is self-correlated. As mentioned, it is also worth constructing the relation across regions because their full cooperation creates a meaningful scene. Similarly, for the feature set $\mathcal{X}=\{{\mathbf{x}}^1,\cdots ,{\mathbf{x}}^N\}$ of all detected regions $\mathcal{O}=\{o_1,\cdots ,o_N\}$, we are looking forward to construct a mapping ${\mathbf{y}}^n=h({\mathbf{x}}^n,\mathcal{X})$, where N is the number of regions. Specifically, like intra-proposal relation, every location or channel is supposed to correspond to other regions. Thus, as shown in Figure 5, the feature of the n-th region can be obtained by

$${\widehat{\mathbf{y}}}^n={\widehat{\mathbf{y}}}^{n\left(\mathrm{position}\right)}+{\widehat{\mathbf{y}}}^{n\left(\mathrm{channel}\right)}+{\mathbf{x}}^n.$$

(4)

To compute ${\widehat{\mathbf{y}}}^{n\left(\mathrm{position}\right)}$ and ${\widehat{\mathbf{y}}}^{n\left(\mathrm{channel}\right)}$, Equation (2) is applied, where their D is equal to $HW$ and d, respectively. However, the dense connections (${\left(ND\right)}^2$) for position and channel are much redundant and result in learning obstacles and even failure training. A simple strategy is used to reduce redundant connection by pooling and sharing (PAS) with IntraRR. We suppose using only one query point to represent a detected region is possible. For convenience, the average pooling in the feature of a region is used to obtain the query point, and the n-th region can be represented by

$${\mathbf{z}}^n=\frac{1}{\mathcal{C}\left({\mathbf{x}}^n\right)}\sum _{j=1}^{D}f(\mathrm{pool}\left({\mathbf{x}}^n\right),{\mathbf{x}}_j^n)g\left({\mathbf{x}}_j^n\right),$$

(5)

where the correlation pairwise function $f(·)$ and the mapping function $g(·)$ share parameters with IntraRR. Thus, Equation (2) can be rewritten as

$${\widehat{\mathbf{y}}}_i=\frac{1}{\mathcal{C}\left(\mathbf{x}\right)}\sum _{n=1}^{N}f({\mathbf{x}}_i,{\mathbf{z}}^n)g\left({\mathbf{z}}^n\right),$$

(6)

where $\mathcal{C}\left(\mathbf{x}\right)={\sum }_{n=1}^{N}f({\mathbf{x}}_i,{\mathbf{z}}^n)$. Thus, the connections are reduced to $N^{2}D$.

3.2.3. RCN for Fully Connected Layer

In some models, detected regions are represented by the vector from a fully connected layer in DNN. In this case, the IntraRR is not computed because the fully connected layer is already an excellent dense embedding method that adds the linear combination for a region. However, the InterRR information is still computed for each region in this situation. In this case, the problem is simplified as $D=1$ and without position and channel element. Therefore, for a query point vector ${\mathbf{x}}^n\in {\mathbb{R}}^{1\times d}$, the InterRR is computed as

$${\widehat{\mathbf{y}}}^n=\frac{1}{\mathcal{C}\left({\mathbf{x}}^n\right)}\sum _{m=1}^{N}f({\mathbf{x}}^n,{\mathbf{x}}^m)g\left({\mathbf{x}}^m\right),$$

(7)

where $\mathcal{C}\left({\mathbf{x}}^n\right)={\sum }_{m=1}^{N}f({\mathbf{x}}^n,{\mathbf{x}}^m)$. Inter-Region Relation (InterRR) for fully connected region features is shown in Figure 6.

4. Experiments

The proposed RCN is implemented in two representative tasks corresponding to three main different levels of DVLU, i.e.,Visual Question Answering (VQA) and Visual Grounding (VG).

4.1. Implementation Detail

Our tracker is implemented in Python, using PyTorch. It runs on a GeForce GTX TITAN X GPU. Following [8] and [9], N regions are represented by $2048D$ features from the fc6 layer of ResNeXt [49], where $K<=100$. Our model is built based on pythia [9] and has the same setting as the best model (without ensemble) presented by their project (https://github.com/facebookresearch/pythia, accessed on 31 July 2022). As an add-in block, our implementation is based on VC (https://github.com/yuleiniu/vc/, accessed on 31 July 2022), and an RCN block is inserted into the neural network. All other hyper-parameters are set to the same as VC. The evaluation metric is accuracy, which means the rate selects the correct regions with the guides from query sentences. An RCN block is inserted before the modal combining operation, combining the image and question features. In other words, it is expected that every combination should consider the picked region and the relationships between this region and others. We use accuracy as our evaluation metric for both VQA and VG, like [11,12,16,17]. The VQA score is calculated by comparing the inferred answer to 10 ground-truth answers. In our method, the running time of the VQA task is about 260 ms, and the running time of the VG task is about 320 ms.

4.2. Bottom-Up Visual Question Answering

The recipe of the winning entries to the VQA challenge [8,9] is the use of the object detector (Faster R-CNN [25] and FPN [28]) to extract detected regions as bottom-up attention. By applying top-down attention to each region in the image and fusing every feature through the Hadamard product, it is easy to predict the answer score for each question. Their glorious prizes show that detected regions of images can significantly improve the performance of traditional methods that only use an image. However, it is a pity that their models never consider the cross-region relation but directly use regions’ linear combination to represent an image. Although it is helpful to assign a factor to every region feature by linearly embedding the concatenation of the question feature and region feature, we think it should consider other regions when computing the relationship between the question and one region. Thus, the proposed RCN is first evaluated on VQA.

4.2.1. Dataset

We evaluate our method on the popular VQA 2.0 dataset. VQA 2.0 is a dataset containing open-ended questions about images. These questions require an understanding of vision, language, and commonsense knowledge to answer. The dataset contains 265,016 images and at least 3 questions (5.4 questions on average) per image. Each question has 10 ground-truth answers. We use split train and validation data as the training set and test-dev as the test set.

4.2.2. Results

Our method is compared with several famous detection-based VQAs, including HieCoAtt-ResNet [50], VQA-Machine-ResNet [51], Ma et al. -ResNet [8], MCB-ResNet [52], up–down [8], and pythia [9]. The experimental results are shown in

Table 1. Single-model VQA 2.0 performance in %. Based on the champion of the VQA challenge 2018, pythia, we insert an RCN block and train the model from scratch.

Model	Test-Dev
HieCoAtt-ResNet [50]	61.80
VQA-Machine-ResNet [51]	63.10
Ma et al. -ResNet [8]	63.80
MCB-ResNet [52]	64.70
up-down [8] (Champion of VQA Challenge 2017)	65.32
pythia [9] (Champion of VQA Challenge 2018)	68.49
pythia + RCN	68.56

. Based on the champion of the VQA challenge 2018, pythia, we insert in an RCN block and train the model from scratch. Compared with a vanilla pythia, an improvement in answer accuracy can be observed (from 68.49% to 68.56%). The results show the proposed RCN can improve further in contrast to the baseline pythia. This shows the proposed RCN does help the VQA task. We believe that before combining image and question features, a question should not only consider but inspect each region independently. A question may come to the relationships among regions, such as “What does the man hold?”—“frisbee”, where the regions of man and frisbee and their relationship should be considered.

4.3. Visual Grounding

Visual Grounding (VG) is a challenging cross-modal task that automatically uses text description to localize an object of interest from an image. The common design of the present VG is formulating the task of selecting the best regions from a set of given proposals [13,53,54,55,56,57] by fusing features from multiple modalities. VC [34] is one of the state-of-the-art methods of VG. It introduces the variational context model where the reciprocity can interpret the variational lower bound between the referent and context.

4.3.1. Dataset

Three popular VG dataset are used, i.e.,ReferCOCO, ReferCOCO+, and ReferCOCOg [58]. The three datasets are based on MS-COCO [59]. In ReferCOCO, the queries are short phrases, while in ReferCOCO+ and ReferCOCOg, their queries are normally declarative sentences, short and long. We have the same data splits as [14,56,60], and the detected bounding boxes are from the ground truth. ReferCOCO and ReferCOCO+ are split into train, val, testA, and testB, where split testA contains only people, while other objects are in split testB. In contrast, ReferCOCOg has no test split.

4.3.2. Results

As shown in

Table 2. VG Comparisons of ReferCOCO, ReferCOCO+, and ReferCOCOg (accuracy in %). VC (ours) means the results of the author’s codes in our environment are retrained. The bold type indicates the state-ot-the-art performance.

Methods	Regions	ReferCOCO			ReferCOCO+			ReferCOCOg
		Val	TestA	TestB	Val	TestA	TestB	Val
Luo et al. [53]	gt	-	74.14	71.46	-	59.87	54.35	63.39
Luo et al. (w2v) [53]	gt	-	74.04	73.43	-	60.26	55.03	65.36
speaker+listener+reinforcer+MMI [61]	gt	79.56	78.95	80.22	62.26	64.60	59.62	72.63
A-ATT-r4 [13]	gt	81.27	81.17	80.01	65.56	68.76	60.63	73.18
MAttN [14]	gt	80.94	79.99	82.30	63.07	65.04	61.77	73.08
VC [34]	gt	-	78.98	82.39	-	62.56	62.90	73.98
VC (Ours)	gt	-	78.08	82.49	-	60.08	61.78	70.21
VC+RCN	gt	82.03	81.37	82.79	66.71	63.19	63.35	74.10

, we compare several state-of-the-art VG methods. VG comparisons on ReferCOCO, ReferCOCO+, and ReferCOCOg. VC (ours) means the results of the author’s code in our environment are retrained. By inserting the RCN block, a VC model can achieve better performances than the baseline, and even surpass the state-of-the-art methods in some splits. VC achieves the best on the split testB of ReferCOCO and ReferCOCO+ and the split val on ReferCOCOg. By adding the RCN block, VC+RCN obtains better results, where 2.39 and 0.63 improvements compared with the claimed results of VC can be observed on the split testA of ReferCOCO and ReferCOCO+ and 0.4 and 0.45 on testB. We find a decline in our implemented VC on three datasets compared with the claimed VC. Compared with the VC that retrained the author’s code in our environment (70.21%), the result of VC+RCN achieves a clear performance improvement. This demonstrates the effectiveness of the proposed RCN; when combined with a VG model such as VC, each region can be enhanced by other regions and further benefit the VG model. This will be more notable when coming up with a complex sentence containing many references. For instance, for “the man holding a frisbee behind a tree”, the regions of frisbee and tree will provide more critical information about the feature of the region of the man.

To reduce the computation, the input feature (d) is embedded into a small-sized one ($d^{\prime }$). We also evaluate how much the embedding size of the embedding feature is more suitable for RCN in VG. As shown in

Table 3. Performances on ReferCOCO with different embedding sizes (from 256 to 2048) of RCN. The bold type indicates the state-ot-the-art performance.

Methods	Embedding Size ${\mathit{d}}^{\prime }$	ReferCOCO
		TestA	TestB
VC [34]	-	78.98	82.39
VC (Ours)	-	78.08	82.49
VC+RCN	256	79.12	82.40
VC+RCN	512	80.07	82.14
VC+RCN	1024	81.37	82.79
VC+RCN	2048	80.95	82.12

, the performances on ReferCOCO with different embedding sizes (from 256 to 2048) of RCN can be seen. It is easy to see that too large or too small an embedding size is not good for the performance, and a proper size (1024) can make a VC model better. A large embedding size means more information but more parameters, and less size is the contrary. We find a small embedding size indeed leads to less computation due to the fewer parameters but results in a slight decline in performance. Moreover, a larger embedding size does not bring about more robust performance but is even worse. A proper embedding size such as 1024 can make the model gain either less computation or high performance (81.37% and 82.79%).

5. Conclusions

Existing Detection-based Vision-Language Understanding (DVLU) methods always process each input image’s detected region separately but ignore that they were an integral whole. This paper proposes a Region Collaborative Network (RCN) to address this problem and an easy-to-add-in RCN block that can be inserted into any DVLU method. Specifically, Intra-Region Relations (IntraRR) are computed inside any detection region by a position-wise and channel-wise joint non-local model. Then, the Inter-Region Relations (InterRR) across all detected regions are computed by pooling and sharing parameters with IntraRR. The proposed RCN makes each region enhanced by all other regions without dimension changes. The proposed RCN can significantly fill the gap between independent regions and DVLU tasks. The RCN is evaluated in two classic DVLU tasks, i.e.,visual question answering and visual grounding. Experimental results show that our method can improve the performance of an existing region-based model. The limitation of our paper is that currently, in our RCN, the regions need to be annotated by the feed, which brings additional human annotation costs. We will combine the one-stage detection method in future work to realize end-to-end model training.