Asymmetric Graph Contrastive Learning

Abstract

Learning effective graph representations in an unsupervised manner is a popular research topic in graph data analysis. Recently, contrastive learning has shown its success in unsupervised graph representation learning. However, how to avoid collapsing solutions for contrastive learning methods remains a critical challenge. In this paper, a simple method is proposed to solve this problem for graph representation learning, which is different from existing commonly used techniques (such as negative samples or predictor network). The proposed model mainly relies on an asymmetric design that consists of two graph neural networks (GNNs) with unequal depth layers to learn node representations from two augmented views and defines contrastive loss only based on positive sample pairs. The simple method has lower computational and memory complexity than existing methods. Furthermore, a theoretical analysis proves that the asymmetric design avoids collapsing solutions when training together with a stop-gradient operation. Our method is compared to nine state-of-the-art methods on six real-world datasets to demonstrate its validity and superiority. The ablation experiments further validated the essential role of the asymmetric architecture.

1. Introduction

Graph structure data are ubiquitous in the real world, such as social networks [1], e-commerce networks [2], traffic networks [3], chemical graphs [4,5,6], and fractal networks [7,8,9]. Graph theory is extensively utilized to analyze these complex networks, studying aspects like degree distribution, cluster coefficients, average path length, degree correlations, and community structures. In addition to these analyses, learning low-dimensional dense embeddings from nodes is widely used in graph analysis tasks, and has been studied in the past few years. The conventional network representation methods include matrix factorization-based methods [10,11] and random walk-based methods [12,13,14]. Recently, graph neural networks (GNNs) [15,16,17,18], which learn node representations by aggregating neighbor information, and have achieved the state-of-the-art performances in many graph analysis tasks, such as node classification [19,20], link prediction [21], and graph classification [22].

GNNs leverage graph theory principles to learn and encode complex relationships within graphs. By combining the theoretical foundations of graph theory with the computational capabilities of neural networks, GNNs provide a versatile and effective approach for understanding and analyzing complex relational data represented as graphs. However, the existing GNN models are mostly defined in a supervised manner, which is not applicable for the data without costly human-annotated labels.

Inspired by recent success in self-supervised contrastive representation learning in computer vision [23,24] and natural language processing [25,26], some works [27,28,29,30,31] attempt to learn graph representations in an unsupervised manner. These methods use GNNs as graph encoders and learn node-level or graph-level representations by maximizing the agreement between similar nodes or graphs. Previous works adopt a cross-scale contrastive mechanism (i.e., node–subgraph or node–graph contrasting), and they contrast the representations of the global graph and local patches by maximizing the mutual information (MI) [32,33]. More recently, some studies based on the same-scale contrastive mechanism [27,28,29] are proposed. Different from contrasting elements in different scales, they directly perform node–node or graph–graph contrasting to learn representations. Generally, these methods first generate two graph views via data augmentation and then learn representations by maximizing the agreement of elements in these two views.

Despite the fact that there are different contrastive mechanisms, avoiding collapsing solutions is always a fundamental challenge in graph contrastive learning. In collapsing solutions, graph models map all nodes into constant outputs to reduce the distance among them and the training loss decreases to zero in a cheating way. Existing graph contrastive learning methods always focus on designing data augmentation strategies to improve their performance on classification tasks [27,28,29,30,31]. None of them address the issue of collapsing solutions; instead, they adopt established techniques from computer vision to circumvent this problem. For example, GRACE [27] uses a large number of negative samples and BGRL [31] uses complex momentum-update encoder together with a predictor network. Recently, SimSiam [34] proposed a hypothesis to investigate which component implicitly works for avoiding collapsing solutions.

Based on the above methods, we perform theoretical analysis and experimental demonstrations. The results show that the asymmetric architecture of encoders is a crucial design to avoid collapsing solutions. According to the observation, an Asymmetric Graph Contrastive learning method (AGC) is proposed by employing a pair of unequal depth GNNs for contrasting. Combined with a stop-gradient operation, AGC can learn node representations without using any negative samples and avoid collapsing solutions. It does not require complex design such as momentum-update encoder or predictor network. Our contributions are summarized as follows:

• We propose a simple asymmetric contrastive learning method for graph representation learning. The proposed AGC method first uses a pair of GNN encoders with unequal depth layers to jointly capture information from different neighbor scales and then learns more comprehensive node representations;
• We are the first to analyze the reasons behind collapsing solutions in graph contrastive learning and make a theoretical analysis on why the asymmetric design working with stop-gradient operation in our AGC can successfully avoid this issue;
• We compare our method with nine baselines on six real-world datasets to demonstrate the superiority of the proposed method over the state-of-the-art graph contrastive learning methods. We empirically demonstrate that batch normalization and residual connections can further improve model performance by adopting a deeper GNN-based encoder.

2. Theoretical Explanation of AGC

In this section, we discuss the reasons behind the collapsing solutions in graph contrastive learning. We explain why our asymmetric model AGC can avoid collapsing solutions even when training without using negative samples.

2.1. Analysis of Symmetric Design

We first explain that even with a stop-gradient operation, a model built up by symmetric encoders will result in trivial solutions. Node representation z is rewritten as $z(\theta ,\tilde{\mathcal{G}})$, denoting that representation z is learned from graph view $\tilde{\mathcal{G}}$ via an encoder with parameter $\theta$. The loss objective is written in the format of L2-norm, which is equivalent to the cosine similarity because the node features are normalized. Consider the loss function as follows:

$$\mathcal{L}=\mathbb{E}\left[{∥z({\theta }_L,\widetilde{{\mathcal{G}}_1})-\mathrm{stopgrad}\left(z({\theta }_S,{\tilde{\mathcal{G}}}_2)\right)∥}_2^2+{∥z({\theta }_L,{\tilde{\mathcal{G}}}_2)-\mathrm{stopgrad}\left(z({\theta }_S,{\tilde{\mathcal{G}}}_1)\right)∥}_2^2\right],$$

(1)

where ${\theta }_L$ and ${\theta }_S$ denote the parameters of two encoders, while ${\tilde{\mathcal{G}}}_1$ and ${\tilde{\mathcal{G}}}_2$ represent two graph views. For symmetric-designed models, ${\theta }_L$ and ${\theta }_S$ are entirely shared, and can be represented by a set of parameter $\theta$ for simplicity. Therefore, the base loss and gradients can be rewritten numerically as:

$$\mathcal{L}=2·\mathbb{E}\left[{∥z(\theta ,{\tilde{\mathcal{G}}}_1)-z(\theta ,{\tilde{\mathcal{G}}}_2)∥}_2^2\right],$$

(2)

where the two norm terms are merged, since they are numerically equal. The terms marked as stopgrad provide no gradient for the model, therefore the gradient is calculated as:

$$\frac{\partial \mathcal{L}}{\partial \theta }=\frac{\partial \mathcal{L}}{\partial \left(z(\theta ,{\tilde{\mathcal{G}}}_1)\right)}·\frac{\partial \left(z(\theta ,{\tilde{\mathcal{G}}}_1)\right)}{\partial \theta }+\frac{\partial \mathcal{L}}{\partial \left(z(\theta ,{\tilde{\mathcal{G}}}_2)\right)}·\frac{\partial \left(z(\theta ,{\tilde{\mathcal{G}}}_2)\right)}{\partial \theta }.$$

(3)

It can be easily shown that the optimizer will adjust the parameters to minimize the loss to zero, because every term in (3) is directly affected by $\theta$ to minimize the objective. As for augmentations in $\mathcal{L}$, $\widetilde{{\mathcal{G}}_1}$ and $\widetilde{{\mathcal{G}}_2}$ are two different augmented graphs randomly sampled from the same distribution in every epoch. With the optimization of parameters, models will give the same constant vector regardless of the different augmented input graph, and the loss decreases to zero.

The collapsing solution is analogous to the overfitting issues in supervised learning without the regularization item. They have the similar problem of paying too much attention on reducing the loss value which leads to abnormal parameters. The difference between the two problems is that supervised training models reduce loss by looking for complex parameters to fit all samples, while in contrastive learning, models prefer simple parameters to output constant embeddings for lowering loss.

Figure 1 illustrates how the symmetric encoders are trained in the latent space. The gradient directions pointing directly at each other lead to the convergence of the two representations in the embedding space and a degradation in learning ability.

2.2. Analysis of Asymmetric Design

Based on the L2-norm loss (1), the gradients of AGC model with asymmetric design can be formulated as:

$$\frac{\partial \mathcal{L}}{\partial \theta }=\frac{\partial \mathcal{L}}{\partial \left(z({\theta }_L,{\tilde{\mathcal{G}}}_1)\right)}·\frac{\partial \left(z({\theta }_L,{\tilde{\mathcal{G}}}_1)\right)}{\partial \theta }+\frac{\partial \mathcal{L}}{\partial \left(z({\theta }_L,{\tilde{\mathcal{G}}}_2)\right)}·\frac{\partial \left(z({\theta }_L,{\tilde{\mathcal{G}}}_2)\right)}{\partial \theta }.$$

(4)

Compared to (3), this loss (1) has two terms based on the parameter ${\theta }_S$, which cannot be optimized directly by the gradient of ${\theta }_L$ but updated by a shared-parameters operation as:

$${\theta }_L\leftarrow \underset{{\theta }_L}{\arg \min }\left(\mathcal{L}\right),{\theta }_S\leftarrow {\theta }_L.$$

(5)

We hypothesize that GNN encoders with unequal depth layers can learn similar embeddings that can be classified into the same category, which is to be proved by experiments in Section 4.2. It can be understood that better embeddings can be learned through aggregating different neighbor scales with proper parameters. Therefore, for each node, the embeddings learned by the two encoders are regarded as a positive pair. Due to the parameters that are not shared in the long encoder, two embeddings in a positive pair will not become a completely identical solution and the loss of AGC will not converge to zero.

The training process in latent space is shown in Figure 1. The gradient of the ${\theta }_L$ directs to the embedding from the short encoder in another view. While in this case, the total loss cannot drop to zero because with the parameters of the long encoder updating, $z({\theta }_S,\tilde{\mathcal{G}})$ keeps moving in the similar scope (marked by the dotted line) around the $z({\theta }_L,\tilde{\mathcal{G}})$ in every epoch. Therefore, the training targets can be considered to keep the two embeddings of augmented views similar but not the same in latent space. The design of the short shared-parameter encoder plays the same role as the regularization term in supervised learning to prevent loss to zero and can protect the model from collapsing.

2.3. Analysis of Predictor-Based Methods

Moreover, we display the predictor-based methods in the embedding space. It has the same principle that prevents loss from falling to zero by using additional parameters to keep the embeddings similar but not the same. In Figure 1, the model tries to use a predictor network to make the embedding approaching to the other view in latent space. Then, we use the similarity objective to measure the distance between one view and the prediction of the other view. The training objective is to enable the predictor to bridge the gap between the two augmentation distributions. By achieving this, the model avoids directly reducing the distance between embeddings, allowing the encoder to learn discriminative representations without collapsing solutions.

2.4. Discussion

Some previous studies adopt single-sampling strategy which calculates the loss with a pair of ($\widetilde{{\mathcal{G}}_1}$, $\widetilde{{\mathcal{G}}_2}$) while our proof is based on dense-sampling with an addition sample of ($\widetilde{{\mathcal{G}}_2}$, $\widetilde{{\mathcal{G}}_1}$). Actually, it makes no difference because the augmentations are based on the same hyperparameters of masking rates in graph contrastive tasks. During the long training procedure, it can be seen as the same sample augmented according to the distribution.

In total, we believe that the asymmetric design of the model changes the optimization goals from making two embeddings coincide to making them similar in latent space and avoids the collapsing solution. Compared to the methods based on predictor network, AGC provides a simpler and intuitive method for graph contrastive learning.

3. Method

In this section, we first explain notations and definitions used throughout the paper. Then, we introduce the Asymmetric Graph Contrastive learning (AGC) in detail including the overall framework and the design of asymmetric encoders.

3.1. Preliminaries

The input graph is defined as $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is the set of nodes ($\left|\mathcal{V}\right|=N$) and $\mathcal{E}$ is the set of edges [35]. Each node $v_i\in \mathcal{V}$ is represented by a feature vector $x_i\in {\mathbb{R}}^d$ and matrix $X\in {\mathbb{R}}^{N\times d}$ denotes all node features in which each row represents a feature vector. The edge set can also be represented by an adjacency matrix $A\in {\mathbb{R}}^{N\times N}$. $A_{i,j}$ is 1 if there is an edge between nodes $v_i$ and $v_j$ or 0 if no edge exists between the two nodes. A graph can also be represented as $\mathcal{G}=(A,X)$. In unsupervised node-level representation learning, the objective is to train an encoder $f(·)$ without using labels. The encoder $f(·)$ can learn node representations $Z\in {\mathbb{R}}^{N\times d^{\prime }}$ with topology A and features X as input, i.e., $Z=f(X,A)$. The learned node representations are low-dimensional ($d^{\prime }\ll d$) and contain structural properties and feature information in graph, which can be used in downstream tasks, such as node classification or link prediction.

3.2. The Framework of AGC

AGC is a simple graph contrastive learning method with asymmetric encoders. The new method is designed to learn comprehensive node representations in a simple and effective way, without using negative samples or complex architecture. At each iteration, AGC first generates two views based on the original graph by using data augmentation. The two graph views are subsequently fed into a pair of asymmetric encoders to learn node representations. Finally, for two representations of a node in the two graph views, AGC employs a cosine similarity objective and maximizes the agreement between them. The flow chart of AGC is shown in Figure 2.

Similar to GRACE [27], we generate two augmented graphs from the original graph at both structure and feature levels. The two augmented graphs can provide different node contexts (structure level) and node content (feature level) for AGC to contrast with. For structure-level augmentation, we randomly remove edges with probability $\alpha$ and denote an augmented topology as an adjacency matrix $\tilde{A}$. For feature-level augmentation, we first select dimensions with probability $\beta$, and then set the feature values in these dimensions to zero for all nodes. The augmented node features are denoted as $\tilde{X}$. In AGC, two augmented graph views are generated: ${\tilde{\mathcal{G}}}_1=({\tilde{A}}_1,{\tilde{X}}_1)$ and ${\tilde{\mathcal{G}}}_2=({\tilde{A}}_2,{\tilde{X}}_2)$.

With two augmented graph views as input, a pair of asymmetric GNN encoders are constructed to learn node representations. The two GNN encoders have unequal depth layers. The asymmetric architecture is a vital component in AGC to avoid collapsing solutions. It can also help us to learn more comprehensive node representations by capturing information from different neighbor scales because a long encoder can aggregate information from high order neighbors while a short encoder can gather information from low order neighbors. A long GNN encoder is denoted by $f_{{\theta }_L}(·)$ and a short GNN encoder is denoted by $f_{{\theta }_S}(·)$. Then, node $v_i$ can be represented as $z_i^L=f_{{\theta }_L}({\tilde{A}}_1,{\tilde{X}}_1)$ and $z_i^S=f_{{\theta }_S}({\tilde{A}}_2,{\tilde{X}}_2)$. The negative cosine similarity is minimized to make the two representations similar:

$$\mathcal{D}(z_i^L,z_i^S)=-\frac{z_i^L}{{∥z_i^L∥}_2}·\frac{z_i^S}{{∥z_i^S∥}_2},$$

(6)

where ${∥·∥}_2$ is L2-norm. We further compute ${\overline{z}}_i^L=f_{{\theta }_L}({\tilde{A}}_2,{\tilde{X}}_2)$ and ${\overline{z}}_i^S=f_{{\theta }_S}({\tilde{A}}_1,{\tilde{X}}_1)$. A symmetrized loss of AGC can be written as the expectation of all nodes

$$\mathcal{L}={\mathbb{E}}_{v_i}\left[\mathcal{D}\left(z_i^L,\mathrm{stopgrad}\left(z_i^S\right)\right)+\mathcal{D}\left({\overline{z}}_i^L,\mathrm{stopgrad}\left({\overline{z}}_i^S\right)\right)\right],$$

(7)

where $\mathrm{stopgrad}(·)$ is a stop-gradient operation. It means that the short encoder $f_{{\theta }_S}(·)$ will not receive gradient. The parameters of $f_{{\theta }_S}(·)$ will not be updated according to gradient, and its parameters are shared by $f_{{\theta }_L}(·)$. The proof that asymmetric encoders together with a $\mathrm{stopgrad}(·)$ can avoid collapsing solutions is described in Section 2.

3.3. The Design of Asymmetric Encoders

In AGC, node representations are learned by designing a pair of asymmetric encoders to avoid collapsing solutions. By using the asymmetric encoders, AGC can learn efficiently without using negative samples, and follow a simple design without a predictor network or a moving-average strategy on parameter updating. The objective without dense contrast (calculating the distance between negative samples) has a lower restraint of memory and computational complexity for large graphs. Meanwhile, the asymmetric GNN encoders can also learn node representation from different neighbor scales. A short encoder is used to capture low-order information and a longer GNN for high-order information. Compared to existing graph contrastive learning methods, which employ symmetric encoders, AGC can learn more comprehensive information by considering different aggregation scales, which is the defect of other models but an important aspect in graph learning field. Generally, we formulate the representation of $v_i$ at the t-th layer as

$$z_i^{\left(t\right)}=\mathrm{COMBIN}{\mathrm{E}}^{\left(t\right)}(z_i^{(t-1)},a_i^{\left(t\right)}),$$

(8)

$$a_i^{\left(t\right)}=\mathrm{AGGREGAT}{\mathrm{E}}^{\left(k\right)}\left(\{z_j^{(t-1)}:v_j\in \mathcal{N}\left(v_i\right)\}\right),$$

(9)

where $\mathcal{N}\left(v_i\right)$ is the neighbor set of $v_i$, $z_i^{\left(0\right)}=x_i$, $\mathrm{COMBINE}(·)$ and $\mathrm{AGGREGATE}(·)$ are component functions with learnable parameters. For the short GNN encoder with m layers in AGC, its parameters of all m layers are shared by the first m layers of the long GNN encoder.

We use batch normalization [36] for stabilizing training and use residual connections [37] in the branch of the long encoder. Results in experiment section demonstrate that batch normalization and residual connections can further improve the performance of our model. The node representation in the long encoder with residual connections can be formulated as

$$z_i^{\left(t\right)}=\mathrm{COMBIN}{\mathrm{E}}^{\left(t\right)}(z_i^{(t-1)},z_i^{\left(m\right)},a_i^{\left(t\right)}),\mathrm{where}t>m,$$

(10)

where $z_i^{\left(t\right)}$ is the representation of $v_i$ at the t-th layer, $z_i^{\left(m\right)}$ is the representation of $v_i$ at the m-th layer, and m is the depth of the short GNN.

4. Experiments

In this section, we empirically evaluate the proposed AGC model on the node classification task, and further perform ablation experiments to investigate the role of each component in AGC.

4.1. Experimental Setup

4.1.1. Dataset

To analyze the effectiveness of AGC, we choose six widely used datasets for the node classification task, including four citation networks (Cora, CiteSeer, CoauthorCS, CoauthorPhy) and two co-purchase networks (Amazon-Computer, Amazon-Photos). For each dataset, nodes are randomly divided into training, validation, and test sets in the ratio of 80%, 10%, and 10% respectively.

4.1.2. Evaluation Protocol

AGC is compared with other baselines by following the original evaluation design of DGI [32], where models are trained in a fully unsupervised way and the learned embeddings are used to train a simple L2-regularized logistic regression classifier. The procedure is repeated for twenty runs with different training and test data splits and the average classification accuracy is reported on each dataset for a fair evaluation.

4.2. Performance on Node Classification

Table 1. Summary of the results of node classification tasks. Performances are measured in terms of classification accuracy along with standard deviations. Available data for each method during training is highlighted in italic font in the second column, where X, A, Y correspond to features, topology, and labels. The best results are in bold, and the second-best results are underlined.

Method	Training Data	Cora	Citeseer	CoauthorCS	CoauthorPhy	Am.Photos	Am.Computers
Raw features	X	64.80 ± 0.78	64.60 ± 0.70	90.37 ± 0.00	93.58 ± 0.00	78.53 ± 0.00	73.81 ± 0.00
node2vec	A	74.80 ± 0.34	52.30 ± 0.42	85.08 ± 0.03	91.19 ± 0.04	89.67 ± 0.12	84.39 ± 0.08
DeepWalk	A	75.70 ± 0.68	50.50 ± 0.50	84.61 ± 0.22	91.77 ± 0.15	89.44 ± 0.11	85.68 ± 0.06
DGI	X, A	82.60 ± 0.40	68.80 ± 0.70	92.15 ± 0.63	94.51 ± 0.52	91.61 ± 0.22	83.95 ± 0.47
GMI	X, A	83.00 ± 0.30	72.40 ± 0.10	OOM	OOM	90.68 ± 0.17	82.21 ± 0.31
GRACE	X, A	83.30 ± 0.40	72.10 ± 0.50	92.93 ± 0.01	95.26 ± 0.02	92.15 ± 0.24	86.25 ± 0.25
GCA	X, A	80.90 ± 0.41	72.14 ± 0.06	93.10 ± 0.01	95.73 ± 0.03	92.53 ± 0.16	87.85 ± 0.31
BGRL	X, A	82.77 ± 0.75	64.45 ± 0.15	93.21 ± 0.18	95.56 ± 0.12	92.87 ± 0.27	89.68 ± 0.31
AGC-L	X, A	83.92 ± 0.79	73.06 ± 0.93	93.75 ± 0.15	95.94 ± 0.03	92.80 ± 0.11	88.75 ± 0.47
AGC-S	X, A	84.88 ± 0.41	73.72 ± 0.24	93.76 ± 0.24	96.09 ± 0.07	93.32 ± 0.10	88.94 ± 0.27
GCN	X, A, Y	82.80 ± 0.30	72.00 ± 0.50	93.03 ± 0.31	95.65 ± 0.16	92.42 ± 0.22	86.51 ± 0.54

presents the result of the node classification accuracy on six datasets. AGC is principally compared with three methods based on the negative samples (GRACE and GCA) or predictor network (BGRL). The performance of two classic unsupervised models (DeepWalk, Node2vec), two superior contrastive models (DGI, GMI), and a semi-supervised two-layers GCN model are also reported. The accuracy results are reused from the reports in [27,28].

According to Table 1, observations are concluded as follows. (1) AGC outperforms the baseline methods on five out of six datasets. The main reason behind the results lies in the asymmetric design of our model. Compared to other models with a fixed-size encoder, AGC enables the model to learn more comprehensive node representations from information carried by different neighbor scales. (2) Our asymmetric-based model shows considerable improvement over the models that depend on negative samples to cover the collapsed solutions. At the same time, our model requires less memory because we find that our 16GB Tesla-P100 GPU cannot stand by the training procedure of GRACE and GCA on CoauthorPhy during experiments while AGC only needs a quarter of it. (3) AGC has an obvious improvement over supervised GCN across all datasets but our performance still falls behind the current supervised state-of-the-art methods. (4) We show that both the output of long (AGC-L) and short encoder (AGC-S) have a competitive performance among all methods, which proves the hypothesis mentioned in Section 2 that the node representations obtained by aggregating neighbors in different scales can represent the same semantics.

4.3. Ablation Study

In this section, we perform a series of ablation studies to show which component may play an important role in avoiding collapsing solution and which component is benefit to outstanding performance.

4.3.1. Asymemetric

As analyzed in Section 2, our proposed model efficiently avoids the collapsing solutions by the design of asymmetric encoders. In this section, we deploy experiments with three kinds of variants in which “w/o asymmetric” represents AGC with symmetrical encoders (two GNNs with same depth layers), “w/o stop-grad” represents AGC without stop-gradient operation, and “w/o share-parameter” denotes that the parameters of short GNN encoder do not update via the share-parameter operation and keep constant.

Experimental results are shown in Figure 3. It is obviously shown that AGC successfully escapes from the collapsing solutions with a non-zero loss throughout the entire training process. Without stop-gradient or asymmetric architecture, the optimizer quickly finds the collapsing places and reaches the minimum loss value of 0. Correspondingly, the linear evaluation accuracy of the trained embeddings is in

Table 2. Classification accuracy of different variants on Cora dataset.

Variants	acc. (%)
AGC	84.88
AGC w/o asymmetric	30.22
AGC w/o stop-gradient	50.19
AGC w/o share-parameter	71.07
AGC w/o bn	78.78
AGC w/o residual	83.09

. AGC achieves top performance with 84.88% accuracy on Cora dataset. While individually eliminating stop-gradient or asymmetric encoders, the accuracy decreases to 50.19% and 30.22%, which shows poor classification ability for the few types of nodes. The results imply that both components make contributions to prevent the model from collapsing.

In addition, if the shared-parameters operation is removed in AGC, the loss is hard to coverage as a normal solution because the contrastive task becomes harder and the embeddings from the completely unrelated encoders are hard to discover their similarities. Actually, 71.07% of the result mostly benefits from the raw features of the graph.

4.3.2. Residual Operating

Table 2 shows the comparison on “with or without residual operator”, where all hyper-parameters keep unchanged and residual connection is the only difference. The reported results clearly show that an additional residual mechanism can improve the representation learning ability of GNNs, while when training without it, the accuracy has dropped about 2 percent on the Cora dataset. Building the extensive connections between the model layers and the intermediate node embedding allows the GNN-based encoder has a deeper structure with no danger of over-smooth caused by the increasing aggregation times. With the help of it, the proposed contrastive method benefits from the joint consideration of effective high-order and closer information to achieve better performance over the models with symmetric-design encoder.

4.3.3. Batch Normalization

To evaluate the effectiveness of the batch normalization layers as an important component of our model, we run experiments with all batch normalization layers removed, which results in a decline of six percent. Normalization methods are designed to make the training procedure more stable by making the different features have the same distribution. In summary, we observe that normalization layers are unrelated to the resolution of the trivial solution, but have a significant boost of about six percent to the learning process by arranging it properly.

4.3.4. Data Augmentation

Data augmentation plays an important role in most of the contrastive learning methods for creating different views. The procedure can be illustrated by the reason that providing different parts of the graph information involving topology and feature can express the same semantics. Previous works (GRACE, GCA) with the same data augmentation strategy as our methods have an exhausting process of finding the optimal solution for the four hyper-parameters that need to be adjusted properly. In our work, we prefer the idea that the two views masking over half of the information on the graph may minimize the overlapping information between each other while carrying the corresponding semantics as the simplest way to define hard positive samples.

Experiments demonstrate that setting rates to 0.6 is a straightforward and effective approach, achieved by training models with augmentation rates ranging from 0.0 to 0.9. First, we fix the rates of removing edges and vary the rates of masking features to see the impact on models including BGRL and AGC with different rate settings. The result in Figure 4 shows that large dropping rates give the model a stronger ability to distinguish nodes, where AGC achieves the best performance at 0.6 and BGRL adopts 0.9 as the best. If we remove the attribute masking, AGC reaches the highest performance at 0.8. Different observations on GRACE show that it prefers to a lower rate of augmentation (0.3, 0.4, 0.2, 0.4, as reported). This is caused by the different contrastive mechanism where methods based on negative samples need low augment rates to improve the quality of representation learned from negative samples. A similar conclusion can be observed in Figure 4, where feature masking rates have a greater impact on accuracy.

5. Related Works

5.1. Graph Neural Networks

Graph neural networks (GNNs) [15,16,17,18] have been proven to be an effective tool for analyzing graph-structural data. Most GNNs follow the principle of message passing. They learn node representations by aggregating information from neighboring nodes based on the guidance of topological structure. Among them, there are lots of semi-supervised methods [35]. GCN [19] simplifies spectral graph convolutions by using a localized first-order approximation. GAT [20] uses an attention mechanism to learn the importance between nodes and their neighbors. GIN [38] aims at distinguishing different graph structures. There are also some designs for unsupervised GNNs. One simple way is to integrate GNNs into an autoencoder framework where an encoder employs GNNs to learn representations in latent space while a decoder is used to reconstruct the graph structure [39]. Another popular way is to use DeepWalk-like objectives [12] to sample a portion of node pairs as negative pairs while existing node pairs with edge in the graphs are positive pairs [40].

5.2. Graph Contrastive Learning

Recently, self-supervised contrastive methods have achieved competitive performance in computer vision (CV) [34,41,42,43]. Benefit from advanced and effective ideas on them, a lot of graph contrastive learning methods have been proposed. DGI [32] is the most representative work which follows the cross-scale contrastive mechanism first purposed by information maximization (InfoMax [44]). DGI maximizes the mutual information (MI) between patch representation and the overall graph representation by using Jenson Shannon divergence as an estimator [45,46]. There are some extended works about DGI. GMI [47] uses MI to measure the correlation between raw features and learned node representations. MVGRL [48] uses different views for comparison such as PPR [49] and heat kernel [50]. Further, for a heterogeneous graph with multiple types of nodes, DMGI [51] follows the InfoMax principle to learn multiple kinds of representations corresponding to multiple relations, and learns the final representations by using a meta-path-based aggregation.

A more general contrastive pattern is to maximize the agreement between two same-scale elements in different augmented data views. Different from augmentation methods in CV [52,53], augmentations on graphs are restricted. For the structure of networks, subgraph sampling and node dropping [29,30] are commonly used for the graph-level task while removing edges [27,28] is suitable for node-level representation learning. Compared with removing edges randomly [27], GCA [28] proposes an adaptive augmentation scheme for both edges and features. Besides data augmentation, the contrastive objective is another important component. SimCLR [41] first adopts an NCE-form [54] loss to contrast among positive and negative samples in different augmented views. Some works on graph follow SimCLR and transfer NCE loss to node-level [27,28] and graph-level [29] representation learning. With the appearance of MoCo [42], a dynamic dictionary is used to preserve more data samples, which enables models to use a larger number of negative samples [55]. GCC [30] follows this idea and designs a graph encoder to capture structural information for pre-training tasks. More recently, some works [34,43] in CV show that a predictor component can effectively avoid collapsing solutions without using negative samples. Similarly, our work aims to avoid collapsing solutions without using negative samples in graph representation learning. Moreover, we consider the properties of graphs and design a simpler and more efficient model than existing methods.

GRACE [27] is a novel framework for graph representation learning that is similar to our model. Both of them generate two graph views at structure and feature levels by removing edges for topology and masking features for node attributes, respectively. Nevertheless, GRACE utilizes a GNN encoder for both two graph views to generate node embedding in low dimensionality. In contrast, our model constructs a pair of asymmetric GNN encoders with unequal depth, which is the key component to avoid collapsing solutions. Furthermore, while GRACE requires a substantial amount of negative samples for training, our model can prevent the occurrence of collapsing solutions even when trained without utilizing negative samples.

6. Conclusions

In this paper, we analyze the reasons why representations in contrastive learning are easy to fall into collapsing solutions when no negative samples are used. We prove that encoders with symmetrical architecture lead to collapsing solutions, while a pair of asymmetric encoders with a stop-gradient operation can avoid this problem. Following the above analysis, we propose a new graph contrastive learning method (AGC). AGC uses two graph neural networks with unequal depth layers to learn node representations from two augmented views. The design of asymmetric architecture makes AGC learn discriminative representations by jointly considering information from different neighbor scales. Based on the proposed AGC, we performed ablation experiments to demonstrate the critical roles of both asymmetric architecture and stop-gradient operation in avoiding collapsing solutions. Experimental results on node classification task also showed the superiority of our proposed model.