Hyperbolic Directed Hypergraph-Based Reasoning for Multi-Hop KBQA

Abstract

The target of the multi-hop knowledge base question-answering task is to find answers of some factoid questions by reasoning across multiple knowledge triples in the knowledge base. Most of the existing methods for multi-hop knowledge base question answering based on a general knowledge graph ignore the semantic relationship between each hop. However, modeling the knowledge base as a directed hypergraph has the problems of sparse incidence matrices and asymmetric Laplacian matrices. To make up for the deficiency, we propose a directed hypergraph convolutional network modeled on hyperbolic space, which can better deal with the sparse structure, and effectively adapt to the problem of an asymmetric incidence matrix of directed hypergraphs modeled on a knowledge base. We propose an interpretable KBQA model based on the hyperbolic directed hypergraph convolutional neural network named HDH-GCN which can update relation semantic information hop-by-hop and pays attention to different relations at different hops. The model can improve the accuracy of the multi-hop knowledge base question-answering task, and has application value in text question answering, human–computer interactions and other fields. Extensive experiments on benchmarks—PQL, MetaQA—demonstrate the effectiveness and universality of our HDH-GCN model, leading to state-of-the-art performance.

1. Introduction

Knowledge base question answering (KBQA) has been a hot task in the field of natural language processing and is very challenging [1]. Several different QA datasets have been proposed, such as the Stanford Question Answering Dataset (SQuAD) [2,3], NarrativeQA [4] and CoQA [5], and the kind of reasoning based on these datasets is termed single-hop reasoning, since it requires reasoning only over a single piece of evidence [6]. For a QA task of single-hop reasoning, the performance of previous work [1,7] has been improved a lot over the last years.

However, in real-world QA tasks, obtaining answers often requires multi-hop reasoning [8], that is, to find a knowledge path consisting of multiple pieces of knowledge in the knowledge base to deduce the answer. Figure 1 shows a two-hop KBQA example. The reasoning path starts from the entity mentioned in the query and consists of the relations at each hop and the intermediate entities. The methods mentioned above that focus only on single-hop reasoning lack the ability to deal with multi-hop reasoning QA tasks. To solve a multi-hop QA task, some work has been proposed recently which can be mainly divided into two categories. One is the neural network-based methods such as models in [9,10] and the other is the graph neural network-based methods such as [11]; this has achieved desirable performance [12].

When we deal with multi-hop tasks, generally, entities extracted from the query begin to be retrieved in the knowledge base, go down to the next hop according to the specific relationship of different hops, repeat this step to form a reasoning path, and finally find the final answer. In this process, we argue that semantic relational information is crucial for multi-hop reasoning, while previous studies have not fully exploited semantic relational information. The work in [13] computes relation-specific transformations by separating different relations, which does not consider semantic relation information. Ref. [14] does not require updating relational information during multi-hop reasoning, but exploits relational information to obtain attention for static graphs.

In addition, the pairwise connections between nodes based on a graph network (GNN) are insufficient with respect to fully representing the higher-order relationships between relationships and entities in the knowledge graph. Recently, some main works on hypergraph convolutional networks (HGNN) have been proposed. HGNN uses hyperedges to connect more than two nodes at the same time, which is conducive to imitating human reasoning and accurately locating a group of entities connected by the same relationship, rather than reasoning entity by entity. The disadvantage is that HGNN is aimed at undirected hypergraphs, while knowledge graphs are directed, and each triplet has a specific directional meaning. HGNN collects potential learning relationships from connected entities, but does not reveal and further utilize them.

Based on the study in the hypergraph neural network introduced above, a directed hypergraph convolutional network-based model for multi-hop KBQA (2HR-DR) was proposed [15]. 2HR-DR models the entities extracted from questions and their related relationships and entities in the knowledge base into directed hypergraphs, and then uses Directed Hypergraph Convolutional Networks (DHGCN) [15] to predict relations hop-by-hop and form a sequential relation path to make the reasoning interpretable. 2HR-DR can explicitly learn and update relation information and dynamically concentrate on different relations at different hops.

Although 2HR-DR can better solve some of the challenges mentioned above, it still has some disadvantages. First, using directed hypergraphs to model entities and relationships may lead to a situation in which an entity can be related to many entities based on some relations while it may be related to a few entities based on some other relations. For example, as shown in Figure 1, when constructing a hypergraph of the query, the number of entities (actors) related to “Child_of_Deaf_Adults” can be very small based on some relations such as “Staring_in” while a large number of entities (actors) do “Act_in” this movie. This results in a large difference in the number of nodes contained by each hyperedge in the modeled directed hypergraph. In that case, the incidence matrices of constructed hypergraphs will be much sparser, and that will have a bad effect on the training efficiency and accuracy of the model. Second, 2HR-DR used the directed hypergraph convolution network, which needs the eigenvalue decomposition of Laplacion matrices when calculating the spectrum convolution of hypergraphs, and that requires that the Laplacian matrices are real symmetric matrices (we are not able to ensure that non-symmetric matrices can certainly perform eigenvalue decomposition). However, as for directed hypergraph convolution networks, since each hyperedge has a direction, the degree of each node is supposed to be divided into in degree and out degree, which are different in most cases; this leads to the fact that Laplacian matrices are often asymmetric matrices.

To solve the problems mentioned above, we firstly propose a Hyperbolic Directed Hypergraph Convolutional Network (HDH-GCN) for a directed hypergraph to take the direction of information transmission into account. We investigate hyperbolic embedding spaces [16] and manage to map the sparse data points and hypergraph to the hyperboloid manifold directly. The rationale is that hyperbolic space has a stronger ability than Euclidean space to accommodate networks with long-tailed distributions and sparse structures [17], which is also verified in our experiments. On that basis, we propose a Hyperbolic Directed Hypergraph Convolutional Network (HDH-GCN)-based framework for multi-hop QA. This framework explicitly updates the relation information and dynamically focuses on specific relations at every hop of the query. In addition, we record the semantic representation of the relationship in each hop, and the representation of the relationship in every hop is influenced by the representation of the relationship in the previous hops, which makes the QA task interpretable to a large extent.

In summary, we make the following contributions:

• For solving the problem of sparse incidence matrices of directed hypergraphs modeled on a knowledge base, we design a method of modeling a directed hypergraph in hyperbolic space.
• Based on the hyperbolic directed hypergraph, we propose a Hyperbolic Directed Hypergraph Convolutional Network (HDH-GCN) for a directed hypergraph and design a framework on this basis that can handle multi-hop knowledge base question-answering tasks well.
• The modules constitute a new model, namely, HDH-GCN for handling the multi-hop knowledge base question-answering task. Through the experiments on several real-world datasets, we confirm the superiority of HDH-GCN over state-of-the-art models.

2. Related Work

2.1. Multi-Hop Question Answering

The multi-hop KBQA model can be basically divided into two types. The first is to apply the neural network mentioned earlier. These models use the previous single-hop question-answering method [18,19,20] for multi-hop question-answering tasks. Xu et al. improved KVMemNet to achieve better results across multiple triples [10]. Zhong et al. used coarse-grained modules and fine-grained modules [21]. Part of the method introduces an end-to-end framework, which is explicitly designed to simulate the step-by-step reasoning process involved in multi-hop QA and MRC. Kundu et al.’s [22] model constructed a path connecting questions and candidate answers, and then scored them through the neural architecture. Jiang et al. [23] also constructed a proposer to propose an answer from each root to leaf path in the reasoning tree, extract a key sentence containing the proposed answer from each path and finally combine them to predict the final answer. However, these methods lack consideration of graph structure information.

The other kind of method is based on graph neural networks. Sun et al. learnt what to retrieve from the KB and corpus and then reasoned over the built graph [24]. Tu et al. employed GCN to reason over heterogeneous graphs [25]. Xiong et al. achieved better performance by applying graph attention networks [14]. Cao et al. proposed a bi-directional attention entity graph convolutional network [26]. These models use r-GCN [13], which does not consider the semantic relation information, or use graph attention networks to assign static weights. Different from these models, ref. [15] proposes a dynamic relation strategy, which dynamically updates relation states during the reasoning process. Documents unrelated to the complex query may affect the accuracy of the model. In the “select, answer, and explain” (SAE) model proposed by Tu et al. [27], BERT [28] acts as the encoder in the selection module. Then a sentence extractor is applied to the output of BERT to obtain the sequential output of each sentence with precalculated sentence start and end indices, to filter out answer-unrelated documents and thus reduce the amount of distraction information. The selected answer-related documents are then input to a model, which jointly predicts the answer and supporting sentences. Concurrently to the SAE model, Bhargav et al. [29] used a two-stage BERT-based architecture to first select the supporting sentence and then used the filtered supporting sentence to predict the answer. The upstream side of Jiang et al.’s [23] proposed model is the Document Explorer to iteratively address relevant documents. Han et al. [15] proposed two-phase hypergraph-based reasoning with dynamic relations which explicitly learns and updates relation information and dynamically concentrates on different relations at different hops.

2.2. Hypergraph Convolutional Networks

Feng et al. [30] proposed a hypergraph neural network, which replaces the general graph with a hypergraph structure, effectively encoding the higher-order data correlation. Bai et al. [31] further enhanced the representational learning ability by using attention modules. Yadati, N. et al. [32] proposed a new method of training a GCN on a hypergraph using tools from the spectral theory of hypergraphs and applying the method to the problems of SSL (hypergraph-based semi-supervised learning) and combinatorial optimization on real-world hypergraphs. Zhang et al. [33] developed a new self-attention-based graph neural network applicable to homogeneous and heterogeneous hypergraphs with variable hyperlink sizes. Han et al. [15] proposed a directed hypergraph convolutional network that incorporates direction information into HGNN to deal with a directed knowledge graph.

2.3. Hyperbolic Neural Networks

Hyperbolic space has always been a popular research domain in mathematics. Some works have been conducted to explore the treelike structure of graphs [34,35] and the relations between hyperbolic space and hierarchical data such as languages and complex networks [36,37]. Such works have demonstrated the consistency between real-world scale-free and hierarchical data and the hyperbolic space, providing a theoretical basis for recent works which apply hyperbolic space to various tasks including link prediction, node classification and recommendation. Some researchers apply hyperbolic space to traditional metric learning approaches such as HyperBPR [38] and HyperML [39]. Some try to adopt hyperbolic space to neural networks and define hyperbolic neural network operations, producing powerful models such as hyperbolic neural networks [40], hyperbolic graph neural networks [41] and hyperbolic convolutional neural networks [17]. Meanwhile, ref. [42] provides a scalable hyperbolic recommender system for industry use. Ref. [43] applies hyperbolic space to heterogeneous networks for link prediction tasks. Ref. [44] applies hyperbolic space to next-POI recommendation. Ref. [45] proposes a path-based recommendation approach with hyperbolic embeddings, etc.

3. Methods

3.1. Hyperbolic Directed Hypergraph Convolutional Networks

In this section, we are going to introduce the directed hypergraph convolutional network constructed on hyperbolic space. Definitions of notations used in the text are shown in the Table 1.

3.1.1. Undirected Hypergraph Convolutional Network

We first introduce undirected hypergraph convolutional neural networks. Different from simple graphs, hyperedges in a hypergraph may contain two or more vertices. A hypergraph can be defined as $\varrho =(V,E,W)$, which includes a vertex set V, a hyperedge set E and each hyperedge is assigned with a weight by W which is a diagonal matrix whose diagonal lines are the weights of each hyperedge. We use a $\left|V\right|\times \left|E\right|$ incidence matrix H to denote a hypergraph $\varrho$, and H can be concretely expressed as:

$$h(v,e)=\left\{\begin{array}{ccc}\hfill 1& \hfill & \hfill v\in e\\ \hfill 0& \hfill & \hfill v\notin e\end{array}\right.$$

(1)

For every vertex $v\in V$, the degree can be defined as $d\left(v\right)={\Sigma }_{e\in E}\omega \left(e\right)h(v,e)$, For every hyperedge $e\in E$, the degree can be defined as $d\left(e\right)={\Sigma }_{v\in V}\omega \left(v\right)h(v,e)$, $D_e$ and $D_v$ denote the diagonal matrices of the hyperedge degrees and the vertex degrees. We let $\mathsf{\Theta }=D_v^{\frac{1}{2}}HW{D}_e^{-1}{H}^T{D}_v^{\frac{1}{2}}$ and $\mathsf{\Delta }=I-\mathsf{\Theta }$ is defined as the hypergraph Laplacian [46], according to the Laplacian expression form, it can be known that it is a symmetric semidefinite matrix. It can be obtained by eigenvalue decomposition of symmetric positive semidefinite matrices that $\mathsf{\Delta }=\mathsf{\Phi }\mathsf{\Lambda }{\mathsf{\Phi }}^T$; [30] use the eigen vectors as the Fourier bases and the eigenvalues as frequencies to express the spectral convolution as:

$$g*X=\mathsf{\Phi }(\left({\mathsf{\Phi }}^{T}g\right)\otimes \left({\mathsf{\Phi }}^{T}X\right))=\mathsf{\Phi }g\left(\mathsf{\Delta }\right){\mathsf{\Phi }}^{T}X$$

(2)

Ref. [30] then approximate the above equation by Chebyshev polynomials, modify inside parameters appropriately and finally formulate hyperedge convolution as:

$$X^{(l+1)}=D_v^{-1}HW{D}_e^{-1}{H}^T{X}^{\left(l\right)}P$$

(3)

where $X^{\left(l\right)}$ is the node feature of the hyperedge at layer l, and P is the learnable parameter.

3.1.2. Hyperbolic Directed Hypergraph Convolutional Neural Network

As shown in Section 3.1.1, the derivation process of hyperedge convolution is based on the eigenvalue decomposition of the Laplacian matrix of the hypergraph. In an undirected hypergraph, the Laplacian matrix is a symmetric positive semidefinite matrix because the degree matrices of its vertices and hyperedges are unique, so the eigenvalue decomposition always works. However, for directed hypergraphs, due to hyperedges having direction, the degree matrices of vertices and hyperedges should be divided into out-degree matrix and in-degree matrix, and according to the random walk explanation of spectral hypergraph partitioning [46], the specific representation of the Laplacian matrix of a directed hypergraph should be as follows:

$$\mathsf{\Delta }=I-\mathsf{\Theta }=I-D_v^{tai{l}^{-1}}{H}^{tail}W{D}_e^{hea{d}^{-1}}{H}^{hea{d}^T}$$

(4)

where $D_v^{tail}$ and $D_e^{head}$ are the diagonal matrices of tail degrees of nodes and head degrees of hyperedges, $H^{tail}$ and $H^{head}$ stand for the tail and head incidence matrices.

$$H^{tail}(i,j)=\left\{\begin{array}{ccc}\hfill 1& \hfill & \hfill v_i\in e_j^{tail}\\ \hfill 0& \hfill & \hfill v_i\notin e_j^{tail}\end{array}\right.$$

(5)

$$H^{head}(i,j)=\left\{\begin{array}{ccc}\hfill 1& \hfill & \hfill v_i\in e_j^{head}\\ \hfill 0& \hfill & \hfill v_i\notin e_j^{head}\end{array}\right.$$

(6)

Since the two incidence matrices are generally different in a directed hypergraph, the Laplacian matrix is often not a symmetric matrix; as a result, the directed hypergraph modeled on the knowledge base may not be able to derive the hyperedge convolution as the undirected hypergraph shown in Section 3.1.1, and this will produce calculation error to some extent.

Aiming at solving the problem above and the sparsity issue in hypergraphs modeled on a knowledge base, we apply the variant forms of GCN on the hyperbolic space [17] to the directed hypergraph and obtain the matrix form of the directed hypergraph convolution network on the hyperbolic space. The directed hypergraph convolution operations on hyperbolic spaces aggregate each representation vector individually in vector dimensions, without the aforementioned problem of symmetry of the Laplacian matrix of directed hypergraphs. The specific process is as follows:

We first transform the initial item features from Euclidean space to hyperbolic space $H^K$, and then we feed the initial hyperbolic item embeddings to learn item embeddings. For the hyperbolic space, we set $\alpha :=\{\sqrt{K},0,0,\dots ,0\}\in H^K$ as the north pole in $H^K$, and the negative curvature of the hyperboloid manifold is $-\frac{1}{K}$. Then the initial item features in hyperbolic space can be deduced from Euclidean space as follows:

$$x^{(0,H)}=ex{p}_{\alpha }^K\left((0,x^{(0,E)})\right)=(\sqrt{K}cosh\left(\frac{\parallel x^{(0,E)}{\parallel }_2}{\sqrt{K}}\right),\sqrt{K}sinh\left(\frac{\parallel x^{(0,E)}{\parallel }_2}{\sqrt{K}}\right)\frac{x^{(0,E)}}{\parallel x^{(0,E)}{\parallel }_2})$$

(7)

where $x^{(0,H)}$ and $x^{(0,E)}$ are the initial hyperbolic embedding and the initial Euclidean embedding, respectively. For the directed hypergraph convolutional network, when updating entity representation through the convolutional layer, we first aggregate the head entities in directed hyperedges to obtain the representation of relations, and then accumulate relation representation containing the same tail entity to obtain the representation of the tail entity so as to continuously update the representation of entities. The representation can be aggregated via the following convolutional operation in hyperbolic space:

$$r_i^{L,H}=ex{p}_{\alpha }^K(\sum _{j\in e_{i_{head}}}{M}_{ij}lo{g}_{\alpha }^K(x_j^{L,H}))$$

(8)

where $r_i^{L,H}$ is the hyperbolic hidden embedding of relation $e_i$ in the L-th layer after aggregation, the node j’s hyperbolic embedding is transformed to Euclidean embedding via $lo{g}_{\alpha }^K$, so the Euclidean-based sum and add operations are available. $ex{p}_{\alpha }^K$ aims to transform the Euclidean-based embedding to hyperbolic embedding. $M_{ij}$ is the projecting weight defined as follows:

$$M_{ij}=softma{x}_{j\in {e_i}_{head}^H}(MLP(lo{g}_{\alpha }^K(x_i^{L,H})\left|\right|lo{g}_{\alpha }^K(x_j^{L,H})))$$

(9)

Accordingly, we can write an expression that evaluates the tail entity representation:

$$x_t^{L+1,H}=ex{p}_{\alpha }^K(\sum _{s\in e_{t_{tail}}}{M}_{ts}lo{g}_{\alpha }^K(r_s^{L,H}))$$

(10)

where $x_t^{L,H}$ is the hyperbolic embedding of tail entity $x_t$ in the L-th layer, $e_{t_{tail}}$ stands for the directed hyperedge containing the tail entity $x_t$.

Because $ex{p}_{\alpha }^K$ and $lo{g}_{\alpha }^K$ are the inverse of each other, the total convolution is as follows:

$$x_t^{L+1,H}=ex{p}_{\alpha }^K(\sum _{s\in e_{t_{tail}}}{M}_{ts}{w}_r^L(\sum _{j\in e_{i_{head}}}{M}_{ij}lo{g}_{\alpha }^K(x_j^{L,H})))$$

(11)

The formula cancels out the adjacent inverse operation $ex{p}_{\alpha }^K$ and $lo{g}_{\alpha }^K$, and applies max pooling to obtain weights for each relation,

$$w_r^L=softmax(MaxPooling(R_i^{L,H}))$$

(12)

To facilitate the formulation of the model, we rewrite it in matrix form:

$$X^{L+1,H}=ex{p}_{\alpha }^K(M_{tail}W{M}_{head}lo{g}_{\alpha }^K(X^{L,H}))$$

(13)

where $X^{L+1,H}$ and $X^{(L,H)}$ are the entity representation matrix in the $L+1$-th and L-th layer, respectively. W stands for a diagonal matrix of hyperedge weights, $M_{head}$ and $M_{tail}$ are the aggregate matrices of the head part and tail part of directed hyperedges.

3.2. Model

In this section, we are going to introduce the concrete model for the multi-hop knowledge base QA task. The overview of how the model works is shwon in Figure 2.

Table 1. Descriptions of notations used in the following parts.

Symbol	Definition
H	a $\left|V\right|\times \left|E\right|$ incidence matrix
$D_e$	a diagonal matrices of the hyperedge degrees
$D_v$	a diagonal matrices of the vertex degrees
$\mathsf{\Delta }$	the hypergraph Laplacian
$\mathsf{\Phi }$	the eigenvectors of the hypergraph Laplacian
⊗	the element-wise Hadamard product
$H^K$	a hyperboloid manifold which negative curvature is $-\frac{1}{K}$
$\alpha$	the north pole in $H^K$
$ex{p}_{\alpha }^K$	the exponential map of the hyperboloid model
$lo{g}_{\alpha }$	the logarithmic map of the hyperboloid model
$cosh$	hyperbolic cosine function
$sinh$	hyperbolic sine function
$M_{head}$	the aggregate matrix of the head-part of directed hyperedges
$M_{tail}$	the aggregate matrix of the tail-part of directed hyperedges

Table 1. Descriptions of notations used in the following parts.

Symbol	Definition
H	a $\left|V\right|\times \left|E\right|$ incidence matrix
$D_e$	a diagonal matrices of the hyperedge degrees
$D_v$	a diagonal matrices of the vertex degrees
$\mathsf{\Delta }$	the hypergraph Laplacian
$\mathsf{\Phi }$	the eigenvectors of the hypergraph Laplacian
⊗	the element-wise Hadamard product
$H^K$	a hyperboloid manifold which negative curvature is $-\frac{1}{K}$
$\alpha$	the north pole in $H^K$
$ex{p}_{\alpha }^K$	the exponential map of the hyperboloid model
$lo{g}_{\alpha }$	the logarithmic map of the hyperboloid model
$cosh$	hyperbolic cosine function
$sinh$	hyperbolic sine function
$M_{head}$	the aggregate matrix of the head-part of directed hyperedges
$M_{tail}$	the aggregate matrix of the tail-part of directed hyperedges

3.2.1. Query-Aware Entity Encoder

The query-aware entity encoder encodes entities and relations in questions and their potential related entities in knowledge base to vector representation. Let $L_q$, $L_e$ and $L_r$ respectively denote the embedding matrices of question, entities and relations; we begin by encoding each question using bidirectional Gated Recurrent Units (GRUs) [47].

$$E_q=BiGRU\left(tanh(W_q{L}_q+b_q)\right)$$

(14)

We then follow the work of [21]; we employ co-attention to learn query-aware entity representation,

$$A_{eq}=L_e\left(H_q^T\right)$$

(15)

$$C_e=softmax\left(A_{eq}\right)E_q$$

(16)

$$C_q=softmax\left(A_{eq}^T\right)L_e$$

(17)

$$D_e=BiGRU\left(softmax\left(A_{eq}\right)C_q\right)$$

(18)

$$E_{attn}=f_c\left([C_e;D_e]\right)$$

(19)

where $softmax$ stands for column-wise normalization, $f_c$ is a linear network which converts 2h dimension to h dimension.

3.2.2. Reasoning over Hypergraph

According to the property that relation embedding can be obtained in the intermediate processes of hyperbolic directed hypergraph convolutional networks, we separate the hyperbolic directed hypergraph convolutional networks into two steps. Specifically, the model first collects the features of nodes onto the connected hyperedges and explicitly represents the learning relationship. Then, it dynamically assigns the weight of the relationship according to the similarity between the problem and the relationship, and predicts the current relation. Finally, the node status is updated through the connection relationship information. The specific process is as follows.

Firstly, assuming that the current hop is l, we use a linear network to concatenate the node status obtained by the previous $l-1$ layer and the input entity representation of the current hop, and then map it onto the hyperbolic space.

$$U_e^{l,H}=ex{p}_{\alpha }^K\left(f_{in}\left([E^{(l-1)};E_{attn}]\right)\right)$$

(20)

where operator [ ; ] is column-wise concatenation. Then the model learns the relation representation $R^{l,H}$ by aggregating the connected head entity feature.

$$U_r^{l,H}=M_{head}lo{g}_{\alpha }^K\left(U_e^{l,H}\right)P_r$$

(21)

where $P_r$ is the relation-specific learnable parameter, we then use a linear network to concatenate the relation representation obtained by the previous $l-1$ layer and $U_r^{l,H}$ to obtain a representation of relations at hop l.

$$R^{l,H}=f_r\left([R^{(l-1),H};U_r^{l,H}]\right)$$

(22)

After that, we apply $w_r^L$ in (12) to obtain weights for each relation and the diagonal matrix W of edge weights is $W=diag\left(w_r^L\right)$. The weight of the dynamic allocation relation depends on the updated relation representation hop-by-hop. This model predicts the current relation based on the relation weight.

Finally, the model adaptively updates entity states by accumulating connected relation feature

$$E^l=ex{p}_{\alpha }^K\left(M_{tail}W{R}^{l,H}{P}_e\right)$$

(23)

where $P_e$ is the entity-specific learnable parameter.

3.3. Training

For an L hop question, we sum the entity representation of each layer to obtain the final representation and use a liner layer $f_{ans}$ to predict the answer distribution.

$$P=\sigma (f_{ans}(\sum _{l=1}^L{E}^{(l)}))$$

(24)

where $\sigma$ is the sigmoid function.

Since the model needs to predict both the answer to the question and the reasoning path, the loss function consists of two parts, one is binary cross-entropy loss of the final answer prediction, the other is the negative log likelihood of the intermediate prediction of the reasoning path. The specific expression of the loss function is as follows:

$$L=-\sum _{i=1}^n(y_{i}log\left(p_i\right)+(1-y_i)log(1-p_i))+\lambda \times \sum _{l=1}^{L+1}(-log\left(w_r^l\left(r_l^{*}\right)\right))$$

(25)

where $y_i$ is the golden distribution over entities. $r_l^{*}$ is the golden relation index at hop L. $\lambda$ is a hyper parameter to balance the two terms.

4. Experiment

This section reports the experiments.

4.1. Experiment Setup

We detail the adopted datasets, evaluation metrics, parameters and baselines.

4.1.1. Datasets

We use two single answer KBQA datasets and two large-scale multi-answer KBQA datasets for the multi-hop KBQA task. We briefly outline these datasets in

Table 2. Statistics of the hypergraph datasets used in the experiments.

Datasets	PQL-2H	PQL-3H	MetaQA-1H	MetaQA-2H
Number of Questions	1594	1031	116,049	148,724
Number of Entities in knowledge base	5034	5034	40,128	40,128
Number of Relations in knowledge base	364	364	9	9

.

• PQL-2H [48]: PQL-2H is a single answer KBQA dataset, which includes a knowledge base containing 5035 entities and 364 relationships, and a two-hop question set containing 1594 two-hop questions. These questions can be answered by following the reasoning path consisting of several relations and intermediate entities. The path has been given.
• PQL-3H [48]: PQL-3H is a single-answer KBQA dataset, which includes the same knowledge base with 5035 entities and 364 relations as PQL-2H, and a three-hop question set with 1031 three-hop questions. The characteristics of questions and the reasoning path are the same as PQL-2H.
• MetaQA-1H [49]: MetaQA-1H contains 116,045 questions for single-hop reasoning QA and the knowledge base in the dataset contains 40,128 entities and nine relations. To test QA systems in more realistic (and more difficult) scenarios, MetaQA-1H also provides neural-translation-model-paraphrased datasets, and text-to-speech-based audio datasets.
• MetaQA-2H [49]: MetaQA-2H contains 148,724 questions for two-hop reasoning and the knowledge base in the dataset contains 40,128 entities and nine relations. MetaQA-2H provides neural-translation-model-paraphrased datasets, and text-to-speech-based audio datasets just like MetaQA-1H.

4.1.2. Metrics and Parameters

We test the effectiveness of the model in four datasets. The total questions in datasets are divided into three parts: 70% for training, 10% for validation and 20% for testing. We evaluate the experiment results via two standard metrics: F1 and Hits@1. The F1 value is an overall evaluation of the precision and recall, which can evaluate the performance of the model well, while Hits@1 measures the proportion of top 1 rankings. The aim of the training is to achieve high F1 and Hits@1.

The reported results are given for the best set of hyper-parameters evaluated on the validation set for each model after grid search on the following values: embedding size ∈ {100, 200, 300, 400, 500}, learning rate ∈ {1, 0.1, 0.01, 0.001}, $\lambda$ and dropout are set to 1 and 0.4.

4.1.3. Baselines

We compare HDH-GCN with the following baselines:

• KVMemNet [50]: This is an end-to-end memory network which divides the memory into two parts, the key memory stores the head entity and relation, and the value memory stores the tail entity.
• IRN [48]: This is an interpretable reasoning network, which uses a hop-by-hop reasoning process and answers questions based on knowledge maps.
• VRN [49]: An end-to-end variational learning algorithm is proposed, which can effectively solve the multi hop reasoning problem and simultaneously deal with the noise in the problem
• GraftNet [12]: Text information and entities are introduced to construct a graph, and GCN is applied to reasoning.
• SGReader [14]: This also combines the unstructured text and knowledge graph to figure out the incompleteness of the knowledge graph. The model employs graph attention to reason effectively.
• 2HR-DR [15]: This models the entities extracted from questions and their related relationships and entities in the knowledge base into directed hypergraphs, then uses Directed Hypergraph Convolutional Networks to predict relations hop-by-hop and form a sequential relation path to make the reasoning interpretable.

4.2. Results of Main Experiment

Table 3. Results of Hit@1 on PQL-2H and PQL-3H.

Model	PQL-2H	PQL-3H
KVMemNet	0.690	0.617
IRN	0.725	0.710
GraftNet	0.707	0.910
SGReader	0.719	0.893
2HR-DR	0.755	0.921
HDH-GCN	0.764	0.933

and

Table 4. Results of Hit@1 and F1 on MetaQA-1H and MetaQA-2H.

Model	MetaQA-1H		MetaQA-2H
	Hit@1	F1	Hit@1	F1
KVMemNet	0.958	-	0.251	-
VRN	0.975	-	0.898	-
GraftNet	0.970	0.910	0.948	0.727
SGReader	0.967	0.960	0.807	0.798
2HR-DR	0.988	0.973	0.937	0.814
HDH-GCN	0.990	0.968	0.951	0.822

show the main experiment results for two kinds of multi-hop KBQA datasets. The highest scores are in bold. As shown in Table 3, we can find out that our proposed HDH-GCN can achieve optimal results under Hit@1 measurement standards (there is only one answer for each question in PQL datasets, so we only adopt Hits@1 for evaluation). For the rest of the datasets, except for the F1 value of HDH-GCN on the MetaQA-1H dataset, which does not exceed the baseline model, other evaluation indexes have been improved to some extent, as shown in Table 4. Specifically, HDH-GCN achieves an improvement for PQL-2H which is 0.9% higher than the second best model. It also obtains good result on PQL-3H, 1.2% higher than the second best one. Table 4 demonstrates the performance of the baseline methods and HDH-GCN on the MetaQA-1H dataset; our model improves Hits@1 by 1.8% and obtains competitive F1. For MetaQA2-Hop, we improve Hits@1 and F1 by 0.3% and 0.8%, respectively.

First of all, compared with models based on a knowledge base modeled on a simple graph, our model reconstructs the knowledge graph to hypergraph structure which fully considers the high-order data correlation. Meanwhile, we dynamically concentrate on relation information at different hops by performing loop operations for each hop of inference to guide the model to follow the golden relation path and select the final answers, so we can introduce the information of the intermediate reasoning path into our model to supervise the model focus on the dynamic relations at different hops.

When comparing with the directed hypergraph-based model 2HR-DR, the improvement of both evaluation values on the two PQL datasets is more obvious than on the other datasets. The reasons why our method performs better include: (1) The directed hypergraph is modeled on hyperbolic space, which effectively reduces the sparsity of the incidence matrix of the directed hypergraph; this will reduce the scale of matrix calculation during training and reduce the inadequacy of training. This also explains why the results on the two PQL datasets are better than on MetaQA-1H and MetaQA-2H, the number of relationships in PQL is much higher than in MetaQA, which leads to more obvious matrix sparsity problems (the number of relations in MetaQA is small, and each relation can relate to many entities). (2) In the convolutional network of a directed hypergraph, there is a computational process of eigenvalue decomposition of the asymmetric Laplacian matrix, but the asymmetric matrix may not be able to carry out the eigenvalue decomposition, which leads to the forced eigenvalue decomposition of the matrix that cannot be diagonalized similarly in the training process, and the training error is always caused. The problem can be solved effectively by deforming the Laplace matrix in hyperbolic space. This problem is evident in the single-answer QA task, while the multi-answer QA task will dilute the influence of this problem to a certain extent when calculating the F1 value, which also explains why the F1 value of HDH-GCN is not improved in MetaQA-1H.

4.3. Parameter Analysis

Embedding size is a significant factor in KBQA models, determining the performance of the model to a large extent. Hence, we will analyze the results obtained by the model on PQL-2H in different embedding sizes to investigate its impact. First, according to the Figure 3a, HDH-GCN outperforms other methods when the dimension is {100, 200, 300, 400}. The Hit@1 of HDH-GCN increases sharply with the early stage of increasing the embedding size and becomes smooth after the embedding size increases to 400. The Hit@1 of 2HR-DR is almost identical to TF-DHP’s from the start; however, because the sparsity issue becomes intense as the dimension increases, it cannot remain smooth like HDH-GCN when the embedding size increases. After the embedding size increases to a certain extent, 2HR-DR’s Hit@1 will decrease slightly. For other methods, since the knowledge base is not modeled on the hypergraph, the sparsity issue has no obvious effect on the training results of higher dimensions; however, due to the reasons mentioned above, its results in each dimension are not as superior as HDH-GCN. We also count the Hit@1 results on PQL-2H of each training session. In the Figure 3b, we compare the Hit@1 between HDH-GCN and 2HR-DR on model training. 2HR-DR is stopped early around 35 epochs because of not updating Hit@1 for 10 epochs, so the line is not complete. HDH-GCN always achieves better performance, and is still updating until around 34 epochs.

4.4. Approximate Training Time Comparison

On the two kinds of datasets PQL and MetaQA, HDH-GCN takes around 75 min and 3 h, respectively, of training time, while 2HR-DR takes around 2 h and 3 h, respectively. All were run on a GeForce GTX 1080 super GPU machine with Python 3.

4.5. Case Study

As Figure 4 shows, we give an exemplar question from PQL-2H and its corresponding reasoning path and triples in KB. It is clear that HDH-GCN has the ability to predict relations hop-by-hop and stop reasoning automatically. For question “Who is the singer of the theme song of the movie “Titanic”?”, the model firstly detects the relation “theme song”, then “singer” successively and finally meets <STOP> to end the reasoning process. From the “path” in Figure 4, we can observe our model’s predicted relation path (Theme song → Singer → <STOP>).

5. Conclusions and Future Work

In this paper, we introduce HDH-GCN, a novel model for multi-hop KBQA tasks. We model the directed hypergraph convolutional network in hyperbolic space, which effectively reduces the influence of the sparsity issue on the model effect. Our model can improve the accuracy of the multi-hop knowledge base question-answering task, and has application value in text question answering, human–computer interactions and other fields. The experimental results verify the advantages of HDH-GCN in both single-answer questions and multi-answer question datasets.

In the future, we will study the multi-hop knowledge base question-answering task in multi-modal data, study the possibility of modeling a multi-modal knowledge base by directed hypergraph and explore the possible application prospect.