Householder Transformation-Based Temporal Knowledge Graph Reasoning

Abstract

Knowledge graphs’ reasoning is of great significance for the further development of artificial intelligence and information retrieval, especially for reasoning over temporal knowledge graphs. The rotation-based method has been shown to be effective at modeling entities and relations on a knowledge graph. However, due to the lack of temporal information representation capability, existing approaches can only model partial relational patterns and they cannot handle temporal combination reasoning. In this regard, we propose HTTR: Householder Transformation-based Temporal knowledge graph Reasoning, which focuses on the characteristics of relations that evolve over time. HTTR first fuses the relation and temporal information in the knowledge graph, then uses the Householder transformation to obtain an orthogonal matrix about the fused information, and finally defines the orthogonal matrix as the rotation of the head-entity to the tail-entity and calculates the similarity between the rotated vector and the vector representation of the tail entity. In addition, we compare three methods for fusing relational and temporal information. We allow other fusion methods to replace the current one as long as the dimensionality satisfies the requirements. We show that HTTR is able to outperform state-of-the-art methods in temporal knowledge graph reasoning tasks and has the ability to learn and infer all of the four relational patterns over time: symmetric reasoning, antisymmetric reasoning, inversion reasoning, and temporal combination reasoning.

1. Introduction

Knowledge graphs (KGs) organize entities and relations in the form of fact triples denoted as $(s,r,o)$, where s and o represent the head and tail entity, respectively, and r is the relation between them, e.g., (Washington, IsCapitalOf, USA). When the facts are associated with a time point or time interval, e.g., (Barack Obama, President of, USA, 2008–2016), they form temporal knowledge graphs (TKGs). TKGs use quadruples $(s,o,r,t)$ to represent valid facts, entities, and relations that will evolve over time. KG reasoning is a very important problem and has been studied extensively. It can infer new facts based on the existing facts in KGs, and complete the KGs, which are also known as KG completion (KGC). Combinatorial reasoning is the most complex and potential situation. Figure 1 gives two examples of combinatorial reasoning. Figure 1a is an example of static KG combination reasoning: given two facts, (David, Wife, Marry) and (Marry, Father, William), we expect to find the fact (David, Father–in–law, William) though reasoning. Figure 1b is an example of TKGs combination reasoning: given two facts, (David, Fall–in–love, Marry, 2013-05) and (Marry, Immigrant, France, 2014-09), for questions (David, Make–a–visit, ?, 2014-10), we expect (David, Make–a–visit, France, 2014-10) to have the highest score.

Most reasoning models are designed for static KGs; the reasoning for TKGs is not yet well understood. RotatE [1] is a typical rotation-based model, which can infer and model various relational patterns. TeRo [2] extends RotatE and applies it to TKGs, but TeRo cannot model temporal combination relations. TeRo defines the temporal evolution of an entity embedding as a rotation of the complex vector space, but does not consider that the embedding of relation will evolve over time. We believe that, in TKGs, the entity exhibits the characteristics of time evolution because the relations connected to it have the characteristics of time evolution. When the relations do not exist, the entity only presents its inherent static characteristics. Therefore, we focus on fusing the temporal information and relations in KGs. OTE [3] and HousE [4] extend the rotation to be high-dimensional, which maintains the ability to model various relations. However, OTE and HousE cannot model the TKGs, and OTE cannot guarantee that the rotation matrix is still an orthogonal matrix after the gradient update; HousE rotates each dimension separately, losing the interactive information between the embedded dimensions. We believe that, in high-dimensional space, rotating the embedded representation as a whole can maximize the retention of interactive information between dimensions.

Inspired by these models, we propose a reasoning model based on the Householder transformation for TKGs. First, we fuse the relation and temporal information in the KGs, then use the Householder transformation to obtain an orthogonal matrix about the fused information, and finally define the orthogonal matrix as the rotation of the head-entity to the tail-entity. We calculate the similarity between the rotated vector and the vector representation of the tail entity. Observing the geometric meaning of matrix multiplication, we can decompose it into two parts: scaling and rotation. The orthogonal matrix mainly changes the direction through rotation, so it can model the symmetry/antisymmetric, inverse, and combination relations. Naturally, we want to construct an orthogonal matrix that can update the gradient and contains temporal and relational information. In this way, it can not only ensure that the rotation matrix is always an orthogonal matrix, but also fuse the temporal information in the TKGs.

Our model overcomes the limitations of existing reasoning models and is able to learn and reason about various temporal relational patterns. Experiments have proven that our model has obtained the most advanced performance. In addition, we compared three methods for fusing relational and temporal information, which proved that our model has a strong generalization ability. The rest of the paper is organized as follows: Section 2 presents related work on reasoning for KGs and TKGs; Section 3 presents preliminary information on Householder transformations; Section 4 presents the HTTR model; Section 5 discusses the details of the experiment and the main conclusions; and Section 6 concludes our work.

The main contributions of this paper are as follows:

• We design a Householder transformation-based reasoning model for TKGs. In contrast to the state-of-the-art static KGE model OTE, which also uses an orthogonal transformation, the orthogonal matrix obtained by the Householder transformation ensures that the orthogonality properties of the matrix can be preserved after the gradient update. Moreover, our approach is to rotate the orthogonal matrix as a whole, which better preserves the effective information. In contrast to the TKG embedding model, HTTR focuses on the features of relations that evolve over time, and thus HTTR can model the temporal combination relation reasoning. The experiments show that our model significantly outperforms the available state-of-the-art methods;
• We defined four temporal relational patterns, temporal symmetric relation, temporal asymmetric relation, temporal inverse relation, and temporal combination relation, and theoretically proved that HTTR can reason over all four temporal relational patterns;
• We propose three methods, $Concatenation$, $Adding$, $Multiplication$, to fuse relational and temporal information, and compare the performance of these fusion methods in detail. At the same time, as long as the dimensions meet the requirements, we allow other fusion methods to replace the current ones, which shows that the model has a strong generalization ability.

2. Related Work

2.1. Static KG Embedding and Completion

Inspired by the translation invariant phenomenon of the word vector, Bordes et al. [5] define the relation as the translation from the head entity to the tail entity and propose the first translation-based model, TransE. Based on TransE, a series of translation-based methods are later proposed, such as TransH [6], TransR [7], and TransD [8]. KG embedding models based on tensor decomposition include DistMult [9], ComplEx [10], etc., which take advantage of tensor factorization to learn the embeddings of entities and relations in KGs. The former restricts the relation matrix to a diagonal matrix, while the latter further introduces complex numbers into the model. Rotation-based KG embeddings have been shown to be an effective method for modeling entities and relations, such as RotatE [1], Rotate3D [11], QuatE [12], OTE [3], HousE [4], etc. RotatE defines each relation as a rotation from the source entity to the target entity in the complex vector space, and proves that rotation is simple and effective for KGC. It can model the combination relation, but it is a static model. Rotate3D and QuatE utilize the quaternion number system, an extension of complex numbers, to embed entities as well as relations and to simulate a 3D rotation in quaternion space. OTE and HousE extend the rotation to a high-dimensional space through orthogonal transformation, and the orthogonal transformation of relation embedding maintains the ability to model various relations. However, OTE cannot guarantee that it is still an orthogonal matrix after the gradient is updated, and all of these methods do not consider the temporal information to be incorporated into the rotation.

2.2. TKG Reasoning and Link Prediction

Based on previous static methods, many TKG reasoning and link prediction methods have been recently proposed. TTransE [13] is the extended version of TransE on TKGs, which learns an independent embedding representation of each entity or relation. HyTE [14] explicitly combines the time information about the entity and relation spaces by associating each time stamp with the corresponding hyperplane. TA-TransE and TA-DistMult [15] regard the combination of r and t in each quadruple $(s,r,o,t)$ as a sequence, which later is taken as an input into the LSTM network to acquire the representation of the combination. DE-SimplE [16] leverages diachronic embedding to extend static methods to TKGs by integrating temporal information to its diachronic embedding. ATISE [17] decomposes a time sequence in a trend component, a seasonal component, and a random component to simulate the temporal evolution in TKGs. TNTComplEx [18] extends static tensor factorization methods to TKGs and expresses the temporal KGE problem as a fourth-order tensor completion problem. TuckERT [19] utilizes the Tucker decomposition of a fourth-order tensor for temporal KGE, and behaves well in several temporal data sets. TeLM [20] uses a linear temporal regularizer and multivector embeddings to decompose TKGs into a fourth-order tensor. TeRo [2] regards the temporal evolution of entity embedding as a rotation in a complex space. TeRo only considers the fusion of the entity embedding representation and time information but not the relation. We believe that, in TKGs, the entity exhibits the characteristics of time evolution because the relations connected to it have the characteristics of time evolution. When the relations do not exist, the entity only presents its inherent static characteristics. RotateQVS [21] represents the temporal entities as rotations in the quaternion vector space and the relations as complex vectors in the Hamilton’s quaternion space. This proves that this method can model key relational patterns in TKGs, such as symmetry, asymmetry, and inverse, and can further capture the time-evolved relations theoretically.

3. Preliminaries

3.1. Householder Transformation

The function of Householder transformation is to perform elementary reflections on any vector $v\in {\mathbb{R}}^n$ on the hyperplane S and the vector v will obtain the vector $v^{{}^{\prime }}$ after Householder transformation; as shown in Figure 2, there is a Householder transformation matrix $\mathbf{H}$ such that $\mathbf{H}v=v^{{}^{\prime }}$, $\omega$ is the normal vector of S. The Householder transformation can transform any vector into multiple times of the modulus length of a coordinate vector, thereby setting other positions of this vector to zero except the coordinate position. For example, for $v={(a_1,a_2,a_3)}^T$, if we select $a_1$ as the transformed principal axis, the coordinate vector is $e_1={(1,0,0)}^T$ and the vector x can be transformed into the form of ${(b_1,0,0)}^T$ through the Householder transformation; that is, there is a Householder transformation matrix H, so that $\mathbf{H}v={(b_1,0,0)}^T=b_1{(1,0,0)}^T$.

Furthermore, if we have a unit vector $\omega \in {\mathbb{R}}^n$, we express $\omega$ in column-block mode, i.e., $\omega ={\left[{\omega }_1,{\omega }_2,\cdots ,{\omega }_n\right]}^T$, so the Householder transformation matrix (or the elementary reflection matrix) can be defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathbf{H}\left(\omega \right)=\mathbf{I}-2\omega {\omega }^T=\left[\begin{array}{cccc}1-2{\omega }_1^2& -2{\omega }_1{\omega }_2& \cdots & -2{\omega }_1{\omega }_n\\ -2{\omega }_2{\omega }_1& 1-2{\omega }_2^2& \cdots & -2{\omega }_2{\omega }_n\\ ⋮& ⋮& \ddots & ⋮\\ -2{\omega }_n{\omega }_1& -2{\omega }_n{\omega }_2& \cdots & 1-2{\omega }_n^2\end{array}\right]\end{array}\end{array}$$

(1)

where ${\omega }^T\omega =1$, and $\mathbf{I}$ is an identity matrix.

3.2. QR Decomposition Based on Householder Transformation

The Householder transformation matrix $\mathbf{H}$ is supposed to satisfy the following properties:

(1)

$\mathbf{H}$ is a symmetric orthogonal matrix, that is, $\mathbf{H}={\mathbf{H}}^T={\mathbf{H}}^{-1}$.

(2)

For any $v\in {\mathbb{R}}^n$, if a vector $v^{{}^{\prime }}$ satisfying $v^{{}^{\prime }}=\mathbf{H}v$, the two-norm value of $v^{{}^{\prime }}$ and that of v will be equal, i.e., $\left|\right|v^{{}^{\prime }}{\left|\right|}_2={\left|\right|v\left|\right|}_2$.

For any matrix $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, we perform a Householder transformation on each column of the matrix in turn, and we can obtain the Householder transformation-based QR decomposition $\mathbf{A}=\mathbf{Q}\mathbf{R}$, where Q is an orthogonal matrix and $\mathbf{R}\in {\mathbb{R}}^{n\times n}$ is the upper-triangular. See Appendix A for a detailed description of the procedure. Each column $r_j$ of $\mathbf{R}$ can be regarded as a set of basis vectors $(e_1,\cdots ,e_j)$, a linear representation of $1\le j\le n$, namely $r_j={\lambda }_1{e}_1+\cdots +{\lambda }_j{e}_j,1\le j\le n.$ The $\mathbf{Q}$ obtained by the QR decomposition is an orthogonal matrix.

The second property of the Householder transformation matrix corresponds to the following theorem:

Theorem 1.

Suppose $v,v^{{}^{\prime }}\in {\mathbb{R}}^n$, $v\ne v^{{}^{\prime }}$, and ${\left|\right|v^{{}^{\prime }}\left|\right|}_2={\left|\right|v\left|\right|}_2$, then there is a Householder reflection matrix H such that $\mathit{H}v=v^{{}^{\prime }}$.

From Theorem 1, we can easily deduce that, for any $v={(v_1,v_2,\cdots ,v_n)}^T\ne 0$, we can always find a Householder reflection matrix $\mathbf{H}$, which makes $\mathbf{H}v=\sigma e_1$, where $\sigma =-sign\left(v_1\right){\left|\right|v\left|\right|}_2$ and $e_1={(1,0,\cdots ,0)}^T$. The basic idea of QR decomposition is to decompose the original target matrix into a normal orthogonal matrix $\mathbf{Q}$ and an upper triangular matrix $\mathbf{R}$, which can simplify the matrix calculations. There are three common methods for QR decomposition: Gram-Schmidt orthogonalization, Householder transformation, and Givens rotation (See Appendix B for more details). Chen et al. [22] pointed out that the classical Gram-Schmidt method is extraordinary sensitive to the errors due to round-off.

Jone R. Rice’s [23] experimental results and Åke Björck’s [24] theoretical analysis indicate that if the matrix A to be decomposed orthogonally is ill-conditioned, using the classical Gram-Schmidt method, the computed columns of Q will lose their orthogonality soon and need to be re-orthogonalized. Householder transformation seems to require less computer memory space because R is stored by overwriting part of A, but Householder transformation algorithms are numerically more accurate and faster than the classical Gram-Schmidt algorithm [22]. Given that this requires more computations compared with the Householder transformation and that multi-step Givens rotation can be replaced by a Householder transformation, Householders have the highest efficiency.

Householder transformations can be left-looking (QR-left), as well as right-looking (QR-right). In the QR-right algorithm, once each Householder reflection matrix is constructed, it operates with A. The QR-left algorithm only applies the Householder reflection matrix to the current k-th column and only one column is calculated at a time, which can be more easily applied to the sparse situation [25]. The QR-left algorithm forms the Householder-based sparse QR decomposition.

We use torch-householder (https://github.com/toshas/torch-householder, accessed on 27 March 2023) repository to implement the Householder transformation algorithm for calculating orthogonal matrices and orthonormal frames. The torch-householder is faster, more precise, and uses less memory than any other solution for orthogonal transformations, bundled in PyTorch (this is true at least for PyTorch 1.9 and NVIDIA P100 16Gb GPU) [26].

4. Our Method

$\mathcal{E}$ denotes the set of entities and $\mathcal{R}$ denotes the set of relations. Usually, we use millions of static triples $(s,r,o)$ to represent KGs, where $r\in \mathcal{R}$ denotes the relation and $s,o\in \mathcal{E}$ are the head entity and the tail entity, respectively. Adding a separate time dimension to previous triples makes the KGs have temporal characteristics, thus representing millions of quadruples $(s,r,o,t)\in {\mathcal{G}}^{+}$, where ${\mathcal{G}}^{+}$ represents the set of all positive quadruples and $t\in \mathcal{T}$ represents the correct time when a fact occurs. We define $e_s,e_r,e_o,e_t\in {\mathbb{R}}^{n\times k}$ as the initial embedding of the head entity, tail entity, relation, and time, respectively.

4.1. OTE

HTTR is inspired by OTE [3]. We first briefly review OTE and then describe the HTTR model in detail. OTE defines the projection from s and r to o as below:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\tilde{e}}_o=f(s,r)=\varphi \left(e_r\right)e_s,\end{array}\end{array}$$

(2)

where $\varphi$ is the Gram Schmidt process applied to the square matrix $e_r$. Then, the distance scoring function in OTE is defined as:

$$d((s,r),o)=\left|\right|{\tilde{e}}_o-e_o\left|\right|.$$

(3)

4.2. HTTR Model

Distinguished from static KGs, TKGs contain temporal information, thus the representation of entities and relations for TKG embedding (TKGE) always change with time. Naturally, we consider fusing the temporal information in TKGs into the embeddings of entities and relations.

Creatively, we fuse the embedding of the relation r and the time steps t in the quadruple in a specific method (the methods of fusion are discussed in Section 5.4, and as long as the dimensions meet the requirements, we allow other fusion methods to replace the current ones), and obtain a fused embedding representation ${\mathbf{A}}_{rt}$. Then, we perform a QR decomposition on ${\mathbf{A}}_{rt}$ based on the Householder transformation to obtain an orthogonal matrix, such as the Formula (4). During the rotation of the orthogonal matrix, the length (norm) of the vector remains unchanged, and only the direction of the vector can be changed.

$${\mathbf{A}}_{rt}={\mathbf{Q}}_{rt}\mathbf{R},$$

(4)

where ${\mathbf{Q}}_{rt}\in {\mathbb{R}}^{n\times n}$ is an orthogonal matrix, which fuses the relation and temporal information, and also satisfies ${\mathbf{Q}}_{rt}{\mathbf{Q}}_{rt}^T={\mathbf{Q}}_{rt}^T{\mathbf{Q}}_{rt}=\mathbf{I}$.

Given a quadruple $(s,r,o,t)$, similar to OTE [3], we define the orthogonal matrix ${\mathbf{Q}}_{rt}$ as the rotation of the head–entity s to the tail–entity o, as shown in Formula (5):

$$\widehat{e_o}=e_s{\mathbf{Q}}_{rt},$$

(5)

• Scoring Function

We use the function $\mathbf{sim}(·,·)$ to measure the similarity between the rotated complex vector $\widehat{e_o}$ and the complex vector representation of the candidate tail entity $e_o$. In our model, we use both cosine similarity (See Appendix C for more details) and inner product to calculate similarity, so the scoring function is calculated as follows:

$$f_{\mathbf{QR}}(s,r,o,t)=\mathbf{sim}(\widehat{e_o},e_o^T)=\mathbf{sim}(e_s{\mathbf{Q}}_{rt},e_o^T).$$

(6)

• Optimization

Similar to the method used in [27], for each quadruple in the training process, we minimize the following instantaneous multi-class log-loss expressed as a cross-entropy loss function:

$$\mathcal{L}\left(f_{\mathbf{QR}}(i,j,k,l)\right)=-f_{\mathbf{QR}}\left(i,j,k,l\right)+log\left(\sum _{k^{{}^{\prime }}}exp\left(f_{\mathbf{QR}}(i,j,k^{{}^{\prime }},l)\right)\right).$$

(7)

where $\left(i,j,k,l\right)$ is a positive sample and $(i,j,k^{{}^{\prime }},l)$ is the kth negative sample. The negative samples are obtained by randomly replacing the correct tail entity in a positive sample with the wrong tail entity. This loss function is only used to train problems similar to $(s,r,?,\tau )$. For a problem of the form $(?,r,o,\tau )$, it can be obtained by reasoning from the corresponding inverse sample $(o,r^{-1},?,\tau )$.

• Regularization

Regularization includes the regularization commonly used in knowledge graph completion and temporal regularization. In this work, we follow the setting of TeLM [20] and use N3 regularization, which has been shown to help improve the performance of (T)KGE models based on tensor factorization [18,27,28,29]. In TKGs, temporal regularization is often used to impose smooth constraints on the temporal representation; that is, the representation of two adjacent timestamps should be close. There are also other approaches. For example, Singer et al. [30] adds a rotation projection when aligning the representations of adjacent timestamps. Yu et al. [31] proposes an autoregressive temporal regularization. Xu et al. [20] proposes a linear temporal regularization method that adds a bias component between adjacent temporal representations. Our framework allows for the replacement of any of these temporal regularization methods, but we only use the most commonly used temporal smoothing constraints, represented by Formula (8) [20]:

$${\mathcal{L}}_t=\sum _{i=1}^{n_t-1}{\left|\right|e_{t_{i+1}}-e_{t_i}\left|\right|}_p^p.$$

(8)

where $n_t$ is the time step and p = 3 since we use N3 regularization. We discuss the effect of regularization on the model performance in detail in Section 5.5, taking ICEWS14 as an example.

4.3. Reasoning across Multi-Relational Patterns

In this section, we demonstrate that our HTTR can reason across multi-relational patterns. Based on previous work [1,11], four kinds of temporal-relational patterns can be defined as follows:

Definition 1.

Temporal symmetric relation: A relation r is symmetric, if $\forall s,o\in \mathcal{E}$, $\forall t\in \mathcal{T}$, $r(s,o,t)\Rightarrow r(o,s,t)$.

Definition 2.

Temporal asymmetric relation: A relation r is asymmetric if $\forall s,o\in \mathcal{E}$, $\forall t\in \mathcal{T}$, $r(s,o,t)\Rightarrow \neg r(o,s,t)$.

Definition 3.

Temporal inverse relation: A relation $r_1$ is the inverse of $r_2$ if $\forall s,o\in \mathcal{E}$, $\forall t\in \mathcal{T}$, $r_1(s,o,t)\Rightarrow r_2(o,s,t)$

Definition 4.

Temporal combination relation: If $\forall s,o,f\in \mathcal{E}$, $\forall t_1,t_2,t_3\in \mathcal{T}$, $r_1(s,o,t_1)\wedge r_2(o,f,t_2)\Rightarrow r_3(s,f,t_3)$, then we define $r_3{t}_3$ as the temporal combination relation of $r_1{t}_1$ and $r_2{t}_2$.

Theorem 2.

HTTR can model all of the temporal-relation symmetric reasoning, asymmetric reasoning, inverse reasoning, and combination reasoning patterns for TKGs.

Proof of Theorem 2.

Since we have Formula (5), the aim of our HTTR is to make $e_o=e_s{\mathbf{Q}}_{rt}$.

For temporal symmetric relation reasoning, we express Definition 1 as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\hfill e_o=e_s{\mathbf{Q}}_{rt}\\ \hfill e_s=e_o{\mathbf{Q}}_{rt}\end{array}\right.\iff {\mathbf{Q}}_{rt}^2=\mathbf{I},\end{array}$$

(9)

where $\mathbf{I}$ is the identity matrix.

For temporal asymmetric relation reasoning, we express Definition 2 as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\hfill e_o=e_s{\mathbf{Q}}_{rt}\\ \hfill e_s=e_o{\mathbf{Q}}_{rt}\end{array}\right.\iff {\mathbf{Q}}_{rt}^2\ne \mathbf{I}.\end{array}$$

(10)

For temporal inverse relation reasoning, we express Definition 3 as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\hfill e_o=e_s{\mathbf{Q}}_{r_{1}t}\\ \hfill e_s=e_o{\mathbf{Q}}_{r_{2}t}\end{array}\right.\iff {\mathbf{Q}}_{r_{1}t}{\mathbf{Q}}_{r_{2}t}=\mathbf{I}.\end{array}$$

(11)

For temporal combination relation reasoning, we express the basic condition in Definition 4 as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\hfill e_o=e_s{\mathbf{Q}}_{r_1{t}_1}\\ \hfill e_f=e_o{\mathbf{Q}}_{r_2{t}_2}\end{array}\right.\iff e_f=e_s{\mathbf{Q}}_{r_1{t}_1}{\mathbf{Q}}_{r_2{t}_2}.\end{array}$$

(12)

Then, if we have $e_f=e_s{\mathbf{Q}}_{r_3{t}_3}$, we obtain $\begin{array}{c}\hfill {\mathbf{Q}}_{r_3{t}_3}={\mathbf{Q}}_{r_1{t}_1}{\mathbf{Q}}_{r_2{t}_2}.\end{array}$ We present a case based analysis of this pattern in Appendix D.

Then, we extend this pattern to multi-hop combination reasoning, e.g., for three-hop combination reasoning we have ${\mathbf{Q}}_{r_4{t}_4}={\mathbf{Q}}_{r_1{t}_1}{\mathbf{Q}}_{r_2{t}_2}{\mathbf{Q}}_{r_3{t}_3}$, which demonstrates that our HTTR has the ability to reason in more complex scenes. □

5. Experiments

5.1. Data Sets

There are four typical TKGs commonly used in previous works: ICEWS [32], GDELT [33], YAGO [34], and WIKI [35]. The time annotations for the first two data sets are time points, such as (Barack Obama, Accuse, North Korea, 20 December 2014), while the latter two are time intervals, such as (Barack Obama, President of, USA, 2008–2016). For time intervals, some are missing the start time or the end time and some time spans are long, while others are short. Prior works such as [2,14,17,18] proposed some methods to discretize the time intervals and we tried them separately in the model verification process; different discretization methods will lead to different results. Therefore, we only evaluate HTTR on ICEWS and GDELT without considering YAGO and WIKI; we will specifically conduct some research on time interval data sets in the future, which is not covered in this work. We use two subsets of ICEWS, ICEWS14 and ICEWS05-15 [15], where the facts correspond to 2014 and 2005–2015, respectively. GDELT (Global Database of Events, Language, and Tone) is the largest, most comprehensive and highest resolution open database ever created by human society. We use a subset of GDELT [16], which corresponds to the facts from 1 April 2015 to 31 March 2016.

We adopt the same method as DE-SimplE [16] to preprocess the training and valididate and test sets to adapt to the TKG completion problem, or the interpolation problem rather than the extrapolation problem. In other words, the time of the fact to be inferred is within the known time span of the data set, not outside the time span.

Table 1. Data set Statistics.

Data Set	Entities	Relations	Time Steps	Train	Valid	Test	Total Triples
ICEWS14	7128	230	365	72,826	8941	8963	90,730
ICEWS05-15	10,488	251	4017	368,962	46,275	46,092	461,329
GDELT	500	20	366	2,735,685	341,961	341,961	3,419,607

shows the statistics of the three data sets. (GDELT is derived from https://github.com/BorealisAI/de-simple/tree/master/datasets/gdelt accessed on 27 March 2023, and ICEWS can be downloaded from https://github.com/soledad921/ATISE accessed on 27 March 2023).

5.2. Experimental Setup

• Hyperparameter

We ran our model in Pytorch and used only one GPU. To seek and find the proper hyperparameters for our HTTR, we utilized a grid search empirically over the following hyperparameters ranges: learning rate in $\{0.001,0.05,0.01,0.1\}$, dropout rate in $\{0.0,0.1,0.2,0.3,0.4,0.5\}$, embedding dimension in $\{36,100,196,324,400\}$. We used $Matrix(·)$ to reshape the fusion of time and relation to obtain a matrix, and rank, which we mention in the following section, refers to the rank of the matrix, e.g., $Matrix(·)$ turns a 400-dimensional vector into a $20\times 20$ matrix. In this case, we say that the rank is 20. Our model was optimized with Adagrad and the batch size was set to 1000 for all three data sets.

• Evaluation Protocol

Our reasoning task is similar to link prediction, and its goal is to complete a time-wise fact with a missing entity under the time-wise filtered settings [2,16]. We use four standard evaluation metrics to evaluate the effectiveness of the model: Hits@1, Hits@3, and Hits@10 (the proportion of correct triples ranked in the top 1, 3, and 10) and Mean Reciprocal Ranking (MRR). For all metrics, a higher value was better. We reported on the average results of five runs for all experiments and ignored the variance, which was usually very low.

• Baselines

We compared our model with several TKG reasoning methods, including TTransE [13], HyTE [14], TA-DistMult [15], DE-SimplE [16], ATiSE [17], TNTComplEx [18], TuckERT [19], TeRo [2], TeLM [20], and RotateQVS [21]. For ICEWS14 and ICEWS05-15, we reported the results of ATISE, TeRo, TuckERT, TNTComplEx, and RotateQVS from the original papers, which used the same evaluation protocol as ours, and other results for some of the baseline from [17]. For GDELT, the results of the baseline came from TuckERT. ChronoR [36] was also inspired by the success of rotation-based models in static KG completion [1,12], which considers general linear transformations on the k-dimensional real space composed of rotation and scaling, and parametrizes the transformation through relation and time. However, there is no openly available source code of ChronoR, so we cannot learn more experimental details for a detailed comparison with our model, and ChronoR was not chosen as the baseline. TeMP [37] is also a recent work on TKGs. The model performs well on ICEWS, a data set with sparse temporal information, while it has no advantage on GDELT, a data set with rich temporal information. We think this is because RGCN can better learn the representation of static KGs, and the results of ablation experiments are not provided in this paper, that is, replacing the embedding input to the temporal processing module with randomly initialized embedding, which is the same as the baseline model we selected.

5.3. Main Results

In this section, we analyze and quantitatively compare our model with previous state-of-the-art models.

Table 2. Results of the experiments. H@N means Hits@1/3/10 and all results are presented as percentages. The best results are written in bold. Dashes: results are not reported in the corresponding literature.

	ICEWS14				ICEWS05-15				GDELT
	MRR	Hits@10	Hits@3	Hits@1	MRR	Hits@10	Hits@3	Hits@1	MRR	Hits@10	Hits@3	Hits@1
TTransE	25.5	60.1	-	7.4	27.1	61.6	-	8.4	11.5	31.8	16.0	0.0
HyTE	29.7	65.5	41.6	10.8	31.6	68.1	44.5	11.6	11.8	32.6	16.5	0.0
TA-DistMult	47.7	68.6	-	36.3	47.4	72.8	-	34.6	20.6	36.5	21.9	12.4
DE-Simple	52.6	72.5	59.2	41.8	51.3	74.8	57.8	39.2	23.0	40.3	24.8	14.1
ATiSE	55.0	75.0	62.9	43.6	51.9	79.4	60.6	37.8	-	-	-	-
TNTComplEx	56.0	74.0	61.0	46.0	60.0	78.0	65.0	50.0	22.4	38.1	23.9	14.4
TuckERT	59.4	73.1	64.0	51.8	62.7	76.9	67.4	55.0	41.1	61.4	45.3	31.0
TeRo ⋆	56.2	73.2	62.1	46.8	58.6	79.5	66.8	46.9	24.5	42.0	26.4	15.4
TeRo-Large	53.4	72.2	59.6	43.2	53.4	80.0	62.7	39.5	25.6	43.7	27.8	16.3
TeLM	62.5	77.4	67.3	54.5	67.8	82.3	72.8	59.9	38.2	54.6	41.0	29.8
TeLM-Small ⋆	59.1	74.5	63.9	50.7	62.6	77.4	67.1	54.7	28.5	45.2	30.7	19.8
RotateQVS	59.1	75.4	64.2	50.7	63.3	81.3	70.9	52.9	27.0	45.8	29.3	17.5
RotateQVS-Small ⋆	57.5	73.7	62.5	48.9	59.1	80.2	68.5	47.3	25.9	42.8	27.0	16.5
HTTR	61.6	77.0	66.6	54.4	66.3	81.5	71.7	58.1	53.1	66.8	56.8	45.6

shows the results of KG reasoning on the three data sets.

For our HTTR, we find that it consistently outperforms all baselines on all metrics over all data sets, except Hits@10 on the ICEWS14 data sets. Especially, as shown in Table 2, compared with the recent TuckERT work, the performance improvement of HTTR on GDELT is very significant, with an increase of 14.6 points, 11.5 points, 5.4 points, and 12.0 points on Hits@1, Hits@3, Hits@10, and MRR, respectively. The improvement of the model in ICEWS05-15 is also more significant than that of ICEWS14. Our model fuses the relation with temporal information so that the characteristic representation of the relations can be better learned and the performance of the model can be effectively improved. We believe that there are two reasons to interpret the above results. First, the ratios of the number of time steps $\left|\mathcal{T}\right|$ to the number of relations $\left|\mathcal{R}\right|$ for GDELT and ICEWS05-15 are 18.3 and 16.0, respectively, while the ratio for ICEWS14 is 1.6; the higher the ratio, the greater the impact of temporal information on the relation. Second, the time steps in GEDLT are very dense. GDELT contains a large number of global news facts in two years, which shows that the facts in GDELT are updated very quickly so GDELT has a stronger temporal property than ICEWS14 and ICEWS05-15. Comparing the two ICEWS data sets, the number of relations between them is very similar, but ICEWS05-15 has more entities than ICEWS14. The most noteworthy difference is that ICEWS05-15 has 4017 time steps, while ICEWS14 has only 365. This further demonstrates that our model can well handle TKG reasoning with rich temporal information.

TeLM and RotateQVS are two recent works on TKGs. RotateQVS’s quaternion representation ($\mathbf{a}+\mathbf{b}i+\mathbf{c}j+\mathbf{d}k$) doubles the embedding parameters of TeRo, which uses complex representation ($\mathbf{a}+\mathbf{b}i$), and TeLM uses two-grade multivectors, which can be written as ($\mathbf{M}=\mathbf{a}+\mathbf{b}{e}_1+\mathbf{c}{e}_2+\mathbf{d}{e}_1{e}_2$); more details are in [20]. For a fairer comparison, the following models are extended: (i) TeLM-Small, TeLM with the embedding dimension as our HTTR; (ii) RotateQVS-Small, RotateQVS with the embedding dimension as our HTTR. We obtained the code of TeLM through open channels (https://github.com/soledad921/TeLM, accessed on 27 March 2023), changed the rank from 2000 to 200, and obtained the TeLM-Small results. The results of RotateQVS-Small are from the original paper. The rows with ^⋆ in Table 2 show the results of similar settings for the initial embedding dimension. Comparing HTTR with these results, our model has obvious advantages in all of the metrics of the three data sets. Compared with TeRo and TeRo-Large, when the dimension was simply changed from 200 to 2000, the model performance did not improve, but decreased. This shows that adding dimensions does not necessarily lead to a performance improvement. For TeLM-Small and RotateQVS-Small, the performance of the model decreased significantly after reducing the initial embedding dimension. This is the same as the conclusion obtained in our experiment, with more details provided in Section 5.6. Increasing the initial embedding dimension within a certain range can effectively enhance the model performance.

5.4. Comparative Analysis of the Fusion Methods of Time and Relation

This section mainly discusses three methods for fusing $e_r$ and $e_t$ to obtain ${\mathbf{A}}_{rt}$ as mentioned in Section 4.2. We first represent time, entities, and relations in the same embedding dimension, and then define $\mathsf{\Phi }\left(t\right)$ to encode the time steps considering the importance of temporal information to the model. $\mathsf{\Phi }(·)$ captures the temporal dependencies between entities based on Bochner’s theorem and Mercer’s theorem [38]. For more details, refer to [38,39,40]. The embedding of time and relation can be performed through Concatenation (Formula (13)), Adding (Formula (14)), and Multiplication (Formula (15)). Here, we only explore the combination of these three ways, but other explorations can be taken further, such as some existing fusion methods used in TA-DistMult [15].

• Concatenation

In the concatenation fusion mode, for each relation $r\in \mathcal{R}$ and each time step $t\in \mathcal{T}$, we design a way of integration $r\left|\right|t$. For example, the temporal fact (Barack Obama, Make a visit, New Zealand, 2014-06-20) encodes as (Barack Obama, Make a visit $\left|\right|$ 2014-06-20, New Zealand). When both the initial embedding dimensions of $e_r$ and $e_t$ are set to 200, through concatenation, a 400-dimensional vector is obtained, and then $Matrix(·)$ turns it into a $20\times 20$ matrix. For each relation $r\in \mathcal{R}$ and each time step $t\in \mathcal{T}$, suppose $e_r={[v_r^1,v_r^2,\cdots ,v_r^n]}^T$, $e_t={[v_t^1,v_t^2,\cdots ,v_t^n]}^T$, by concatenation, we obtain $e_r\left|\right|e_t={[v_r^1,v_r^2,\cdots ,v_r^n,v_t^1,v_t^2,\cdots ,v_t^n]}^T$.

$${\mathbf{A}}_{rt}=Matrix(e_r\left|\right|e_t)$$

(13)

• Adding

Adding, that is point-wise addition, for this fusion method in order to obtain a $20\times 20$ matrix, the initial embedding dimension of $e_r$ and $e_t$ is set to 400. For each relation $r\in \mathcal{R}$ and each time step $t\in \mathcal{T}$, suppose $e_r={[v_r^1,v_r^2,\cdots ,v_r^n]}^T$, $e_t={[v_t^1,v_t^2,\cdots ,v_t^n]}^T$, by adding fusion, we obtain $e_r+e_t={[(v_r^1+v_t^1),(v_r^2+v_t^2),\cdots ,(v_r^n+v_t^n)]}^T$.

$${\mathbf{A}}_{rt}=Matrix(e_r+e_t)$$

(14)

• Multiplication

For the multiplication fusion method, a time step is used as a coefficient affecting the relation embedding, and the initial embedding dimension of $e_r$ and $e_t$ is also set to 400 to obtain a $20\times 20$ matrix. For each relation $r\in \mathcal{R}$ and each time step $t\in \mathcal{T}$, suppose $e_r={[v_r^1,v_r^2,\cdots ,v_r^n]}^T$, $e_t={[v_t^1,v_t^2,\cdots ,v_t^n]}^T$, by multiplication fusion, we obtain $e_r\ast e_t={[(v_r^1\ast v_t^1),(v_r^2\ast v_t^2),\cdots ,(v_r^n\ast v_t^n)]}^T$.

$${\mathbf{A}}_{rt}=Matrix(e_r\ast e_t)$$

(15)

Comparative Experiment Results

Table 3. Comparative results of three methods for fusing relational and temporal information. The metrics are expressed as percentages. The best results are shown in bold.

Data Sets	${\mathit{A}}_{\mathbf{rt}}$	MRR	Hits@10	Hits@3	Hits@1
	$Matrix\left(e_r\right|\left|e_t\right)$	61.6	77.0	66.6	54.4
ICEWS14	$Matrix(e_r+e_t)$	59.4	74.8	65.3	51.2
	$Matrix(e_r\ast e_t)$	58.0	73.2	63.6	49.2
	$Matrix\left(e_r\right|\left|e_t\right)$	66.3	81.5	71.7	58.1
ICEWS05-15	$Matrix(e_r+e_t)$	64.6	81.3	71.0	55.3
	$Matrix(e_r\ast e_t)$	63.0	79.6	70.7	53.5
	$Matrix\left(e_r\right|\left|e_t\right)$	53.1	66.8	56.8	45.6
GDELT	$Matrix(e_r+e_t)$	52.1	64.4	55.2	45.3
	$Matrix(e_r\ast e_t)$	51.0	64.0	54.6	43.7

shows the results of the three methods for fusing relational and temporal information on three data sets. The results on all three data sets show that in HTTR, the way of Concatenation is significantly better than that of Adding and Multiplication. In fact, Concatenation is just an operation to connect features. It does not play the role of feature fusion. It allows the network to learn how to fuse features without losing information in the process. The real feature fusion happens in the Householder transformation immediately following. Through the Adding and Multiplication, new features will be obtained and these new features can reflect some characteristics of the original features, but some information about the original features will also be lost in the process. We can think of point-wise addition as a special form of concatenating, but the computation of addition is much smaller than that of concatenation, which saves parameters and computation. When computing resources are limited, Adding is the second best choice.

5.5. Ablation Study

• w. HT vs. w/o HT

To verify the effect of Householder transformation, we design a variant of HTTR that removes the Householder transformation module from the model, and conduct further experiments on ICEWS14, using the validation set every 10 epochs. The model tends to converge at 150 epochs. In order to express the comparison more intuitively and clearly, Figure 3 only shows the results of the 60–150 epochs. The results show that the Householder transformation module significantly improves the performance of the model and that the improvement is basically consistent in all four metrics, indicating that HTTR can model temporal information well and is stable.

• Regularization

N3 regularization has been shown to help improve the performance of tensor factorization-based (Temporal) Knowledge Graph embedding models, often referred to as embedding regularization [18,27,28]. In general, it is assumed that the embedding representations of adjacent time steps should be as close as possible. Therefore, in the work of TKGs, it is common practice to use time as a regularization term to impose smooth constraints on the temporal embedding, also commonly referred to as temporal regularization [18,20,29]. Lacroix et al. [18] studied the impact of regularization on ICEWS05-15, finding that adding embedding regularization and temporal regularization to TNTComplEx can bring an improvement of 5% on MRR. Similar to TNTComplEx and TeLM, we carefully added N3 regularization and temporal regularization to HTTR, and verified the effect of regularization on HTTR on ICEWS14 and ICEWS05-15. At the same time, we obtained the source code of TNTComplEx and TeLM on Github, set the regularization parameter to 0, and obtained the result without regularization (w/o reg.). For TNTComplEx, on ICEWS05-15, the reproduced MRR results were consistent with those reported in the original paper. From

Table 4. Effect of regularization.

	ICEWS14				ICEWS05-15
	MRR	Hits@10	Hits@3	Hits@1	MRR	Hits@10	Hits@3	Hits@1
TNTComplEx	56.0	74.0	61.0	46.0	60.0	78.0	65.0	50.0
TNTComplEx (w/o reg.)	49.6	65.4	53.4	41.1	54.9	73.7	60.0	45.9
improvement	−6.4	−8.6	−7.6	−4.9	−5.1	−4.3	−5.0	−4.1
TeLM	62.5	77.4	67.3	54.5	67.8	82.3	72.8	59.9
TeLM (w/o reg.)	54.3	67.6	57.6	47.2	58.8	72.6	62.4	51.5
improvement	−8.2	−9.8	−9.7	−7.3	−9.0	−9.7	−10.4	−8.4
HTTR	61.6	77.0	66.6	54.4	66.3	81.5	71.7	58.1
HTTR (w/o reg.)	61.8	76.4	67.0	53.6	66.1	81.8	72.0	58.1
improvement	+0.2	−0.6	+0.4	-0.8	−0.2	+0.3	+0.3	0

, there are two main conclusions. First, adding regularization can improve the model performance. Second is the case that our method significantly outperforms the other two methods without considering regularization. The effect of regularization on HTTR is almost negligible, while for the other two methods, removing regularization greatly reduces the performance of the model. Among them, the performance of the TNTComplEx drops by 4–9% and TeLM drops by 7–11%, so the model is prone to overfitting. This further shows that our model can model temporal information well.

5.6. Effect of Rotation Dimension

In order to verify the effect of the rotation dimension on model performance, we conducted a detailed analysis on ICEWS14 (Figure 4). We set the training epochs to 300, verified it every 60 epochs, and the other parameters were the same as the previous main experimental settings. Then, we set the rank to 6, 10, 16, 18, and 20, respectively for verification. On ICEWS14, HTTR tended to converge after 100 epochs of training, so the broken lines in the chart were relatively stable. When the rank was set to 16 and 18, the two lines were very close, so we omitted the case of rank18 from the figure. On the whole, with the increase in rank, the performance of the model improved, and the improvement amplitude of the four metrics was similar. When the rank increased from 6 to 10, the performance of the model improved. When the rank continued to increase to 16, the performance coulld still be greatly improved. However, when the rank further increased from 16 to 18 and 20, the training time improved greatly, but the improvement was not obvious. This indicates that, within a certain range, the model performance could be improved by increasing the rotation dimension.

6. Conclusions

In this work, we propose a novel TKG reasoning model HTTR. We introduce three methods to fuse relational and temporal information, then use the QR decomposition based on Householder transformation to obtain an orthogonal matrix containing relation and temporal information. We define the orthogonal matrix as the rotation of the head-entity to the tail-entity, and calculate the similarity between the rotated vector and the representation of the tail entity to discover new facts. The experimental results on reasoning show that our model can outperform all of the baselines over all of three commonly used benchmark data sets. In addition, HTTR can effectively model the temporal combination reasoning of TKGs. Our model mainly improves reasoning performance by better obtaining knowledge representation so our method can be extended to more complex temporal reasoning tasks, which will be reflected in future work.