ProvGRP: A Context-Aware Provenance Graph Reduction and Partition Approach for Facilitating Attack Investigation

Abstract

Attack investigation is a crucial technique in proactively defending against sophisticated attacks. Its purpose is to identify attack entry points and previously unknown attack traces through comprehensive analysis of audit data. However, a major challenge arises from the vast and redundant nature of audit logs, making attack investigation difficult and prohibitively expensive. To address this challenge, various technologies have been proposed to reduce audit data, facilitating efficient analysis. However, most of these techniques rely on defined templates without considering the rich context information of events. Moreover, these methods fail to remove false dependencies caused by the coarse-grained nature of logs. To address these limitations, this paper proposes a context-aware provenance graph reduction and partition approach for facilitating attack investigation named ProvGRP. Specifically, three features are proposed to determine whether system events are the same behavior from multiple dimensions. Based on the insight that information paths belonging to the same high-level behavior share similar information flow patterns, ProvGRP generates information paths containing context, and identifies and merges paths that share similar flow patterns. Experimental results show that ProvGRP can efficiently reduce provenance graphs with minimal loss of crucial information, thereby facilitating attack investigation in terms of runtime and results.

1. Introduction

Enterprises face threats from covert and persistent multi-step attacks, such as Advanced Persistent Threats (APT). These sophisticated attacks employ complex strategies and state-of-the-art techniques to infiltrate target systems and remain lurking for months, stealthily gathering confidential information. To counter these attacks, attack investigation comprehensively analyzes audit logs to identify attack entry points and previously unknown attack traces through comprehensive analysis [1,2,3,4]. Audit logs are generated by monitoring system activity using kernel-level audit log collection tools such as Auditd, Sysmon, and Dtrace. These logs record system events and status, capturing information about system entities (e.g., processes, files, and sockets), dependency relationships between entities (e.g., process-file interactions), event timestamps, and other system information.

Attack investigation based on audit logs often employs provenance analysis, which constructs provenance graphs from audit logs, where nodes represent system entities and directed edges represent dependency relationships. Analyzing these provenance graphs enables investigation of attack behaviors and reconstruction of attack scenarios. Several works have demonstrated the effectiveness of utilizing provenance analysis in attack investigation [4,5,6,7] and detection [8,9,10]. However, it is complex and costly to analyze audit logs. To effectively detect Advanced Persistent Threat (APT) attacks with long durations and cover the complete attack process, it is necessary to collect audit logs over large time spans through system monitoring, resulting in a significant volume of audit data. Due to the dependency explosion problem [11,12], the size of the graphs generated from these audit logs is complex and huge, typically containing 200,000 edges. This complexity makes it challenging for attack investigation approaches to identify attack-related edges and entities within the graphs.

To address this challenge, recent research has proposed methods for provenance graph reduction and compression [12,13,14]. These approaches are based on the observation that audit logs record a large number of system call events that are redundant and not strictly necessary for attack investigation. Therefore, these methods decrease the size of the provenance graphs by eliminating redundant and irrelevant system events while preserving crucial attack-related information. CPR/PCAR [13] merges repeated low-level events between two OS objects, in order to extract a high-level abstraction of system activities. NodeMerge [12] and LogApprox [15] identify redundant events using predefined templates. For instance, they merge events that involve a process reading or writing to multiple similar files in a short period. These reduction techniques are compared and illustrated in Figure 1.

Such reduction techniques have demonstrated promising outcomes in enhancing the efficiency of attack investigation. However, there are two major limitations. (1) Audit logs record coarse-grained system operations, which can lead to unrelated dependencies among long-running processes. The main reason for this is that some processes have a long lifetime and iterative input/output processes. In Case 1 of Figure 1, the nodes and edges in gray have no relationship with the black nodes in high-level behavior, but are associated to the same graph. Existing methods fail to partition the two behaviors in the graph. (2) Existing methods cannot identify redundant patterns that have not been defined by schemes and templates. Furthermore, they do not consider the information flow patterns implicit in the deep causal relationships and contextual information of system events. Although the existing techniques can effectively reduce the graph in Case 1, they cannot deal with the redundant events in Case 2 of Figure 1 because they do not consider the context information. Indeed, Case 2 can still be reduced without loss of critical information, and the reduction result is shown on the right of Figure 1.

To address the above limitations, this paper proposes ProvGRP, a novel context-aware provenance graph reduction and partition approach. ProvGRP partitions provenance graphs into subgraphs describing different behaviors and eliminates redundant and behavior-unrelated events, thereby facilitating the efficiency of attack investigation. Firstly, ProvGRP constructs platform-independent provenance graphs by parsing audit logs. Subsequently, it proposes three features to calculate dependency similarity and utilizes clustering technique HDBSCAN [16] to cluster events with high similarity into an execution partition. The above step is implemented to partition a provenance graph into multiple subgraphs to eliminate erroneous dependencies due to the coarse-grained nature of logs. A key insight of merging redundant events based on context information is that information paths belonging to the same high-level behavior share similar information flow patterns. Therefore, ProvGRP obtains context information through forward and backward tracing based on information tributaries, and generates information paths that effectively represent information flow patterns by identifying information paths with similar flow patterns and merging them to reduce redundant system events. Specifically, the information path representation is naturally fit with sequence distance calculation methods. Thus, an improved Levenshtein edit distance is designed to calculate the distance between two information paths. Finally, ProvGRP merges the information paths clustered into one class to achieve the reduction of the provenance graphs.

ProvGRP is evaluated on the publicly available datasets DAPRA TC and ATLAS, which contain labeled attack entities and events. We employ four metrics to conduct a comprehensive and quantitative analysis of the method’s effectiveness. Specifically, the reduction factor evaluates the efficiency of the reduction, while Behavior Partition Accuracy evaluates the accuracy of the graph partitioning. Attack Information Loss and Causal Information Loss quantify the information loss situation caused by the reduction method. The experimental results show that the reduction factors achieved by ProvGRP on these two datasets are 10.10X and 42.21X, respectively. Taking into account both the reduction factor and information loss, ProvGRP achieves further data reduction with losing much less information than previous methods. The result of Behavior Partition Accuracy shows that ProvGRP can accurately partition the provenance graph to remove false dependencies. Furthermore, the evaluation results demonstrate that the use of ProvGRP-reduced graphs can facilitate the efficiency of attack investigation approaches while maintaining high performance.

2. Background and Motivation

2.1. Audit Logs and Provenance Analysis

Audit Logs: Audit logs are collected by system monitoring tools from various operating systems, which record in detail the actions and status of the system layer. Each audit log is an encapsulation of a specific system event or system call, containing details about system objects, relationships between objects, timestamps, event IDs, and other requisite system information. Audit logs capture three types of system entities: Process, File, and Network. The types of relationships are various according to different object operations. (1) The relationships between recorded processes include process execution, fork, and close, etc. (2) The relationship between processes and files records the operation from processes, such as read, write, and delete. (3) The relationships recorded in the logs of the network object include link, close, etc. Thereby, an audit log can be represented as a quadruple <event= Subject, operation, Object, timestamp>. For instance, the quadruple <WORD.EXE, write,Desktop\my_report.docx, 2023/4/22 9:31:32> represents the process, $WORD.EXE$ writes a file named $my\_report.docx$ in the path $Desktop\backslash$, and the corresponding high-level user behavior saves the $my\_report.docx$ file on the Desktop.

Provenance Analysis: Provenance analysis has been widely applied to attack investigation and detection. Provenance analysis analyzes the audit logs to infer the dependencies and construct a directed labeled graph known as the provenance graph. By performing comprehensive causality analysis on the graphs, provenance analysis can identify attack behavior patterns and traces. In fact, the causal relationships and contextual data found within audit logs provide valuable insights into the tactics and goals of attackers, which are inherently difficult to hide. In the provenance graph $G=(V_{Entity},E_{dep})$, a node $v\in V_{Entity}$ represents a system entity (such as a process, a file, or an IP address). An edge $e\in E_{dep}$ represents a dependency relationship (such as process write\read file) with its direction indicating the relationship between two entities.

2.2. Motivating Example

Scenario. Consider the simulated APT attack scenario implemented by an APT group named Kimsuky [17]. This scenario encompasses both malicious activities and regular user actions. For instance, a user habitually checks emails and downloads attachments. Among these emails, there is a phishing attempt containing a .zip file carrying a malicious .scr file. It writes a DLL file and sets the registry key to establish persistence. Subsequently, the DLL file utilizes the technique of process hollowing to inject malware code into explorer.exe to avoid Anti-Virus detection. Finally, the attacker transmits encrypted information concerning the compromised machine. Concurrently, the user performs activities such as downloading documents, working with data samples, installing Python, and executing Python codes, etc. Figure 2 illustrates a simplified provenance graph depicting the aforementioned process. The gray boxes and lines in Figure 2 indicate the key nodes and steps of the attack.

CPR/PCAR [13] merges repeated low-level events between two objects, in order to extract a high-level abstraction of system activities. CPR/PCAP can merge the python.exe writing and deleting pyc files in the graph, and the reduced graph is still noticeably complex. NodeMerge [12] only merges a set of read-only files that are always accessed together in the system events. The system event of the encryptor.exe process reading 241 files in the graph can be merged to one node by NodeMerge. LogApprox [15] solely merges similar files read and written by a process, employing regular expression learning to assess their similarity. Compared to NodeMerge, it can achieve better reduction results because it can merge python.exe to write and delete all pyc files into a single event. However, these methods can only reduce redundant events for specific schemes and templates, and cannot effectively deal with undefined redundant patterns. More importantly, not considering the context information limits these methods from further identifying multi-step redundant events.

It can be seen from Figure 2 that chrome.exe is a long-running process, and it associates multiple high-level behaviors related to it within the same graph. For example, there is no causal relationship between the behavior of download mail attachment <chrome.exe write *.zip, T0.2> and the behavior of downloading installation packages <chrome.exe write *.exe, T3.2> during the lifetime of the process. This makes it difficult to determine the correct information flow path. However, some previous work did not consider this situation. Existing methods do not solve this problem.

Using ProvGRP. ProvGRP first partitions the original provenance graphs into multiple subgraphs, thereby removing false dependencies between different high-level behaviors. Here, ProvGRP partitions chorme.exe into multiple execution partitions, so that high-level behaviors such as downloading malware, downloading code, and downloading python packages are separated in different subgraphs. The error dependencies between attack event <chrome.exe write *.zip, T0.2> and other normal events are removed correctly. Then, ProvGRP constructs information paths based on context information, and realizes the identification of redundant events by calculating the similarity between these information paths. System events describing the same behavior in Figure 2 are accurately and efficiently merged. For example, the system events that python.exe generates and deletes a large number of pyc files are merged into a single event <python.exe read py file, T4.2> to represent the high-level behavior of running the python code.

3. Approach Overview and Threat Model

3.1. ProvGRP Overview

The overall workflow of ProvGRP is shown in Figure 3. ProvGRP acts as the predecessor work of attack investigations to reduce provenance graphs. It takes the provenance graphs generated by the audit logs from different operating systems as input and outputs the reduced provenance graphs for attack investigation. ProvGRP consists of three components: (1) Provenance Graph Construction, (2) Provenance Graph Partition, (3) Behavior-unrelated Events Elimination. The Provenance Graph Construction component constructs platform-independent provenance graphs from collected audit logs by performing causality correlation (Section 4.1). The Provenance Graph Partition component implements the partitioning of long-running process nodes by event features (Section 4.2). The Behavior-unrelated Event Elimination component generates information paths and merges similar paths to reduce the behavior-unrelated events (Section 4.3).

3.2. Threat Model

Our threat model is similar to the threat model of previous work [7,8,18,19,20], which assumes that the underlying operating system, audit engine, and monitoring data are part of the trusted computing base (TCB). Kernel-level attacks that compromise the audit monitoring and log collection systems are beyond the scope of this work. This paper also assumes that the integrity of audit logs is maintained, meaning that attackers cannot modify or delete them. Additionally, the usage of existing software and secure provenance systems can ensure the secure capture and storage of logs.

4. Approach Design

4.1. Provenance Graph Construction

System monitoring tools are utilized to collect audit logs from various operating systems, such as Windows, Linux, and Unix. ProvGRP then parses the collected logs to construct platform-independent provenance graphs, which are directed labeled graphs. The nodes represent system entities, and the directed edges represent system events. Specifically, ProvGRP extracts information about system events recorded in audit logs including node attributes and dependency attributes. ProvGPR focuses on three types of node entities, process, file, and network. The dependency attributes include operations and timestamps, and ProvGRP uses a quadruple to represent system events $<Subject,operation,Object,timestamp>$.

Table 1. Attributes of extracted node attributes and dependency attributes.

Subject Entity		Object Entity		Denpendency (Edge)
Type	Attributes	Type	Attributes	Operations	Attributes
Process	Name, PID, User, Type, Subject ID	Process	Name, PID, User, Type, Object ID	[execute, fork, clone, close]	Timestamp, Cmd lines, Event ID
		File	Name, Path, Type, Object ID	[read, write, delete]	Timestamp, Event ID
		Network	IP, Port, URL, Protocol, Object ID	[connect, connected session, sock send]	Timestamp, Evnet ID

displays the complete extraction information of system events. ProvGRP performs causality correlation on the extracted data to construct provenance graphs. The node entities include the file name (c:/windows/system32/es.dll), process name and PID (svchost.exe_896), and IP address (192.168.223.2:53). The labels of the directed edges include the operations (read, execute, and connect, etc.) and the timestamp. Finally, ProvGPR eliminates unknown nodes and isolated nodes which are the noise produced by the system monitoring.

4.2. Provenance Graph Partition

The goal of this component is to partition the provenance graphs into multiple subgraphs to remove false dependencies caused by the coarse-grained nature of logs. The subgraphs depict different high-level behaviors, which have no relationship with each other. To achieve this, ProvGRP proposes a new provenance graph partition algorithm. This algorithm abstracts three behavior features to judge which system events belong to the same high-level behavior. Based on these features, incoming and outgoing edges are clustered separately, and each class cluster corresponds to an execution partition. Subsequently, ProvGRP merges the execution partitions of the incoming edges and the execution partitions of the outgoing edges based on the time of the events in the clusters. Through the above steps, the original provenance graph is partitioned into multiple subgraphs.

4.2.1. Feature Definition

ProvGRP extracts three features to calculate the similarity between dependencies. These three features can comprehensively quantify the similarity between dependencies from multiple dimensions.

Time Interval Feature $f_{T\left(e\right)}$. Intuitively, dependencies belonging to the same behavior have much smaller time intervals compared to those belonging to different behaviors. To validate this intuition, we conducted a statistical analysis of the time intervals between system events belonging to the same behavior in two publicly available audit log datasets (the results are depicted in Section 5.2). Thus, the feature $f_{T\left(e\right)}$ is designed to model this intuition.

$$f_T(e_i,e_j)=2\times atan(\frac{T_{end}{-T}_{start}}{|t_{e_i}-t_{e_j}|+\alpha }-\beta )/\pi$$

(1)

where $t_{e_i}$ and $t_{e_j}$ represent the timestamp of the events (edges) $e_i$ and $e_j$, which are either incoming or outcoming from the same node. $T_{end}$ is the latest timestamp among the edges, and $T_{start}$ is the start timestamp. $atan\left(t\right)={tan}^{-1}\left(t\right)$ is an arctangent function, and $\alpha$ (we set $\alpha =0.001$) is a positive number used to make sure the denominator is not 0. When two events occur at the same time, the value of $f_T(e_i,e_j)$ is close to 1. $\beta =\frac{T_{end}{-T}_{start}}{T_{end}{-T}_{start}+\alpha }$ to ensure the value of $f_T(e_i,e_j)$ is 0 when the time interval between events is maximum. The value of the formula ranges from 0 to 1.

Entity Name Feature $f_{E\left(e\right)}$. The subjects or objects of system events that belong to the same behavior have high similarity. This intuition is based on observing the habits of users and computers. For example, when unzipping a zip file, the files are typically extracted to the same directory path. Then, the feature $f_{E\left(e\right)}$ is designed to model this intuition. Depending on the type of comparison nodes, $f_{E\left(e\right)}$ uses different formulas.

If the two entity nodes are different types, $f_{E\left(e\right)}=0$. If the type of two entity nodes is process, the value of $f_{E\left(e\right)}$ is calculated by process name and PID, and

$$f_E(n_{e_i},n_{e_j})=\left\{\begin{array}{c}1if{n}_{e_i}=n_{e_j}\\ 0otherwise\end{array}\right.$$

(2)

where $n_{e_i}$ and $n_{e_j}$ are entity node names of the events (edges) $e_i$ and $e_j$, and $n_e$ is represented as process_PID. If the type of two entity nodes is IP address, the value of $f_{E\left(e\right)}$ is calculated by counting the number of the same initial bits of IP address, and

$$f_E(n_{e_i},n_{e_j})=same\_bit\left(binary\right(n_{e_i}),binary(n_{e_j}\left)\right)/32$$

(3)

where $binary\left(n_e\right)$ converts the decimal IP address to binary, and $same\_bit$ counts the number of the same initial bits. If the type of two entity nodes is file, path, or URL, the value of $f_{E\left(e\right)}$ is calculated by comparing path and file name. For the similarity between two paths, we treat each directory name as a token. Compare each token from the root directory and end when the token is different. Thus, the feature $f_{T\left(e\right)}$ is designed as follows:

$$f_E(n_{e_i},n_{e_j})=same\_token\left(token\right(n_{e_i}),token(n_{e_j}\left)\right)/max\_len$$

(4)

where $token\left(n_e\right)$ splits a file path into tokens, and $max\_len=max\left(len\right(token\left(n_{e_i}\right),token\left(n_{e_j}\right)\left)\right)$. These formulas all yield values in the range [0, 1].

Operation Feature $f_{O\left(e\right)}$. Operations belonging to the same behavior have a potential relationship, meaning that within the same behavior, there are only a few fixed types of operation. For example, the same operation most likely belongs to the same behavior, and write and delete operations often co-occur within the same behaviors. Therefore, we define the Operation Feature as follows:

$$f_O(o_{e_i},o_{e_j})=bool\_oper(o_{e_i},o_{e_j})$$

(5)

where $bool\_oper$ is a bool function that the value is set to 1 when the operations $o_{e_i}$ and $o_{e_j}$ match the rule that belongs to the same behavior, otherwise 0.

Finally, the confidence value that two system events $e_i$ and $e_j$ belong to the same behavior is obtained by weighting the three aforementioned feature values. The formula is defined as follows:

$$F(e_i,e_j)={\omega }_T{f}_{T\left(e\right)}+{\omega }_E{f}_{E\left(e\right)}+{\omega }_O{f}_{O\left(e\right)}$$

(6)

${\omega }_T$, ${\omega }_E$, and ${\omega }_O$ are the weight coefficients of each feature, and their sum is 1. In this work, the weight coefficients are empirically determined as ${\omega }_T=0.5$, ${\omega }_E=0.4$, and ${\omega }_O=0.1$. The value range of $F(e_i,e_j)$ is [0, 1].

4.2.2. Provenance Graph Partition

ProvGRP uses the above features to calculate the confidence matrices $F_{in(i,j)}\in {\mathit{F}}_{in}^{N\times N}$ and $F_{out(i,j)}\in {\mathit{F}}_{out}^{M\times M}$ for the in-edges and out-edges of each long-running node. Since the number of partitions is uncertain, ProvGRP uses HDBSCAN to implement our clustering task, which does not need to declare the number of clusters in advance and has good robustness to outliers. HDBSCAN can receive the all pairs matrix, and the code published by McInnes L et al. [21] is adopted to implement the clustering task. For long-running process nodes, incoming and outgoing edges are gathered separately. The incoming and outgoing edges are then associated according to information reachability, with the requirement that the information flow follows the chronological order of events. This means that the incoming edge should exist before any of the outgoing edges.

Figure 4 illustrates the partitioned subgraphs. By partitioning the long-running process (chrome.exe), ProvGRP divides the provenance graph into subgraphs that describe different behaviors. Specifically, ProvGRP calculates the dependency distance between dependencies associated with the process chrome.exe and performs clustering, as shown in the upper part of Figure 4. Subsequently, based on the reachability of the information flow, the incoming and outgoing dependencies belonging to the same information flow are associated within the same execution partition. Finally, by splitting the nodes, the original graph is divided into multiple subgraphs, as depicted in the lower part of Figure 4.

Based on the clustering results, ProvGRP merges the leaf nodes (nodes with in-degree or out-degree of 0) within the same cluster. This reduces the number of information paths that are subsequently generated, thus improving the efficiency of log reduction.

4.3. Behavior-Unrelated Events Elimination

The low-level and verbose nature of audit logs makes the presence of behavior-unrelated events severely impact the efficiency of the attack investigation. In behavioral instances, many events are redundant, and removing or merging them does not alter the information flows. Previous approaches to event merging and elimination have focused on individual event characteristics without considering contextual information and information transfer paths. As a result, a large number of redundant events still exist in the reduced data. To identify these redundant and behavior-unrelated events, ProvGRP obtains the context information of events based on the information flow and generates information paths of a specific length. By merging these paths, ProvGRP can remove or merge behavior-unrelated events effectively.

Information Path. Information flows represent the transfer path of information between system entities in provenance graphs. As can be seen from Figure 5A, information from one node will pass through multiple information tributaries and merge at a certain node. These information tributaries can be merged into one information flow without loss of critical information describing high-level behavior, because they belong to the same behavior instance and have a similar flow pattern. An information path, denoted as P, indicates an information tributary that is a chain of events in which all events record the same flow of information. It is an ordered sequence of dependency events and represented as $P={\{e}_1,e_2,\dots ,e_n\}$. For example, a specific information path $P=\{<chrome.exe,write,report.pdf>,<AcroRd32.exe,read,report.pdf>,<AcroRd32.exe,write,report*.pdf>\}$ has a length of 3. First, the information is transferred into report.pdf by chrome.exe. Then, the information flows to the AcroRd32.exe by reading the report.pdf. Finally, AcroRd32.exe writes a new file report*.pdf, which represents the information flows into the file. It should be noted that the flow direction of information is not always consistent with the direction of edges. Furthermore, events may appear in multiple paths, indicating the existence of multiple information flows through these events.

Merging Information Paths. ProvGRP employs an iterative process to merge similar paths based on their length. This has the advantage of reducing the number of long information paths and improving merge efficiency. Our goal is to derive an algorithm that accurately quantifies information path similarity based on these three features. The information path representation is naturally fit with sequence distance calculation methods. Thus, an improved Levenshtein edit distance is designed to calculated the distance between two information paths $P_i$ and $P_j$:

$$LP(m,n)=\left\{\begin{array}{c}max(m,n)ifmin(m,n)=0\\ min\left\{\begin{array}{c}LP(m-1,n)+1\\ LP(m,n-1)+1\\ LP(m-1,n-1)+(1-F(e_m,e_n\left)\right)\end{array}\right.otherwise.\end{array}\right.$$

(7)

where $1-F(e_m,e_n)$ calculate the distance between $e_m$ and $e_n$ in $P_i$ and $P_j$, respectively. For information path pair $P_i^L$ and $P_j^L$ of a certain length L, the normalization distance between them can be expressed as ${LP}_{P_i^L,P_j^L}(L,L)/L$, the value range of which is [0, 1]. We empirically determined the threshold to be 0.2. If the distance is lower than the threshold, the information paths $P_i^L$ and $P_j^L$ are merged. ProvGRP only calculates the distance of information paths that contain at least one identical event. Moreover, it is important to note that process nodes are not merged because most processes are directly associated with behaviors, and merging them would result in the loss of crucial behavior information.

Figure 5 visually illustrates the path merging process. Figure 5A is a provenance subgraph generated from Provenance Graph Partition, where the dotted box indicates information paths of length 1. These paths are subjected to similarity calculation and merged, resulting in Figure 5B. when the path length reaches 4, the two paths within the red dotted box are merged, as shown in Figure 5C. Subsequent iterations do not identify path pairs that exceed the similarity threshold, and the merging process terminates when the path length reaches 6, which is the maximum length. Algorithm 1 outlines the iterative calculation of path similarity for path merging.

Algorithm 1: Behavior-unrelated Event Elimination

Input: Provenance subgraphs  $G_1,G_2,\dots G_n$

Result:$\mathit{Merged}\mathit{Provenance}\mathit{subgraphs}{G}_1^{\prime },G_2^{\prime },\dots ,G_n^{\prime }$

for $\mathrm{each}{G}_i$ do

$\mathrm{Set}max\_len\leftarrow G_i$;

$\mathrm{Empty}\mathrm{list}P$;

while $len\_path<max\_len$ do

$\mathrm{Add}{P}_n$$\mathrm{to}P$;

$\mathrm{Calculate}\mathrm{each}\mathrm{pair}\mathrm{paths}{P}_i,P_jϵP$$\mathrm{with}{S}_{path}(P_i,P_j)$

$\mathbf{if}{S}_{path}(P_i,P_j)>0.8$ then

$\mathrm{Merge}\mathrm{paths}{P}_i$$\mathrm{and}{P}_j$;

$\mathrm{Return}{G}_i^{,}$

5. Evaluation

5.1. Datasets and Evaluation Metrics

5.1.1. Datasets

The effectiveness and generalizability of ProvGRP are evaluated using two publicly available audit log datasets, ATLAS Dataset [1] and DAPRA CADETS dataset [22]. These datasets are widely used for evaluating attack investigation methods and contain ground truth information about attack scenarios.

ATLAS Dataset. The ATLAS Dataset, as provided in reference [1], consists of 10 repeated APT attacks with different vulnerabilities and attack strategies. The dataset covers a time span of 24 h, and each attack is completed within one hour. During the 24 h emulation, an average of 20,784 unique entities and 249 K events are generated for each attack. This paper constructs provenance graphs based on the attack scenarios in the dataset. In the previous work, these attack scenarios were numbered. For ease of reference, this paper uses the dataset name followed by the corresponding number to represent the attack scenarios in the evaluation. For example, ATLAS. S-1 represents Strategic web compromise, which is an APT campaign.

DAPRA CADETS dataset. The DAPRA CADETS dataset [22] is released by the DARPA Transparent Computing program. This dataset was collected during DARPA’s two-week red team vs. blue team Engagement 3. The dataset includes attacks against the FreeBSD system, with the red team executing multiple attack methods four times during different time periods. Throughout the attack, normal behaviors such as SSH login may also occur on the host. A total of 44,404,339 OS-level audit logs were collected over the two-week period.

5.1.2. Evaluation Metrics

Reduction factor. The reduction factor measures the rate of edges (events) removed or merged. Therefore, the reduction factor is defined as ${Num}_{orig}/{Num}_{redu}$, where ${Num}_{orig}$ is the number of edges in the original provenance graphs and ${Num}_{redu}$ is the number of edges after Reduction and Partition by ProvGRP.

Behavior Partition Accuracy. This metric evaluates the accuracy of the provenance graph partition. Here, we evaluate the accuracy of ProvGRP in separating attack behaviors from normal behaviors, i.e., the proportion of events related to the attack in the subgraph describing the attack scenario. Thus, Behavior Partition Accuracy is defined as follows.

Given a ground truth provenance graph of an attack scenario $G_{att}=\left(V,E\right)$ and partitioned provenance subgraph $G_{att}^{\prime }=\left(V^{\prime },E^{\prime }\right)$, a lossless subgraph implies that $E^{\prime }=E$ and $V^{\prime }=V$. The accuracy of provenance graph partition can be measured using the formula.

$${\mathcal{A}\mathcal{c}\mathcal{c}}_{\mathcal{B}\mathcal{P}}=\frac{|E^{\prime }-E|+|V^{\prime }-V|}{\left|E\right|+\left|V\right|}$$

(8)

Attack Information Loss. This metric evaluates the loss of critical information related to the attack after removing behavior-unrelated events. The key attack information is crucial in the attack investigation. Mistakenly removing key attack information (nodes or events) may disrupt the attack path, making it challenging for context-based attack investigation methods to obtain complete context information of the attack events. It may also hinder matching-based attack investigation methods in effectively matching similar attack graphs. Therefore, the metric is used to describe the attack information loss.

Given a ground truth provenance graph of an attack scenario $G_{att}=\left(V,E\right)$, where ${\epsilon }_iϵ\mathbb{E}$ represents attack-related system events in $G_{att}$, $G_{rm\_att}$ represents a provenance graph that contains attack events and has removed behavior-unrelated events, and ${\epsilon }_i^{\prime }ϵ{\mathbb{E}}^{\prime }$ represents attack-related system events in $G_{rm\_att}$. The loss of the attack information can be measured using the formula:

$${\mathcal{L}}_{AI}=1-\frac{|\mathbb{E}\cap {\mathbb{E}}^{\prime }|}{\left|\mathbb{E}\right|}$$

(9)

Causal Information Loss. Graph reduction methods may mistakenly remove events related to normal behaviors, impacting the effectiveness of attack investigation methods that learn from both attack and normal behaviors.

Give all ground truth events ${\mathbb{E}}_{GT}=\{{\epsilon }_1^{att},\dots ,{\epsilon }_m^{att},{\epsilon }_1^{nor},\dots ,{\epsilon }_n^{nor}\}$ that are directly related to behaviors. $\mathbb{G}=\{G_1,G_2,\dots ,G_n\}$ represents a set of subgraphs after removing behavior-independent events. Loss is defined as the proportion of any event ${\epsilon \in \mathbb{E}}_{GT}$ that does not appear on any graph $G\in \mathbb{G}$, which can be measured using the formula:

$${\mathcal{L}}_{CI}=1-\frac{|{\mathbb{E}}_{GT}\cap \mathbb{G}|}{\left|{\mathbb{E}}_{GT}\right|}$$

(10)

5.2. Feature Verification

The features used in ProvGRP are proposed based on intuition and statistical analysis. Below, we illustrate the validity of these features through statistical analysis results on two publicly available datasets used in our experiments.

Time Interval Feature. Figure 6 shows the number of events belonging to the same behavior for different time densities. The time density represents the relative time interval, that is, the proportion of the time interval in the total lifetime of the node. It is calculated by $\frac{|t_{e_i}-t_{e_j}|+\alpha }{T_{end}{-T}_{start}}$, and a lower value indicates that two events are closer to each other. In the time density distribution, events are mainly concentrated in the 0–0.05 and 0.95–1 ranges. Events in the 0–0.05 range belong to the same behavior of long-running processes, while events in the 0.95–1 range are executed by short-lived processes, where each process carries out a single behavior, resulting in event intervals approximately equal to the process lifecycle.

Entity Name Feature. Figure 7 shows the number of event pairs with different entity similarities within the same behavior. The value is calculated by $f_{E\left(e\right)}$. In the similarity distribution, events within the same behavior exhibit entity similarities greater than 0.6, whereas in attack-related behaviors, entity similarity is mostly distributed in the range of 0.8–1.

Operation Feature. Figure 8 represents the distribution of operations in the same behavior. We can see that the distribution is similar in both datasets. In the operation distribution, there are potential relationships between different operations within the same operation.

5.3. Performance Evaluation

Reduction Performance. The effectiveness of ProvGRP is evaluated using the reduction factor.

Table 2. The scale of the reduction graph and reduction factor obtained by CaDR in different attack scenarios.

Attack Scenarios	Prov. Graph		Reduction Graph		Reduction Factor
	|V|	|E|	|V|	|E|
ATLAS. S-1	7468	14,820	1265	1330	11.14X
ATLAS. S-2	33,990	42,126	2749	3981	10.58X
ATLAS. S-3	8975	19,545	1985	3248	6.02X
ATLAS. S-4	13,021	23,669	1025	1730	13.68X
ATLAS. M-1	17,633	38,131	1842	2965	12.86X
ATLAS. M-2	24,489	45,775	2246	4782	9.57X
ATLAS. M-3	24,472	40,040	2312	4293	9.33X
ATLAS. M-4	15,405	32,217	987	1382	23.31X
ATLAS. M-5	35,716	52,934	3268	6092	8.69X
ATLAS. M-6	26,685	37,722	3109	4562	8.27X
ATLAS. Total	207,854	346,979	20,788	34,365	10.10X
CADETS. case-1	237,722	519,657	10,731	14,194	36.61X
CADETS. case-2	298,722	730,717	12,889	13,094	55.81X
CADETS. case-3	175,607	296,206	5106	9350	31.68X
CADETS. Total	712,051	1,546,580	28,726	36,638	42.21X

shows the reduction factors for the attack scenarios in the publicly available datasets. It can be observed that ProvGRP achieves reduction factors ranging from 6.02X to 55.81X for different attack scenarios. On average, ProvGRP reduces the size of provenance graphs generated from the ATLAS dataset by 10.10X, and from the CADETS dataset by 42.21X. The variation in reduction factors is a result of the distinct data characteristics between the two datasets, which utilize different system monitoring tools. It is evident that the reduction factors are similar within the same dataset. The reduction factors of different attack scenarios within the same dataset are similar. Additionally, certain attack scenarios involve user actions that generate a large number of dependencies in a short period of time, such as installing software and running codes. These dependencies are effectively merged and removed by ProvGRP.

Effectiveness Evaluation. The effectiveness of ProvGRP is evaluated using the three metrics: Behavior Partition Accuracy, Attack Information Loss, and Causal Information Loss. The two public datasets contain multiple attack scenarios, with marked information related to the attacks such as entities and events. For each attack scenario, quantitative evaluations of the reduction effectiveness are conducted. The results of the evaluation are presented in Figure 9.

For Behavior Partition Accuracy, taller bars indicate better partition accuracy, meaning that our method correctly divides different behaviors into separate subgraphs. The partition results obtained by our method range from 0.805 to 0.954 across different attack scenarios. These results indicate that the provenance subgraphs describing the attack behaviors retain the majority of attack-related events while containing only a small number of normal system events. This provides greater accuracy for some attack investigation methods. As shown in Figure 9, the partition accuracy for the initial six attack scenarios is generally higher than that of the last three. This difference arises from the presence of numerous normal behaviors and system calls in CADETS that occur closely with the attack behaviors, resulting in their inclusion in the same subgraph. However, the predominantly leaf nodes representing normal behaviors do not significantly affect the topology of the subgraph that describes the attack behavior.

The quantification of loss includes Attack Information Loss and Causal Information Loss. The loss of attack information is controlled within the range of 0.031 to 0.098 across all attack scenarios. This demonstrates that our method avoids substantial loss of attack information. The analysis reveals that the lost attack information mainly consists of repeated or failed attack events in the initial stages, such as port scanning and unsuccessful SSH connections. The loss of such information does not significantly impact subsequent attack investigations. Causal Information Loss includes both normal and attack information loss. The loss of causal information is controlled within the range of 0.033 to 0.1 across all attack scenarios.

5.4. Comparative Evaluation

ProvGRP is compared with other reduction methods using the publicly available CADETS dataset. Since the source code of these methods is unavailable, we attempted to recreate them based on their descriptions in the original papers. Figure 10 shows the change in reduction ratios for the generated graphs compared to the original provenance graphs as the original provenance graphs are reduced. ProvGRP generates the smallest graphs, removing 97.63% of the system events from the provenance graphs of CADETS. In contrast, LogApprox produces the largest graphs, removing 74.05% of the system events. ProvGRP still achieves the best results on ATLAS dataset, and the reduction performance of the other two methods is equal to that achieved by the method proposed by ATLAS in data preprocessing. The experimental results show that ProvGRP can achieve better data reduction results than the existing methods. Additionally, we compare the performance of our re-implemented method with the original work. Although the re-implemented method does not reach the peak of the previous work, the difference is not significant. The main reason for this difference is that the datasets used are different.

The existing methods are also evaluated using the three proposed metrics and compared with ProvGRP. The results are shown in Figure 11. Since LogApprox and NodeMerge do not partition provenance graphs, the graphs they generate contain numerous normal events and entities, resulting in lower values for Behavior Partition Accuracy. LogApprox exhibits nearly no loss of attack information, which aligns with the results from the original work. However, LogApprox experiences higher causal information loss compared to ProvGRP, while its reduction factor is the smallest. Considering the comprehensive evaluation, ProvGRP can achieve further data reduction with losing much less information than previous methods, and can accurately partition the provenance graph to remove false dependencies.

5.5. Attack Investigation Facilitation

To evaluate the facilitation effect of ProvGRP for attack investigation, two attack investigation methods are selected for analysis, which, respectively, used the above two public datasets. ATLAS [1] provides the ATLAS dataset and uses the dataset for the experimental evaluation. Although this work optimized the provenance graph, the average reduction factor is 5.21X. Since the primary goal of this work is to perform attack investigations, its graph optimization method is relatively simple and does not consider context information. The LogKernel [23] method performs experimental evaluation on the CADETS dataset. This work only partitions the long-running process nodes and does not reduce the graphs. In our experiments, we replaced the original graphs used as inputs in the two methods with the graphs reduced by ProvGRP, and compared the changes in runtime and their impact on the results.

Table 3. The changes of the above three indicators and running time after using the reduced graphs.

Methods	Graphs	Time (h:m:s)	Precision	Recall	F1-Score
ATLAS	Original GOA	1:10:37	99.88%	99.89%	99.88%
	Reduction GRA	0:46:21	99.73%	99.68%	99.70%
LogKernel	Original GOC	1:18:08	3\(3 + 0)	3\(3 + 0)	1
	Reduction GRC	0:30:29	3\(3 + 0)	3\(3 + 0)	1

shows the comparison results. When utilizing the reduced graphs G_RA and G_RC, the runtime of the attack investigation is reduced by 34.4% (ATLAS) and 61.0% (LogKernel), respectively.

Precision, recall, and F1-score are employed to evaluate the impact of using ProvGRP-reduced graphs on attack investigation results. As can be seen in Table 3, when using the reduced G_RA, the precision of ATLAS reached 99.73%, the recall reached 99.68%, and the F1-score reached 99.70%. These metrics are slightly lower compared to using the original graph G_OA. A more in-depth analysis of the attack results shows that ATLAS is able to more accurately identify attacks that share processes with normal users when performing attack investigation in the reduced graphs G_RA. ProvGRP can help identify this type of attack more accurately by removing false dependencies caused by long-running processes. This ensures that the origin graphs describing attacks are devoid of unrelated normal behavior instances. As a result, the number of false positives (FP), i.e., the misclassification of normal events as attack events, is reduced. However, some attack information was lost during the reduction process, leading to attack-related events being incorrectly classified as normal events. By utilizing the causal relationships between events to construct attack scenarios, these false negatives (FN) can be associated with the corresponding attack scenarios.

Overall, the evaluation results demonstrate that the use of ProvGRP-reduced graphs significantly reduces the runtime of attack investigations while maintaining high precision. Although a slight decrease in recall and F1-score was observed, the improved representativeness of the obtained sequence and the ability to associate false negatives with attack scenarios compensate for this loss. Thus, ProvGRP effectively promotes the efficiency and accuracy of attack investigations.

5.6. Running Time Performance

The time consumption of ProvGRP is measured on two publicly available datasets. The size of these two datasets is comparable to real-world data. The runtime overhead of ProvGRP consists of three phases. The first phase is to construct the audit logs as provenance graphs, the second phase is provenance graphs partition, and the third phase is behavior-unrelated event elimination and semantic extraction. We perform the experiments on a server with an Intel(R) Xeon(R) Silver 4215R CPU (with 8 cores and 3.20 GHz of speed each) and 256 GB of memory running on Ubuntu 18.04.5 LTS.

In this setting, the processing speed of constructing provenance graphs from ATLAS and CADETS datasets are on average 169 MB/min and 483 MB/min, respectively. The runtime overhead of provenance graph partitioning was measured on both datasets, where our approach was able to handle an average of 38 and 45 long-running nodes per second, respectively. Finally, we analyze each provenance subgraph to remove events that are not related to behaviors. Although the removal was performed in a circular iteration, the size of the graphs decreased and the number of generated paths decreased as the iteration progressed. As a result, the time overhead for this phase was not as high as initially anticipated. Specifically, it took 8 min and 35 s and 12 min and 19 s to process the graphs generated by the two datasets, respectively.

From the reading of log data to construct the provenance graphs to the generation of the reduced graphs after semantic extraction, the total time cost by ProvGRP was 17 min and 43 s and 25 min and 39 s, respectively. These results are close to or even less than the data preprocessing time in multiple attack investigation methods, indicating that the time overhead of our method is acceptable. It will not significantly impact the time efficiency of attack investigations.

6. Related Work

Data Reduction. Various approaches have been proposed to address the challenges posed by large-scale provenance graphs in provenance analysis. Prior work has focused on targeted reduction methods based on the characteristics of redundant data. One category of approaches, including LogGC [24], FD-SD [25], KCAL [26], and Nodemerge [12], adopts a lossy reduction strategy by removing logs based on predefined patterns. While these methods have demonstrated effectiveness through case experiments, their reliance on predefined patterns may limit their adaptability to diverse scenarios, raising concerns about the completeness and correctness of results. In contrast, other reduction techniques such as CPR [13], DPR [27], and ProTracer [12] take a more selective approach, retaining only those system events deemed essential for constructing an accurate information flow graph. Similarly, methods like PCAR [13], DFA [27], and LogApprox [15] explore the bounded approximation of audit logs, accepting a controlled loss of accuracy to gain improved space efficiency. These techniques have shown promise in enhancing the efficiency and accuracy of forensic analysis, especially in scenarios where specific schemes and templates can be applied effectively. It is worth noting that lossless compression methods, typified by SEAL [14], prioritize the preservation of all information for detailed causal analysis. SEAL achieves this by generating a dependency graph from system logs and subsequently losslessly compressing the graph structure and attributes, including timestamps. This approach ensures accurate query results while maintaining efficiency in query processing.

Although these methods have introduced innovative ideas for log reduction and achieved good results, they overlook the erroneous dependency relationships caused by the coarse-grained nature of logs. The erroneous dependency relationships are also a significant factor leading to the generation of exceptionally large and complex provenance graphs. Moreover, the method is unable to identify redundant patterns not defined by schemes and templates. In fact, this paper’s approach demonstrates that leveraging context information can more accurately and efficiently reduce audit data.

Attack Investigation. Sophisticated attacks disguise their behavior to evade the monitoring of intrusion detection systems when infiltrating information systems [28]. Provenance analysis is commonly used to investigate attacks and uncover hidden attack behaviors at the system layer. Nodoze [18] proposes attack investigation methods based on statistical characteristics. They prioritize abnormal events and causal dependencies, considering frequency and topological metrics. However, Priotracker focuses on individual event anomalies, while Nodoze considers anomalies across event chains and employs statistical low-frequency path mining. The statistical low-frequency path mining method is proposed to solve the dependence explosion problem, so as to restore the subgraph of the traceability data corresponding to the alarm generation more accurately. However, the IP address of abnormal transmission cannot be accurately located, and this method based on statistics may lead to unstable results.

Other approaches, Holmes [7] and RapSheet [29], treat multi-stage attacks as chains of causal events that conform to the TTP specification. WATSON [30] uses context information based on knowledge graph of system audit logs to realize semantic inference, expresses different behavior semantics through vectors, and uses semantically similar behaviors for clustering. The results show that benign and malicious behaviors can be accurately abstracted. OmegaLog [2] can accurately coordinate application events with system layer access by identifying and simulating application layer log behavior. Then, it intercepts application runtime log activities and migrates these events to the system layer traceback graph, so that investigators can more accurately infer the nature of attacks.

Hercule [31], Tiresias [32], Attack2vec [33], ATLAS [1], and IDERES [34] use machine learning techniques to model attack behaviors. Hercule uses community detection algorithms to correlate attack events, identifying clear behavioral divisions between threat events and normal events. Tiresias and Attack2vec are limited to identifying and reporting attack events within a single log. ATLAS aims to locate attacks through sequence learning and reconstructing attack paths based on known attack features.

Overall, these previous works contribute valuable insights and techniques to attack investigation and data reduction in provenance analysis, but each has its limitations and focuses on specific aspects of the problem.

7. Discussion

7.1. Limitation of ProvGRP

Although ProvGRP has achieved good results in data reduction, the method still has limitations. Our approach can identify more redundant events using contextual information, but it may produce errors when dealing with lengthy and intricate information paths. The extended length of information paths may accumulate minor differences, leading to the risk of the method incorrectly categorizing similar information paths as dissimilar. Another limitation is that ProvGRP cannot accurately partition two behaviors into different subgraphs when they perform operations using the same process almost simultaneously. If both of these behaviors are normal, they will not affect the subsequent investigation of the attack. However, if one of them is an attack behavior, it may impact the assessment of that behavior. Fortunately, the probability of this scenario occurring in practice is very low. In addition, ProvGRP has effectively divided most behaviors into subgraphs, a practice that is often overlooked in current methods for reducing audit data and investigating attacks. The last obvious limitation is that ProvGRP consumes more computational resources than previous methods based on predefined templates. This level of increased complexity is acceptable compared to the efficiency of facilitating attack investigation.

7.2. Security Analysis of ProvGRP

ProvGRP is generally robust to attacks including APT attacks. Attackers typically evade detection through spoofing or camouflage techniques, such as spoofing the IP address, modifying the time information, double extension on file names, encapsulation techniques, etc. These modifications do pose significant challenges to intrusion detection systems, especially rule-based and matching IOC approaches. To address this challenge, attack investigation methods are widely used to perform a comprehensive causal analysis on audit data. Causal analysis is effective in identifying attacks because even if the attacker conceals or alters some of their indicators of compromise (IOCs) at certain stages, the behavioral patterns and objectives displayed throughout the attack process are markedly different from normal behavior. ProvGRP, by generating information paths based on contextual information, aids in recognizing similar behavior patterns, mitigating the impact of deceptive techniques that alter information at various steps.

APT attacks typically change their behavior gradually to evade detection by template-based and complex attack detection methods. However, this alteration does not significantly impact the ProvGRP outcomes. Our approach does not require matching predefined behavioral pattern templates, which means it does not need constant updates as attackers change their strategies and behaviors. ProvGRP generates information paths that describe all the paths that the information transfers pass through in the same attack instance. These paths share similar flow patterns that describe this attack example. We identify comparable information pathways and combine them to achieve efficient data reduction. Thus, ProvGRP is capable of effectively handling APT attacks that frequently alter their behavior.

8. Conclusions

Existing audit data reduction methods generally face the following two limitations: (1) false dependencies caused by the coarse-grained nature of logs, and (2) the information flow patterns implicit in the deep causal relationships and contextual information of system events are not taken into account. In order to solve the above problems, this paper proposes ProvGRP, a novel context-aware provenance graph reduction and partition approach that partitions provenance graphs into subgraphs describing different behaviors and eliminates redundant and behavior-unrelated events. It proposes three comprehensive features to determine whether the system events belong to the same event, so as to realize the segmentation of the provenance graph to remove the wrong dependencies. Subsequently, it uses information paths containing rich context information to represent information flow patterns, and removes redundant system events by identifying similar information paths. Experimental results show that ProvGRP can effectively reduce audit data while retaining key information of attacks. Furthermore, ProvGRP outperforms state-of-the-art data reduction methods and reduces the runtime of attack investigation methods.

Due to the wide use of deep learning, it is possible to use deep learning models to automatically learn information flow patterns. Therefore, exploring the use of deep learning models to realize information path merging will be one of the future works. In addition, we plan to explore new methods for segmenting graphs, such as analyzing the underlying logic of audit logs to discover deeper features.