Data-Driven Field Representations and Measuring Processes

Abstract

Natural mathematical objects for representing spatially distributed physical attributes are 3D field functions, which are prevalent in applied sciences and engineering, including areas such as fluid dynamics and computational geometry. The representations of these objects are task-oriented, which are achieved using various techniques that are suitable for specific areas. A recent breakthrough involves using flexible parameterized representations, particularly through neural networks, to model a range of field functions. This technique aims to uncover fields for computational vision tasks, such as representing light-scattering fields. Its effectiveness has led to rapid advancements, enabling the modeling of time dependence in various applications. This survey provides an informative taxonomy of the recent literature in the field of learnable field representation, as well as a comprehensive summary in the application field of visual computing. Open problems in field representation and learning are also discussed, which help shed light on future research.

1. Introduction

Fields are useful notions that are ubiquitous in all scientific areas. In generic terms, a field can be treated as a mapping from spatial locations to a physical variable of interest. Given a reference frame, a location is specified by a set of real-valued coordinates, and the most common form of a field function is $f:{\mathbb{R}}^3\mapsto \mathcal{Y}$. $\mathcal{Y}$ represents the domain of the quantities of interest.

The concept of fields serves as a fundamental cornerstone in various scientific disciplines, offering a versatile framework to understand and describe diverse phenomena. Across scientific areas, field representations take on distinct forms tailored to the unique demands of each discipline. In the related physics and mathematics subjects, fields are often construed as formal mathematical entities, inviting analytical scrutiny to uncover their intricate characteristics. In the domain of quantum field theory (QFT) [1,2,3], fields cease to be mere abstractions and emerge as dynamic entities permeating space and time. These fields—whether scalar, vector, or tensor—are not fixed values; instead, they are dynamic and fluctuating entities that embody the fundamental nature of quantum interactions. Transitioning from the microscopic to the macroscopic, the gravitational field introduces a different facet of field representation. In the study of gravitation [4], fields manifest as gravitational fields, bending the geometry of space–time itself. In contrast to the analytical focus of QFT, gravitational fields lead toward a geometrical interpretation, weaving a narrative where masses dictate the curvature, and objects move along paths carved by this curvature. In applied mathematics, the study of partial differential equations (PDEs) [5] further diversifies our perspective on field representations. PDEs are ubiquitous in describing a myriad of physical phenomena, from heat diffusion to wave propagation.

While the theoretical underpinnings of fields find elegance in formal mathematics, the practical application of field concepts in engineering demands a shift in perspective. When grappling with real-world engineering problems, computational aspects take center stage. In the area of representing and performing computations on fields, a prevalent approach involves discretization, or quantization, of space, offering a practical bridge between theory and application [6,7]. Depending on the application fields, various quantization schemes have been found effective. For example, Eulerian quantization [8] finds practical application in finite element analysis, where structures are divided into smaller, discrete elements, and computations are performed at each node or element. Contrasting Eulerian quantization, Lagrangian quantization [9] follows the motion of individual particles or elements through space and time. In simulating the behavior of fluids, Lagrangian methods track the movement of fluid particles, allowing for a detailed examination of fluid flow, turbulence, and interactions. Similarly, in simulating the dynamics of solids, Lagrangian quantization enables the study of deformations, collisions, and structural responses under varying conditions.

Recently, the development of data-driven approaches has revolutionized various scientific and engineering domains. The emergence of data-driven techniques, powered by advancements in machine learning and computational capabilities, offers a promising ability to handle complex, high-dimensional data. Specifically, a useful technique involves employing flexible parameterized representations to model the family of appropriate field functions, particularly through neural networks. Originating from computational geometry and vision, this technique aims to recover fields for subsequent computational vision tasks, such as representing light-scattering fields. Despite the rapid development of data-driven field modeling techniques, there is currently no comprehensive review summarizing the advancements in this field. Therefore, this review aims to provide an overview of state-of-the-art learnable computational models for 3D field functions. Specifically, we focus on models that utilize neural networks as function approximators, showcasing the capabilities and flexibility of neural networks in handling various fields.

The structure of this review is as follows: In Section 2, we present the technical definition of field functions and neural networks. In Section 3, we review neural-network-based and parametric-grid-based approximations for practical field functions. In Section 4, we discuss the developments of the measurement process in the data-driven frameworks by reviewing existing works contributing to the observation process. This section aims to reveal the challenges in this domain and suggests emerging research problems. Finally, we conclude the review.

2. Background

A field function describes a spatially (and temporally) distributed quantity

$$\begin{array}{c}\hfill f:\mathbf{x}\mapsto \mathbf{u}\end{array}$$

(1)

where $\mathbf{x}\in \mathcal{X}$ represents the domain in which the interested physical process is defined, usually an extent of 3D space with an optional time span. $\mathbf{u}$ represents the range of the target physical properties of interest.

We consider a parameterized function space:

$$\begin{array}{c}\hfill \mathcal{H}:\left\{h\right(\mathbf{x},\mathbf{o};\theta \left)\right|\theta \in \mathrm{\Theta }\}\end{array}$$

(2)

from which a candidate function $h^{*}$ is selected to approximate the underlying field f in Equation (1). In Equation (2), h represents a candidate approximator for f. $\mathbf{x}$ specifies a space–time point. The symbol $\theta$ represents learnable parameters. There is an extra input variable to the learning field model, $\mathbf{o}$, which specifies extra conditions to make observations of the underlying field (The observation condition in Equation (2) should be considered without loss of generality. In practice, useful modeling of a physical system can be formulated by specifying the observation process. For example, a quantum mechanical system with a state vector $|x\rangle$ can often be formulated with an observation operator $H|x\rangle$).

The field model is determined in a data-driven manner. In the design stage, the parametric model family is constructed, typically by selecting a specific neural network structure. Once the network structure is fixed, specific parameter values define the particular field function. In the data-driven modeling approach, these parameters are set during an optimization process (learning). The optimization objective is formulated to ensure that the field model aligns with a set of observed variables. The modeling process involves specifying how the observable quantities used to constrain the model were generated.

Neural Networks as Function Approximator

One of the most successful families of function approximators is the neural network models [10]. It is not surprising that multiple-layer perceptrons (MLPs) have become a prevalent framework for generic function approximation to represent a target field function. An MLP is composed of interconnected nodes, referred to as neurons, organized in layers. Each neuron is linked to every neuron in the preceding and succeeding layers. The output of a neuron is adjusted by weights and biases to approximate complex functions, allowing the capture of intricate patterns and relationships in the data.

Generally, the computation of an MLP involves multiple primary units, where, for a unit $a\in \mathcal{A}$,

$$\begin{array}{c}\hfill f^a\left(\mathit{x}\right)=\phi \left(\sum _{i=1}^k{w}_i^a{x}_i+b^a\right)\end{array}$$

(3)

where f denotes a specific layer with input vector $\mathit{x}$. The parameters $\left\{w_i^a\right\}$ and $b^a$ determine the behavior of the particular unit. $\phi (·)$ is the activation function applied to the summation of weighted inputs and bias. This design draws inspiration from biological neurons, where synaptic weights modulate excitatory and inhibitory stimuli, akin to the weights $\left\{w\right\}$ and bias b in Equation (3). The status of a neuron is determined by its response to the stimuli, akin to the activation function $\phi$. The unit in Equation (3) is commonly referred to as a neuron. MLP represents a specific class of computational models containing multiple units $a\in \mathcal{A}$. In an MLP, the units $\mathcal{A}$ are organized in an acyclic graph, and the computations are hence sorted in multiple stages. Units belonging to the same computational step, which can be evaluated simultaneously, are often referred to as forming a layer in the network. To be specific, the inputs to a certain layer are given by:

$$\begin{array}{c}\hfill x_i^l=f^{l-1,i}(\dots )\end{array}$$

(4)

where the superscripts on the right-hand side specifies a neural in $\mathcal{A}$: the ith neuron output in the $(l-1)$th layer serves as the ith input to the neurons in the lth layer. The inputs to the lth layer expand recursively as in Equation (4), and the inputs to the first layer, ${\mathit{x}}^0$, are the raw inputs of the MLP model.

MLP can serve as a generic computational model as a function approximator. Given a function $g:{\mathbb{R}}^n\mapsto \mathbb{R}$ that is Borel-measurable, Hornik et al. [10] established that for any $ϵ>0$ and for any compact subset $K\subset {\mathbb{R}}^n$, there exists a two-layer neural network f such that the supremum norm of the difference between g and f over K is less than $ϵ$. The neural network can be represented as:

$$\begin{array}{c}\hfill f^{\left(2\right)}\left(x\right)=\sum _{i=1}^N{w}_i^{\left(2\right)}{f}^{\left(1\right)}\left(x\right)+b^{\left(2\right)}\end{array}$$

Theoretical investigations of neural network expressivity have been conducted since then.

3. Computational Model of Learnable Field

This section concerns the use of the universal approximation capacity of neural network models to represent physical fields. Learnable models have been gaining attention in the areas of fluid simulation [11], electromagnetics [12], climate science [13], acoustics field [14], medical imaging [15,16,17,18,19], etc.

The brief overview in the preceding section demonstrated that the neural network family contains sufficient expressivity and can be used to approximate most practical field functions. However, most theoretical investigations have been either nonconstructive or have considered the properties of the model asymptotically. In practical field modeling, specific designs and techniques need to be developed to ensure effective modeling. The inputs to a field function specify a location in the spatial domain where the field of interest is evaluated. Although the field can also evolve over time and the function differentiates one time from another [20,21,22], in this section, we focus on formulating the computations and specifically consider a static field for simplicity. We explore recent methodologies and advancements in analytics for neural-network-based field learning by sampling works in a burgeoning research topic, such as the field of light scattering [23]. However, the principles of computational design for field models extend beyond the exemplary cases and are generally applicable to other tasks involving field function approximation using neural networks as learning models.

3.1. Neural Network-Based Approximator

In [23], the authors present a field f that maps a location $p\in {\mathbb{R}}^3$ to interested values such as density $\sigma$ and color c (Equation (5)). Furthermore, the variable p continues to represent the location or position in the remainder of this section.

$$f:(p,d)\mapsto (\sigma ,c)$$

(5)

where d represents the observing direction, which is selected based on the specific interest and is not limited to the particular observing direction.

The researchers employed a neural network to approximate two field functions using shared parameters. Firstly, they sample locations along the ray from observing direction. Then, each location’s coordinates are fed into the neural network, which output density values and geometry feature vectors. In this stage, the neural network approximates the scene’s geometry as the geometry function denoted as $f_{\sigma }$, where $f_{\sigma }\in \mathcal{H}$. In the second stage, the inputs of the neural network are the observing direction and geometry feature vectors. The neural network approximates the appearance function denoted as $g_c$, where $g_c\in \mathcal{H}$, and outputs visual attributes such as color at that location. Then, the overall field approximation can be written as a composition of a geometry field and an appearance field:

$$f=g_c(f_{\sigma }\left(p\right),d)$$

(6)

In the implementation, the two field functions are approximated by the same neural network, see Figure 1a. This approximator type has also been utilized in other studies, such as [24,25,26]. However, all the parameters of the approximator are shared, which means each field cannot be updated and adjusted individually. Additionally, combining two fields results in an increased number of layers in the network, leading to longer computation times during training and evaluation.

Garbin et al. [27] implemented two field approximations by two separate neural networks and achieved a faster and more flexible approximation as shown in Equation (5). The geometry and appearance fields have their own parameters, which are not shared with each other, as depicted in Figure 1b. When a change happens, the two approximations can be updated more flexibly, such as storing one approximation and adjusting the other.

3.2. Eulerian Representation via Parameters on Spacial Grids

The above field approximated using neural networks is a global approximation where all small regions within the field are interconnected. When a change is observed at a specific location (see Section 4), the neural-network-based approximation needs to update all parameters of the field function accordingly. However, in most cases, a change only affects a small region surrounding it. An alternative computational model associates function approximations with particular regions in the domain. The most straightforward specification of a location is via Cartesian coordinates. In the case of a 3D field, the ‘regions’ can be arranged by a grid with Cartesian coordinates [16,28,29,30,31,32].

Liu et al. [28] developed a grid-based approximator to express local properties in specific regions (Figure 2). The grid was utilized to spatially organize the field, where each grid cell represents a small region within the field. They defined a grid as $G=\{V_1,\dots ,V_N\}$ containing N small cells V. ${\mathcal{G}}_{\theta }^i\in \mathcal{H}$ represents the local field function within the ith cell. We use the same example of geometry representation as mentioned above. The geometry field approximator can be written as follows: cite number

$${\mathcal{G}}_{\theta }^i:p\mapsto \sigma ,p\in V_i$$

(7)

It can be seen that ${\mathcal{G}}_{\theta }^i$ has the same function as the neural network mentioned above, which maps a location to the corresponding values of interest. However, it has a different structure from neural networks. Unlike neural networks, the parameters are shared between layers, and the learnable parameters are associated with the location of the cell. For example, if the cell is a small cube, the parameters can be assigned to the eight vertices, and the local properties are bound within the volume of the cube. When p falls in a small region $V_i$, its feature ${\mathcal{V}}_i\left(p\right)$, where $\mathcal{V}\in \mathcal{H}$, is calculated based on the features stored for that region. This implies that only local parameters are utilized to represent and observe p.

The grid approximation can also be combined with other approximating functions, such as $g_c$:

$$\begin{array}{cc}\hfill \mathcal{G}& =\{{\mathcal{G}}_{\theta }^0,{\mathcal{G}}_{\theta }^1,\dots ,{\mathcal{G}}_{\theta }^i,\dots \}\hfill \\ \hfill f& =g_c(\mathcal{G},d)\hfill \end{array}$$

The authors demonstrated the effectiveness of utilizing a grid-based strategy for approximating a field.

Yu et al. [33] also employed similar grid-based approximators $\mathcal{G}$. However, they did not utilize a neural network to compose the other fields. They leveraged local priors such as spherical harmonic (SH), which assumes light scatters on a small area of the target [34]. They demonstrated that grid-based approximations can work without neural networks, and each region can output different interests directly.

As grid-based approaches associate the parameters with spatial location, it is possible to use a more efficient structure to arrange parameters with fluctuation in a region. One common choice is using an octree.

Octree is a hierarchical data structure used to efficiently represent spatial information. When applying an octree for a grid-based field, the empty or full property of a tree node (Figure 3) is associated with the parameters of the approximator. Octree also represents the property of the target function. When the target function changes very quickly in a region, more samples are needed for approximation, which leads to a higher density of parameter distribution in that region and vice versa [28,35]. Octree can be used to prune or skip uninterested regions in the field and recursively partition the interested ones into smaller cells. By distributing parameters according to the function’s fluctuation in each region, the octree can effectively capture the complex region of the field, allowing for a more precise approximation of the target function.

Different from previous researchers focusing on grid-based approaches, Chen et al. [36] factorized the field function through compact tensors for scene modeling. Taking a geometry field as an example, the original field function maps a location to a density interest and is denoted as G. The tensor representation T aims to approximate G through low-rank factorization. The rank $r=[1,2,\cdots ,R]$ is the number of tensors or matrices required to recreate the original elements of G (Figure 4).

The three mode tensors can be indexed using $(i,j,k)$, and $v_i^X$ is the ith value of the vector $v^X$. For CP decomposition, a tensor element $T_{ijk}$ is computed by summing the R outer products of vectors ($v_r^X,v_r^Y,v_r^Z$) as follows:

$$T_{ijk}=\sum _{r=1}^R{v}_{r,i}^X\otimes v_{r,j}^Y\otimes v_{r,k}^Z$$

where ⊗ is the outer product.

For VM decomposition, an element is computed by vectors and matrix:

$$T_{ijk}=\sum _{r=1}^R{v}_{r,i}^X\otimes M_{r,jk}^{Y,Z}+v_{r,j}^Y\otimes M_{r,ik}^{X,Z}+v_{r,k}^Z\otimes M_{r,ij}^{X,Y},v_r^X\in {\mathbb{R}}^I,v_r^Y\in {\mathbb{R}}^J,v_r^Z\in {\mathbb{R}}^K$$

Also, a tensor function can be combined with other interesting functions like the appearance function discussed above [38,39,40].

In summary, neural-network-based and parametric-grid-based approximations were discussed for representing fields. The brief overview demonstrated the capabilities and flexibility of using neural networks to handle various fields. The learning parameters can belong to a single field or be shared by multiple-field approximations, which approximate global information well but may not efficiently handle local changes. On the other hand, a grid-based approximator can effectively capture local changes but may have limitations in capturing global information. To address these limitations, researchers have explored the combination or fusion of neural networks and grid-based approximators [28,30]. This approach allows for leveraging the strengths of each method and potentially achieving a more comprehensive approximation.

4. Computational Models of the Measurement Process

Learning models are constructed using data-driven methods. In addition to the spatial and temporal distribution of the interested physical properties, it is also necessary to incorporate the observable variables in the models, i.e., we need to describe the process of how data are measured from the physical fields, such as images from a scattering field of light. The computational observation model allows the information of the physical fields to be extracted from the sensory data and hence learn the underlying model, for example, learning a magnetic field from sensory data [41]. In this section, we discuss several challenges and design choices in this aspect and illustrate the ideas using examples of learning light-scattering fields from sensory observations. The relationship between 2D observations and field functions is shown in Figure 5.

Formally, recall the field approximator function family discussed in previous sections. An example instantiating Equation (2) was discussed in Section 3,

$$\begin{array}{c}\hfill F:({\mathbb{R}}^3,S^2)\to ({\mathbb{R}}^3\times \mathbb{R})\end{array}$$

where F specifies the light scattering at a location in 3D space ${\mathbb{R}}^3$; $S^2$ represents the observation condition of the light field, i.e., the view perspective from the two-sphere (indicating all possible viewing directions). The discussion so far focused on expressing F in an adequate function family.

However, despite the conceptual clarity of the model, the field cannot be directly learned from measurements. In other words, a function approximation model ${\mathcal{H}}^a:\left\{h({\mathbb{R}}^3,S^2;\theta )\right\}|\theta \in \mathrm{\Theta }$ alone cannot be used to represent practical fields. This is because learning (data-driven) models aim to find an optimal approximation to F from observations. Typically, this is achieved by minimizing the difference between the observed quantities and what the field function has specified. In the task above, actual measured sensory data are not directly related to F; instead, the data are obtained from a process of forming observations by reducing the 3D field to variables distributed on a 2D plane. The result of the observation process is 2D images.

To learn from data, constructing a computational model that mimics the process of deriving observations from the field is essential. The computational observation model facilitates inferring field information from the data. To faithfully, efficiently, and effectively formulate this process for learning, attention must be paid to several key aspects.

Specifically, we consider three aspects that underscore the significance of the observation process: the integral in 3D space, the discretization of the integral, and the diverse computational processes involved in executing the integral.

In measuring the field, quantifiable values depend on the field’s extent rather than a specific point, making direct empirical verification of field equations impractical. Thus, constructing a computational model to depict the observation process becomes necessary. Given the observational nature across a region, a logical approach is to integrate the field over that region, assuming the observation aligns with the expected value of the field function across the field.

The assumption is only valid when the interested physical quantities are reasonably homogeneous. In practical scenarios where this condition is not met, adjustments to the computational model become imperative. For instance, in the observation process within the neural radiance field, the mathematical expression involves the volumetric rendering equation, as shown in Equation (8) [42].

$$C\left(r\right):={\int }_{t_{near}}^{t_{far}}T\left(t\right)\sigma \left(r\left(t\right)\right)c(r\left(t\right),d)dt$$

(8)

Here, the integral encapsulates the integration over the entire depth range from $t_{near}$ to $t_{far}$, where t refers to the distance between two points along a line. The variable d corresponds to the observing direction. $\sigma$ is the quantity value illustrating the propensity of the volume to scatter photons—essentially, quantifying “how many” photons can be scattered. For example, a volume on the surface can scatter more photons and contribute more to the final observations. c denotes the color at the specific position concerning observing direction. $T\left(t\right)$ represents the probability that photons do not contribute to the volume within the range $t_{near}$ to t. This introduces a probabilistic element, accounting for the likelihood that photons traverse the specified distance without interaction, formally defined as:

$$T\left(t\right):=e^{-{\int }_{t_{near}}^t\sigma \left(r\left(s\right)\right)ds}$$

The computational model can be altered based on changes in the observation process. For instance, if the observation emphasizes the geometric structure of scene objects, necessary modifications to the computational model are warranted [43,44,45,46,47,48,49,50]. This might entail a transition to a model that first reconstructs the surface before undertaking the observation process [51,52,53,54,55,56,57,58,59].

The second aspect involves the discretization process within the computational framework. A conventional method used to estimate the integral in Equation (8) is a process known as quadrature [24,25,27,29,30,33,36,60,61,62,63,64]. This entails discretizing the continuous line into a set of discrete samples, obtained via a spatial sampler. These discrete samples function as a representative approximation of continuous volumetric data.

Specifically, in the example of NeRF, a coarse-to-fine strategy is adopted [65]. In the coarse stage, for each line integral, a stratified sampling approach is implemented to uniformly partition the depth range [$t_{near}$, $t_{far}$] into N bins. The mathematical definition of the sampling process within each bin is as follows:

$$t_i\sim \mathcal{U}[t_{\mathit{near}}+\frac{(i-1)}{N}(t_{\mathit{far}}-t_{\mathit{near}}),t_{\mathit{near}}+\frac{i}{N}(t_{\mathit{far}}-t_{\mathit{near}})]$$

where $t_i$ represents the sampled points within each bin, ensuring a stratified distribution along the line integral. This strategic sampling approach aims to capture the varied characteristics of the scene at different depths, providing a more comprehensive representation for the subsequent integral computation.

The estimation of the integral in Equation (8) can then be expressed in a discrete form as follows [66]:

$$\widehat{C}\left(r\right)=\sum _{i=1}^N{T}_i(1-e^{(-{\sigma }_i{\delta }_i)})c_i$$

(9)

In this formulation, $\widehat{C}$ represents the estimated observing color, which is computed by summing over all the sampled points within the stratified bins. Each term in the summation accounts for the contribution of a specific sampled point i, considering its transmission probability $T_i$, the volumetric scattering term $(1-e^{(-{\sigma }_i{\delta }_i)})$, and the color value $c_i$ at that point. ${\delta }_i=t_{i+1}-t_i$ refers to the distance between adjacent samples.

Specifically, the transmission probability $T_i$ is calculated using the discrete transmission function, which encapsulates the cumulative effect of volumetric absorption along the line, influencing the contribution of each sampled point to the final observation:

$$T_i=e^{(-{\sum }_{j=1}^{i-1}{\sigma }_j{\delta }_j)}$$

In the fine stage, points are sampled based on the volume that is predicted in the coarse stage, which samples more points in the regions the model expects to contain visible content. To achieve this, Equation (9) is rewritten based on the result from the coarse stage as:

$$\widehat{C_c}\left(r\right)=\sum _{i=1}^{N_c}{w}_i{c}_i$$

$$w_i=T_i(1-e^{(-{\sigma }_i{\delta }_i)})$$

The points are then sampled based on the normalized weights: $\widehat{w_i}=w_i/{\sum }_{j=1}^{N_c}{w}_j$.

The sampling strategy needs to be adjusted based on different observation processes. For instance, when the integral range encompasses an unbounded scene rather than a bounded one [67], a stratified sampling strategy may not be appropriate. In cases where the observation prioritizes the geometric structure of scene objects, the sampling strategy may have less significance in the model, given that the surface can be approximated.

4.1. Integration Adjustments

In Zhang et al. [68], the observation model tackles the challenge regarding the scene range. In the previous discussion, Equation (8) integrates attributes from $t_{near}$ to $t_{far}$. When the dynamics range of the depth in the scene is small, a finite number of samples is enough for estimating the integral. However, when the scene is an outdoor scene or a 360-degree unbounded scene, the distance range could be large (from nearby objects to distant elements like clouds and buildings). Specifically, for expansive scenes, where distances vary significantly, the integration range becomes extensive.

The challenge arises when the observation is determined by the field evaluation over a large extent. For example, when forming images in a NeRF model, if the device is focusing outward, the distance range is large. The observation model, rooted in Equation (8), faces complexity, as highlighted in [68], due to the need for sufficient resolution in both foreground and background areas within the integral. However, employing a sampling strategy in a Euclidean space, as demonstrated [23], proves challenging to meet this requirement.

Alternatively, when all cameras are forward-facing toward the scene, NeRF utilizes normalized device coordinates (NDCs) [69] for 3D space reparameterization. While effective for forward-facing scenes, this reparameterization strategy limits possible viewpoints and is unsuitable for unbounded scenes.

To overcome this challenge, NeRF++ [68] employs the inverted sphere parameterization. This approach maps coordinates in Euclidean space to a new space, as illustrated in Figure 6, offering a solution to the limitations imposed by the traditional Euclidean and NDC approaches.

The fundamental concept behind this parameterization strategy involves a division of 3D space into two distinct regions: an inner unit sphere encapsulating the foreground and an outer space accommodating the background. The re-parameterization process is selectively applied solely in outer space. To model these two regions effectively, the framework employs two MLPs separately, each dedicated to capturing the characteristics of the inner sphere and outer space. In practical terms, to determine the color of a pixel, Equation (4) is applied independently for the two MLPs, with the results then merged in a final composition step.

Specifically, as shown in Figure 6, the inverted sphere parameterization in outer space reparameterizes a 3D point $(x,y,z)$, where $r=\sqrt{x^2+y^2+z^2}>1$, into a quadruple $(x^{\prime },y^{\prime },z^{\prime },1/r)$. The resulting unit vector $(x^{\prime },y^{\prime },z^{\prime })$ satisfies ${x\prime }^2+{y\prime }^2+{z\prime }^2=1$, indicating its orientation along the same direction as the original point $(x,y,z)$ and representing a direction on the sphere.

In contrast to Euclidean space, where distances can extend infinitely, the parameterized quadruple imposes bounds: $x^{\prime },y^{\prime },z^{\prime }\in [-1,1]$, and $1/r\in [0,1]$. This finite numerical range encapsulates all values within manageable limits. Furthermore, the parameterization acknowledges that objects at greater distances should exhibit lower resolution. By incorporating $1/r$ into the scheme, it naturally accommodates reduced resolution for farther objects. This not only preserves the scene’s geometric characteristics but also establishes a foundation for subsequent modeling and computational processes.

With the inverted sphere parameterization, the integral Equation (8) can be updated as follows:

$$\begin{array}{cc}\hfill C\left(r\right)=& {\int }_{t_0}^{t^{\prime }}\sigma (\mathbf{o}+t\mathbf{d})·c(\mathbf{o}+t\mathbf{d},\mathbf{d})·e^{-{\int }_{s=0}^t\sigma (\mathbf{o}+s\mathbf{d})ds}dt\hfill \\ \hfill & +e^{-{\int }_{s=0}^{t^{\prime }}\sigma (\mathbf{o}+s\mathbf{d})ds}·{\int }_{t^{\prime }}^{\infty }\sigma (\mathbf{o}+t\mathbf{d})·c(\mathbf{o}+t\mathbf{d},\mathbf{d})·e^{-{\int }_{s=t^{\prime }}^t\sigma (\mathbf{o}+s\mathbf{d})ds}dt\hfill \end{array}$$

where $\mathbf{o}+t\mathbf{d}$ refers to the line to be integrated; $\mathbf{o}$ is an observation point; $\mathbf{d}$ is the viewing direction.

Mip-NeRF [60] introduces a fundamental transformation to the integral process, the starting point in the computational observation process. The main change provided with Mip-NeRF is integration along 3D cones originating from each pixel instead of a line, as shown in Figure 7. The integration along these cones captures a broader set of spatial relationships, providing a distinctive characteristic. This adjustment in the observation process results in images with reduced aliasing artifacts. By integrating along cones, Mip-NeRF extends its reach to a richer set of spatial relationships within the scene. This departure from line integration allows for a more comprehensive observation of the field representation.

Traditionally, the field was integral along each line, as shown in Equation (8), capturing information discretely along a one-dimensional path. Mip-NeRF, however, redefines this approach by integrating along a 3D cone originating from every pixel. With the alteration from line to cone, the subsequent point sampling process undergoes a corresponding shift. Instead of uniformly sampling points along a line, Mip-NeRF adapts its sampling strategy to efficiently capture information within these 3D cones. This adjustment is reflected in the use of multivariate Gaussian distributions to approximate intervals within the conical frustums. Additionally, instead of mapping a point in 3D space to a positional encoding feature, Mip-NeRF maps each conical frustum to an integrated positional encoding (IPE).

The discretization process of Mip-NeRF involves dividing the cast cone into a series of conical frustums, essentially cones cut perpendicular to their axis. However, the utilization of a 3D conical frustum as input is not a straightforward task. To address this challenge, Mip-NeRF employs a multivariate Gaussian to adeptly approximate the characteristics of the conical frustum. A Gaussian is employed to fully characterize the conical frustum with three essential values: the mean distance along the line ${\mu }_t$, the variance along the line ${\sigma }_t^2$, and the variance perpendicular to the line ${\sigma }_r^2$. A midpoint $t_{\mu }=(t_0+t_1)/2$ and a half-width $t_{\delta }=(t_1-t_0)/2$ are utilized to parameterize the quantities. This process is as follows:

$${\mu }_t=t_{\mu }+\frac{2t_{\mu }{t}_{\delta }^2}{3t_{\mu }^2+t_{\delta }^2}$$

$${\sigma }_t^2=\frac{t_{\delta }^2}{3}-\frac{4t_{\delta }^4(12t_{\mu }^2-t_{\delta }^2)}{15{(3t_{\mu }^2+t_{\delta }^2)}^2}$$

$${\sigma }_r^2=r^2(\frac{t_{\mu }^2}{4}+\frac{5t_{\delta }^2}{12}-\frac{4t_{\delta }^4}{15(3t_{\mu }^2+t_{\delta }^2)})$$

where $t_0$ and $t_1$ correspond to the start and end depth of a conical frustum, as shown in Figure 7, and r refers to the width of the pixel in world coordinates scaled by $2/\sqrt{12}$.

Crucially, as shown in Equations (10) and (11), the coordinate frame of the conical frustum, now represented by the Gaussian, transforms world coordinates. After the coordination transformation, IPE is applied to obtain the encoding of each conical frustum. Since this is not our focus, we do not provide further details here.

$$\mu =\mathbf{o}+{\mu }_t\mathbf{d}$$

(10)

$$\Sigma ={\sigma }_t^2\left(\mathbf{d}{\mathbf{d}}^T\right)+{\sigma }_r^2(\mathbf{I}-\frac{\mathbf{d}{\mathbf{d}}^T}{{\parallel \mathbf{d}\parallel }_2^2})$$

(11)

In the earlier discussion, the field measuring process was designed around the distribution of physical properties. Conversely, when the observation emphasizes the geometric structure of scene objects, necessary adjustments should be implemented in the computational model. When observing a field with a specified surface, leveraging geometric information can improve the accuracy of the observation.

An example in the radiance field is the NeuS model and its follow-ups [43,70,71,72,73,74,75,76,77,78]. NeuS extracts the surface by approximating a signed distance function (SDF) in the radiance field. With the extracted surface, the observation process is different from that of the original NeRF.

Before delving into the measuring process in NeuS, it is crucial to comprehend how a surface is represented in the radiance field. The surface S of the objects in a scene is represented by a zero-level set of its SDF. As previously mentioned, $\sigma$ in Equation (8) denotes the quantity value illustrating the volume’s propensity to scatter photons. For surface reconstruction, NeuS introduces a weight function $w\left(t\right)$ that signifies the contribution of a point’s color to the final color of a pixel by combining $\sigma$ and T. The measuring process can then be expressed as Equation (12). NeuS asserts that the weight function should satisfy two conditions for surface approximation: unbiased and occlusion-aware.

$$C\left(r\right):={\int }_0^{+\infty }w\left(t\right)c(r\left(t\right),d)dt$$

(12)

The “unbiased” criterion necessitates a weight function that ensures the intersection of the integrating line with the zero-level set of the SDF contributes the most to the pixel color. The “occlusion-aware” property of the weight function requires that when a ray passes multiple surfaces sequentially, the integration process correctly utilizes the color of the surface nearest to the device to compute the output color. NeuS formulates the weight function as Equation (13) to make the function unbiased in the first-order approximation of the SDF.

$$w\left(t\right)=\frac{{\varphi }_s\left(f\left(\mathbf{p}\left(t\right)\right)\right)}{{\int }_0^{+\infty }{\varphi }_s\left(f\left(\mathbf{p}\left(u\right)\right)\right)du}$$

(13)

where $\mathbf{p}$$\left(t\right)$ refers to a point in an integral line.

To render the weight function occlusion-aware, NeuS introduces an opaque density function $\rho \left(t\right)$ to replace $\sigma \left(t\right)$ in the standard observation process. The weight function can then be expressed as follows:

$$w\left(t\right)=T\left(t\right)\rho \left(t\right)$$

(14)

$$T\left(t\right)=e^{(-{\int }_0^t\rho \left(u\right)du)}$$

(15)

By combining Equations (13)–(15), $\rho \left(t\right)$ can be obtained:

$$\rho \left(t\right)=\frac{-\frac{d{\varphi }_s}{dt}\left(f\left(\mathbf{p}\left(t\right)\right)\right)}{{\Phi }_s\left(f\left(\mathbf{p}\left(t\right)\right)\right)}$$

4.2. Discretization of the Integration

As mentioned previously, NeRF++ proposes a reparameterization strategy achieved by extending the concepts in NDC to handle unbounded scenes. However, the efficacy of this strategy relies on points sampled from a one-dimensional line. With Mip-NeRF, this line evolves into a 3D cone, necessitating an extension from NeRF++’s original parameterization. The challenge arises because the reparameterization strategy designed by NeRF++ is not straightforward for the extended line-to-cone transition in Mip-NeRF. In response to this challenge, Mip-NeRF360 [67] introduces a parameterization strategy and a sampling scheme. Departing from the conventional approach of NeRF++, this strategy harmonizes with the 3D cone structure, offering an adaptive and optimized strategy for sampling distances, extending the integral range compared with that of Mip-NeRF.

Technically, Mip-NeRF360 contributes to the observation model in two main aspects. Firstly, it introduces a parameterization method for the 3D scene, mapping 3D coordinates to an alternate domain. Secondly, it provides an approach for sampling t in the discretization process.

To achieve Gaussian reparameterization, Mip-NeRF360 leverages the classical extended Kalman filter. The coordinate transformation is outlined in Equation (16). Applying this operator to a 3D space maps the coordinates of an unbounded scene to a ball with a radius of two. Similar to NeRF++, points within a radius of one remain unaffected by this transformation.

$$\begin{array}{c}\hfill contract\left(x\right)=\left\{\begin{array}{cc}x\hfill & \parallel x\parallel \le 1\hfill \\ (2-\frac{1}{\parallel x\parallel })\left(\frac{x}{\parallel x\parallel }\right)\hfill & \parallel x\parallel >1\hfill \end{array}\right.\end{array}$$

(16)

Additionally, the sampling scheme is designed by introducing a parameterization in the 3D space. In a bounded scene, NeRF employs uniform point sampling from the near to far planes during the coarse stage. When employingn NDC parameterization, the uniformly spaced series of samples in the sampling process becomes uniformly spaced in inverse depth, fitting well with forward-facing scenes. However, this approach proves unsuitable for an unbounded scene in all directions. Therefore, Mip-NeRF360 suggests a sampling strategy, linearly sampling distances t in disparity, as proposed by Mildenhall et al. [79]. Barron et al. achieved this by mapping the Euclidean distance t to another distance s, establishing the relationship as follows:

$$s\triangleq \frac{g\left(t\right)-g\left(t_n\right)}{g\left(t_f\right)-g\left(t_n\right)}$$

$$t\triangleq g^{-1}(s·g\left(t_f\right)+(1-s)·g\left(t_n\right))$$

where $g(·)$ represents an invertible scalar function. To attain samples that are linearly distributed in disparity within the t space, Mip-NeRF360 chooses $g\left(x\right)=1/x$. This choice ensures that the samples become uniformly distributed in s space.

In practical implementations, Mip-NeRF360 modifies the coarse-to-fine sampling strategy. Unlike NeRF, which calculates weights based on coarse-stage results and subsequently utilizes them repeatedly in the fine-stage sampling procedure, Mip-NeRF360 adopts a distinct approach. It recommends incorporating an additional MLP exclusively for predicting weights, excluding color prediction. The predicted weights are then used to sample s intervals.

Significantly, the supplementary MLP in Mip-NeRF360 is smaller than the NeRF MLP, specifically designed to constrain the weights generated by the NeRF MLP. As a result, the resampling process in Mip-NeRF360 involves reduced computational complexity compared to NeRF, owing to the considerably smaller size of the additional MLP compared to the entirety of the NeRF MLP.

4.3. Computational Process of Summing over Sampled Points

Numerical integration is commonly used for computing integrals in practice, including Riemann sums, quadratures, or Monte Carlo methods [80]. Based on these, authors [61] introduced a framework called AutoInt, which utilizes neural networks to approximate integrals in the evaluation stage. As shown in Equation (8), the integral involves two nested integrations: line integration and transmission, Equation (4), which is not trivial to approximate. Therefore, AutoInt divides each line into piecewise sections. Following this division, the observing integral can be refined as follows:

$$C\left(r\right):=\sum _{i=1}^N\overline{{\sigma }_i}\overline{c_i}\overline{T_i}{\delta }_i$$

$$T\left(t\right):=e^{-{\sum }_{j=1}^{i-1}\overline{{\sigma }_j}{\delta }_j}$$

where

$$\overline{{\sigma }_i}={\delta }_i^{-1}{\int }_{t_{i-1}}^{t_i}\sigma \left(t\right)dt$$

$$\overline{c_j}={\delta }_i^{-1}{\int }_{t_{i-1}}^{t_i}c\left(t\right)dt$$

where ${\delta }_i=t_i-t_{i-1}$ is the length of each section. After dividing the line into sections, AutoInt is applied to evaluate each integral.

5. Conclusions

In conclusion, this review focused on data-driven field models, exploring diverse implicit fields grounded in various computational structures. Different field approximators were explored, including neural-network-based and parametric-grid-based approximations for practical field functions, demonstrating the capabilities and flexibility of using neural networks to handle various fields. Specifically, we summarized two approaches based on how the field models are updated:

• Global model with shared parameters: The most common choice in this way is neural networks, which update the entire field. It tends to smooth out local changes or details in the data and may face challenges in capturing and representing small variations. Also, its time consumption has been shown to be heavy in the studies mentioned above.
• Local model with parametric grid: Regardless of the smallest unit of a grid, such as an indexed cube or an element in tensor, in the grid-based method, the field space is arranged, offering efficient spatial querying, allowing for the rapid retrieval of objects or data points within a specific region of interest. When a change is observed, only its surrounding area needs to be updated. Furthermore, this approach allows for skipping or pruning unimportant areas and the adaptive concentration of parameters in areas containing more significant and informative data. However, this approach may not be suitable for certain physical phenomena, such as light phenomena, which often require additional techniques like spherical harmonics for accurate representation [33]. Additionally, the resolution limitations of the grid can restrict its ability to capture fine-grained details, and the memory requirements can grow rapidly as the complexity of the problem increases.

Our examination also extended beyond the theoretical frameworks, delving into the intricate details of measuring models designed for the field functions. Furthermore, the critical challenges and design choices of observation processes were discussed, including the integration adjustments, discretization of the integration, and computational strategies. Following our classification framework, we identified key research problems, with each aspect representing a distinct avenue for further exploration:

• In the area of integration adjustments, our synthesis of the literature revealed the diverse reparameterization strategies employed to accommodate varying scene scopes. This encompasses scenarios ranging from a limited depth of the scene to an infinite depth and extends to the intricate challenges posed by a 360-degree unbounded scene.
• When it comes to the discretization of integration, our exploration encompasses a discussion on how existing methods strategically sample from space. This sampling is intricately tied to the specific design choices made in integration adjustments, forming a crucial aspect of the computational framework.
• In describing the computational processes, we delved into the widely adopted method and the nuanced numerical integration approach embraced by AutoInt.

In essence, our review not only identifies existing research problems but also serves as a guide for future investigations, offering a roadmap for researchers to navigate and contribute to the evolving field of spatial attribute representation.