Image Composition Method Based on a Spatial Position Analysis Network

Abstract

Realistic image composition aims to composite new images by fusing a source object into a target image. It is a challenging problem due to the complex multi-task framework, including sensible object placement, appearance consistency, shadow generation, etc. Most existing researchers attempt to address one of the issues. Especially before compositing, there is no matching assignment between the source object and target image, which often leads to unreasonable results. To address the issues above, we consider image composition as an image generation problem and propose a deep adversarial learning network via spatial position analysis. We target the analysis network segment and classify the objects in target images. A spatial alignment network matches the segmented objects with the source objects, and predicts a sensible placement position, and an adversarial network generates a realistic composite image with the shadow and reflection of the source object. Furthermore, we use the classification information of target objects to filter out unreasonable image compositing. Moreover, we introduce a new test set to evaluate the network generalization for our multi-task image composition dataset. Extensive experimental results of the SHU (Shanghai University) dataset demonstrate that our deep spatial position analysis network remarkably enhances the compositing performance in realistic, shadow, and reflection generations.

1. Introduction

Image composition [1,2,3] aims to combine two different images or parts of images into an image that conforms to human visual common sense. It is a very complex task, involving a lot of details, such as image harmonization, object placement, object shadow generation, etc. We focus on the methods of placing (a foreground object) the source object onto the target image to make it look realistic. The main problems to be solved include placing the source object into the target image and adjusting the position and size to make it reasonable, coordinating the illumination and color between the source objects and the target images, and generating shadows and reflections that the source object does not have.

The main issues in image composition can be divided into two categories, i.e., appearance inconsistency and geometric inconsistency. The appearance inconsistency mainly includes (1) an unnatural boundary between the source object and the target image after compositing; (2) inconsistency in color, illumination, and contrast between the source object and the target image; (3) a lack of shadows and reflections in the target image as a backdrop. Image normalization aims to adjust the color and lighting statistics of the source object and target images to make them more compatible. In addition, shadow and reflection generations aim to train deep learning-based networks to generate shadows and reflections to better integrate the source object into the target image. Geometric inconsistency mainly includes (1) the size of the source object appearing too large or small in the target image; (2) the position of the source object not being associated with other objects in the target image; (3) different perspectives between the source object and the target image; (4) unreasonable occlusion. To address geometric inconsistency, geometric corrections are performed on the source object, such as displacement, scaling, perspective transformation, etc. Object placement performs geometric transformations on the source object to adapt to the target image. The prediction through models aims to analyze the target image and predict the position and size of the source object through a deep learning-based network.

The image composition task is decomposed into several sub-tasks to solve separately. The object placement [4,5,6] task aims to find the reasonable location of the source object in the target image and give the appropriate size. The image harmonization task [7,8,9] is to solve the problem of color harmony between the source object and target image based on reasonable placement and size assumptions. The image blending task [10,11] mainly solves the problem of appearance inconsistency between the source object and the target image, especially in boundary regions. In addition, there are some studies that address the shadow and reflection [12,13,14] generations of the source object. MT-GAN [15] attempts to utilize image generation methods to simultaneously address object placement, appearance consistency, and shadow generation issues.

However, these studies were conducted on the premise that the two images are suitable for composition, and there are few studies on whether the two images are actually suitable for image composition. Before we perform the composition task, one important task is to estimate whether the target image contains reasonable spatial positions that are suitable for the placement of the source object. If not, then the two images are not suitable for image composition. Therefore, we consider analyzing the spatial position of target images before making synthetic images. One purpose of the analysis is to filter the images, and the other is to choose a suitable location for source object placement. Hence, we propose a generation method based on target image resolution to solve the above problems.

Our main contributions are listed as follows:

1.

Deeply learned spatial position analysis for image composition—We propose a multi-task deep learning network for target object screening. We first analyze the spatial position of the target image, and then find an appropriate placement location for the source object. Furthermore, the source object is generated in the position box to obtain the image compositing result. Qualitative analysis shows that our method can generate clear composite images for source objects with different shapes and patterns, and quantitative analysis shows that the proposed framework achieves the highest score of authenticity.

2.

The largest multi-task image composition dataset with spatial position annotations—We establish a large multi-task image composition dataset with 400 images with position masks for spatial position analysis. Moreover, a new generalization test set is collected as a supplement to the SHU dataset. It includes various target objects suitable for placing the source object, including stools, chests of drawers, tables, bedside cabinets, etc. We also add new annotations of source objects’ locations for spatial position analysis.

2. Related Work

In this section, we will discuss the related works on image composition.

Image composition has attracted wide attention in recent years. It aims to cut the source object from one image and process it or paste it seamlessly on another target image, resulting in a composite image. So far, the image composition task has been explored from a variety of perspectives. To complete a realistic-looking composite image, it is necessary to address issues such as geometric inconsistency, appearance inconsistency, and shadow and reflection generations. Deep learning-based methods have achieved widespread application, with some researchers [16,17] utilizing CNN frameworks in their computational methods for recognition tasks in the biological field. Similarly, many deep learning methods have emerged in image composition tasks. For example, Realism CNN [18] fine-tuned the VGG network to distinguish the authenticity of images, which assists in making high-quality synthesis images and provides a criterion for evaluating the authenticity of composite images. Tsai et al. [9] proposed an effective method to collect large-scale and high-quality training data and designed an end-to-end deep convolutional neural network that can capture the contextual and semantic information of composite images during the coordination process. They proposed a solution to the image harmonization task. Then, Cong et al. [7] provided a dataset (iHarmony4) suitable for image harmony tasks and proposed an attention mechanism network to improve the effectiveness of image composition. Afterward, some researchers [19,20] conducted a series of studies on the basis of the iHarmony4 dataset, further improving the image composition effect.

During this period, the target placement task also made some progress with the emergence of ST-GAN [21]. ST-GAN integrates the STN and GAN frameworks to find realistic geometric corrections to the source object, so that it looks natural when composited into the target image. On this basis, Chen et al. [1] proposed a similar shape object replacing method. A GAN architecture with a pair of discriminators and a segmentation network was used to adjust the color of the source object for automatic image compositing. A transformation network and a refinement network were used to improve geometric consistency and polish the boundary of the composite image, respectively. Spatial fusion GAN [22] combines the spatial transformation network (STN) and style transformation network to achieve realism in both geometric and appearance spaces of the synthesized image. Tripathi et al. [5] presented a task-aware approach containing a trainable image synthesizer that can assess the strengths and weaknesses of a network to generate meaningful training samples. Subsequently, Zhang et al. [6] proposed PlaceNet, which can predict a diverse distribution of reasonable locations for source object placement. This had the benefit of a self-learning framework that could generate necessary training data without any manual labeling. Li et al. [23] presented a fast OPA model including foreground dynamic filters, background prior transfer, and composite feature simulation. Zhou et al. [24] proposed a graph completion module (GCM) with a dual-path framework to address the object placement problem. This GCM can fully exploit annotated composite images. In recent years, encouraging progress has been made in object placement and image harmonization, and novel solutions for shadow and reflection generations have emerged [12,13,25,26].

However, there are some limitations to the above methods. Image harmonization tends to adjust the color and illumination statistics of the source object to make the whole image harmonious, while shadow or reflection generation aims to generate plausible shadow or reflection for the source object to make the image more realistic. But they need to specify the location and size of the source object in advance. Object placement focuses on seeking information such as the appropriate size, shape, and position of the source object, but lacks the ability to adjust the appearance consistently.

Therefore, we propose a multi-task framework that can solve image harmonization, object placement, and shadow generation in a one-stop manner. In addition, existing image composition approaches do not judge whether the two images are suitable for compositing; our approach is an exception. The proposed method filters data by analyzing the target image.

3. Materials and Methods

In this section, we propose a spatial position analysis network (SPAN) for image compositing, matching positions before generating images. The overall process of the proposed SPAN is synthetically described in Figure 1. The network consists of five main components, including a target analysis network, spatial alignment network, composite generator, global discriminator, and local discriminator. Symbols and their representations in this section are shown in

Table 1. Related mathematical symbols and their representation.

Symbol	Representation
$I_t$	target image
$I_s$	source object
$I_a$	analyzed data; semantic segmentation data
$I_m$	mask map
$I_{mr}$	ground truth mask map
$I_c$	composite image
$I_r$	ground truth image
$I_{cm}$	source object cropped from the composite image
$I_{rm}$	source object cropped from the ground truth image
$L_p$	position loss function
$L_c$	compositing loss function
$L_{Dg},L_{Gg}$	global discriminator loss function
$L_{Dl},L_{Gl}$	local discriminator loss function
$L_{final}$	final loss function
$P_p$	predicted position
$P_g$	ground truth position
${\lambda }_p,{\lambda }_g$	super parameters

.

As the first step of the SPAN, the target analysis network is used to analyze the target image. It segments different objects in the target image and corresponds to different values in semantic segmentation image $I_a$. Then, we concatenate the analyzed data $I_a$ with the source object $I_s$ and feed it to the spatial alignment network. The spatial alignment network predicts the locations of the source object based on the analysis results of the target image and provides the most reasonable placement position. The position is represented by the diagonal coordinates of the corresponding box, namely, the four numerical values of the upper left corner coordinate and the lower right corner coordinate, e.g., ($x_0,y_0,x_1,y_1$). These coordinate values are represented by a mask map $I_m$ and fed to the subsequent composition generator. The values within the box are set to 1, and the values in other positions are set to 0. Then, the target image $I_t$, source object $I_s$, and position mask map $I_m$ are concatenated and fed to the composite generator to predict the synthesized image $I_c$. The whole image is obtained through the adversarial generative learning process, and the generation process focuses on the region of the source object. Therefore, in addition to identifying the authenticity of the whole image, attention should also be paid to the details of the generated source object. We use two discriminators to train the generator: a global discriminator and a local discriminator. The global discriminator identifies the authenticity of the whole image between composite and real images. We extract the source objects from the composite image and the real image and then feed them to the local discriminator to verify the detailed authenticity of the generated source object. For all the experiments, we use the Adam solver with a batch size of 1. We set the learning rate as 0.00005.

3.1. Target Analysis Network

There are usually several different objects in an image. The relative size and position of each object in a real image are consistent with common sense. If an object in the image is placed in an inappropriate position, the whole image will look unreal. For example, normally, the train is running on the track, but if the train appears on the road, the picture will look fake.

The target analysis network first pre-processes the input data by analyzing the instances in the target image and labeling different target objects with different values. Then, the analyzed data are fed to the subsequent network to assist in training the image compositing. Through iterative training, the network learns to find the suitable location and scale of the target object that matches the source object. If the object attributes in the target image match the source object, image compositing can be performed; if there is no match, image compositing is not possible. The purpose is to filter the target images.

We adopt SegNeXt [27] as our target analysis network. We call the model parameters pre-trained on the ADE20K dataset and fine-tune them on our target images based on a semantic analysis task. The output of the target analysis network is the semantic information, in which different values represent different target objects using a three-channel RGB format.

3.2. Spatial Alignment Network

After the semantic analysis of the target image, the analyzed data ($I_a$) and the source object ($I_s$) are fed to the spatial alignment network to predict the location and size. Based on the input information, the spatial alignment network predicts the mask map of the source object, which corresponds to the position and scale distribution of the source object in the target image to be embedded. The mask map is set to a rectangle, and the network gives two coordinates of the diagonal of the rectangle: the coordinates in the upper left corner and the lower right corner.

The analyzed data $I_a$ are three-channel and the source object $I_s$ data are four-channel, in RGBA format. We resize both groups of data to 256 × 256 and feed the concatenation data to the spatial alignment network. The architecture is shown in Figure 2. Each blue block represents a sub-sampling convolution layer following a normalized layer and a ReLU activation function. The yellow block represents a ResNet module containing two convolutional layers. The input data first go through three sub-sampling convolution layers followed by three ResNet schemes for feature extraction. Next, the data go through another three sub-sampling convolution modules. Finally, the data are flattened, and four coordinate values are output through two fully connected layers. In order to facilitate the data concatenation in the subsequent image composite generator, we introduce the position mask map $I_m$ according to the coordinate values by aligning the predicted position of the source object to 1 in a 256 × 256 all-zero map.

In the spatial alignment network, we define the position loss using the MSE between the predicted position $P_p$ and the ground truth position $P_g$. The position loss function can be expressed via Equation (1).

$$L{}_p=MSE(P_p,P_g)$$

(1)

3.3. Composite Generator

Although the reasonable placement of the source object has been determined, the resulting composite image may still be unrealistic. The reason for this is that the source object and target images are collected under different lighting conditions and shooting angles. Therefore, the source object should be further processed to make it consistent with the color of the target image. We use a generative network for image compositing. The generator takes the input data as a condition to generate a brand new image. The output of the generator tends to be plausible as long as the input of the discriminator is a real image. One original task of an adversarial generative network is generating real face images based on random noise [28], which is almost independent of the initial input data. So, the generated source objects are more easily integrated with the target image in terms of shape and color.

We use the merge data, source object $I_s$, mask map $I_m$, and target image $I_t$ as the input of the composite generator. Where $I_s$ is 256 × 256 × 4, mask map is 256 × 256 × 1, and $I_t$ is 256 × 256 × 3. So, the input data are 256 × 256 × 8.

The structure schematic of the composite generator is shown in Figure 3a. We borrow the encoder–ResNet–decoder structure of the CycleNet [29] backbone network. We expect the encoder attention map to pay more attention to the target image of the encoder feature. The for this reason is that the source objects the of encoder feature may not be fully harmonized yet. Inspired by the attention module in DoveNet [7], we add a feature attention mechanism to the CycleNet. The encoder, ResNet, and decoder are represented as blue, yellow, and green blocks, respectively.

Each blue block corresponds to a convolution module, containing a convolutional layer, a regularization layer (InstanceNorm) [30], and a LeakyReLU activation function. Each yellow block corresponds to a ResNet module. The green block corresponds to a deconvolution module, containing a transposed convolution layer, a regularization layer (InstanceNorm), and a ReLU activation function. The attention module is shown in Figure 3b.

Firstly, we concatenate the corresponding encoder features (the output of the blue block) with the decoder features (the output of the green block). Then, the full attention maps containing the spatial attention and channel attention are learned. Specifically, we apply a 1 × 1 convolution layer following Sigmoid activation on the concatenation. Next, we perform element-wise multiplication on the encoder (resp., decoder) attention map and the encoder (resp., decoder) feature.

We compare the compositing image $I_c$ and the ground truth image $I_r$ at pixel level. The $L1$ loss is used to promote the network to generate plausible images, and the compositing loss function can be expressed as follows:

$$L_c=\left|\right|I_c-I_r{\left|\right|}_1$$

(2)

3.4. Discriminator

The purpose of the composite generator is to generate realistic source objects in the corresponding region in the mask map $I_m$. Two discriminators are used to discriminate the generated composite images. The global discriminator mainly focuses on the overall characteristics of the composite image, making the entire image close to the real image. Meanwhile, the local discriminator pays more attention to the details of the composite image in the corresponding region of the mask map. In addition, the global discriminator distinguishes the harmony of the whole image, and the local discriminator identifies the consistency of the generated source object with the ground truth. The two discriminators assist each other in the training process.

The global discriminator identifies the realism of the composite image. We feed the ground truth image and the composite image to the discriminator with different annotations. This can train the discriminator’s ability to distinguish authenticity, and continuously improve the generator’s ability to synthesize realistic images. The architecture of the global discriminator is shown in Figure 4, with fixed input data of 256 × 256. There are five convolution modules. The first one contains a convolution layer and a LeakyReLU activation function, and the subsequent three convolution modules contain a convolution layer, a regularization layer (InstanceNorm), and a LeakyReLU activation function. The last convolution module consists of only one convolution layer.

Figure 5 shows the architecture of the local discriminator, which is almost the same as the global discriminator. The difference is that the size of the input data is related to the mask map, which is not fixed. Specifically, due to the small size of some source objects, the local discriminator may experience a size smaller than 1 during the down-sampling process. So, we increase the degree of padding in the convolution layer to avoid this situation in the local discriminator.

The loss function of the global discriminator is shown as follows:

$$\begin{array}{c}{L}_{D_g}=E\left[max(0,1-D_g\left(I_r\right))\right]+E\left[max(0,1+D_g\left(I_c\right))\right]\\ L_{G_g}=-E\left[D_g\left(G(I_t,I_m,I_s)\right)\right]\end{array}$$

(3)

We minimize $L_{D_g}$ to train global discriminator $D_g$, which is encouraged to produce large scores for real images and small scores for generated images. In the same way, we minimize $L_{G_g}$ to train G, and the generated samples are expected to fool global discriminator $D_g$ into taking them for large scores.

The loss of the local discriminator is given by the following:

$$\begin{array}{c}{L}_{D_l}=E\left[max(0,1-D_l(I_r,I_m))\right]+E\left[max(0,1+D_l(I_r,I_m))\right]\\ L_{G_l}=-E\left[D_l(G(I_t,I_m,I_s),I_m)\right]\end{array}$$

(4)

We train $D_l$ by minimizing $L_{D_l}$, and train G by minimizing $L_{G_l}$.

Overall, the final loss function for training the whole network is shown in the following:

$$L{}_{final}=L_c+{\lambda }_p{L}_p+{\lambda }_g(L_{G_g}+L_{G_l})$$

(5)

where ${\lambda }_p$ and ${\lambda }_g$ are super parameters. We set ${\lambda }_p$ as 1, and ${\lambda }_g$ as 0.1 in our experiment.

4. Results

We evaluate the SPAN model on the SHU dataset, and achieve optimal performance in the literature. All training and testing of the proposed model was completed based on the PyTorch framework.

4.1. Experiment Setup

Our SPAN method contains three sub-networks: the target analysis network, the spatial alignment network, and the composite generator. The target analysis network uses the parameters pre-trained on the ADE20K dataset, while the spatial alignment network and the composite generator are trained and evaluated on the proposed multi-task image composite dataset, named the SHU dataset.

ADE20K dataset is a publicly available dataset published by the Computer Vision team at the Massachusetts Institute of Technology (MIT) in 2016. This dataset fully marks the targets in the image, with more than 3000 object categories, which can be used for semantic segmentation, instance segmentation, scene analysis, and other tasks. Currently, the data publicly provide 25,574 training images and 2000 validation images. We utilize the SegNeXt network parameters, which are obtained by training on this dataset.

The SHU dataset is proposed by MT-GAN and contains a total of 7756 images across eight source object categories, including 6206 images in the training set and 1550 for the test. The source objects are different in color, texture, and pattern. The data are taken in different scenes with diverse angles, distances, and lighting conditions. The paired samples are a target image without the source object and a ground truth image with the source object under the same photographic conditions. Each ground truth sample also marks the RoI containing the source object and corresponding shadows. In this paper, the spatial alignment network inputs the source object region, excluding the shadow part. Thus, we add a bounding box annotation containing only the source objects. The spatial alignment network and the composite generator are trained and tested on the SHU dataset, of which 20% is used for the test set and 80% for training. Further, we add new test data to test the generalization of the network, called the generalization test set (GTS), to distinguish them from the previous test set (PTS). The images in the PTS exhibit different angles and different lighting conditions compared to the training data, but the same scene. To test the compositing effect of our method in multiple brand-new scenarios, we collect a total of 400 images from 15 completely different scenes in the GTS, using different angles and distances to those used with the PTS. Considering that the GTS needs to be adapted for image composition with source objects, we adopt a similar shooting method with the SHU dataset. The difference between the SHU dataset and other datasets is that there are multiple paired images corresponding to a source object. Each pair of data contains a target image (without the source object) and a ground truth image (with the source object) under the same shooting condition. To our knowledge, there are currently no other datasets with large quantities of paired sample data in different scenes. In addition, we name the eight categories of source objects, which are shown in Figure 6.

4.2. Results Comparison with Other Methods

We evaluate our method and report the results of the comparison with other methods, including quantitative results, qualitative results, ablation study results, and the objective estimation of other objects of the SHU dataset. We used mean squared error (MSE), peak signal-to-noise ratio (PSNR), intersection over union (IoU), the user study, and the objective estimation score [18] as the evaluation metrics.

4.2.1. Quantitative Experimental Results

We compare our approach with other deep learning-based approaches.

Table 2. Comparison results of objective estimation with other methods. The best results are denoted in boldface.

Method	MSE ↓	PSNR ↑	Mask-IoU Score ↑	PTS Score ↑	GTS Score ↑
Arbitrary Composite	423.98	20.86	-	0.1510	0.1391
ST-GAN [21]	200.87	23.52	-	0.6429	0.4639
AGCP [31]	198.40	24.56	-	0.6887	0.4954
DoveNet [7]	92.80	27.83	-	0.9021	0.6861
MT-GAN [15]	82.30	29.51	-	0.9453	0.7561
Ours (two-stage)	115.45	28.00	0.6103	0.9751	0.8688
Ours (proposed)	103.51	28.69	0.8971	0.9958	0.8933

shows the comparison of the composite results of source object S1. Following MT-GAN [15], we use MSE, PSNR, and the objective estimation score of the compositing images. The objective estimation scores are tested on the PTS and GTS, respectively. In addition, in order to obtain the objective estimation of the spatial alignment network, we introduce IoU to evaluate the accuracy of mask map generation and obtain a Mask-IoU score.

With the structure of the network backbone anchored, we first try a two-stage approach, in which the spatial alignment network and the composite generator are trained separately. Then, we attempt an end-to-end approach, in which the spatial alignment network and the composite generator are trained simultaneously. Importantly, the spatial alignment network is trained with only one constraint of the bounding box in a two-stage method. Meanwhile, in the end-to-end method, there are two more constraints for training, e.g., the global image ground truth constraint and adversarial constraint. We can see that the end-to-end method is superior to other methods. Specifically, the end-to-end method is 46.99% higher than the two-stage method in terms of the Mask-IoU score.

Our method achieves the best performance in both PTS score and GTS score, but is slightly worse than MT-GAN and DoveNet in terms of MSE and PSNR. The reason for this is the insufficient accuracy of the generated target position resulting in the deviation between the generated and the ground truth position of the source target.

Samples of the mask map are shown in Figure 7. We can see that the differences between the predicted mask maps and the generated source objects and their corresponding ground truths are mainly in size, while the position, color, texture, pattern, and shadow are very realistic. This resulted in our method not achieving optimal results for MSE and PSNR. However, the position and size of the source object are reasonable. Moreover, the objects far from the camera are smaller, while those closer are larger. Thus, the composite images still look authentic.

4.2.2. Qualitative Experimental Results

Figure 8 shows samples of our composite results of the eight categories of the source objects, which demonstrate that the composite images are very similar to the real images in terms of location, size, shape, texture, color, shadow, and detail. This indicates that our method can generate realistic compositing images for different source objects in different scenes. Particularly, both the geometric pattern details and the characters are consistent with the ground truth source objects. For glossy color source objects, our method can generate appropriate luster results through integration with the target images. In addition, we can achieve satisfactory shadows of the source objects.

Figure 9 shows that the source objects have been resized and placed on the matching target objects, making the composite images appear reasonable. In addition, the network adjusts the angle of the source objects. Taking the composite image of source object s1 as an example, the white lid of the cup has an inclination angle. This indicates that the source object in the composite image has an angle adjustment in the Z-axis direction relative to the original source object.

We use data augmentation to avoid overfitting. We annotate the bounding boxes of the source objects in each real image, and randomly crop the samples while ensuring the inclusion of the source objects.

4.3. Ablation Study

We investigate the effectiveness of each learning objective in our method, and

Table 3. Comparison results of the SHU dataset with different learning objectives. The best results are denoted in boldface.

Method	PTS Score ↑	GTS Score ↑
$L_c$	0.3563	0.0601
$L_c+L_{G_g}$	0.9279	0.5960
$L_c+L_p+L_{G_g}$	0.9821	0.7417
$L_c+L_p+L_{G_g}+L_{G_l}$	0.9958	0.8933

shows the results. Because of the inherent deviation between the predicted position of the source object and the ground truth, the non-uniqueness of the reasonable location of the source object in the composite image, and no ground truth on the generalization dataset, we utilize score evaluation when measuring the compositing effect.

In Table 3, we can see that the objective evaluation score is lower without the spatial alignment network in the PTS and the GTS. The scores rise noticeably after adding a global discriminator. The addition of a spatial alignment network significantly surges the score of the GTS, indicating an enhanced authenticity and visibility of the compositing images. The local discriminator mainly improves the detail and clarity of the generated source object. Our proposed method reaches the highest score of objective authenticity and also reaches a peak in the GTS score.

To pursue the optimal performance of our method, we conduct the key parameter study. When adjusting the weight of one loss function, the weights of others are fixed. Firstly, when ${\lambda }_p=1$, we set ${\lambda }_g$ as four different options: 0.001, 0.01, 0.1, and 1. As shown in

Table 4. Comparison results of the SHU datasets with different values of the key parameter ${\lambda }_g$, which is used to keep the balance of the loss functions. ${\lambda }_g$ is set to 0.1 in training. The best results are denoted in boldface.

${\mathit{\lambda }}_{\mathit{g}}$	PTS Score ↑	GTS Score ↑
0.001	0.9787	0.6086
0.01	0.9923	0.8231
0.1	0.9958	0.8933
1	0.9943	0.8555

, the optimal result is under ${\lambda }_g=0.1$. The PTS scores are similar in other cases, but the GTS scores fluctuate wildly. Then, when ${\lambda }_g=0.1$, we set ${\lambda }_p$ as five different options: 0.1, 0.5, 1, 5, and 10.

Table 5. Comparison results of the SHU datasets with different values of the key parameter ${\lambda }_p$, which is used to keep the balance of the loss functions. ${\lambda }_p$ is set to 1 in training. The best results are denoted in boldface.

${\mathit{\lambda }}_{\mathit{p}}$	PTS Score ↑	GTS Score ↑
0.1	0.9826	0.8564
0.5	0.9936	0.8754
1	0.9958	0.8933
5	0.9947	0.8807
10	0.9921	0.8641

shows that the optimal performance occurs under ${\lambda }_p=1$.

4.4. The Objective Estimation of Other Source Objects of the SHU Dataset

We conduct the objective estimation of the PTS score and GTS score on other source objects of the SHU dataset. In addition, we perform a user study to further evaluate the perceptual quality of our method, the results of which are shown in

Table 6. Comparison results of the evaluation of different source objects. The error of the same evaluation metric for different source objects does not exceed 0.1, indicating that the proposed method achieves similar results in image compositing tasks with different source objects.

${\mathit{I}}_{\mathit{s}}$	Objective Evaluation		User Study
	PTS Score ↑	GTS Score ↑	Ground Truth Score ↑	PTS Score ↑	GTS Score ↑
S1	0.9958	0.8933	0.7205	0.6128	0.5630
S2	0.9313	0.8281	0.7304	0.6032	0.5652
S3	0.9219	0.8140	0.7135	0.5937	0.5562
S4	0.9653	0.8981	0.7469	0.6265	0.5648
S5	0.9385	0.8672	0.7173	0.6586	0.5913
S6	0.9537	0.8749	0.7821	0.6071	0.5714
S7	0.9150	0.8075	0.7124	0.5965	0.5632
S8	0.9886	0.8927	0.7549	0.6202	0.5826

. For different source objects, the PTS score of objective evaluation is higher than 0.9 points, and the GTS score of objective evaluation is higher than 0.8 points. The user study scores have strong subjectivity, since different observers and different environments affect the score results. We can see that the ground truth samples receive scores between 0.7821 and 0.7124. The mean relative error between the PTS score of different source objects and the ground truth is 16.24%. The mean relative error between the GTS score of different source objects and the ground truth is 22.40%.

User study: This user study involves 30 participants. During this study, each participant was shown 240 samples, consisting of the ground truth, the composite image in the PTS, and the GTS in a 1:1:1 ratio. They watch for 3 s and rate the authenticity of each sample. Participants were asked to rate the results based on two levels: 1 for real and 0 for fake. Firstly, we obtain the scores of each source object from participants. Then, we sum them and divide them by the number of participants. The average scores (higher the better) are calculated via Equation (6).

$$S_{av}=\frac{1}{N_p}\sum _{j=1}^{N_p}(\frac{1}{N_s}\sum _{i=1}^{N_s}x)$$

(6)

where x is the score of the images of each source object from participants, $N_s$ is the number of samples of each source object, $N_p$ is the number of participants, and $S_{av}$ is the average score of images with the corresponding source object.

5. Conclusions

In this work, we supplemented the multi-task image composition dataset, including position masks for spatial position analysis and a generalization test set for generalization ability testing. We also propose a novel image composition method using spatial position analysis. The target analysis network can effectively segment and classify target objects. The purpose is to filter the target image and choose suitable samples for image composition. According to the target analysis results, the spatial alignment network effectively matched the source object with the target image, and predicted regions with appropriate position and reasonable size. As the experimental results of the SHU dataset and our newly proposed test set show, the trained network achieves good performance in terms of the image composition task, and generates realistic composite images. However, there are some limitations. When compositing a new source object, it is necessary to collect a lot of relevant data and train the model again. That is to say, our method relies on training data, and obtaining a more realistic source object compositing result requires targeted data support. In the future, we will aim to optimize the model. Furthermore, we would like to apply spatial position analysis to improve video compositing performance.