Zero-Shot Image Harmonization with Generative Model Prior

Jianqi Chen Zhengxia Zou Yilan Zhang Keyan Chen Zhenwei Shi*
Beihang University
https://github.com/WindVChen/Diff-Harmonization
arXiv:2307.08182v1 [cs.CV] 17 Jul 2023

Foreground
Stable Diffusion
(Frozen)


Composite Image Harmonized Image
Background
Figure 1. Given a composite image, our method can achieve its harmonized result, where the color space of the foreground is aligned with
that of the background. Our method does not need to collect a large number of composite images for training, but only utilizes a pretrained
text-to-image generative model. The first column from the left in the upper row is the source image of the foreground (“house”), and the
others are the harmonized results of the foreground in different backgrounds. In the lower row, we take one of the composite images as an
example to show the entire harmonization process.

Abstract Constraint Text which is optimized to well illustrate the

Recent image harmonization methods have demon-
strated promising results. However, due to their heavy re-
liance on a large number of composite images, these works
are expensive in the training phase and often fail to gen-
eralize to unseen images. In this paper, we draw lessons
from human behavior and come up with a zero-shot image
harmonization method. Specifically, in the harmonization
process, a human mainly utilizes his long-term prior on har-
monious images and makes a composite image close to that
prior. To imitate that, we resort to pretrained generative
models for the prior of natural images. For the guidance
of the harmonization direction, we propose an Attention-
* Corresponding author.
image environments. Some further designs are introduced
for preserving the foreground content structure. The result-
ing framework, highly consistent with human behavior, can
achieve harmonious results without burdensome training.
Extensive experiments have demonstrated the effectiveness
of our approach, and we have also explored some interest-
ing applications.
1. Introduction
Image harmonization is a technique for aligning the
color space of foreground objects with that of the back-
ground in a composite image. Recent state-of-the-art meth-
ods [32, 61, 26, 20, 9, 24, 10, 54, 18, 66, 5, 50, 7] are mostly
based on deep learning networks which have achieved more
promising results than traditional ones [29, 62, 42, 38, 53].
However, the performance of these methods, either per-
forming image harmonization in a supervised manner with
paired inharmonious/harmonized images [61, 26, 20, 9, 24,
10, 54, 18, 7] or constructing a zero-sum game by means
of GAN [17] technique [66, 5], is highly correlated with
the quality of the collection of composite images. If the
collected composite images could not cover real-world sit-
uations, then the trained network will fail to obtain satisfac-
tory results in practical use.
Compared with real image collection, the production of
a real-world composite image needs more labor costs, as
one should crop a foreground object from an image and
then paste it at a reasonable place in another one. To avoid
the unbearable cost of making a large real composite image
dataset, the current mainstream alternative [54, 5, 10] is to
“synthesize” composite images. Specifically, by applying a
variety of color transforms [42, 59, 39, 14, 30] on the se-
mantic regions of existing large-scale datasets [4, 33, 67],
we can obtain enough “synthesized” composite images, to-
gether with their paired harmonized images (i.e. the origi-
nal ones). The more transformations considered, the more
likely to get close to the real situation. However, this will
also result in a rather large dataset [10], which will place
much burden on the training phase. Such a costly approach
is in fact inconsistent with human behavior.
Let us think about how we humans harmonize an image:
Suppose we are familiar with the basic controls of an image
editing software (e.g., Photoshop). Given a composite im-
age, we first locate the inconsistency between foreground
and background (with the help of a mask). Then, based
on the foreground/background regions, we coarsely iden-
tify their possible imaging environment, which guides our
editing direction. By editing the foreground region and re-
peating the previous steps, we can often get a satisfactory
harmonious result at last. This process, without training on
large amounts of inharmonious data, harmonizes images ac-
tually in a zero-shot manner, and we attribute its success and
feasibility to human’s long-term strong prior. Specifically,
most of what we observe in our lives are real and harmo-
nious images, and if asked to imagine something, we would
tend to give a harmonious answer. Thus, an inharmonious
image is an outlier to the prior (possibly the reason why we
define it as “inharmonious”), and the above harmonization
process is to gradually pull this outlier closer to our prior.
There is just no need to know the distribution of composite
images. We display the human process in Fig. 2.
So, can we imitate human behavior in harmonizing im-
ages without relying on composite image collection? To
answer this question, we need to find out the “strong
prior” first, and recent large-scale text-to-image models
[47, 40, 64, 43] come into our sight. Trained on ex-
2

Long-Term Prior
Outlier Composite Image
Figure 2. Human behavior of harmonizing a composite image. We
humans can perform image harmonization relying only on our
long-term prior, without seeing many composite images in ad-
vance. E.g., we can harmonize the “over-bright” dog above.
tremely huge amounts of data, these generative models have
demonstrated to the world their extraordinary generative
power, and are intrinsically embedded with the prior of
real image distribution. Among them, the Stable Diffusion
[43], featuring its open source and fabulous results, gains
rather unprecedented attention, thus we base our method
on it. To guide the harmonization direction, we propose
an Attention-Constraint Text. This synthesized text, in-
spired by the Textual Inversion technique [16, 45], is op-
timized to restrict attention to foreground/background area,
thus well delineating their respective environments (just like
the aforementioned human behavior). We can then combine
the synthesized text with the existing text-guided local edit-
ing approaches [21, 36] to conduct the image harmoniza-
tion process. However, direct combination will cause severe
content distortion, making harmonization lose its meaning.
To this end, we exploit the self-attention maps in the Sta-
ble Diffusion and also leverage supplemental content text
to preserve the structure. The final framework is an itera-
tive harmonization loop (again similar to human behavior),
which can achieve high content retention and satisfactory
harmonized results. Fig. 1 displays some of our results.
In summary, our contributions are as follows:
• We introduce a new perspective on image harmoniza-
tion. By thinking about human behavior, we analyze
the feasibility of zero-shot image harmonization based
on the generative model prior.
• We propose a zero-shot image harmonization frame-
work. With the designed Attention-Constraint Text
and some operations for content retention, we can
achieve satisfactory results without relying on burden-
some composite image collection. This framework is
also highly consistent with human behavior.
• We conduct extensive experiments on various cases to
 demonstrate the effectiveness of the method. Further
applications are also explored such as artwork harmo-
nization, user auxiliary, etc., where our method has
great potential.
2. Related Works
Image harmonization. Compared with traditional
methods [29, 62, 42, 38, 53] that rely on matching low-
level statistics between foreground and background, Tsai et
al. [54] pioneered leveraging deep learning in image harmo-
nization and demonstrated the superiority of its powerful se-
mantic representation capability. To suffice the data-hungry
training of deep networks, Cong et al. [10] constructed a
large synthesized image harmonization dataset, by applying
a variety of color transforms [42, 59, 39, 14, 30] on segmen-
tation regions of the existing datasets [4, 33, 28]. Many of
the following works [61, 26, 9, 18, 50, 7] then based their
methods on this large synthesized dataset. Some other ap-
proaches have either tried to leverage different color trans-
forms [20, 24] or leverage the GAN [17] technique to ex-
plore harmonization with unpaired image data [5, 66], but
still heavily rely on transforming the segmented regions for
composite image collection. Although some promising re-
sults have been achieved, whether the synthesized compos-
ite images can reflect the real-world situation determines
the final generalization and performance of these methods.
With more color transformations considered, the trained
network can be more robust, yet the training cost also be
more unaffordable.
Text-to-image synthesis. Recently, we have witnessed
the incredible generative power of many large-scale text-
to-image models [41, 13, 47, 40, 64, 43]. Trained on vast
amounts of image-text data, these models have the real data
distribution embedded in themselves and gain the ability to
generate stunning and fabulous images that well follow the
given textual guidance. Among them, the diffusion mod-
els [47, 40, 43], an especially interesting class of genera-
tive model that performs denoising process iteratively from
a random Gaussian noise initial [51, 22], have achieved
rather unprecedented attention. Many downstream meth-
ods emerge and have implemented the diffusion model into
a variety of tasks, such as image restoration [57, 63], image
inpainting [31, 46], adversarial attack [6], and even recogni-
tion [1, 8]. While, the image editing application still covers
a great proportion. With the guidance from text [23], trained
classifier [12], etc., these methods [21, 36, 58, 3, 11, 60, 2]
achieve manipulating local content of the image while pre-
serving the remained area. Since they mainly aim to change
the local content structure, these methods are not suitable
for harmonizing an image. Methods of style transfer [27]
also cannot be adopted, as they tend to transfer the style be-
tween two images, whereas we focus on making one com-
posite image harmonious.
3
3. Preliminaries
Before introducing our proposed method, in this section,
we briefly review the involved techniques to better under-
stand what follows.
Diffusion model. Denoising diffusion probabilistic
models (DDPMs) [22] are a class of generative models that
perform an iterative image denoising process on a random
Gaussian noise initial. A U-Net [44] like neural network is
trained based on the minimization of the following objective
function:
L = Ex0 ,ϵ∼N (0,I),t ∥ϵ − ϵθ (xt , t)∥22 (1)
where ϵθ is the network for predicting the noise ϵ that is ap-
√ the input√clean image x0 with different intensities:
plied on
xt = αt x0 + 1 − αt ϵ, where αt are a series of fixed
hyper-parameters. By gradually denoising for T timesteps,
the diffusion models can then generate images following the
real data distribution from a randomly sampled noise image.
These models can further generalize to conditional genera-
tions, guided by image classes or texts, and the predicted
noise changes into ϵθ (xt , t, C), where C denotes the addi-
tional guidance.
Since the denoising process is conducted on the high-
dimensional image space repeatedly, the training and in-
ference of the diffusion models are unbearably expensive.
Thus Rombach et al. [43] proposed to map the input im-
age to a latent space with an autoencoder network before
conducting the forward and reverse process. They further
introduce the cross-attention mechanism [56] into the net-
work besides the self-attention one which is normally used
[51, 22, 37, 12], thus facilitating the guidance from various
modalities. By training on large amounts of image-text data
pairs [48], this work is later developed into the well-known
Stable Diffusion, which is open source and displays amaz-
ing generative performance.
DDIM and DDIM inversion. To accelerate sampling
of the diffusion models, Song et al. [52] generalize DDPMs
from a particular Markovian process to non-Markovian pro-
cesses (DDIM), which can lead to a deterministic sampling
given an initial noise xT :
 √ 
p xt − 1 − αt ϵθ (xt ) p
xt−1 = αt−1 √ + 1 − αt−1 ·ϵθ (xt )
αt
(2)
Since DDIM can be further taken as the Euler integration
for a particular ODE [52], it provides the feasibility of map-
ping a real image back to its corresponding initial latent by
reversing Eq. (2). This operation, named DDIM inversion,
paves the way for later editing of real images [11, 36].
Real image local editing. Recently, Hertz et al. [21]
deeply exploited the semantic strength of the cross-attention
maps in the large-scale text-to-image model [47] and pro-
posed Prompt-to-Prompt (abbreviated as P2P). Leveraging
 Iterative Harmonization Loop
Composite image Iteration 1 Iteration 𝑺

Harmonization Harmonization
Operations  Operations
𝒙𝑻 𝒙𝑻−𝟏 𝒙𝟏 𝒙𝟎 
DDIM Inversion

Optimization Harmonization Operations

“Autumn*”
Intermediate self-attention maps

“Autumn” “Girl winter*”

(Foreground) Text Guidance

Cross attention Mask

𝒙𝒕 Replace 𝒙𝒕−𝟏
Optimization on
“Winter” “Winter*” Intermediate self-attention maps
(Background) Background Mask
from “Girl autumn ” *
(a) Attention-Constraint Text (b) Content Retention
Figure 3. The pipeline of our method. The harmonization process is an iterative loop, which is aligned with human behavior. Starting
from the composite image, we invert the last harmonized result into its diffusion latent and then leverage some harmonization operations
to obtain the next harmonized result. Among the harmonization operations, (a) Attention-Constraint Text optimization is designed to get
texts that can describe the foreground/background environment better. For the (b) foreground content retention, we leverage self-attention
maps and supplemental content text. (We also utilize P2P [21] and Null-Text [36] technologies. For the sake of brevity, the above figure
does not explicitly draw that part, and we refer to their work for more details.)

the strong correlations between the image pixels and the text
tokens, they achieved local image editing by fixing the spe-
cific cross-attention maps while changing the prompt texts.
However, when applied to real images, it fails to recon-
struct the original image well. This problem was later ex-
plored by Mokady et al. [36], where they successfully trans-
ferred P2P to real images by optimizing the null token em-
beddings in the classifier-free guidance [23] (we abbreviate
their method as Null-Text).
4. Method
With the techniques mentioned in Sec. 3, given an input
real image, we can leverage DDIM inversion [52] to obtain
its initial latent, optimize the unconditional token embed-
dings to well reconstruct the input [36], and finally resort to
P2P for real image local editing [21]. Thus, we can come
up with a straightforward approach for image harmoniza-
tion: First describe the environments of the foreground and
the background, each with one text. Based on P2P [21],
we can fix the cross-attention maps between the foreground
text and the image pixels, and change the foreground text to
background text. Then, it is expected that the background
text can guide the editing of the foreground area environ-
ment, thus achieving image harmonization.
However, from our experiments (see Sec. 5.2), the
above straightforward approach suffers from two problems.
Firstly, it is arduous to find two words that can accurately
represent the environments of the foreground and the back-
ground. For an image composed of rich elements, even an
experienced and well-literated person has to make some ef-
fort on it. Since only two words are quite limited, one might
consider using more words to improve the representation,
yet it would introduce a new problem of different matching
orders between foreground and background texts. The other
problem is content distortion. When leveraging P2P [21]
for image harmonization, there are usually nontrivial dis-
tortions in the foreground content structure. This would go
against the purpose of image harmonization, where we hope
to harmonize the image while preserving the foreground
content structure as much as possible.
To tackle the problems, we propose a framework (as de-
picted in Fig. 3), where two Attention-Constraint Texts are
optimized to well describe the image environments, and
some operations are exploited to keep the content struc-
ture. The pipeline can achieve harmonious results by iterat-
ing the harmonization loop. With the diffusion model itself
frozen and only the text embeddings optimized, the method
is mainly based on the generative model prior without re-
lying on a large number of composite training images, thus
is highly consistent with the human behavior (see Sec. 1).
Here, we turn to describe more details of the framework.
4.1. Attention-Constraint Text
From Fig. 4, we can observe the amazing generative
power of the Stable Diffusion [43] that can create images

4
 t=T t=0
Diffusion
timestep
“ Bear ”
“ Mars ”
“A bear walking on Mars”
Figure 4. Given the prompt text “A bear walking on Mars”, the
Stable Diffusion can generate an image well aligned with the se-
mantic information. Visualizing the intermediate cross-attention
maps between text and images, we can observe that there are clear
correlations even at the earlier steps.
well reflecting the text guidance. If visualizing the cross-
attention maps in the diffusion process, we would be sur-
prised to find that even at the earlier timesteps with quite
noisy images, there are obvious correlations between the
text and the image pixels. The model seems to have de-
cided which image area for which text just from the ini-
tial diffusion timestep. Thus based on this, Hertz et al.
[21] can achieve local image editing by changing text while
maintaining cross-attention maps. Then, to achieve image
harmonization, we can leverage a word whose attention is
mainly constrained to the foreground area of the composite
image, and replace it with another word that can illustrate
the background environment.
Let’s leave the foreground word alone for now and fo-
cus on the background one first. In the text-to-image pro-
cess, the Stable Diffusion does not leverage the guidance
from texts directly, but in fact utilizes the text embeddings
that the texts correspond to. Thus, we may not be able to
well illustrate the environment with only one word even
consulting the entire dictionary, yet it is possible to find a
high-dimensional representative text embedding. Accord-
ing to the aforementioned correlations between texts and
image pixels, here we propose a coarse quantitative indi-
cator, based on cross-attention maps, to evaluate whether a
text embedding can represent the environment. Specifically,
we can take the embedding as an approximation of the envi-
ronment if the attention is well focused on the background
region, neither overflowing nor underfilling. To achieve
that, we can optimize a text embedding as follows:
Att(Emb)
LEmb = ∥M − ∥2 . (3)
max(Att(Emb)) 2
where Emb denotes the optimized text embedding, M rep-
resents the background mask, and Att is the cross-attention
map between the embedding and the image pixels. To align
5
with the value range of the mask, we normalize the atten-
tion map with its maximum value. In practice, we can just
coarsely describe the background environment with an ini-
tial word and then minimize Eq. (3) to get a more represen-
tative refined embedding, thus facilitating a lot. To avoid
the overfitting of the optimization, we add a regularization
term to get the refined embedding close to its initial:
min(LEmb + w × ∥Emb − Embinit ∥22 ). (4)
Emb
where Embinit is the embedding of the initial text that
coarsely illustrate the environment, and w is a hyper-
parameter to balance the two terms. We coin the obtained
embedding as Attention-Constraint Text.
With the optimized embedding for the background, we
can then replace the word whose attention is fixed on the
foreground area. One may consider leveraging a noun word
describing the foreground content, such as “girl”, “car”,
“dog”, etc., as these words focus well on foreground regions
while being easier to think of than environment words.
However, we found that substitutions of these words fail
to achieve harmonious results (see Fig. 7), probably due to
the lack of environmental information as guidance. Thus,
practically we also optimize the Attention-Constraint Text
for the foreground, based on an initial coarse description of
its environment.
As inspired by Textual Inversion [16, 45], our method
shares a similarity with them in that the Attention-
Constraint Text also comes from the optimization of the
text embedding. However, there are essentially many dif-
ferences. From the perspective of goals, [16, 45] aim to
abstract specific objects from multiple images and further
generate richer images with the optimized text, while our
method is based on only one composite image and the tar-
get is to harmonize the image. From the perspective of im-
plementations, [16, 45] optimize the text by reconstructing
multiple input images, while ours constrains the attention
of the text to specific regions (foreground/background) to
better represent the image environments.
4.2. Content retention
To achieve image harmonization while avoiding much
content distortion, we here preserve the content structure
from two aspects.
Self-attention exploitation. Although Hertz et al. [21]
have been successful in local editing of images, we found
that even if we changed the text of the environment but
not the object itself, the content structure would distort a
lot (see Fig. 8). Different from P2P [21] dependent on the
cross attention, we here leverage the self-attention maps to
achieve high content retention. As demonstrated in [49, 55]
previously, the self-similarly-based descriptors can achieve
structure capturing while ignoring appearance information.
By fixing the self-attention maps in the Stable Diffusion
firstly for the foreground environment text and then replac-
ing the text while keeping the cross-attention maps as in
[21], we can achieve the goal of better content retention.
By utilizing the self-attention mechanism, we in fact restrict
each iteration of editing to small changes (see Fig. 3), just
like human behavior. Thus, in practical use, we apply the
diffusion editing several times until getting a satisfactory
harmonized result.
Supplemental text for foreground content. As men-
tioned in Sec. 4.1, we currently only require one envi-
ronment text for foreground and background respectively.
However, in our experiments (Fig. 8), we found that only
the environment text failed to well preserve the foreground
structure, even when we have leveraged the self attention.
Furthermore, we also noticed that leveraging only the en-
vironment text would pose more challenges (need longer
optimization of unconditional embedding with [36]) on the
reconstructing of the foreground area.
We attribute these phenomena to the lack of text guid-
ance. Specifically, since the environment text did not con-
tain information of image content, when we forced the
DDIM-inversed latent to reconstruct its input as [36], we
actually relied on the optimized unconditional embedding
to represent the content information. This optimized un-
conditional embedding, only specific to its paired environ-
ment text (foreground), has no generalization to another
environment text (background). Thus, when replaced the
foreground environment text with the background one, we
fail to preserve the foreground content information. This
then causes the content distortion. Therefore, we supple-
ment another text for the foreground content besides the en-
vironment one. Since the context text embedding is well
trained before with a huge number of images [43], it can
easily adapt to pair with other texts, and thus achieve high
content retention (see Fig. 8).
It should be also noted that, the resulted content reten-
tion here with the supplemental text has no relation with the
fixation of the cross-attention maps in P2P [21]. To verify
this, we conduct experiments in Fig. 8 where only the self
attention is utilized. It can be observed that the content is
still highly preserved without fixing cross-attention maps.
5. Results
We here start by comparing our method with the exist-
ing image harmonization methods, both qualitatively and
quantitatively. Then, we conduct the ablation study of the
designs in our method. Some further applications are ex-
plored at last. More implementation details of our method
can be found in the Supplementary Material.
5.1. Comparisons
To demonstrate our superiority, in Fig. 5 we display the
harmonized results of our method and the current state-of-
6
the-art harmonization methods [18, 61, 26, 19]. Since our
method is not trained on a large number of synthesized
composite images like theirs, for fairness and practicality,
comparisons are performed on 158 real composite images
collected from Flickr by ourselves (some examples are dis-
played in Fig. 1) and from [9, 20]. From the qualitative
results, we can observe that our method, even without re-
lying on heavy training, has achieved comparable or even
better (e.g., Row 3 in Fig. 5) performance to other methods,
demonstrating the effectiveness of our zero-shot approach.
As ground-truth harmonized results are unavailable for
the real composite images, following [16, 36, 2, 58, 60],
we conduct a subjective user study for the quantitative
comparisons. Specifically, we constructed a questionnaire
where 29 participants are invited to select visually harmo-
nious images. Each time, participants will be given two
images, either the results of the harmonization methods
or the original composite image. The displaying order is
randomly shuffled and the participants are to pick up the
more harmonious one between the two. A total of 22,910
votes are collected. From the voting percentages shown in
Fig. 6, we can observe that our zero-shot approach success-
fully achieves comparable results to existing harmonization
methods trained on a large number of composite images,
demonstrating the effectiveness and superiority of our work.
5.2. Ablation Study
We here conduct ablation studies on the designs in
Sec. 4. For more ablation experiments and analyses, please
refer to the Supplementary Material.
Ablation of Attention-Constraint Text. In Fig. 7,
we have demonstrated the effectiveness of the proposed
Attention-Constraint Text. Without the optimization, it can
be seen from the attention maps that the initial text fails to
well illustrate the environments of foreground/background
areas, resulting in a deviated harmonization direction. The
presence of initial text is also important, otherwise we can-
not obtain satisfactory results with text embeddings opti-
mized from scratch. Furthermore, we verified the infeasi-
bility of substituting environment text with content text for
the foreground mentioned in Sec. 4.1 which fails to achieve
harmonious results.
Ablation of content retention designs. From Fig. 8,
we can observe that with the self attention leveraged, the
content structure is largely preserved. This is further im-
proved when adding supplemental content text. To verify
that the improvement does not come from the fixation of
cross-attention maps in P2P [21], we show the results with
and without fixed cross-attention. As shown in Fig. 8, we
can see that there is almost no difference before and after,
validating our statement in Sec. 4.2.
 Composite Mask Intrinsic IHT Harmonizer DCCF Ours

Summer*
Spring*

Bright*
Dim*

Bright*
Warm*

Cold*
Sunset*

Bright*
Cloudy*

Bright*
Dark*

Bright*
Autumn*

Figure 5. Qualitative comparisons with other state-of-the-art harmonization methods. From left to right are composite images, foreground
masks, and the harmonized results of Intrinsic [19], IHT [18], Harmonizer [26], DCCF [61], and ours. Please zoom in for a better view.
The rightmost column is the foreground/background environment text, where the asterisk in the upper right corner indicates the Attention-
Constraint optimization. (For simplicity, we here ignore the supplemental content text.)

Voting Percentage
𝟕𝟓. 𝟒𝟓% 𝟐𝟒. 𝟓𝟓% Composite

𝟒𝟒. 𝟏𝟏% 𝟓𝟓. 𝟖𝟗% IntrinsicIH Girl sunny* Girl sunny Girl (empty)* Girl girl
Foreground Text
and Cross-attention:
Ours 𝟒𝟓. 𝟖𝟑% 𝟓𝟒. 𝟏𝟕% IHT
Background Text: “Girl dark*” “Girl dark” “Girl (empty)*” “Girl dark*”

𝟓𝟗. 𝟒𝟓% 𝟒𝟎. 𝟓𝟓% Harmonizer

𝟓𝟔. 𝟑𝟑% 𝟒𝟑. 𝟔𝟕% DCCF

Figure 6. User study of the harmonization performance. Users are

invited to select the more harmonious one between two images.
Here, we display the voting percentages of different methods [61, Foreground Text
Girl autumn* Girl autumn Girl (empty)* Girl girl

26, 19, 18]. “Composite” denotes the original composite image. and Cross-attention:

Background Text: “Girl winter*” “Girl winter” “Girl (empty)*” “Girl winter*”
5.3. Applications (a) Final (b) W/o text (c) W/o initial (d) Double
optimization text content text

In this subsection, we go a further step to explore the ap-
plications of our method. We mainly focus on two: artwork
harmonization, and user auxiliary.
Artwork harmonization. Besides the harmonized re-
sults on the natural composite images displayed in Sec. 5.1,
we try to generalize our method to composite images with
a large style gap between the foreground and background.
Here, we evaluate the performance on some artistic pic-
Figure 7. Ablation of the Attention-Constraint Text. The
Attention-Constraint optimization and the initial text are ablated.
We further verify the infeasibility of utilizing double content text
for the foreground (mentioned in Sec. 4.1). Please zoom in for a
better view.
tures. To achieve better results, we slightly relax the con-
straint on content retention by utilizing self-attention maps

7

(a) Composite image (b) W/o self attention (c) W/o supplemental
W/o supplemental text text
Figure 8. Ablation of the operations for content retention. We ab-
late the utilization of self-attention maps and the supplemental text
here. We also ablate the utilization of cross-attention maps in P2P
[21] to verify the effectiveness of our operations (mentioned in
Sec. 4.2). Please zoom in for a better view.
“Cat/House/Bird/Plane photo* painting*”
Figure 9. Further application of artwork harmonization. We ex-
plore generalizing our method on artwork. The small image in the
lower left corner of each image is the original composite image.
Please zoom in for a better view.
only in certain diffusion timesteps (for natural images all
timesteps are considered). From Fig. 9, it can be seen
that our method still succeeds in harmonizing the artwork
images. We attribute this success to the powerful gen-
erative model prior. As mentioned in Sec. 1, a human
mainly relies on his long-term prior on harmonious im-
ages to conduct the image harmonization process, and can
often lead to harmonious results. Similar to that, thanks
to the extremely large amount of real-world training data
[41, 13, 47, 40, 64, 43], the pretrained large-scale genera-
tive models inherently have that prior. Thus, our method,
based on the generative model prior, can gain better gener-
3D LUT
(d) Final (e) W/o cross attention
Figure 10. Further application as user auxiliary. With our iterative
harmonization process, the intermediate operation of the network
can be extracted, which can be leveraged as guidance for the hu-
man process.
alization to a variety of image types, while the existing ap-
proaches [18, 9, 61, 20, 26, 24, 7] are limited to be applied
to the specific scope of the synthesized composite images
seen in their training phase.
User auxiliary. As depicted in Fig. 3, our method is an
iterative harmonization process. Each time the composite
image is harmonized a little. Thus, compared with the one-
step methods [20, 18, 10, 54], the user observes all the in-
termediate results, which can bring many benefits and lead
the method to be a white-box approach. Specifically, due to
the trivial content distortion between the results of two adja-
cent iterations, we can optimize a 3D LUT [65, 25], which
is a lookup table that maps RGB values between two im-
ages. By predicting either a global 3D LUT for the entire
foreground region or multiple local LUTs for foreground
meaningful parts such as faces, we can reveal the latent op-
erations of the model, which in turn can provide guidance
for an image editor to harmonize images. The process is
illustrated in Fig. 10.
6. Conclusions
In this work, we introduce the feasibility of imitating hu-
man behavior and propose a zero-shot image harmonization
method. Without relying on burdensome training of a huge
amount of composite image data, we can achieve satisfac-
tory harmonized results only with the prior of a pretrained
generative model and the guidance of the text describing the
image environment. The effectiveness of our method has
been validated through a variety of cases. Even for com-
posite images with large inconsistencies between the back-
ground and the foreground such as artistic ones, our method
can also lead to a harmonious result. Furthermore, we have
deeply discussed the limitations of the method and future
work (please refer to the Supplementary Material). We hope
that our method can pave the way for further research on
zero-shot image harmonization.
8
References
[1] Tomer Amit, Eliya Nachmani, Tal Shaharbany, and Lior
Wolf. Segdiff: Image segmentation with diffusion proba-
bilistic models. arXiv preprint arXiv:2112.00390, 2021. 3
[2] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended
diffusion for text-driven editing of natural images. In Pro-
ceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 18208–18218, 2022. 3, 6
[3] Manuel Brack, Patrick Schramowski, Felix Friedrich, Do-
minik Hintersdorf, and Kristian Kersting. The stable artist:
Steering semantics in diffusion latent space. arXiv preprint
arXiv:2212.06013, 2022. 3
[4] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo
Durand. Learning photographic global tonal adjustment with
a database of input/output image pairs. In CVPR 2011, pages
97–104. IEEE, 2011. 2, 3
[5] Bor-Chun Chen and Andrew Kae. Toward realistic image
compositing with adversarial learning. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 8415–8424, 2019. 1, 2, 3
[6] Jianqi Chen, Hao Chen, Keyan Chen, Yilan Zhang, Zhengxia
Zou, and Zhenwei Shi. Diffusion models for impercep-
tible and transferable adversarial attack. arXiv preprint
arXiv:2305.08192, 2023. 3
[7] Jianqi Chen, Yilan Zhang, Zhengxia Zou, Keyan Chen,
and Zhenwei Shi. Dense pixel-to-pixel harmonization
via continuous image representation. arXiv preprint
arXiv:2303.01681, 2023. 1, 2, 3, 8
[8] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffu-
siondet: Diffusion model for object detection. arXiv preprint
arXiv:2211.09788, 2022. 3
[9] Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong
Gao, Qihao Sun, and Liqing Zhang. High-resolution im-
age harmonization via collaborative dual transformations. In
Proceedings of the IEEE/CVF Conference on Computer Vi-
sion and Pattern Recognition, pages 18470–18479, 2022. 1,
2, 3, 6, 8
[10] Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling,
Weiyuan Li, and Liqing Zhang. Dovenet: Deep image
harmonization via domain verification. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 8394–8403, 2020. 1, 2, 3, 8
[11] Guillaume Couairon, Jakob Verbeek, Holger Schwenk,
and Matthieu Cord. Diffedit: Diffusion-based seman-
tic image editing with mask guidance. arXiv preprint
arXiv:2210.11427, 2022. 3
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models
beat gans on image synthesis. Advances in Neural Informa-
tion Processing Systems, 34:8780–8794, 2021. 3
[13] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng,
Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao,
Hongxia Yang, et al. Cogview: Mastering text-to-image gen-
eration via transformers. Advances in Neural Information
Processing Systems, 34:19822–19835, 2021. 3, 8, 12
[14] Ulrich Fecker, Marcus Barkowsky, and André Kaup.
Histogram-based prefiltering for luminance and chromi-
nance compensation of multiview video. IEEE Transactions
9
on Circuits and Systems for Video Technology, 18(9):1258–
1267, 2008. 2, 3
[15] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun
Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang,
and William Yang Wang. Training-free structured diffusion
guidance for compositional text-to-image synthesis. arXiv
preprint arXiv:2212.05032, 2022. 13
[16] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash-
nik, Amit H Bermano, Gal Chechik, and Daniel Cohen-
Or. An image is worth one word: Personalizing text-to-
image generation using textual inversion. arXiv preprint
arXiv:2208.01618, 2022. 2, 5, 6
[17] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing
Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and
Yoshua Bengio. Generative adversarial nets. In Z. Ghahra-
mani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Wein-
berger, editors, Advances in Neural Information Processing
Systems, volume 27. Curran Associates, Inc., 2014. 2, 3, 13
[18] Zonghui Guo, Zhaorui Gu, Bing Zheng, Junyu Dong, and
Haiyong Zheng. Transformer for image harmonization and
beyond. IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence, 2022. 1, 2, 3, 6, 7, 8
[19] Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu,
and Bing Zheng. Intrinsic image harmonization. In Pro-
ceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 16367–16376, 2021. 6, 7
[20] Yucheng Hang, Bin Xia, Wenming Yang, and Qingmin Liao.
Scs-co: Self-consistent style contrastive learning for image
harmonization. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 19710–
19719, 2022. 1, 2, 3, 6, 8
[21] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman,
Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im-
age editing with cross attention control. arXiv preprint
arXiv:2208.01626, 2022. 2, 3, 4, 5, 6, 8, 12
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu-
sion probabilistic models. Advances in Neural Information
Processing Systems, 33:6840–6851, 2020. 3
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion
guidance. In NeurIPS 2021 Workshop on Deep Generative
Models and Downstream Applications, 2021. 3, 4, 12
[24] Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe
Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi,
Sarah Kong, and Zhangyang Wang. Ssh: A self-supervised
framework for image harmonization. In Proceedings of the
IEEE/CVF International Conference on Computer Vision,
pages 4832–4841, 2021. 1, 2, 3, 8
[25] Hakki Can Karaimer and Michael S Brown. A software
platform for manipulating the camera imaging pipeline. In
European Conference on Computer Vision, pages 429–444.
Springer, 2016. 8
[26] Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, and Rynson WH
Lau. Harmonizer: Learning to perform white-box image and
video harmonization. In European Conference on Computer
Vision, pages 690–706. Springer, 2022. 1, 2, 3, 6, 7, 8
[27] Gihyun Kwon and Jong Chul Ye. Diffusion-based image
translation using disentangled style and content representa-
tion. arXiv preprint arXiv:2209.15264, 2022. 3
[28] Pierre-Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian,
and James Hays. Transient attributes for high-level under-
standing and editing of outdoor scenes. ACM Transactions
on graphics (TOG), 33(4):1–11, 2014. 3
[29] Jean-Francois Lalonde and Alexei A Efros. Using color com-
patibility for assessing image realism. In 2007 IEEE 11th
International Conference on Computer Vision, pages 1–8.
IEEE, 2007. 2, 3
[30] Joon-Young Lee, Kalyan Sunkavalli, Zhe Lin, Xiaohui Shen,
and In So Kweon. Automatic content-aware color and
tone stylization. In Proceedings of the IEEE conference on
computer vision and pattern recognition, pages 2470–2478,
2016. 2, 3
[31] Wenbo Li, Xin Yu, Kun Zhou, Yibing Song, Zhe Lin, and
Jiaya Jia. Sdm: Spatial diffusion model for large hole image
inpainting. arXiv preprint arXiv:2212.02963, 2022. 3
[32] Jingtang Liang, Xiaodong Cun, Chi-Man Pun, and Jue
Wang. Spatial-separated curve rendering network for effi-
cient and high-resolution image harmonization. In European
Conference on Computer Vision, pages 334–349. Springer,
2022. 1
[33] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
European conference on computer vision, pages 740–755.
Springer, 2014. 2, 3
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay
regularization. arXiv preprint arXiv:1711.05101, 2017. 12
[35] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan
Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffu-
sion probabilistic model sampling in around 10 steps. arXiv
preprint arXiv:2206.00927, 2022. 13
[36] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch,
and Daniel Cohen-Or. Null-text inversion for editing real
images using guided diffusion models. arXiv preprint
arXiv:2211.09794, 2022. 2, 3, 4, 6, 12
[37] Alexander Quinn Nichol and Prafulla Dhariwal. Improved
denoising diffusion probabilistic models. In International
Conference on Machine Learning, pages 8162–8171. PMLR,
2021. 3
[38] Francois Pitie, Anil C Kokaram, and Rozenn Dahyot. N-
dimensional probability density function transfer and its ap-
plication to color transfer. In Tenth IEEE International Con-
ference on Computer Vision (ICCV’05) Volume 1, volume 2,
pages 1434–1439. IEEE, 2005. 2, 3
[39] François Pitié, Anil C Kokaram, and Rozenn Dahyot. Au-
tomated colour grading using colour distribution transfer.
Computer Vision and Image Understanding, 107(1-2):123–
137, 2007. 2, 3
[40] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu,
and Mark Chen. Hierarchical text-conditional image gen-
eration with clip latents. arXiv preprint arXiv:2204.06125,
2022. 2, 3, 8, 12
[41] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray,
Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.
Zero-shot text-to-image generation. In International Confer-
ence on Machine Learning, pages 8821–8831. PMLR, 2021.
3, 8, 12
10
[42] Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter
Shirley. Color transfer between images. IEEE Computer
graphics and applications, 21(5):34–41, 2001. 2, 3
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz,
Patrick Esser, and Björn Ommer. High-resolution image
synthesis with latent diffusion models. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 10684–10695, 2022. 2, 3, 4, 6, 8, 12, 13
[44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-
net: Convolutional networks for biomedical image segmen-
tation. In International Conference on Medical image com-
puting and computer-assisted intervention, pages 234–241.
Springer, 2015. 3
[45] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch,
Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine
tuning text-to-image diffusion models for subject-driven
generation. arXiv preprint arXiv:2208.12242, 2022. 2, 5
[46] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee,
Jonathan Ho, Tim Salimans, David Fleet, and Mohammad
Norouzi. Palette: Image-to-image diffusion models. In
ACM SIGGRAPH 2022 Conference Proceedings, pages 1–
10, 2022. 3
[47] Chitwan Saharia, William Chan, Saurabh Saxena, Lala
Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed
Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi,
Rapha Gontijo Lopes, et al. Photorealistic text-to-image
diffusion models with deep language understanding. arXiv
preprint arXiv:2205.11487, 2022. 2, 3, 8, 12
[48] Christoph Schuhmann, Romain Beaumont, Richard Vencu,
Cade Gordon, Ross Wightman, Mehdi Cherti, Theo
Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts-
man, et al. Laion-5b: An open large-scale dataset for
training next generation image-text models. arXiv preprint
arXiv:2210.08402, 2022. 3
[49] Eli Shechtman and Michal Irani. Matching local self-
similarities across images and videos. In 2007 IEEE Con-
ference on Computer Vision and Pattern Recognition, pages
1–8. IEEE, 2007. 5
[50] Konstantin Sofiiuk, Polina Popenova, and Anton Konushin.
Foreground-aware semantic representations for image har-
monization. In Proceedings of the IEEE/CVF Winter Confer-
ence on Applications of Computer Vision, pages 1620–1629,
2021. 1, 3
[51] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan,
and Surya Ganguli. Deep unsupervised learning using
nonequilibrium thermodynamics. In International Confer-
ence on Machine Learning, pages 2256–2265. PMLR, 2015.
3
[52] Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models. arXiv preprint
arXiv:2010.02502, 2020. 3, 4, 12, 13
[53] Kalyan Sunkavalli, Micah K Johnson, Wojciech Matusik,
and Hanspeter Pfister. Multi-scale image harmonization.
ACM Transactions on Graphics (TOG), 29(4):1–10, 2010.
2, 3
[54] Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli,
Xin Lu, and Ming-Hsuan Yang. Deep image harmonization.
 In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 3789–3797, 2017. 1, 2, 3, 8
[55] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali
Dekel. Splicing vit features for semantic appearance transfer.
In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 10748–10757, 2022.
5
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. Advances in neural
information processing systems, 30, 2017. 3
[57] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot im-
age restoration using denoising diffusion null-space model.
arXiv preprint arXiv:2212.00490, 2022. 3
[58] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya
Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and
Shiyu Chang. Uncovering the disentanglement capabil-
ity in text-to-image diffusion models. arXiv preprint
arXiv:2212.08698, 2022. 3, 6
[59] Xuezhong Xiao and Lizhuang Ma. Color transfer in cor-
related color space. In Proceedings of the 2006 ACM in-
ternational conference on Virtual reality continuum and its
applications, pages 305–309, 2006. 2, 3
[60] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun
Zhang. Smartbrush: Text and shape guided object inpaint-
ing with diffusion model. arXiv preprint arXiv:2212.05034,
2022. 3, 6
[61] Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia, Binqiang
Zhao, and Xing Tang. Dccf: Deep comprehensible color fil-
ter learning framework for high-resolution image harmoniza-
tion. In European Conference on Computer Vision, pages
300–316. Springer, 2022. 1, 2, 3, 6, 7, 8
[62] Su Xue, Aseem Agarwala, Julie Dorsey, and Holly Rush-
meier. Understanding and improving the realism of image
composites. ACM Transactions on graphics (TOG), 31(4):1–
10, 2012. 2, 3
[63] Yueqin Yin, Lianghua Huang, Yu Liu, and Kaiqi Huang.
Diffgar: Model-agnostic restoration from generative arti-
facts using image-to-image diffusion models. arXiv preprint
arXiv:2210.08573, 2022. 3
[64] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun-
jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin-
fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres-
sive models for content-rich text-to-image generation. arXiv
preprint arXiv:2206.10789, 2022. 2, 3, 8, 12
[65] Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, and Lei Zhang.
Learning image-adaptive 3d lookup tables for high perfor-
mance photo enhancement in real-time. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 2020. 8
[66] Fangneng Zhan, Shijian Lu, Changgong Zhang, Feiying Ma,
and Xuansong Xie. Adversarial image composition with
auxiliary illumination. In Proceedings of the Asian Confer-
ence on Computer Vision, 2020. 1, 2, 3
[67] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi-
dler, Adela Barriuso, and Antonio Torralba. Semantic under-
standing of scenes through the ade20k dataset. International
Journal of Computer Vision, 127(3):302–321, 2019. 2

11
A. Overview
In the appendix, we will first introduce more implemen-
tation details of our proposed method in Appendix B. Then,
in Appendix C, we conduct more ablation studies, includ-
ing the existence of the regularization term in the optimiza-
tion, the prompt text format and order, and the number of
harmonization iterations. We give a deep analysis of our
method’s limitations and future work in Appendix D. Some
failure cases are displayed in Appendix E, and further trials
on content retention are demonstrated in Appendix F at last.

B. Implementation Details
(a) Composite image (b) W/o regularization (c) Final
We base our method on the pretrained Stable Diffusion Figure 11. Ablation of the regularization term in the Attention-
[43] and images with 512×512 resolution. The DDIM sam- Constraint Text optimization.
pling schedule [52] is leveraged for a deterministic diffusion
process, and the number of the diffusion timesteps is set to “A girl in the style of
bright* sunset*”
“Bright* girl” “Girl bright*”
sunset*
Sunset*
50. We remove the conditional guidance in DDIM inversion
by setting the guidance scale to 0, while in the reverse pro-
cess, the guidance scale in the classifier-free guidance [23]
is set to 2.5.
The optimization of the Attention-Constraint Text is “A girl in the style of “Autumn* girl” “Girl autumn*”
autumn* winter*” Winter* winter*
based on the cross-attention maps in the diffusion process,
and we have explored two kinds of implementations: opti-
mizing style and training style. The former is to optimize
the text embedding in each diffusion timestep, similar to
[36]. In each step, we optimize the embedding several times
based on the embedding out of the last step. Thus we will
get a series of optimized embeddings when finishing the fi- (a) Composite image (b) Formal format (c) Order reverse (c) Final
nal timestep. Different from this, the latter implementation Figure 12. Ablation of the format and order of the initial text. The
is to train one embedding. By taking all the timesteps as a effect of formal/simple text is explored, together with whether the
dataset, we can iteratively sample from the dataset a batch to environment text is placed at the end. Please zoom in for a better
train the embedding. When adopting AdamW [34] with an view.
initial learning rate 1e−3 (1e−2 for training style), setting w
in Eq.(4) in the main paper to 5000 (1000 for training style),
and optimizing two times in each timestep (train 50 epochs tion of the text embedding. We ablate the regularization in
with batch size set to 4), we found these two implementa- Fig. 11, and the results have demonstrated its effectiveness.
tions can both achieve satisfactory harmonized results. If With the regularization, the text embedding will not go far
not specified, we leverage the optimizing style in the con- away from the initial and thus play a better guiding role.
text.
We also adopt AdamW [34] in the reconstruction process C.2. Prompt text format
[36] with the learning rate 1e−1 . Following [36], we utilize Currently, we just leverage one environment text and one
an increasing strategy, starting from one iteration with an content text for the foreground/background. However, it can
increment for every ten timesteps, to decide the iterations of be observed that the given examples of the large-scale text-
the unconditional embedding optimization in each diffusion to-image models [41, 13, 47, 40, 64, 43] often leverage a
timestep. All the experiments are run on a single RTX 3090 more complete and formal sentence to describe a picture.
GPU. Thus, we here explore the effect of different prompt text
formats. As shown in Fig. 12, we can see that the cur-
C. More Ablation Studies rently used simplified texts can achieve better results. We
believe that this is due to the restrictions imposed by long
C.1. Regularization of Attention-Constraint Text
prompt texts. When utilizing cross-attention maps in P2P
As mentioned in Sec. 4.1 of the main paper, we leverage [21], more texts will bring more constraints on image con-
an additional regularization term to restrict the optimiza- tent, while the simple text format will facilitate the harmo-

12
 1 2
6 7
“Girl autumn*”
winter*
Harmonization Iterations
Figure 13. Ablation of the number of harmonization iterations. More iterations allow for more impact from the background environment,
yet also more content distortion. The number at the upper right corner of the image denotes the iteration number.
nization a lot.
C.3. Prompt text order
Interestingly, we found that different orders of prompt
texts lead to some differences in the results. When the envi-
ronment text is placed before the content text, there is a drop
in the performance (see Fig. 12). Deep into the structure of
the Stable Diffusion [43], we found that this is because of
the usage of the casual attention mask, with which the latter
text token will be blended with the semantics of the formal
token. Thus, if place the content text behind, it will be af-
fected by the optimization of the former environment text,
which will then lead to the failure of the content retention
mentioned in Sec. 4.2 of the main paper. This phenomenon
is also consistent with the findings by [15].
C.4. Harmonization iterations
As our method is an iterative process, here we explore
how many iterations can lead to a stable harmonious result.
From Fig. 13, we can see that there is a tradeoff between
harmony and content retention. More iterations allow for
more impact from the background environment, yet also
more content distortion. Practically, we set the number of
iterations to 10 and select the best result from them.
D. Limitations and Future Work
While we can achieve image harmonization without
training on a large number of composite images, our method
is still subject to some limitations to be addressed in follow-
up work.
13
3 4 5
8 9 10
The first limitation comes from the time cost. Due to the
progressive denoising process of the diffusion model where
a full sampling involves multiple timesteps, the diffusion it-
self is much more time-consuming [52, 43] compared with
other generative models such as GAN [17]. As mentioned
in Sec. 4 in the main paper, our method can be viewed as a
hyper-diffusion with iterative diffusion samplings, harmo-
nizing the composite image a little each time. Thus, the
harmonization process will cost a lot of time. By leveraging
recent efficient sampling strategies [52, 35], we can largely
alleviate the burden of time cost, yet it still struggles to ap-
ply the method for real-time applications. Further acceler-
ation of the diffusion process needs to be explored for this
problem.
Another limitation is content retention. For image har-
monization, the hope is to harmonize an image while keep-
ing its content structure. Thus in Sec. 4.2 in the main paper,
we have exploited the self attention and the supplemental
text which can achieve high content retention. However, we
also found that for some specific cases (see Appendix E),
the distortion may still remain severe. A few further ap-
proaches have been tried, yet with little success (see Ap-
pendix F).
Moreover, as mentioned in Sec. 4.1 in the main paper,
the Attention-Constraint Text is optimized based on an ini-
tial text, which is provided by users to coarsely illustrate the
environments of the image. Although for many images we
can easily come up with the initial text, there are still some
that we may need to think a bit more, which is then incon-
venient. Also, if the prompt text given by the user deviates
greatly, the result will not be satisfactory (see Appendix E).
The harmonization results depend to some extent on the ini-
 (a) Failure case: Content distortion
Sunny* Dim* Bright* Sunset*
(b) Failure case: Misleading text
Figure 14. We here display two representative failure cases. One is
the content distortion when faced with huge inconsistency between
foreground and background. The other is the deviated result by
misleading text.
tial text.
One avenue for future work is to provide initial text au-
tomatically. This can be achieved by training a network
for generating environment texts. By constructing a dataset
with pairs of images and texts describing the environment,
the whole harmonization process can be freed from human
intervention and thus facilitated.
In addition, more objective quantitative indicators need
to be explored in the future. As described in Sec. 5.1 in the
main paper, we currently resort to subjective user surveys
to quantitatively evaluate the performance of the harmo-
nization methods on real-world composite images. Such an
evaluation approach is labor-intensive and time-consuming.
Therefore, future research should also focus on this part.
E. Failure Cases
In Fig. 14, we have demonstrated the limitations of our
method mentioned in Appendix D. We can observe that for
images with huge inconsistencies between the foreground
and the background, the content structure distorts more eas-
ily. Also, if the initial text provided is inaccurate, the result
will be misleading and inharmonious.
F. Further Trials on Content Retention
As mentioned in Appendix D, we have also tried some
other approaches for preserving content, yet with little suc-
cess. We list these trials here as potentially valuable refer-
ences for further research.
Reliance on 3D LUT. In Sec. 5.3 of the main paper, we
described the further applications on user guidance by opti-
14
(a) Composite image (b) 3D LUT (c) Content text (d) Final
optimization
Figure 15. We here display our two further trials for content reten-
tion. One is to optimize a 3D LUT, while the other is to apply the
Attention-Constraint optimization also on the content text.
mizing a 3D LUT. Thus we have also considered optimizing
a 3D LUT between the initial composite image and the fi-
nal harmonized result. However, from the results in Fig. 15,
we can see that it does not usually work (seems fine for the
“handbag”, while bad for the “girl”). We attribute the failure
to the content distortion. Since the 3D LUT is intrinsically
an RGB-to-RGB global transformation, the content struc-
tures of two images need to be close enough to get good
LUT parameters, a condition our method cannot satisfy due
to the content distortion that remained in our method. Fur-
thermore, as shown in Fig. 15, our method achieves more
local editing, which cannot be fully represented by a global
3D LUT.
Optimization of content text. As we leverage the
Attention-Constraint Text to well illustrate the environment
of the images in Sec. 4.1 of the main paper, a natural
thought is whether we can apply the same to the content
text for content retention. Motivated by this, we try to opti-
mize the content text to focus on the foreground area of the
cross-attention maps. However, the content retention fails
to improve and gets even worse (see Fig. 15). Similar to
the statement to the supplemental text effect (Sec. 4.2 of
the main paper), we believe this is because the optimization
of content text breaks the text embedding’s generalization,
making it specific to only one context and not generalizable
to another, thus leading to more serious content distortions.