896 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO.
5, SEPTEMBER 2011
A Review of Adaptive Image Representations
Gabriel Peyré
Abstract—Improving the modeling of natural images is impor-
tant to go beyond the state-of-the-art for many image processing
tasks such as compression, denoising, inverse problems, and tex-
ture synthesis. Natural images are composed of intricate patterns
such as regular areas, edges, junctions, oriented oscillations, and
textures. Processing efficiently such a wide range of regularities re-
quires methods that are adaptive to the geometry of the image. This
adaptivity can be achieved using sparse representations in a redun-
dant dictionary. The geometric adaptivity is important to search
for efficient representations in a structured dictionary. Another
way to capture this geometry is through non-local interactions be-
tween patches in the image. The resulting non-local energies can be
used to perform an adaptive image restoration. This paper reviews
these emerging technics and shows the interplay between sparse
and non-local regularizations.
Index Terms—Adaptivity, approximation, best basis, denoising,
dictionary learning, inverse problems, non-local regularization,
sparse regularization, texture synthesis, triangulations.
Finding efficient representations for natural images is rele-
vant for many image processing tasks. Sparse and non-local reg-
ularizations have recently emerged as unifying concepts to for-
malize this notion of geometric representations. These methods
have been proved useful to perform compression, denoising, in-
version of linear operators, and texture synthesis. This paper re-
views these technics and shows that the concepts of sparsity and
variational minimization are at the heart of the success of these
methods.
I. INTRODUCTION
A
of
LARGE class of image processing problems can be for-
malized as recovering a high-resolution image
pixels from noisy observations , where .
The forward model assumes that is a degraded version of an
unknown high-resolution image , where . The
linear operator models the acquisition process.
If , the recovery of a good approximation of cor-
responds to removing noise. When is ill-posed, the inverse
problem of recovering a high-resolution image is more difficult.
Manuscript received October 05, 2010; revised December 31, 2010; ac-
cepted February 21, 2011. Date of publication February 28, 2011; date of
current version August 17, 2011. This work was supported by the French
“Agence Nationale de la Recherche” (ANR), under Grant NatImages
(ANR-08-EMER-009), “Adaptivity for natural images and texture represen-
tations”. The associate editor coordinating the review of this manuscript and
approving it for publication was Dr. Jean-Luc Starck.
The author is with the CEREMADE CNRS-Université Paris-Dauphine,
75775 Paris, Cedex 16, France (e-mail: gabriel.peyre@ceremade.dauphine.fr).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/JSTSP.2011.2120592
1932-4553/$26.00 © 2011 IEEE
Typical inverse problems are super-resolution, inpainting (see
for instance [5], [69]) and compressed sensing [11], [34].
Non-Adaptive Regularization: Classical regularization
theory makes use of a prior that is intended to
be low for the images one is interested in. Priors based on
derivatives of the image impose some smoothness on the
recovered image. The simplest prior is the norm of the gra-
dient , where is a finite difference
approximation of the gradient. This corresponds to a Sobolev
prior that favors uniformly smooth images. The total variation
prior, introduced for denoising by Rudin, Osher, and Fatemi
[89], reads . It allows one to take into
account discontinuous images, and favors piecewise constant
images with edge discontinuities of small perimeters. Other
classes of priors include sparsity in orthogonal or redundant
representations, which are discussed in Section II.
A prior is taken into account during the inversion of
using a variational minimization
(1)
where weights the fidelity to the observations versus the
regularity of the solution, and should be adapted to the noise
level.
Adaptive Regularization: Instead of using a fixed prior ,
it is possible to enhance the regularization by using a family
of priors . The parameter should be adapted to match
the geometric features of the images of interest. In the practical
situations considered in this paper, the parameter constrains
the local position and direction of edges and textures. We will
thus loosely refer to this parameter as the “geometry” of the
image.
This parameter takes different forms across the sections of
this review paper. For instance, it is a quadtree in Section III,
an association field in Section IV-B, a triangulation in
Section IV-C, a manifold in Section V-A, and a weighted
graph in Section V. This parameter always caries some geomet-
rical information about the location of edges or the direction of
textures.
The non-adaptive regularization (1) is turned into an adaptive
recovery by minimizing
where
(2)
The additional term allows one to introduce constraints on
the set of admissible geometries.
For a fixed , the energy is usually convex, but is
non-convex jointly on . One thus has to resort to iterative
PEYRÉ: REVIEW OF ADAPTIVE IMAGE REPRESENTATIONS 897

schemes that progressively estimate both the image and the ge-
ometry, and converge to a stationary point of .
The following sections describe several instances of this
adaptive regularization framework (2), each time using a dif-
ferent parameterization of the prior adapted to several kinds
of edge or texture features.

II. SPARSE AND ADAPTIVE REPRESENTATIONS

Computing a sparse decomposition corresponds to approxi-
mating an image of pixels as
using a small number of
atoms chosen in a dictionary of atoms
.

A. Nonlinear Approximation
Best Approximation and Thresholding: A sparse approxima-
tion is efficient if the error is small. The best ap-
proximation with nonzero coefficients in the dictionary
can be found by solving a variational minimiza-
tion where Fig. 1. Examples of images within several complexity classes.

them to capture the directionality of regular edges. The curvelet

(3) frame of Candès and Donoho [10] enjoys an error decay of
for images that are outside edges, thus
In the case where is an orthogonal basis enhancing over the wavelet approximation. A discrete imple-
of atoms, the problem (3) is re-written as mentation of the curvelet transform corresponds to the projec-
tion of the image on a tight frame [12]. This curvelets
approximation has also been used for inverse problems regular-
ization [13]. For smoothly varying textures, a fixed waveatom
where (4) frame [29] provides a small approximation error.
To approximate efficiently images with complicated edges
for a value of adapted so that . This is the vari- and textures, more redundant dictionaries are needed. To keep
ational regularization (1) for the identity operator , no the computation of the approximation tractable, it is how-
noise , , and the prior . Although this prior ever important to use a dictionary with some structure. Many
is non-convex, a global minimizer is found using a hard classical image processing dictionaries (such as Fourier and
thresholding that selects the inner products with largest ampli- wavelets) are structured because the atoms are parameterized
tudes, see for instance [65] by their frequency, position, and size. Imposing similar struc-
tures in redundant dictionaries is challenging but is important
to obtain fast and efficient algorithms.

and (5) B. Sparse Approximation Over Redundant Dictionaries

Computing sparse approximations in orthogonal bases
is at the heart of many efficient compression algorithms. For
instance JPEG-2000 uses a nearly orthogonal wavelet basis and
is a state-of-the-art coding scheme; see [65].
Decay of Approximation Error: For several classes of
images, one can prove the asymptotic optimality of such a non-
linear approximation scheme (5) in well chosen ortho-bases.
Uniformly images enjoy an optimal asymptotic approx-
imation error decay in a Fourier
basis. Piecewise regular images with discontinuity curves
of finite perimeters enjoy an optimal asymptotic decay
in a wavelet basis. Fig. 1 shows
some examples of smooth and discontinuous images.
Wavelets are however not optimal for piecewise regular im-
ages whose edges are regular. The error for such a cartoon image
decays as . The square support of wavelet atoms forbid
For an arbitrary redundant dictionary, computing the best
-term approximation (3) is however NP-hard [27], and one
thus needs to resort to greedy schemes [68] or minimization.
Sparse Approximation: A sparse approximation is ob-
tained by convexifying the pseudo norm in (3), and solving
the following basis pursuit denoising convex problem [15],
, where
(6)
where is adapted so that and where the
norm is defined as . Many algorithms have
been proposed to solve the convex problem (6), for instance it-
erative thresholding methods [24], [25]. Sparse coding in re-
dundant dictionaries finds application to compression of sounds
[52] and images [46].
898 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 5, SEPTEMBER 2011

if the sorted magnitudes decay like ,

then the coding-decoding error satisfies

(7)

see [65]. Note that this decay of the coefficients implies

Fig. 2. Image compression using wavelet support coding.
Such sparse approximation can be shown to be close to op-
timal for a restricted class of dictionaries, see for instance [98].
Most image processing dictionaries are however too redundant
to fit within this theoretical analysis. Furthermore, for highly re-
dundant dictionaries, solving (6) is computationally demanding.
Redundant and Structured Dictionaries: To avoid these pit-
falls, one needs to use structured dictionaries which enable a
theoretical analysis of the approximation performance for re-
stricted class of images, and also enable a fast search for nearly
optimal approximations. Sections III and IV detail several struc-
tured dictionaries, together with adaptive strategies to perform
this search.
C. Sparse Approximation for Processing
Building ortho-bases or frames for which the approximation
error decays fast is important for many applications. One can
show that for several image processing problems, the efficiency
of the processing is related to the nonlinear approximation error.
This section only sketches the main results for compression, de-
noising, and inverse problem regularization. It gives informal
statements that require additional technical hypotheses to be
mathematically rigorous.
Compression: Efficient image compression schemes per-
form a binary coding of quantized coefficients , where
is an orthogonal basis. An image is coded
with bits, and the decoder computes an approximation of
. is ill-conditioned. To recover an approximation
To exploit the sparsity of the decomposition of the image,
a support coding scheme codes independently the locations of
the nonzero coefficients and their amplitudes. Fig. 2 shows an
example of compression using this simple coding scheme.
Under some conditions on the basis, which are satisfied by
many classical ortho-bases such as wavelets, one can shows that
. This shows that a basis that is
efficient to perform nonlinear image approximation is able
to efficiently perform image compression by support coding.
State-of-the-art coders such as JPEG-2000 [65] use advanced
conditional coding schemes that exploit statistical redundan-
cies among the coefficients. This improves the decay of the
distortion with respect to a naive support coder.
This result does not extend to an arbitrary dictionary, since it
might be too costly to code the coefficients. It extends to dic-
tionaries of ortho-bases detailed in Section III, if the number of
atoms within the dictionary is not too large.
Denoising: Denoising corresponds to estimating an image
from a noisy observation , where is a
deterministic, unknown, clean image. A standard assumption is
that is a Gaussian white noise of variance .
A popular nonlinear denoising method is the hard thresh-
olding over an orthogonal basis , where
is defined in (5). This corresponds to the minimization of (1)
for and the prior. It was introduced by Donoho
and Johnstone [36], and uses a well chosen proportional
to the noise level . In an orthogonal basis , the decay
implies
(8)
if one uses the universal threshold ; see [36].
Here denotes the expectation with respect to the noise
random variable , and is the best terms approximation
of in . This shows that the average denoising error decays
fast to zero with if is able to efficiently approximate .
Denoising by thresholding is extended to redundant frames,
in which case is reconstructed using the dual frame
which is the dictionary defined by , where
. Denoising artifacts are reduced by using a
translation invariant redundant frame [23]. Denoising of images
is also enhanced by using statistical models that take into ac-
count the non-independence of coefficients of natural images;
see for instance [88].
Inverse Problems: For inverse problems, one measures
, where is an additive noise, and where
of
from , one replaces the sparse approximation by a regular-
ization , where
(9)
PEYRÉ: REVIEW OF ADAPTIVE IMAGE REPRESENTATIONS 899

Each sub-space has dimension and is equipped

with one or several orthogonal bases indexed by a token

Having multiple bases at each node increases the expressive-

ness of the dictionary, and is required to perform directional ap-
proximation of cartoon images, that are regular outside a set of
regular edge curves.
Fig. 3. Left: example of dyadic subdivision of in squares ; right:
corresponding quad-tree . A basis is obtained by aggregating bases for that
are leaves of , and for a specific choice of token
at each node of the tree
This is equivalent to computing a sparse decomposition of in
the redundant dictionary of using basis
pursuit denoising (6).
In the special case, where is an orthogonal basis, the
minimization (9) can be rewritten as the variational problem (1) When one does not care about the location of the basis elements
using an prior . in the tree, the basis is written as , where the index
A few theoretical results exist, but they are restricted is with and .
to the case where is a well-conditioned frame [98], This dictionary defines a highly redundant set of atoms
or for compressed sensing [11], [34] when is random. . One could use directly to compute
These results roughly show that in these restricted cases, a sparse approximation using for instance minimization (6).
, where is the variance of the noise, if This is however numerically expensive for large images. Re-
decays fast and if is well chosen. This means that stricting the sparsity to orthogonal bases leads to fast algorithms
the noise is kept under control during the inversion. for structured dictionaries.

B. Best Basis Selection

III. TREE-STRUCTURED REPRESENTATIONS
Generalizing (4) to an adaptive setting, the approximation in
To improve the representation of geometrical images, one can a best basis is defined as being a solution of
use a family of bases instead of a fixed basis. A dictionary of
orthogonal bases is a set of orthogonal bases
of , where is the number of pixels in
the image. Instead of using an a priori fixed basis such as the where (10)
wavelet basis, one chooses a parameter adapted to the
structures of the image to process and then uses the optimized Note that this minimization fits into the general framework (2)
basis . with , , , and . Note also that
We insist on the fact that the parameter has a different is the best terms approximation of with any basis in ,
meaning across the sections of this paper. In this section, is where is the number of nonzero coefficients of in .
a quadtree, but this will not be the case anymore in the other According to (5), for each , the optimal image that mini-
sections. mizes is . The minimization (10) can thus
be simplified as , where

A. Quadtree-Based Dictionaries (11)

To enable the fast optimization of a parameter adapted to
a given image to process, each is constrained to be where . This kind of Lagrangian is effi-
a quadtree. The quadtree that parametrizes a basis defines ciently optimized using a dynamic search algorithm, originally
a dyadic segmentation , where each presented by Coifman et al. [20]. It is a particular instance of the
is a square of size . Fig. 3 shows an example of Classification and Regression Tree (CART) algorithm of Brei-
such a dyadic subdivision. This segmentation is used to partition dman et al. [7] as explained by Donoho [35].
either the spacial domain (local cosine packets and bandlets) or The complexity of the algorithm is proportional to the com-
the frequency domain (wavelet packets). plexity of computing the whole set of inner products
The full tree indexes sub-spaces of for scales in the dictionary. For several dictionaries, such as
and position , that obey a refinement rela- those considered in this section, a fast algorithm performs this
tionship , the sum being orthogonal. computation in operations, where is the total number of
To enrich the representation parameterized by a quadtree, a atoms in . For tree structured dictionaries, this complexity is
token is attached to each leave of the tree. thus , where is the number of tokens asso-
We call such a tree with its token a tagged quadtree, and use the ciated to each leaf of the tree. This is much smaller than the total
notation to refer to both the quadtree and its tokens. number of bases in , which grows exponentially with .
900 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 5, SEPTEMBER 2011

C. Spacial and Frequency Adaptive Dictionaries

Local cosine and wavelet packets bases are parameterized
by a quadtree tree that segments respectively the image
plane and the frequency plane [20], [22], [65].
The wavelet packet construction is generalized by consid-
ering non stationary (NS) wavelet packets [18], that apply dif-
ferent quadrature mirror filters at each scale of the tree. An NS
wavelet packet basis is parameterized by a tagged quadtree
, where each node is tagged by a filter. The selection of a best Fig. 4. Example of subdivision adapted to a cartoon image . The basis
NS wavelet packet basis is obtained by a dynamic program- minimizes .
ming algorithm detailed in [76] that generalizes the best basis
algorithm. scale of the wavelet transform. The orthogonal bases are
computed using discrete oriented orthogonal bases. If is a
D. Geometry Adaptive Dictionaries function that is outside a set of -regular edge curves, the
approximation in the best bandlet basis minimizing (11)
Geometric images with regular edges are difficult to approx-
satisfies , where is the number of
imate using a wavelet basis. Wedglets approximations and ban-
bandlets composing . Such an orthogonal
dlets bases compute a quadtree segmentation of the image
bandlet transform finds applications to image [66] and surface
together with a local estimate of the orientation of the edges that
[84] compression and image denoising [85].
is encoded in the tokens .
Adaptive Wedgelets Approximation: A geometric approxi-
E. Adaptive Tree-Structured Processing
mation is obtained by considering for each node a collec-
tion of discontinuous approximation spaces for the square The processing scenarios described in Section II-C extend
of the segmentation. The wedgelets [33] uses a piecewise con- to the best basis setting. For compression and denoising, the
stant approximation on each side of a straight edge that follows best basis parameter is computed from the (possibly noisy)
the direction indexed by . observed image, as a pre-processing. For inverse problems such
The best wedgelets algorithm optimizes the Lagrangian (11) as inpainting or compressed sensing, where the data might be
over all possible wedgelets segmentation , where the severely degraded, it is not possible to estimate directly from
thresholding approximation is replaced by a piecewise constant the observations. One thus has to use an iterative algorithm that
approximation. This scheme is efficient to approximate a piece- progressively refines the best basis parameter as the quality of
wise constant image whose edges are curves. For such a the reconstructed image increases.
cartoon image, the approximation error with wedgelets de- Adaptive Tree-Structured Compression: For compression ap-
cays like , see [48]. It is also possible plications, one computes the best basis adapted to the image
to consider approximation spaces with higher order polynomials to compress by minimizing the corresponding Lagrangian
to capture arbitrary cartoon images [94]. The computation of the (11), and then binary codes the quantized coefficients
low-dimensional projection can be significantly accelerated; see together with the parameter of the best basis, see [65].
[47]. The coding error decay (7) extends to a dictionary of or-
Adaptive Spatial Bandlet Approximation: The bandlet bases thognonal basis if for some ; see
dictionary was introduced by Le Pennec and Mallat [58], [59]. [65]. The total number of bits is reduced by splitting it as
Bandlets perform an efficient adaptive approximation of images , where is the number of bits to code the
with geometric singularities. This transform has been refined by nonzero coefficients among coefficients, and is the
Mallat and Peyré [66], [84] to obtain a dictionary of regular and number of bits to code the quadtree .
orthogonal basis functions. For compressing cartoon images, the best wedgelet
A directional orthogonal basis of elongated directional sub-space provides a distortion rate
wavelets is defined over each square . Keeping only a few . The best bandlet basis further improve
bandlet coefficients and setting the others to zero performs an this rate to for cartoon im-
approximation of the original image that follows the local di- ages that are outside edges.
rection indexed by . The optimal segmentation and tokens Adaptive Tree-Structured Denoising: To perform denoising,
are those that minimize the Lagrangian (11). Fig. 4 shows an the best basis is computed from the noisy observations
example of such a segmentation adapted to a geometric image. by minimizing an empirical Lagrangian ;
Adaptive Approximation Over the Wavelet Domain: Ap- see [57] and [65]. The denoised image is defined by thresholding
plying such an adaptive geometric approximation directly on the noisy observations . The asymptotic op-
the image leads to unpleasant visual artifacts. To overcome timality result (8) extends to a dictionary of orthogonal bases
this issue, one applies a tree structured approximation on the containing atoms by selecting [65].
discrete set of wavelet coefficients. The wedgeprints of Wakin best sub-space was originally designed to perform denoising.
et al. [99] use a vector quantization to extend the wedgelet For piecewise constant function with edges, Wedgelets
scheme to the wavelet domain. The orthogonal bandlets of leads to an optimal decay of the estimation error of
Peyré and Mallat [66] use an adaptive bandlet basis for each , up to logarithmic factor, where is the
PEYRÉ: REVIEW OF ADAPTIVE IMAGE REPRESENTATIONS 901

Fig. 5. (a) ( ) original image. (b) ( ) compressed sensing reconstruction using a translation invariant wavelet frame ( dB and 22.1 dB). (c) ( )
reconstruction using iteration in a best bandlet basis ( 39.3 dB and 23.9 dB).

variance of the noise [31], [33]. The best bandlet basis thresh-
olding extends this result to a decay of
for cartoon images; see [85].
Adaptive Tree-Structured Regularization of Inverse Prob-
lems: The resolution (9) of the inverse problem
is extended to a best basis recovery by solving over both the
image and the quadtree
using a dictionary of bandlet basis. The operator projects an
image of pixels on random vectors.
Note that for some specific inverse problems, such as super-
resoluton, it is possible to estimate directly the local direction of
bandlets-like atoms directly from without resorting to a global
optimization problem such as (12). This allows to regularize the
operator inversion with a fast algorithm that produces state-of-
the-art super-resolution results; see [67].

IV. SEMI-STRUCTURED REPRESENTATIONS

where (12) A. Lifting-Based Dictionaries
To enhance the wavelet representation, it is possible to adapt
which corresponds to the general framework introduced in (2) the wavelet filters to the image content, using spatially adaptive
with . filters to avoid creating large wavelet coefficients near edges.
Peyré proposes in [80] an iterative algorithm to compute a The lifting scheme, introduced by Sweldens [96], is an unifying
stationary point of . Starting from some initial , at a step framework to design adaptive biorthogonal wavelets, through
, the estimate is modified as the use of spatially varying local interpolations.
Lifting Scheme: The lifting scheme has been introduced by
Sweldens [96] as a way to decompose wavelet filtering into el-
ementary lifting steps. This allows one to define wavelets on
The best basis parameter is then updated non-translation invariant grids using spatially varying filters. It
can thus be used to define wavelets on surfaces [61], [91] and
on complicated planar triangulations; see Section IV-C.
The initial sampling grid for some is recursively
if , split, for each as a disjoint union ,
where
otherwise. where is a coarser grid, and is a detail grid intended to
store wavelet coefficients. On an image, one can use a traditional
Note that this is a Lagrangian optimization similar to (11), which dyadic refinement so that , or to build non-
can be solved efficiently using the same dynamic programming tensor product wavelets, one can use quincunx sub-sampling
algorithm [20]. The next iterate is defined using a soft thresh- [45], where , but this splitting scheme is general
olding and holds for instance in the case of triangulation refinement;
see Section IV-C.
A biorthogonal wavelet basis is implic-
where itly defined through lifting by computing the wavelet coeffi-
cients for . though the application
If , one can show that is decaying of elementary lifting steps. The lifting algorithm also maintains
through the iterations, and thus converging; see [80]. temporary variables, which are inner products with scaling func-
Fig. 5 shows an example of application of this algorithm to tions for . At the finest scale ,
the recovery from compressed sensing measurements [11], [34] the scaling coefficients are initialized using the data .
902 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 5, SEPTEMBER 2011

Then, for each , it successively applies a local predictor

to compute the wavelet coefficients, for all

(13)

and an update operator to maintain stability of the transform, for

all

(14)

The association field connects each point of the

detail grid to a set of points on the coarse grid. This
association field is the only degree of freedom of the scheme,
since the weights are usually computed as a polyno-
mial interpolation on the set of points selected by the field, while
is obtained by solving a linear system to impose that
has the same mean and higher order moments as . The
lifting wavelet transform is easily inverted by successively in-
verting steps (14) and (13).
One can chain more than one predict and one update oper- Fig. 6. Examples of grouplet vectors for several positions and scale .
ators, to construct arbitrary biorthogonal wavelets on uniform
1-D grid [26]. Unfortunately, on non-uniform grids, it is impos-
association fields, where some singular point might be-
sible to control properties such as vanishing moments, orthogo-
long to many different fields for various .
nality, and stability of more than one step of predict and update.
Grouplets Over the Image: The grouplet transform also
Lifting Processing: A nonlinear approximation is obtained
makes use of an association field, but it replaces the lifting
by thresholding the lifted wavelet coefficients before re-
computation of wavelet coefficients by an extended Haar trans-
construction. This can be used to perform compression and de-
form, where points in are processed in sequential order to
noising following the schemes detailed in Section II-C. Note
maintain orthogonality.
however that, for arbitrary association fields , the lifted
A grouplet family is an orthogonal basis, or a redundant
wavelets can be unstable and far from orthogonality. This re-
tight frame adapted to the processing of geometric images or
sults in a loss of performance for compression and denoising
regularly oscillating textures, introduced by Mallat [64]. It has
using thresholding.
been used to perform image denoising and super-resolution [64]
Adaptive Predictions: One can adaptively design the associ-
as well as texture inpainting and synthesis [81]. The geometric
ation fields used during the lifting pyramid. One can compute
parameter is a multiscale association field, where
these associations to reduce the length of the wavelet filter near
each is a vector field that follows the geometry of the image
edges [17]. For compression applications, this estimation is per-
at a scale .
formed from already coded coefficients, so that the fields do
For each scale , the flow links the point to , where
not need to be transmitted to the decoder. Such a filter adaptation
. A modified Haar transform progressively
is however no sufficient to capture fully the geometric informa-
extract grouplets detail coefficients along this flow. This corre-
tion. Such schemes are related to adapted subdivision [70].
sponds to the projection of the image on a orthogonal or redun-
To further reduce the distortion of geometric images, the ori-
dant tight frame of . The grouplets atoms
entation of the association fields can be optimized though
at scale have a width of 1 pixel and follow the geometric flow
the scales. Because of the lack of structure of the set of bases
on a width of pixels. Fig. 6 shows examples of grouplets
, computing the fields that minimizes (10) is intractable.
at several scales.
The lifting operators are thus usually computed using heuristics
The selection of an optimal grouplet flow adapted to
to detect the local orientation of edges. The association fields
an image by minimizing a Lagrangian such as (11) is in-
are obtained by minimizing the amplitude of wavelet coeffi-
tractable, because the grouplet dictionary does not have a tree
cients; see for instance [14] and [50] or using local gradient in-
structure. One thus uses a greedy scheme that performs block
formation [54]. These adaptive lifting schemes can be extended
matching to select the patch around that is the closest
to perform adaptive video transform [92] where the lifting op-
to the patch around . Alternative methods compute by
erates through in time by following the optical flow.
integrating a texture flow along a distance of ; see [81].
The grouplet transform is related to the curved wavelet trans-
B. Grouplets
form [100] and the easy path [86]. A distinctive features of grou-
A difficulty with lifted transforms is that they do not guar- plets is that the fields can converge to a singular point, which
antee the orthogonality of the resulting wavelet frame. The sta- means for . This is useful to represent
bility of the transform thus tends to degrade for highly adaptive turbulent textures with complicated singularities.
PEYRÉ: REVIEW OF ADAPTIVE IMAGE REPRESENTATIONS 903

Fig. 8. A few iterations of the grouplet inpainting algorithm.

Fig. 7. Example of texture synthesis by statistical modeling of grouplet coef-

ficients. spline function (hat functions centered on the vertices ),
which has dimension .
The best approximation with vertices of is obtained by
Grouplets Over Wavelets: To process image with both ori- an orthogonal projection
ented texture and edges, this grouplet transform can also be ap-
plied over a wavelet transform, so that the wavelet coefficients
at each wavelet scale are projected on a grouplet frame . This
corresponds to the projection of on a multiscale grouplet
frame composed of atoms with varying widths and lengths.
This grouplet transform has been applied to texture synthesis
[81]; see Fig. 7. The synthesis is obtained by sampling grouplet where (15)
coefficients with the same distribution as the coefficient of an
exemplar, and by randomizing the geometry of the flow .
Since is not orthogonal, computing the coefficients re-
Sparse grouplet regularization can be used to perform geo-
quires the resolution of a sparse linear system.
metric inpainting of large missing regions by minimizing (12)
Smooth Image Approximation With Triangulations: For
over the image while adapting progressively the association
uniformly regular functions, one can build isotropic triangula-
field ; see [81]. Fig. 8 shows an example of iterations of the
tions that generate an error ,
grouplet inpainting algorithm, that performs iteratively a soft
where is the Hessian of the image .
thresholding and an estimation of an adapted grouplet flow .
For a image, one should use anisotropic triangles
whose aspect ratio matches the anisotropy of the image
[4]. A triangle near a point should be aligned
C. Adaptive Triangulations with the principal eigenvector of , and the aspect
ratio should match the ratio of the eigen-
A finite elements approximation first builds a segmentation values of . The approximation error with such
of the image domain into elementary cells, and then perform a adapted anisotropic approximations of a image satis-
low-dimensional polynomial approximation on each cell. The fies ; see for
adaptivity comes from the selection of this segmentation, that instance [74]. This shows that anisotropic approximations
should be adapted to the features of the image. have a better constant than isotropic for the decay of the error
Image Approximation by Triangulations: The parameters .
to adapt a triangulation dictionary is a set of Cartoon Image Approximation With Triangulations: For a
vertices and a set of connections cartoon image, which is a function outside contours, one
that prescribes the connectivity between the points. can also diminish the approximation error using
Triangles are convex hulls of the elements of , and they form anisotropic triangles. Near edges, the anisotropic triangles
a partition of the image domain . should obey the same parabolic scaling as curvelet atoms,
One performs approximation with continuous piecewise . Fig. 9 shows an example of such an adapted
affine function on the triangles, which corresponds to using triangulation. Using such anisotropic triangulations for a car-
an adapted basis of the space of linear toon image leads to an error decay ;
904 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 5, SEPTEMBER 2011

Fig. 9. Adaptive triangulations for piecewise linear approximations of a

-cartoon image. Fig. 10. Comparison of the adapted triangulation scheme [6] with JPEG-2000,
for the same number of bits, triangles.

see [65]. This improves significantly over the wavelet approxi-

mation decay . from to , and impose that is the Delaunay triangula-
Heuristics and Greedy Algorithms: The issue is that com- tion of . This can be shown experimentally to produce highly
puting the best triangulation that minimizes anisotropic meshes, which can be used to perform compression;
over a discretized domain is NP-hard [1]. There is currently see Demaret et al. [30].
no algorithm that guarantees the error decay Multiscale Approximations on Triangulations: Embedded
for an arbitrary cartoon image. One thus needs to rely triangulations with an increasing number of
on subobtimal greedy schemes. These schemes generate a se- vertices satisfy . This is achieved by imposing that
quence of embedded triangulation by either and that each face of is split into one of more
refinement (increasing values) or coarsening. faces of . The set of triangulation defines a hi-
A greedy refinement scheme starts by a simple fixed triangu- erarchical segmentation of the image that can be encoded using
lation of the squares , and iteratively add one or several a tree, generalizing the quadtree construction of Section III-A.
vertices to to obtain a triangulation that minimizes the A biorthogonal wavelet basis is constructed
error . The faces needs also to be updated using a basis of the detail space that satisfies
to obtain . . The decomposition of an image on a
Delaunay refinement introduced by Ruppert [90] and Chew biorthogonal wavelet basis defined on embedded triangulations
[16], proceeds by inserting a single point, that is imposed to be can be computed using lifting steps; see [61] and [91].
a circumcenter of one triangle, and also impose that is a De- Regular splits of orthogonal triangles lead to isotropic adap-
launay triangulation of . This constraint accelerates the search tive triangulations [32]. Splitting triangles according to a well-
and also leads to triangles with provably good isotropic aspect chosen median leads to anisotropic triangulations that exhibit
ratio, which might be useful to compute the approximation of optimal aspect ratio for smooth images; see [19].
the solution of an elliptic PDE on the mesh grid. For image ap-
proximation, one however needs to design aniosotropic trian-
V. PATCH-BASED REPRESENTATIONS
gulations, which requires to modify the notion of circumcenter
using an anisotropic metric [6], [49]. Fig. 10 shows an example A new class of adaptive image processing have emerged re-
of adaptive triangulation produced by the greedy anisotropic re- cently by considering non-local interactions between patches
finement scheme of [6]. Such a refinement scheme can be used in an image. These interactions are used to perform non-local
to perform image compression, and gives results superior to filtering, sparse coding with dictionary learning, and non-local
JPEG-2000 for cartoon images with regular edges. Other refine- regularization of inverse problems.
ment schemes are possible, such as for instance edge bisection,
for which the optimal error decay is known to hold for smooth A. Manifold Geometry of Patches
convex functions [74].
Triangulation thinning algorithms start with a fine scale trian- Manifold of Patches: A patch of size is extracted
gulation of and progressively remove either a vertex, around a pixel in an image by defining for all
an edge or a face to increase as slowly as possible the approxi-
mation error until vertices remain, see for
instance [37]. One can for instance remove a single vertex to go (16)
PEYRÉ: REVIEW OF ADAPTIVE IMAGE REPRESENTATIONS 905

Fig. 12. Example of recovery from compressed sensing measure-

ments, using sparsity in a translation invariant wavelet frame, and using (17) for
the manifold of edges.

Fig. 11. Display of the patch manifold that has the topology of a cylinder for
binary edges.

The set of patches extracted from natural images

often depends on a small number of hidden variables. This is for
instance the case near geometric edges or directional textures.
For some simple classes of geometric images and textures,
this set of patches can be shown to be close to a low-dimen-
sional smooth manifold of dimension . This
is for instance the case for binary cartoon images, where edge
patches can be parameterized by the orientation of the closest
edge, and the distance to this edge. The resulting manifold has
the topology of a cylinder, as shown in Fig. 11. Similar mani-
folds can be use to model more general cartoon images and ori-
ented textures; see [78].
Manifold Regularization: The manifold model assumes that
the patches are close to the manifold , and a variational
energy quantifies the deviation from this model
where
For simple geometric images and textures, where is explic-
itly known, the energy is a fixed, non-adaptive prior. It can
be used to regularize an inverse problem by min-
imizing
(17)
A stationary point of this non-convex energy is computed
using an alternative minimization on and on the patch
of the manifold; see [78]. Fig. 12 shows an
example of compressed sensing reconstruction using a regu-
larization with a manifold of edges. If the manifold is also
optimized in (17), it corresponds to an adaptive regularization
that fits into the framework (2).
Fig. 13. Local Fourier analysis of an image containing edges and textures.
Locally parallel textures are oscillating in a direction indi-
cated at a point by a smooth vector field . An adaptive
texture norm makes use of the regularity of the patterns and en-
hances the separation with respect to non-adaptive norms that
favor arbitrary oscillations. Fig. 13 shows the local Fourier ex-
pansion of an image mixing edges and textures. This local
analysis is obtained by projecting on a local Fourier frame
. Each atom is localized in a patch around a
point and has a frequency .
Following Maurel et al. [71], an adaptive texture energy
is defined as a weighted quadratic sum of the local
Fourier coefficients
where is a weighing function that is close to zero when
and is rapidly increasing with . This weighting function
forces the local frequencies of the texture to be nonzero around
only for frequencies close to . This prior is well suited
to model locally parallel textures.
The energy captures the oscillating content of the image.
The cartoon content of the image can be captured using the total
variation , where is a
finite difference approximation of the gradient of at . The
cartoon and the texture priors are integrated into a variational
energy minimization to perform a separation from noisy
low-dimensional measurement

B. Geometric Texture Regularization

Separating a texture content from the cartoon content of
a noisy image has been emphasized by Meyer
[72] that introduces several norms for oscillating patterns. Sev-
eral numerical algorithms have proposed to perform this sepa-
ration; see for instance [3].
(18)
where should be adapted to the noise level and where can
be used to encode constraints on the frequency localization of
the texture. This separation problem requires the computation of
906 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 5, SEPTEMBER 2011

Fig. 15. Some vectors of a dictionary learned from a set of patches

extracted from natural images.

point of this minimization; see for instance [83]. When the dic-
tionary is fixed, one performs a sparse coding to find the
coefficients that solve, for each the basis pursuit denoising (6)
Fig. 14. Variational separation using an energy adapted to locally parallel tex-
tures.
(21)
both the components and the adapted geometric flow . It gen-
eralizes the adaptive regularization (2) to the case where several When the coefficients are fixed, the dictionary is the
priors are used. An iterative minimization algorithm computes solution of
a stationary point of ; see [71]. Fig. 14 shows an example of
decomposition obtained by solving (18), using the iden-
tity operator.

which is the minimization of a quadratic convex functional

C. Dictionary Learning
under quadratic convex constraints and can be solved using a
Dictionary Learning for Patch Approximation: Sparse mod- projected gradient descent [83].
eling of the patches requires the design of a dictionary Fig. 15 shows an example of dictionary learned with this al-
adapted to the local geometry of the images gorithm. The exemplars are patches of 16 16 pixels
to process. The dictionary can be learned from a large set of extracted from a library of natural images, and the redundancy
patches . As first proposed by Olshausen of the dictionary is set to .
and Field [75], is optimized to maximize the sparsity of these Application to Denoising and Inverse Problems: This
exemplars; see also [2] and [43] and references therein. learning can be performed jointly with the resolution of an
The optimal dictionary minimizes the average approximation inverse problem by minimizing
error using the basis pursuit sparse coding (6)
(22)

where is defined in (20). The prior enforces the sparsity in

of each patch extracted at some location as defined
in (16)

(19)

and where the constraint on the dictionary is defined as (23)

if ,
(20) Note that this corresponds to the adaptive regularization frame-
otherwise.
work (2) with the replacement of the geometry by a patch
This problem is convex over the dictionary (under convex con- dictionary .
straints) and on the coefficients, but not jointly convex. One can The resolution of (22) requires the minimization of a non-
use an iterative coordinate descent that converges to a stationary convex energy over both the image to recover and the dictionary
PEYRÉ: REVIEW OF ADAPTIVE IMAGE REPRESENTATIONS
Fig. 16. Comparison of inpainting using sparsity in a fixed dictionary of
wavelets and local DCT (MCA), and using a wavelet basis and a learned
dictionary (adaptive MCA).
to learn. This minimization can be performed alternatively on ,
the coefficients involved in (23), and , which are all
convex problems. One can show the convergence of this scheme
toward a stationary point of the energy; see [83].
An in-depth review of the applications of dictionary learning
to image processing can be found in [41]. For , the min-
imization (22) performs state-of-the-art denoising [40]. When
is a masking operator, the minimization (22) performs in-
painting [63], in which case the dictionary can be learned from
exemplar images, and also updated at each step of the minimiza-
tion. One can also use several dictionaries and perform a joint
image separation and regularization. This separation process
extends the original morphological component analysis to an
adaptive setting [83]. Fig. 16 shows the enhancement of the in-
painting obtained using a learned dictionary instead of a fixed
local cosine dictionary.
Application to Computer Graphics and Vision: Texture syn-
thesis is performed by computing a random synthesized image
whose patches are sparse in a given dictionary ; see [79].
This dictionary is learned from patches extracted from an
exemplar . The synthesis is obtained by initializing to be
a Gaussian white noise, and then iterating between the sparse
patch coding (21) in using and averaging of the
reconstructed patches
where is the number of patches that overlap at a point .
Dictionary learning over sparse patches also allows to per-
form texture classification and segmentation [79], [95]. A dic-
tionary is learned for each texture class, and a pixel is assigned
to the class that performs the best sparse approximation around
this pixel. A similar idea is applied to perform image recognition
907
Fig. 17. Example of non-local spectral vectors for several value of .
[62], where the dictionary is trained by enforcing both sparsity
and discrimination of the classes. See [102] for a review of the
application of dictionary learning to computer vision, including
its application to face recognition.
Structured Dictionary Learning: To obtain faster algorithms
and enhance the efficiency of the methods on some classes of
images, it is possible to consider structured dictionaries. The
simplest structure is a union of ortho-bases ,
where , as already considered in Section III. It
is possible to extend the optimization problem (19) to include a
block-orthogonality constraint; see for instance [60] and [93]. It
is also possible to replace the sparsity energy by a weighted
norm, which corresponds to a Gaussian mixture prior. The
resulting dictionary can be used to perform image compression
[28] and provide state-of-the-art result for super-resolution and
deblurring inverse problems [53]. Let us mention that there are
more complicated way to structure a dictionary, for instance by
enforcing tree structures in the sparsity patterns of the coeffi-
cients; see [55].
D. Non-Local Methods
Non-Local Filtering: Non-local filtering have been intro-
duced by Buades et al. [8] as an efficient denoising method.
A spatially varying filter parameterized by computes
, where characterizes the
interaction between the features around point and around
point .
An adaptive non-local filtering computes the weights
from the image to process
where
(24)
where and controls the shape of the weights.
908 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 5, SEPTEMBER 2011
Fig. 18. Examples of inpainting with 80% of pixels.
As tends to zero, the filtering becomes related to texture
synthesis method [38], [101]; see Fig. 7, third image from
the left. Indeed, given an input image , and starting from
a random noise image , applying the recursive scheme
for a small produces a random image
visually similar to .
Non-Local Spectral Bases: This adaptive filtering is con-
nected to adapted decomposition by introducing an adaptive
non-local spectral basis [77], [97]. It is defined as the orthog-
onal eigenvectors basis of the filtering oper-
ator
and where is the corresponding eigenvalue, assumed
to be sorted in increasing order. The eigenvectors are a gen-
eralization of Fourier atoms to a spatially adaptive setting.
Fig. 17 shows examples of non-local spectral atoms adapted to
the geometry of a cartoon image. They are oscillating near the
boundary of the shape to allow a better reconstruction of this
singularity.
An adaptive denoising of a noisy observations
is obtained by computing the non-local graph and then
thresholding the coefficients in this basis ; see
[77].
Non-Local Regularization: The non-local filtering
is related to the minimization of a non-local energy
; see for instance [21], [42], and [51].
To tackle the regularization of an inverse problem , VI. CONCLUSION
this energy is replaced by
(25)
see [44] and [82]. For , it is a generalization of the Sobolev
prior to the non-local setting. For , it extends the total
variation.
For some inverse problems such as deblurring
[56], [73] or inpainting small holes [9], it is possible to estimate
the graph from the observations . For generic inverse
problems, the graphs needs to be optimized in parallel to the
reconstruction. In this setting, the weights are constrained to
be a probability distribution, so that for each .
Following [44], the spread of the weights is controlled using a
negative entropy with weight
.
Recovery from the noisy measurements is
obtained by minimizing
(26)
as introduced in [44] for inpainting and in [82] for generic
inverse problems. Note that this fits into the general adaptive
framework (2).
The minimization of the non-convex problem (26) is per-
formed by iteratively minimizing with respect to and to .
For a fixed , the optimal weights minimizing (26) are defined
as in (24). For a fixed , the optimal image is computed by min-
imizing a convex problem. One can shows that for , this
scheme converges to a stationary point of defined in (26); see
[82].
Fig. 18 shows a comparison between this approach and total
variation and wavelet sparsity for the recovery from missing
random pixels.
To conclude this paper, we would like to recall the main
strengths and limitations of the methods detailed in this review.
— Wedglets (Section III-D) are efficient to approximate car-
toon images, but might lead to visual artifacts because of
the discontinuous segmentation of the image domain.
— Bandlets (Section III-D) adopt a strategy similar to
wedgelets, but can be applied over a wavelet domain to
PEYRÉ: REVIEW OF ADAPTIVE IMAGE REPRESENTATIONS
produce an orthogonal expansion with reduced spacial
artifacts. This is useful in particular for compression
applications. Bandlets and wedgelets are however highly
constrained constructions, and might not be the best
choice to process complicated natural images.
— Lifting (Section IV-A) is a simple way to produce adap-
tive directional wavelets. The large number of degrees of
freedom in these transforms makes the use of heuristic al-
gorithms mandatory, and might be an obstacle for their use
beyond compression applications.
— Grouplets (Section IV-B) share similarities with lifting
transforms, but provide a conservation of energy, which
makes them suitable for inverse problems and texture
synthesis.
— Triangulations (Section IV-C) are popular for the resolu-
tion of PDEs using mesh adaptation. Several algorithms
have been proposed to generate anisotropic triangulations,
which are efficient to approximate cartoon images. Trian-
gulations are however mostly restricted to compression ap-
plications.
— Patch manifolds (Section V-A) provide highly constrained
models that are only suitable for simple images. While
manifolds cannot be used for compression they are able
to regularize inverse problems.
— Texture norms (Section V-B) offer image models tailored
for oscillating and oriented patterns. They are conceptually
simpler than grouplets models, but are restricted to locally
parallel textures. Their use is mostly restricted to texture/
cartoon separation and inpainting, and they should be used
in conjunction with an other model to capture the cartoon
content.
— Dictionary learning (Section V-C) introduces the impor-
tant concept of training the representation from examples.
This is a promising area of research, with state-of-the-art
results in denoising and possible extensions to inverse
problems and texture/cartoon separation. Their use is until
now restricted to small patches because of the difficult
optimization process that is required.
— Non-local filtering (Section V-D) is easy to use and give
denoising results close to the state-of-the-art. It can be ex-
tended to inverse problems regularization which leads to
good results on some problems but requires a slow opti-
mization process over the set of patches.
We hope that this review has helped the reader to understand the
connections between these different ways of capturing the ge-
ometry of images. The large diversity of approaches comes both
from the richness of the geometrical features in natural images
and from the variety of applications in image processing, com-
puter vision, and graphics. We believe there are many exciting
avenues of future works, starting with the difficult problem of
bridging the gap between geometrical representations and auto-
matic learning of sparse models.
REFERENCES
[1] P. Agarwal and S. Suri, “Surface approximation and geometric parti-
tions,” SIAM J. Comput., vol. 27, no. 4, pp. 1016–1035, 1998.
909
[2] M. Aharon, M. Elad, and A. M. Bruckstein, “The K-SVD: An algo-
rithm for designing of overcomplete dictionaries for sparse represen-
tation,” IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311–4322,
Nov. 2006.
[3] J.-F. Aujol and A. Chambolle, “Dual norms and image decomposition
models,” Int. J. Comput. Vis., vol. 63, no. 1, pp. 85–104, Jun 2005.
[4] I. Babuška and A. K. Aziz, “On the angle condition in the finite element
method,” SIAM J. Numer. Anal., vol. 13, no. 2, pp. 214–226, Apr. 1976.
[5] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image in-
painting,” in Proc. Siggraph’00, 2000, pp. 417–424.
[6] S. Bougleux, G. Peyré, and L. Cohen, “Image compression with
geodesic anisotropic triangulations,” in Proc. ICCV’09, 2009, pp.
2343–2348.
[7] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classifica-
tion and Regression Trees. Belmont, CA: Wadsworth, 1984.
[8] A. Buades, B. Coll, and J. M. Morel, “A review of image denoising
algorithms, with a new one,” SIAM Multiscale Modeling Simul., vol.
4, no. 2, pp. 490–530, 2005.
[9] A. Buades, B. Coll, J.-M. Morel, and C. Sbert, “Self similarity driven
demosaicking,” IEEE Trans. Image Process., vol. 18, no. 6, pp.
1192–1202, Jun. 2009.
[10] E. Candès and D. Donoho, “New tight frames of curvelets and optimal
representations of objects with piecewise singularities,” Commun.
Pure Appl. Math., vol. 57, no. 2, pp. 219–266, 2004.
[11] E. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles:
Exact signal reconstruction from highly incomplete frequency infor-
mation,” IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489–509, Feb.
2006.
[12] E. J. Candès, L. Demanet, D. L. Donoho, and L. Ying, “Fast discrete
curvelet transforms,” SIAM Multiscale Modeling Simul., vol. 3, no. 5,
pp. 861–899, 2003.
[13] E. J. Candès and D. L. Donoho, “Recovering edges in ill-posed inverse
problems: Optimality of curvelet frames,” Ann. Statist., vol. 30, no. 3,
pp. 784–842, 2000.
[14] V. Chappelier and C. Guillemot, “Oriented wavelet transform for
image compression and denoising,” IEEE Trans. Image Process., vol.
15, no. 10, pp. 2892–2903, Oct. 2006.
[15] S. S. Chen, D. L. Donoho, and M. A. Saunders, “Atomic decomposition
by basis pursuit,” SIAM J. Sci. Comput., vol. 20, no. 1, pp. 33–61, 1999.
[16] L. P. Chew, “Guaranteed-quality mesh generation for curved surfaces,”
in SCG ’93: Proc. 9th Annu. Symp. Comput. Geom., New York, 1993,
pp. 274–280, ACM.
[17] R. L. Claypoole, G. M. Davis, W. Sweldens, and R. G. Baraniuk, “Non-
linear wavelet transforms for image coding via lifting,” IEEE Trans.
Image Process., vol. 12, no. 12, pp. 1449–1459, Dec. 2003.
[18] A. Cohen and N. Dyn, “Nonstationary subdivision schemes and
multiresolution analysis,” SIAM J. Math. Anal., vol. 27, no. 6, pp.
1745–1769, 1996.
[19] A. Cohen, N. Dyn, F. Hecht, and J.-M. Mirebeau, “Adaptive multires-
olution analysis based on anisotropic triangulations,” Math. Comput.,
2011, preprint: arXiv:1101.1512, to be published.
[20] R. Coifman and V. Wickerhauser, “Entropy-based algorithms for
best basis selection,” IEEE Trans. Inf. Theory, vol. IT-38, no. 2, pp.
713–718, Mar. 1992.
[21] R. R. Coifman, S. Lafon, A. B. Lee, M. Maggioni, B. Nadler, F. Warner,
and S. W. Zucker, “Geometric diffusions as a tool for harmonic anal-
ysis and structure definition of data: Diffusion maps,” Proc. Nat. Acad.
Sci., vol. 102, pp. 7426–7431, May 2005.
[22] R. R. Coifman, G. Matviyenko, and Y. Meyer, “Modulated Malvar-
Wilson bases,” J. Appl. Comput. Harmon. Anal., vol. 4, no. 1, pp.
58–61, 1997.
[23] R. R. Coifman and D. L. Donoho, “Translation-invariant de-noising,”
in Wavelets and Statistics, Lecture Notes in Statistics. New York:
Springer-Verlag, 1995, pp. 125–150.
[24] P. L. Combettes and V. R. Wajs, “Signal recovery by proximal forward-
backward splitting,” SIAM Multiscale Model. Simul., vol. 4, no. 4, pp.
1168–1200, 2005.
[25] I. Daubechies, M. Defrise, and C. De Mol, “An iterative thresholding
algorithm for linear inverse problems with a sparsity constraint,”
Commun. Pure Appl. Math., vol. 57, pp. 1413–1541, 2004.
[26] I. Daubechies and W. Sweldens, “Factoring wavelet transforms into
lifting steps,” J. Fourier Anal. Appl., vol. 4, no. 3, pp. 245–267, 1998.
[27] G. Davis, S. Mallat, and M. Avellaneda, “Greedy adaptive approxima-
tion,” J. Construct. Approx., vol. 13, no. 1, pp. 57–98, 1997.
[28] X. Delaunay, M. Chabert, V. Charvillat, and G. Morin, “Satellite image
compression by post-transforms in the wavelet domain,” Elsevier
Signal Process., vol. 90, no. 2, pp. 599–610, 2010.
910 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 5, SEPTEMBER 2011
[29] L. Demanet and L. Ying, “Wave atoms and sparsity of oscillatory pat-
terns,” J. Appl. Comput. Harmon. Anal., vol. 3, no. 23, pp. 368–387,
2007.
[30] L. Demaret, N. Dyn, and A. Iske, “Image compression by linear splines
over adaptive triangulations,” Signal Process. J., vol. 86, no. 7, pp.
1604–1616, 2006.
[31] L. Demaret, F. Friedrich, H. Führ, and T. Szygowski, “Multiscale
wedgelet denoising algorithms,” in Proc. SPIE, Wavelets XI, 2006,
vol. 5914, p. X1-12.
[32] R. Distasi, M. Nappi, and S. Vitulano, “Image compression by
B-tree triangular coding,” IEEE Trans. Commun., vol. 45, no. 9, pp.
1095–1100, Sep. 1997.
[33] D. Donoho, “Wedgelets: Nearly minimax estimation of edges,” Ann.
Statist., vol. 27, no. 3, pp. 859–897, 1999.
[34] D. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52,
no. 4, pp. 1289–1306, 2006.
[35] D. L. Donoho, “Cart and best-ortho-basis: A connection,” Ann. Statist.,
vol. 25, no. 5, pp. 1870–1911, 1997.
[36] D. L. Donoho and I. Johnstone, “Minimax estimation by wavelet
shrinkage,” Ann. Statist., vol. 26, pp. 879–921, 1998.
[37] N. Dyn, D. Levin, and S. Rippa, “Data dependent triangulations for
piecewise linear interpolation,” IMA J. Numer. Anal., vol. 10, no. 1,
pp. 137–154, Jan. 1990.
[38] A. A. Efros and T. K. Leung, “Texture synthesis by non-parametric
sampling,” in Proc. ICCV ’99: Proc. Int. Conf. Comput. Vis.-Vol. 2,
1999, p. 1033, IEEE Comput. Soc..
[39] A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis
and transfer,” in Proc. Siggraph’01, Aug. 2001, pp. 341–346.
[40] M. Elad and M. Aharon, “Image denoising via sparse and redun-
dant representations over learned dictionaries,” IEEE Trans. Image
Process., vol. 15, no. 12, pp. 3736–3745, Dec. 2006.
[41] M. Elad, M. A. T. Figueiredo, and Y. Ma, “On the role of sparse and
redundant representations in image processing,” Proc. IEEE, vol. 98,
no. 6, pp. 972–982, 2010.
[42] A. Elmoataz, O. Lezoray, and S. Bougleux, “Nonlocal discrete regular-
ization on weighted graphs: A framework for image and manifold pro-
cessing,” IEEE Trans. Image Process., vol. 17, no. 7, pp. 1047–1060,
Jul. 2008.
[43] K. Engan, S. O. Aase, and J. H. Husoy, “Method of optimal directions
for frame design,” in Proc. ICASSP’99, Washington, DC, 1999, pp.
2443–2446, IEEE Comput. Soc..
[44] G. Facciolo, P. Arias, V. Caselles, and G. Sapiro, “Exemplar-based
interpolation of sparsely sampled images,” IMA Preprint Series # 2264,
2009.
[45] J. C. Feauveau, “Analyze multirésolution avec un facteur de résolution
,” J. Traitement du Signal, vol. 7, no. 2, pp. 117–128, 1990.
[46] R. M. Figueras i Ventura, P. Vandergheynst, and P. Frossard, “Low-
rate and flexible image coding with redundant representations,” IEEE
Trans. Image Process., vol. 15, no. 3, pp. 726–739, Mar. 2006.
[47] F. Friedrich, L. Demaret, H. Fuhr, and K. Wicker, “Efficient moment
computation over polygonal domains with an application to rapid
wedgelet approximation,” SIAM J. Sci. Comput., vol. 29, no. 2, pp.
842–863, 2007.
[48] H. Fuhr, L. Demaret, and F. Friedrich, “Beyond wavelets: New image
representation paradigms,” Survey article, in Document and Image
Compression, 2006, M. Barni and F. Bartolini.
[49] P. L. George, H. Borouchaki, P. J. Frey, P. Laug, and E. Saltel, “Mesh
generation and mesh adaptivity: Theory, techniques,” in Encyclopedia
of Computational Mechanics, E. Stein, R. de Borst, and T. J. R. Hughes,
Eds. New York: Wiley, 2004.
[50] O. N. Gerek and A. E. Cetin, “Adaptive polyphase subband decompo-
sition structures for image compression,” IEEE Trans. Image Process.,
vol. 9, no. 10, pp. 1649–1660, Oct. 2000.
[51] G. Gilboa, J. Darbon, S. Osher, and T. F. Chan, “Nonlocal convex func-
tionals for image regularization,” UCLA CAM Rep. 06-57, 2006.
[52] R. Gribonval and E. Bacry, “Harmonic decomposition of audio signals
with matching pursuit,” IEEE Trans. Signal Process., vol. 51, no. 1, pp.
101–111, Jan. 2003.
[53] G. Yu, G. Sapiro, and S. Mallat, “Solving inverse problems with piece-
wise linear estimators: From Gaussian mixture models to structured
sparsity,” 2011, preprint: arxiv.org/abs/1006.3056, to be published.
[54] H. J. Heijmans, B. Pesquet-Popescu, and G. Piella, “Building nonre-
dundant adaptive wavelets by update lifting,” J. Appl. Comput.
Harmon. Anal., vol. 18, no. 3, pp. 252–281, May 2005.
[55] R. Jenatton, J. Mairal, G. Obozinski, and F. Bach, “Proximal methods
for hierarchical sparse coding,” Preprint HAL inria-00516723, 2010.
[56] S. Kindermann, S. Osher, and P. W. Jones, “Deblurring and denoising
of images by nonlocal functionals,” SIAM Mult. Model. Simul., vol. 4,
no. 4, pp. 1091–1115, 2005.
[57] H. Krim, D. Tuckier, S. Mallat, and D. Donoho, “Near-optimal risk
for best basis search,” IEEE Trans. Inf. Theory, vol. 45, no. 7, pp.
2225–2238, 1999.
[58] E. Le Pennec and S. Mallat, “Bandelet image approximation and com-
pression,” SIAM Multiscale Model. Simul., vol. 4, no. 3, pp. 992–1039,
2005.
[59] E. Le Pennec and S. Mallat, “Sparse geometric image representations
with bandelets,” IEEE Trans. Image Process., vol. 14, no. 4, pp.
423–438, Apr. 2005.
[60] S. Lesage, R. Gribonval, F. Bimbot, and L. Benaroya, “Learning unions
of orthonormal bases with thresholded singular value decomposition,”
in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP),
2005, vol. 5, pp. v293–v296.
[61] M. Lounsbery, T. D. DeRose, and J. Warren, “Multiresolution analysis
for surfaces of arbitrary topological type,” ACM Trans. Graphics, vol.
16, no. 1, pp. 34–73, 1997.
[62] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, “Supervised
dictionary learning,” in Proc. Adv. Neural Inf. Process. Syst., 2008, pp.
1033–1040.
[63] J. Mairal, M. Elad, and G. Sapiro, “Sparse representation for color
image restoration,” IEEE Trans. Image Process., vol. 17, no. 1, pp.
53–69, Jan. 2008.
[64] S. Mallat, “Geometrical grouplets,” J. Appl. Comput. Harmon. Anal.,
vol. 26, no. 2, pp. 161–180, 2009.
[65] S. Mallat, A Wavelet Tour of Signal Processing, 3rd ed. Amsterdam,
The Netherlands: Elsevier, 2009.
[66] S. Mallat and G. Peyré, “Orthogonal bandlets bases for geometric
image approximation,” Commun. Pure Appl. Math., vol. 61, no. 9, pp.
1173–1212, 2008.
[67] S. Mallat and G. Yu, “Super-resolution with sparse mixing estimators,”
IEEE Trans. Image Process., vol. 19, no. 11, pp. 2889–2900, 2010.
[68] S. Mallat and Z. Zhang, “Matching pursuits with time-frequency dictio-
naries,” IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397–3415,
Dec. 1993.
[69] S. Masnou, “Disocclusion: A variational approach using level lines,”
IEEE Trans. Image Process., vol. 11, no. 2, pp. 68–76, Feb. 2002.
[70] B. Matei and A. Cohen, Nonlinear Subdivison Schemes : Applications
to Image processing, in Tutorials on Multiresolution in Geometric
Modelling, A. Iske, E. Quak, and M. S. Floater, Eds. New York:
Springer Verlag, 2002, pp. 93–97.
[71] P. Maurel, J. F. Aujol, and G. Peyré, “Locally parallel texture mod-
eling,” SIAM J. Imaging Sci.., vol. 4, no. 1, pp. 413–447, 2010.
[72] Y. Meyer, Oscillating Patterns in Image Processing and Nonlinear
Evolution Equations. Boston, MA: American Mathematical Society,
2001.
[73] M. Mignotte, “A non-local regularization strategy for image deconvo-
lution,” Pattern Recogn. Lett., vol. 29, no. 16, pp. 2206–2212, Dec.
2008.
[74] J.-M. Mirebeau and A. Cohen, “Greedy bisection generates optimally
adapted triangulations,” Laboratoire Jacques-Louis Lions, 2008, Tech.
Rep..
[75] B. A. Olshausen and D. J. Field, “Emergence of simple-cell receptive-
field properties by learning a sparse code for natural images,” Nature,
vol. 381, no. 6583, pp. 607–609, Jun. 1996.
[76] N. Ouarti and G. Peyré, “Best basis search in a non-stationary wavelet
packets dictionary,” in Proc. ICIP 2009, 2009.
[77] G. Peyré, “Image processing with non-local spectral bases,” SIAM Mul-
tiscale Model. Simul., vol. 7, no. 2, pp. 703–730, 2008.
[78] G. Peyré, “Manifold models for signals and images,” Comput. Vis.
Image Understand., vol. 113, no. 2, pp. 249–260, Feb. 2009.
[79] G. Peyré, “Sparse modeling of textures,” J. Math. Imaging Vis., vol.
34, no. 1, pp. 17–31, 2009.
[80] G. Peyré, “Best basis compressed sensing,” IEEE Trans. Signal
Process., vol. 28, no. 5, 2010.
[81] G. Peyré, “Texture synthesis with grouplets,” IEEE Trans. Patt. Anal.
Mach. Intell., vol. 32, no. 4, pp. 733–746, Apr. 2010.
[82] G. Peyré, S. Bougleux, and L. D. Cohen, “Non-local regularization of
inverse problems,” Inverse Problems Imag., 2011, to be published.
[83] G. Peyré, J. Fadili, and J.-L. Starck, “Learning the morphological di-
versity,” SIAM J. Imag. Sci., vol. 3, no. 3, pp. 646–669, 2010.
[84] G. Peyré and S. Mallat, “Surface compression with geometric ban-
delets,” ACM Trans. Graphics, (SIGGRAPH’05), vol. 24, no. 3, pp.
601–608, Aug. 2005.
PEYRÉ: REVIEW OF ADAPTIVE IMAGE REPRESENTATIONS
[85] G. Peyré, E. Le Pennec, C. Dossal, and S. Mallat, “Geometric estima-
tion with orthogonal bandlet bases,” in SPIE Wavelet XII, 2007.
[86] G. Plonka, “The easy path wavelet transform: A new adaptive wavelet
transform for sparse representation of two-dimensional data,” SIAM
Multiscale Model. Simul., vol. 7, no. 3, pp. 1474–1496, 2009.
[87] J. Portilla and E. P. Simoncelli, “A parametric texture model based on
joint statistics of complex wavelet coefficients,” Int. J. Comput. Vis.,
vol. 40, no. 1, pp. 49–70, 2000.
[88] J. Portilla, V. Strela, M. Wainwright, and E. P. Simoncelli, “Image de-
noising using a scale mixture of Gaussians in the wavelet domain,”
IEEE Trans. Image Process., vol. 12, no. 11, pp. 1338–1351, Nov.
2003.
[89] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based
noise removal algorithms,” Phys. D, vol. 60, no. 1–4, pp. 259–268,
1992.
[90] J. Ruppert, “A Delaunay refinement algorithm for quality 2-dimen-
sional mesh generation,” J. Algorithms, vol. 18, no. 3, pp. 548–585,
1995.
[91] P. Schröder and W. Sweldens, “Spherical wavelets: Efficiently repre-
senting functions on the sphere,” in Proc. SIGGRAPH 95, 1995, pp.
161–172.
[92] A. Secker and D. Taubman, “Lifting-based invertible motion adap-
tive transform (LIMAT) framework for highly scalable video compres-
sion,” IEEE Trans. Image Process., vol. 12, no. 12, pp. 1530–1542,
Dec. 2003.
[93] O. G. Sezer, O. Harmanci, and O. G. Guleryuz, “Sparse orthonormal
transforms for image compression,” in Proc. IEEE Int Conf. Image
Process. (ICIP), 2008, pp. 149–152.
[94] R. Shukla, P. L. Dragotti, M. Do, and M. Vetterli, “Rate distortion op-
timized tree structured compression algorithms for piecewise smooth
images,” IEEE Trans. Image Process., vol. 14, no. 3, pp. 343–359, Mar.
2005.
[95] K. Skretting and J. H. Husoy, “Texture classification using sparse frame
based representations,” EURASIP J. Appl. Signal Process., vol. 2006,
no. 1, p. 102, 2006.
[96] W. Sweldens, “The lifting scheme: A custom-design construction of
biorthogonal wavelets,” J. Appl. Comput. Harmon. Anal., vol. 3, no. 2,
pp. 186–200, 1996.
911
[97] A. D. Szlam, M. Maggioni, and R. R. Coifman, “Regularization on
graphs with function-adapted diffusion processes,” J. Mach. Learn.
Res., vol. 9, pp. 1711–1739, 2008.
[98] J. A. Tropp, “Just relax: Convex programming methods for identifying
sparse signals in noise,” IEEE Trans. Inf. Theory, vol. 52, no. 3, pp.
1030–1051, Mar. 2006.
[99] M. Wakin, J. Romberg, H. Choi, and R. Baraniuk, “Wavelet-domain
approximation and compression of piecewise smooth images,” IEEE
Trans. Image Process., vol. 15, no. 5, pp. 1071–1087, May 2006.
[100] D. M. Wang, L. Zhang, A. Vincent, and F. Speranza, “Curved wavelet
transform for image coding,” IEEE Trans. Image Process., vol. 15, no.
8, pp. 2413–2421, Aug. 2006.
[101] L.-Y. Wei and M. Levoy, “Fast texture synthesis using tree-structured
vector quantization,” in Proc. 27th Annu. Conf. Comput. Graphics In-
teract. Tech. SIGGRAPH’00: , 2000, pp. 479–488, ACM Press/Ad-
dison-Wesley.
[102] J. Wright, Y. Ma, J. Mairal, G. Sapiro, T. S. Huang, and S. Yan,
“Sparse representation for computer vision and pattern recognition,”
Proc. IEEE, vol. 98, no. 6, pp. 1031–1044, Jun. 2010.
Gabriel Peyré graduated from Ecole Normale
Supérieure de Cachan, Cahan, France, in 2003 and
received the Ph.D. degree in applied mathematics
from École Polytechnique, Paris, France, in 2005.
His research interests include geometrical repre-
sentations for image processing with applications
in computer and biological vision. He worked with
S. Mallat on image compression with bandlets
during the Ph.D. degree. Since 2006, he has been
a Researcher at the Centre Nationale de Recherche
Scientifique (CNRS), working in Ceremade, Univer-
sity Paris-Dauphine. Since 2005, he has coauthored 22 papers in international
journals, 35 conference proceedings in top vision and image processing
conferences, and two books. He is the main contributor of the third edition of
the book Wavelet Tour Of Signal Processing (Academic, 2008) by S. Mallat.
He is the creator of the “Numerical tours of signal processing” (www.numer-
ical-tours.com), a popular online repository of Matlab/Scilab resources to teach
modern signal and image processing.