SET THEORETIC SIGNAL PROCESSING*
P. L. Combettes
Department of Electrical Engineering
City College and Graduate School
City University of New York
New York, NY 10031, USA
ABSTRACT
In signal processing, estimation and design problems
have traditionally been solved via optimization meth-
ods. Such an approach often leaves little flexibility to
incorporate constraints. In the set theoretic frame-
work, each constraint on the problem is associated
with a set in the solution space and their intersec-
tion represents the set of solutions. The main feature
of this framework, which places emphasis on feasibility
rather than optimality, is to provide great flexibility in
the incorporation of statistical as well as nonstatistical
constraints. In signal processing, set theoretic meth-
ods have been applied to several classes of estimation
and design problems. In this paper we give a general
overview of the field and discuss recent advances. We
conclude the paper with areas of on-going research and
open problems.
INTRODUCTION
Most signal processing problems fall into two main
categories: estimation and design problems. In esti-
mation problems, the goal is to produce estimates of
an object h from the measurements of signals physi-
cally or mathematically related to it and some a priori
knowledge. Such problems include:
System identification: h characterizes a system,
e.g., impulse response, coefficients of a parametric
model (ARMA, Volterra, etc.), poles and zeros.
Bearing estimation: h is a vector containing the
directions of arrivals of sources impinging on an
array of sensors.
Signal restoration: h is the original form of a signal
which has been blurred and corrupted by noise.
*This work was supported by the National Science
Foundation under grant MIP-9308609.
- 1 -
0 Signal reconstruction: h is the original form of a
signal which is partially known, e.g., from Fourier,
Radon, or Wavelet transform information.
e Spectral estimation: h is the spectral distribution
of a random process which has been observed over
a finite support.
On the other hand, in design problems, the objective
is to create an object h that satisfies certain specifica-
tions. Let us mention in particular:
0 Filter design: h is the set of coeflhcients of a filter
(or filter bank).
0 Pulse design: h is a pulse shape for a digital com-
munication system.
0 Window design: h is a window for data analysis.
0 Beamforming: h contains the parameters of an ar-
ray of sensors that produces a desired beam pat-
tern.
On an abstract level, estimation and design problems
can be reduced to the synthesis of an (object in a space
Z according to some criterion describing what is, in the
user’s belief, a good solution. Traditionally, the crite-
rion has been optimality with respect to some objec-
tive. The systematic quest for optimal solutions, which
is now well rooted in the scientific culture, originated
in the late 1940’s. It has been motivated to a large
extent by the conjunction of technological advances in
computing machinery as well as progresses in branches
of applied mathematics such optimization theory, nu-
merical analysis, and statistics. Naturally, optimal es-
timators and designs also rule in signal processing and
we can mention in particular the prominence of least-
squares filter designs, maximum likelihood estimates
in system identification or array processing, maximum
entropy spectral estimates, Bayesian signal recoveries.
Optimal estimation and design procedures have un-
doubtedly provided satisfactory solutions in a wide
range of applications. However, a reservation that can
be formulated vis-&-vissuch approaches is that the cri-
terion of optimality is inherently subjective and that
different criteria may yield different solutions. More-
over, the resulting optimization problem is usually not
easily modifiable to incorporate whatever a p o r i in-
formation or constraint may be available about the
problem. In practice, one tends to select an optimality
criterion on grounds which are seldom related to ra-
tional and practical goals reflecting the specificities of
the problem at hand. One generally favors standard
procedures that provide, without too much effort, a
solution. For instance, the maximum likelihood (ML)
estimator has enjoyed widespread use on account of
its “desirable” properties such as normality and effi-
ciency. Nonetheless] the practical value of these prop-
erties is quite debatable and, moreover, they are only
asymptotic. In addition] the ML estimator may not
be defined or may lead to irrelevant results. Finally,
computational tractability generally imposes standard
assumptions, such as normality of the random pro-
cesses involved, and leaves almost no room for the in-
corporation of a priori knowledge. On that last point,
the Bayesian framework is better suited. However, it
requires to model the object h to be estimated as the
realization of a random variable to which a distribution
function is assigned (in most situations, there is how-
ever nothing random about h, which may for instance
be a fixed but unknown physical constant). Moreover,
not all U przori information can be easily described in
probabilistic terms and the resulting prior distribution
is usually too complex t o yield a tractable minimiza-
tion of the resulting conditional expectation. At all
events, a Bayesian technique is fundamentally subjec-
tive since the specification of the prior distribution and
of the loss function is left to the user. In the extreme,
the user could reach any conclusion he desires through
biased choices of these specifications.
In the set theoretic framework, the governing principle
is to produce solutions which satisfy all the constraints
about the problem [8]. These constraints arise from
data measurements and a priori knowledge in estima-
tion problems, and from technical specifications in de-
sign problems. Each constraint 9,defines a so-called
property set Si in the solution space Z, namely
S; = { U E Z [ a satisfies qi}. (1)
The problem is then to produce a feasible solution,
i.e., to find a point in the intersection of all the prop-
erty sets. The set theoretic framework departs radi-
cally from the conventional approach which relies on
optimal decision theory and in which feasibility is ei-
- 2 -
ther ignored or of secondary importance. Its main fea-
ture is to provide great flexibility in the incorporation
of statistical as well as nonstatistical constraints. Of
course, the requirement for a single estimate is relaxed
and the solution is a set of equally valid solutions. In
this regard, it has been charged that the validity of set
theoretic methods could be questioned on the grounds
that they do not yield a unique solution but a set of
solutions and are therefore suboptimal. Let us first
remark that, although it may be gratifying to have ob-
tained the “optimal” solution, optimality claims were
seen to be subjective. At best, they result in a qualita-
tive selection of a feasible solution. Moreover, from a
conceptual standpoint, demanding that only one solu-
tion be acceptable for problems which are notoriously
affected by uncontrollable factors (e.g. noise, uncer-
tain data formation model) seems somewhat unwise.
Let us also note that methods which yield unique solu-
tions are usually iterative and their solution depends
on the stopping rule. Certainly, such a criterion does
not define a unique solution.
The set theoretic framework has been applied to nu-
merous estimation and design problems in signals pro-
cessing, e.g., signal restoration, image reconstruction,
signal extrapolation, array processing, seismic signal
processing, spectral estimation, system identification,
deconvolution, filter design, window design, antenna
array design, pulse shape design [3, 4, 8, 17, 18, 21,
23, 261. More recent applications can be found in the
areas of geophysical signal processing [a], signal recon-
struction from wavelet maxima [6], radar signal anal-
ysis [19] and biomedical image processing [ZO].
The purpose of this paper is to give a general overview
of the field of set theoretic signal processing and to dis-
cuss recent advances in specific areas. We shall first
discuss set theoretic formulations and then the prob-
lem of the synthesis of set theoretic solutions. The
last section is devoted to areas of on-going research
and open problems.
SET THEORETIC FORMULATIONS
Let ( @ i ) i e ~be the family of constraints available on
the problem and let Z be the solution space. According
to (l),these constraints define a collection of property
sets (Si)iEr in Z. The pair (E,(Si)iEr) is called a set
theoretic formulation. In this set theoretic format, the
problem can be stated as
Find u E S = n S , . (2)
;€I
It should be noted that the set theoretic formulation is
not unique since several options are usually available
to select the solution space, For instance, the identifi-
cation of an autoregressive system can be carried out
in the space of regression coefficients or in the space of
reflection coefficients; a signal deconvolution problem
can be posed in the time domain or in the frequency
domain. In general one chooses the solution space in
which the available constraints are most easily repre-
sented and give “nice))sets, in particular convex ones.
In the case of deterministic constraints, the construc-
tions of sets is fairly straightforward. For instance, if
one wants t o synthesize a signal whose energy does not
exceed a bound E , the corresponding set is the closed
ball
si
= { a E 2 I lla112 5 E } . (3)
If the signal is constrained to have band-limit ] - B , B [ ,
the corresponding set is the closed vector space
Si = (U E 8 I (VY E R) ( F u ) ( v )= 0 if [VI 2 B},
(4)
where F denotes the Fourier transform operator. Like-
wise, in the design of an N-tap low-pass filter, the lin-
ear phase constraint leads to the subspace
= { U E R~ 1 (vi E { I , + - . , N ) )ai = aN+I-i}.
(5)
In signal processing problems, stochastic information
is often available in the form of probabilistic attributes
of the uncertainty that surrounds the data formation
model, e.g., modeling errors or measurement noise.
These attributes can be amplitude bounds, moments,
mixed moments, distributions, spectral and polyspec-
tral properties, etc. In this case, the constraint \ki
takes the form of a statistical test and the resulting
set is a confidence region in the solution space [8]. As
an example, suppose that the observed data t are re-
lated to the true object h via the model
2 = T ( h )+ U , (6)
where T is a known operator and U the uncertainty.
For a probabilistic property 4
i of U , the set of interest
is’
Si = { a E B I 2 - T(a) is consistent with Qi}.(7)
The problem is then to determine the confidence level
to be placed on each such set. As discussed in [lo],
this involves three conflicting objectives:
0 Since the complexity of algorithms for finding a
feasible solution increases with the number of sets,
few should be used.
~
0 The smallest feasibility set should be obtained in
(2) to guarantee the quality of solutions.
0 Since each set is a confidence region, the solution
set S should contain the true object h with a rea-
sonable level of confidence.
As in [lo], let us note that in the caste of deterministic
information, the more sets the bett,er since the fea-
sibility sets becomes smaller in (2). However, in the
case of statistical information, this iis not necessarily
true. Indeed, for a given probability that S contains
h , the confidence level on (and therefore the size of)
the individual sets much increase as tjhe number of sets
increases; in some cases intersecting more larger sets
may not yield a smaller intersection than intersecting
fewer smaller sets, This leads to the rather counter-
intuitive conclusion that some statistical constraints
may have to be discarded in order to obtain a small
set of solutions at a given probability that h E S .
FEASIBILITY PROBLEM
Overview. Solving a signal processing problem with
set theoretic formulation (Z, ( S i ) i E I amounts
) to solv-
ing the feasibility problem (2). This basic problem
arises in many branches of applied mathematics and
it has therefore a long history [5, 8, 151. Because it
can usually not be solved in one step:, most feasibility
algorithms are iterative:
0 Projection methods [l,8, 111: a,+l is computed in
terms of the simultaneous projections of a, onto
some of the sets. Although such methods exist for
metric spaces, they have been most developed in
the context of a Hilbert space in, which the sets
(Sj)iEr are closed and convex.
0 Subgradient projection methods [l, 51: the sets
take the form Si = { a E Z 1 $ $ ( U ) 5 0}, where
gi is a convex functional and 2 a Hilbert space. At
iteration n , the set Si is activated lby projecting a,
onto the hyperplane Hi(u,), which is the intersec-
tion of E with a hyperplane of E x IR tangent to
the graph of g; at the point (U,,, g,i(a,,)).
0 Minimization method: In a Banach space Z, the
feasibility problem (2) can be posed as that of min-
imizing a functional CP whose set (of minimizers is
contained in S. For instance, if I is finite, take
( w i ) i e I C]O, 11 with CiEr
w; = 1. Then the func-
tional

‘In t h e so-called bounded error models, the information

on takes the form of amplitude bounds and yields simple
U
sets when T is linear [14, 22, 251.

- 3 -
 admits S as a set of minimizers (d denotes the
distance of 5 ) . Another example is the functional
(VU E E) @ ( a ) = sup d(a, Si). (9)
iEJ
Standard optimization methods can then be used
to solve (a), depending on the properties of Q, (con-
vexity, continuity, subdifferentiability) and of E.
Random search methods [13]: For 5 = Rk,this
method consists in generating points at random
over an adaptively reduced search region until a
feasible point is found. The main advantage of
this Monte Carlo approach is its basic simplicity
and the lack of assumptions about the geometrical
complexity of the sets. Because of computational
cost, it is not appropriate for large scale applica-
tions.
Approximation methods: One builds a sequence
(Sn),yo of supersets of the feasibility set S con-
verging in some sense to S . This approach has
been taken is certain problems with bounded error,
where the only information used to construct prop-
erty sets is a magnitude bound on the perturbation
of the system (noise, modelling errors) 114, 22, 251.
The feasibility set is refined at each new observa-
tion of the data process and S, is usually taken to
be a bounding ellipsoid or hyperparallelepiped.
Recent developments. The theory of projection al-
gorithms has been the focus of a lot of research recently
in the case when ( S i ) i c ~are closed and convex subsets
of a Hilbert space 2. Throughout this section, Z has
-
scalar product < .(.>, norm (1 (1, and distance d. A
sequence (an),20 C E is said to converge to a E E
- - converges t o zero and weakly
strongly if (Ilu,, a((),>o
if, for every b in Z, (< a,-alb >)n>O converges to zero.
Strong convergence implies weak convergence and, in
finite dimensional spaces, the converse is true.
Let P;(a) denote the projection of a point a onto Si,
-
i.e., the unique point in Si such that ] ! U Pi(a>ll =
inf(lla - bl\ 1 b E Si}. In most signal processing appli-
cations, the method of successive projections (POCS)
(Vn E N) a,+l = Pi,(a,), i, = n(modm)-t-1, (10)
where m is the total number of sets (assumed finite
here) has been used to solve (2). In this method, the
sets are activated cyclically: a1 is the projection of
a0 onto Sl, a2 is the projection of a1 onto Sz, ..., am
is the projection of a,-1 onto Sm, a,+l is the pro-
jection of a, onto SI, ..., akm+i is the projection of
ukm+;-1 onto Si, etc. POCS converges weakly to a
- 4 -
point in S for every a0 E Z (strong convergence also
holds if additional conditions are imposed on the sets)
[8]. A straightforward extension of (10) is obtained by
introducing relaxation coefficients (A,),?o in [ E , 2 - E ]
(0 < E 5 1) as follows
(Vn E N) an+l = an + L ( P i , ( a n ) - an). (11)
In other words, instead of taking the exact projec-
tion P;,,(an)of a, onto Sin, one takes a relaxed one,
i.e., a point an+l on the segment between a, and
its reflection 2P;,,(a,) - a , with respect to Sin.If
1 < An < 2, u,+l is called an overrelaxation, if A, 1,
a,+l is called a projection (unrelaxed iteration), and,
if 0 < A, < 1, a,+i is called an underrelaxation. Re-
laxed POCS (11)possesses the same convergence prop-
erties as (10).
Despite its widespread popularity in signal processing
applications, POCS has three major shortcomingsn2
0 Serial structure: A salient feature of POCS is its
serial algorithmic structure: at each iteration only
one of the sets can be activated. Clearly, such a
structure does not lend itself naturally to an im-
plementation on an architecture with parallel pro-
cessors.
0 Inconsistent formulations: In some problems, the
set theoretic formulation may be inconsistent m
a result of two or more incompatible constraints.
In such instances, it would be desirable to obtain
an approximate feasible solution. On that score,
POCS falls short. First of all, it is clear that
(11) cannot converge if S = 0 since the tail of
the sequence will keep oscillating from one set to
the next indefinitely. At best, the periodic subse-
quence (amn+i)n>o (1 5 i 5 m) converges weakly
to a point in Si. This solution is unsatisfactory
since it is guaranteed to satisfy only the ith con-
straint, which constitutes a poor approximation of
a feasible solution.
0 Speed of convergence: A long recognized problem
with POCS is its slow convergence. Conceptually,
POCS can be accelerated by properly relaxing the
projections at each iteration. Unfortunately, even
for simple set theoretic formulations, there is no
systematic method for determining so as
to speed up the iterations in (11).
'Alternative serial projection schemes have been de-
vised in which the sets are activated in a noncyclic manner
[5, 81. They will not be discussed here since they have
rarely been used in signal processing and have essentially
the same inherent shortcomings as POCS.
Recently, several authors have investigated the follow-
ing method of parallel projections (MOPP) that over-
comes the shortcomings of POCS [l, 9, 11, 121
(vn E N) a n + l = an + An(xwi,npi,n(an)-an),
;€I,
( 12)
(0 < E 5 1) and
where (An)n,o C]E,+a[
C wi,n = 1 and (Vt E I ) Wi,n > 0. (13)
iEI,
Practically, iteration n of MOPP in (12) is performed
as follows. First, one selects the sets to be activated;
In contains the indices of these sets. Then, for every i
in In,one computes the projection P;(on) of a,, onto
Si and forms any strictly convex combination cn =
xiGI* wi,,Pi(a,) of these projections. The position
of the new iterate a,+l on the ray emanating from a,
and going through c, is determined by the relaxation
parameter A,. By comparing (11) and (12), it is clear
that POCS is a particular case of MOPP, obtained by
letting I, = {n (mod m) 1). + A conceptual concern with this approach, however, is
The superiority of MOPP over POCS is threefold:
Parallel structure: at each iteration, the projec-
tions onto the selected sets can be computed si-
multaneously on concurrent processors. It should
also be pointed out that MOPP is more flexible
than a method in which all the sets would be acted
upon at each iteration (i.e., In = I in (12)). This
allows a better distribution of the computational
load among the processors.
Inconsistent problems: It is shown in [9] that when
S = 8,(A,), c [ ~ , -2 E], and the weights are
constant over the iterations, say wi,n = w;,then
MOPP converges weakly to a point that minimizes
@ in (8), that is, a weighted least-squares solution
to the inconsistent feasibility problem. From a sig-
nal recovery point of view, such a solution is clearly
more acceptable and useh1 than that generated by
MOSP, whose properties are elusive.
Convergence: As shown [12], MOPP converges for
relaxations in the adapted range
(Vn EM) E 5 An 5 ( 2 - ~ ) L n , (14)
where the extrapolation coefficient Ln is defined
by
- 5 -
Since the extrapolation parameter L, in (15) is
never less than 1, the condition A, < 2 is by-passed
and large overrelaxations are allowed. These ex-
trapolated relaxations contribute to significantly
accelerating convergence to a feasible solution in
numerical applications. Thus, (14) was found to
accelerate convergence by sever#alorders of magni-
tude in [12].
NEW DIRECTIONS
In this section we point out certain areas of on-going
research as well as open problems that deserve further
investigation.
Alternative set theoretic frameworks. In the
above framework, we use sets to represent informa-
tion and classes of signals. There m<aybe more flexible
ways to represent and exploit information, especially
when it is imprecise or inaccurate. The first alterna-
tive set theoretic framework that cornes to mind is that
of fuzzy sets [16], which has already been used in [7].
that it attempts to model inaccurate information with
an exact mathematical being, namely a membership
function. Another concern is that it essentially brings
the problem back to the optimization of some aggre-
gation of membership functions, the very type of sub-
jective problem we were trying to get away from in the
first place. More general set theories could be investi-
gated such as semi-sets theory [24].
Optimal use of information. In many problems a
lot of constraints are available, especially of the statis-
tical nature. It is usually unclear, however, how these
constraints are interrelated and how to select those
that will prove the most efficient in effectively reduc-
ing the size of the solution set. This problem should
be studied further.
Algorithms. With MOPP, exact projections must
be computed at each iteration. In many applications
this constitutes an obstacle as exact projections can
be difficult if not impossible to compute. We are cur-
rently investigating algorithms in which the exact pro-
jections can be replaced by approximate ones a t each
iteration. This approach is expected to bring together
projections methods as well as subgradient projection
methods. Another point that deserves investigation
is the extension of MOPP to nonconvex problems, a
context in which local convergence iresults should be
establishable.
Applications. Although set theoretic methods have
been widely used in certain areas of signal process-
ing such as signal and image recovery, they have only
enjoyed limited use in others such as spectral estima-
tion a n d array signal processing. It should also be
pointed out that the set theoretic approach has not
been used in detection problems, which represent an
important field of applications besides estimation and
design problems.
REFERENCES
[l] H. H. Bauschke and J. M. Borwein, “On projection
algorithms for solving convex feasibility problems,”
submitted for publication.
[a] I. T. Bjarnason and W. Menke, “Application of the
POCS inversion method to cross-borehole imaging,”
Geophysics, vol. 58, no. 7, pp. 941-948, July 1993.
[3] 0. M. Bucci, G. Franceschetti, G. Mazzarella, and
G. Panariello, “Intersection approach to array pattern
synthesis,” IEE Proceedings-H, vol. 137, no. 6, pp.
349-357, December 1990.
[4] J. A. Cadzow, “Signal enhancement - a composite
property mapping algorithm,” IEEE Transactions on
Acoustics, Speech, and Signal Processing, vol. 36, no.
1, pp. 49-62, January 1988.
[5] Y. Censor, “Iterative methods for the convex feasibil-
ity problem,” Annals of Discrete Mathematics, vol.
20, pp. 83-91, 1984.
[6] A. E. Qetin and R. Ansari, ”Signal recovery from
wavelet transform maxima,” IEEE Transactions on
Signal Processing, vol. 42, no. 1, pp. 194-196, Jan-
uary 1994. .
[7] M. R. Civanlar and H. J. Trussell, “Digital signal
restoration using.fuzzy sets,” IEEE Transactions on
Acoustics, Speech, and Signal Processing, vol. 34, no.
4, pp. 919-936, August 1986.
[8] P. L. Combettes, “The foundations of set theoretic
estimation,” Proceedings of the IEEE, vol. 81, no. 2,
pp. 182-208, February 1993.
[9] P. L. Combettes, ”Inconsistent signal feasibility prob-
lems: Least-squares solutions in a product space,”
IEEE Transactions on Signal Processing, accepted
for publication.
[lo] P. L. Combettes and T. J. Chaussalet, “Selecting sta-
tistical information in set theoretic signal processing,”
these proceedings.
[ll] P. L. Combettes and H. Puh, “Iterations of paral-
lel convex projections in Hilbert spaces,” Numerical
Functional Analysis and Optimization, vol. 15, nos.
3-4, pp. 225-243, 1994.
[la] P. L. Combettes and H. Puh, “A fast parallel pro-
jection algorithm for set theoretic image recovery,”
Proceedings of the IEEE ICASSP Conference. Ade-
laide, Australia, April 19-22, 1994 (also submitted for
publication).
- 6 -
[13] P. L. Combettes and H. 3. Trussell, “Set theoretic
estimation by random search,” IEEE Transactions on
Signal Processing, vol. 39, no. 7, pp. 1669-1671, July
1991.
[14] J. R. Deller, Jr., M. Nayeri, and S . F. Odeh, “Least-
square identification with error bounds for real-time
signal processing and control,” Proceedings of the
IEEE, vol. 81, no. 6, pp. 815-849, June 1993.
[15] F. Deutsch, “The method of alternating orthogonal
projections,” in Approximation Theory, Spline Func-
tions and Applications, (S. P. Singh, Editor), pp. 105-
121. The Netherlands: Kluwer, 1992.
[16] D. Dubois and H. Prade, Fuzzy Sets and Systems:
Theory and Applications. New York: Academic Press,
1980.
[17] H. Elmikati and A. A. Elsohly, “Extension of pro-
jection method to nonuniformly linear antenna ar-
rays,” IEEE Transactions on Antennas and Propa-
gation, vol. 32, no. 5, pp. 507-512, May 1984.
[18] N. E. Hurt, “Signal enhancement and the method of
successive projections,” Acta Applicandae Mathemat-
icae, vol. 23, no. 2, pp. 145-162, May 1991.
[19] 0. P. Kenny and B. Boashash, “Time-frequency anal-
ysis of backscattered signals from diffuse radar tar-
gets,” IEE Proceedings-F, vol. 140, no. 3, pp. 198-208,
June 1993.
[20] S. Oh, C. Ramon, R. J. Marks 11, A. C. Nelson, and M.
G. Meyer, “Resolution enhancement of biomagnetic
images using the method of alternating projections,”
IEEE Transactions on Biomedical Engineering, vol.
40, no. 4, pp. 323-328, April 1993.
[21] R. D. Palmer, 3. R. Crus, and D. S. Zrnid, “Enhanced
autoregressive moving average spectral estimation a p
plied to the measurement of Doppler spectral width,”
IEEE Transactions on Geoscience and Remote Sens-
ing, vol. 29, no. 3, pp. 358-368, May 1991.
[22] F. C. Schweppe, Uncertain Dynamic Systems. Engle-
wood Cliffs,NJ: Prentice-Hall, 1973.
[23] H. J. Trussell and M. R. Civanlar, “The feasible so-
lution in signal restoration,” IEEE Transactions on
Acoustics, Speech, and Signal Processing, vol. 32, no.
2, pp. 201-212, April 1984.
[24] P. Vop5nka, Mathematics in the Alternative Set The-
ory. Leipzig: Teuber, 1979.
[25] E. Walter (Guest Editor), “Parameter identifications
with error bounds,” Mathematics and Computers in
Simulation, vol. 32, no. 5-6, December 1990.
[26] D. C. Youla and H. Webb, “Image restoration by the
method of convex projections: Part 1 - theory,” IEEE
Transactions on Medical Imaging, vol. 1, no. 2, pp.
81-94, October 1982.