COST EFFECTIVENESS VIGNETTES FOR COMMUNITY

ENGAGEMENT

A paper prepared by the NICE secretariat for the Community Engagement

Programme

September 2007

1
Foreword

This paper brings together two vignettes or scenarios that bear on the cost
effectiveness of community engagement approaches. Very rarely, data that
compares different approaches (e.g. peer educators, community champions) of
community engagement, including resource usage, have ever been gathered.
The first of these, a study by Tudiver et al (1992), compared three approaches
of providing information on safer sex to gay and bisexual men in a city in the
USA. The question of whether the approach would be cost effective if they had
the same success rates in the UK but with current UK prevalence of HIV and
current UK costs has been addressed by adapting a model of cost
effectiveness by Pinkerton (1998).

The second vignette, of community engagement as part of the implementation

of a flood prevention scheme, comes not from the literature but from
contemporary practice within the UK. The vignette was suggested by one of
the members of the NICE Programme Development Group on Community
Engagement, who has had ongoing involvement in the scheme. The
engagement, carried out by or on behalf of the Environment Agency of
England and Wales, is nearing completion at the time of writing. The vignette
shows the difficulty for analysis and interpretation that arises when no ready
comparator for the engagement process has been employed. Despite that, it
shows that some analysis that aids decision makers about the usefulness of
various approaches of community engagement is still possible. Decision
makers must use more judgement in interpreting the results of such analysis
than they would if an analysis using a recognised comparator had been
possible. They must also take equity issues into account – for example,
disadvantaged groups are more likely to be slower to adapt in the face of
adversity.

This vignette has been made possible by the close cooperation of staff of the
Environment Agency, in particular Michael Guthrie, Megan Rimmer and Cath
Brooks, and by Lindsey Colbourne, a member of the Community Engagement
PDG, who suggested the topic and provided some of the information about it.

2
A potential third study by Kumpusalo et al (1996), of 6 villages in Finland which
conducted community engagement to promote healthy eating, using a different
set of villages as a control, was remarkable in that not only did it report cost
data and resource usage, but also reported “hard” outcomes: changes in blood
pressure, and cholesterol and vitamin C concentration in the blood. As data by
age were not reported and were not available for this study, it was not possible
to undertake sufficiently meaningful modelling of this potential vignette.
(Because of this, the study was not followed up in detail in this set of
vignettes.) However, the existence of such a study shows that estimation of the
cost effectiveness of some community engagement approaches would be
possible for carefully designed and executed studies. The requirements for
meaningful economic analysis in such a study are thus (a) a suitable control
group or comparator has been employed (b) the collection of “hard” outcomes
has been undertaken. By “hard outcomes” is meant accurately measured
intermediate health-related outcomes that can be mapped to quality-adjusted
life years (QALYs) or a similar generic index of health gain, by relating the
measured outcomes to the probability of death and to future expected quality
of life, and (c) either costs or resource usage (or both) are carefully collected.

Within NICE, Lorraine Taylor, Antony Morgan and Mike Kelly have helped with
the preparation of this paper.

Alastair Fischer

Reference

Kumpusalo E et al (1996) “Finnish Healthy Village Study: Impact and

outcomes of a low-cost local health promotion programme”. Health Promotion
International. 11:105-15.

3
 Cost effectiveness of two interventions in community
engagement

I Trained volunteer peer educators for the promotion of safer sex

Introduction

The Health Promotion effectiveness review identified a paper by Tudiver et al

(1992) that provides details of resource use in different interventions. This
paper describes how unpaid trained volunteer peer educators working with gay
and bisexual men were utilised to promote safe or safer sex, as an alternative
to utilising experienced paid leaders for this task or to doing nothing. From the
descriptions in the studies of resource use and health outcomes, it is possible
to construct an approximate cost effectiveness analysis which would pertain at
today’s prices in the UK. This is carried out below.

Intervention: A single group session (of three hours’ duration) of 8-12 men
conducted by 2 unpaid trained volunteer peer educators regarding safe sex.
They ran 45 different groups. To estimate training costs of volunteer peer
educators, we assume that there are 30 different volunteer peer educators
(that is, the volunteers do on average 3 sessions each – the paper says that
each volunteer led several different sessions). For brevity, call this alternative
A

Comparator: Four sessions (of two hours’ duration) for groups of 8-12 men
conducted by experienced paid leaders. They ran 21 groups. Call this
alternative B

Second comparator: 211 men on the waiting list for the intervention and main
comparator. Call this alternative C

Results of study: Effects

A: Reduction from 25% to 15% in percentages of men practising unsafe sex

after 3 months (N = 201)

B: Reduction from 24% to 20% (N = 88)

4
C: Reduction from 21% to 17% (N = 211)

It can be seen that the outcome of unpaid trained volunteer peer educator
intervention was of greater magnitude than that of either comparator:
experienced paid leaders or “do nothing”. On the assumption that there was no
clustering into groups, A was statistically significantly more effective than C (at
the p = 0.05 level), and more effective than B, but not statistically significantly
so at the 5% level. When the clustering effect is taken into account, 1 we
cannot say whether the difference between any intervention is statistically
significant, as the number of degrees of freedom 2 is reduced from 200 to a
number between 200 and 44 for alternative A, and from 87 to somewhere
between 87 and 20 for alternative B.

Costs. In order to compare the two alternatives on an equal footing, we

assume 45 groups for each of A and B, because the alternative to 45 groups
of volunteers would be 45 groups of experienced paid leaders. Assume 3
trainers for one day for the training of volunteers at a cost of £400 per day,
plus 3 days per trainer to prepare or assimilate training materials at the same
rate, yielding a total of 12 person-days). (This cost would be lower if training
courses were to be repeated at a later time.) Training therefore costs £4,800 if
volunteers are not paid. If volunteers are also paid honoraria or expenses for
training at £200 per day, there would be an additional cost of £6,000. Total
cost would be either £4,800 or £10,800, plus costs of recruitment and other
materials.

For B, each of 45 groups meets 4 times, at a cost of 500 Canadian dollars

expressed in terms of 1990 prices per meeting (given in paper), which would
translate to about £400 per meeting in 2007. This cost totals £72,000.

For C, there are no meeting or training costs.

Comparative costs for a similar study using peer educators by Pinkerton and
Kelly in southern USA in 1989 for 449 in the treatment group: $6,700 staff cost,

1
That is, because there may be an interaction effect within the group, in the extreme case there may only
be one observation per group rather than the 8 to 12 individual observations assumed in the paper
2
Degrees of freedom are the number of effectively independent observations.

5
$5,300 in “incentives” for volunteers and $4,100 for materials, a total of
$17,150. In 2007 prices, these costs would be about 1.66 times as high, or in
total $28,600, and in terms of comparable wage rates, almost exactly twice as
high, or in total $34,300. At the current exchange rate of £1 = $US2 (and in
terms of comparable wage rates), the original costs in $ equal the current cost
in £, viz staff costs of £6,700, £12,000 for total “wage” costs and total costs of
£17,150. Thus the figures assumed in the Tudiver study of £4,800 for staff
costs and £10,800 for total “wage” costs are of a similar order of magnitude.

Cost effectiveness.

A is more effective and less costly than B and so dominates B. That is, unpaid
trained volunteer peer educators are a cost effective option against
experienced paid leaders in promoting safer sex among gay and bisexual men.

C is as effective as B and less costly, so C effectively dominates B. That is, in

this study, not providing any informational sessions from experienced paid
leaders was cost effective against their provision.

A versus C: In this case, are unpaid volunteer peer educators better value-for-
money than not providing any service? Since the training of volunteers is not
free, we must estimate a cost per QALY. Pinkerton (1998) analysed Kelly’s
(1991) study of a similar group, in which a cost per QALY of $65,000 was
estimated. The costs assumed for the Tudiver study are slightly lower than
those of the Pinkerton study. However, the main differences in applying the
result to the UK are that (i) the prevalence of HIV positive men among gay and
bisexual men in the UK is 3.2 per 100 but was 9 per 100 in Pinkerton’s US
study and (ii) the follow-up period for Tudiver was 3 months compared with 2
months for Pinkerton. These differences, together with allowing for inflation
and the exchange rate, yield a cost per QALY estimate of £200,000 per QALY
for unpaid trained volunteer peer educators against no intervention when
Tudiver’s study is translated to the UK and uses 2007 prices. The reason for
this very high figure is that the intervention effectiveness for the Pinkerton
study was assumed to be only 2 months and for the Tudiver study was
assumed to be only 3 months, the respective times of follow-up. If, for

6
example, the effectiveness were to have lasted for 12 months, the estimated
cost effectiveness would be of the order of £50,000 per QALY when translated
to the UK and if it had lasted 2 years, would yield about £25,000 per QALY. In
the extreme, a change in baseline safer sex from 75% to 85% which was
permanent (rather than last a mere 2 or 3 months) would reduce the first round
of infections among such a community by 40% (that is, the proportions
practising unsafe sex reduce from 25% to 15%, a 40% reduction, thus leading
to a reduction in initial HIV transmission by that amount). In Pinkerton’s study
in a steady state, the 40 assumed infected with HIV (9% of 449) would pass
the disease on to 24 rather than 40, approximately, and assuming a
community where no-one enters or exits from outside. The number of
discounted QALYs gained would be 180, at a cost per QALY of about $130.
Translated to the UK in 2007 and using similar inferred costs for Tudiver, this
would yield £320 per QALY.

Conclusions.

The study, taken in isolation, tells us that

• Unpaid trained volunteer peer educators should be used rather than

experienced paid leaders.

• It would be better not to intervene at all rather than use experienced paid
leaders to do so.

• The reduction in unsafe sex using unpaid trained volunteer peer

educators would not be cost effective against doing nothing if the
changes to safer sex behaviour were to last only 2 or 3 months, but would
be highly cost effective if they were to last a lifetime.

Considerations

Taken in isolation, the cost effectiveness of trying to persuade gay and

bisexual men to practise safer sex depends crucially on how long the changed
behaviour patterns persist for. If the study by Tudiver is pertinent to the UK of
15 years later, the use of unpaid trained volunteer peer educators to promote

7
safer practice is likely to be cost effective. However, this study cannot be
generalised to say anything about the cost effectiveness of this community
engagement approach in other circumstances.

However, there are a number of other caveats to consider in the Tudiver study.
First, there is considerable uncertainty in the estimates of cost effectiveness.
Second, the conclusion that experienced paid leaders are not more effective
than doing nothing in delivering safer sex messages is not borne out more
generally (see references in Ellis et al (2003) section 4.1.2.2). In this respect,
the conclusion obtained above could truly be considered an isolated one.
Third, it is almost impossible in any situation where information is being
broadcast to a particular group to confine the information to that group. In a
relatively small population of gay and bisexual men, giving out information to
one group within that population will inevitably result in its spread to the rest of
the population, and that spread is likely to be rapid. In time, it can be assumed
that there will be a convergence in behaviour between those who initially
received the information from untrained peers and those who did not. In one
scenario, we could assume that the effect of the safer sex message wears off
over time, in which case the benefits of the initial safer sex sessions, however
they were conducted, might not be very beneficial. In another scenario, we
could assume instead that the effect of the safer sex message on behaviour
lasts through time. In this case, convergence in safer sex practice between
groups occurs by the diffusion of the safer sex message to everyone else in
the population. In this scenario, the increase in safer sex practices is
transferred and maintained throughout the whole population, in which case the
sessions would be extremely beneficial. The fact that convergence between
groups over time occurs is, in itself, not sufficient to conclude anything about
the cost effectiveness of the mode of delivery of the original message.

Since any informational message (such as a safer sex message) cannot be

contained within a group that is exposed to it, the effectiveness and cost
effectiveness of the mode of the delivery of this message cannot readily be
measured, even if an RCT is available to undertake the attempted
measurement. This has serious implications for the appraisal or evaluation of

8
almost all community engagement approaches, because the delivery of
change normally involves an informational message or messages. This
explanation gives succour to those who have said that community engagement
approaches are not amenable to the normal methods of analysis of
effectiveness and cost effectiveness.

References

Ellis, S et al (March 2003) HIV prevention: a review of reviews assessing the

effectiveness of interventions to reduce the risk of sexual transmission
Evidence briefing, Health Development Agency, National Health Service, UK
(now obtainable at http://www.nice.org.uk/page.aspx?o=502573)

Kelly JA, St Lawrence JS et al (1991) “HIV risk behavior reduction following

intervention with key opinion leaders of population: an experimental analysis
American J of Public Health, 81, 2, 168-71

Pinkerton SD, Holtgrave DR et al (1998) “Cost-effectiveness of a community-

level HIV risk reduction intervention” Amer J of Public Health, 88, 8, 1239-42

Tudiver F, Myers T et al (1992) “The talking sex project. Results of a

randomised controlled trial of small-group AIDS education for 612 gay and
bisexual men” Evaluation and the Health Professions 15, 4, 26-42

9
 II Community Engagement in a flood mitigation scheme

Introduction

The village of Shaldon in Devon, England, lies on the southern side of the
Teign estuary. Much of the village lies in a basin in which the level of the land
is below the level of some tides in the estuary. Parts of the village are therefore
at risk of flooding from high tides and local runoff, although the most severe
flooding will result from high tides.

The existing tidal defences take the form of relatively low flood walls which are
not continuous due to access openings between village and the estuary. High
tides alone, or in combination with wind-generated wave action, pose a risk to
almost 400 properties, and would cause damage estimated to be up to £17m
for a single episode. In a careful study of property damage expected for
various levels of floodwaters and an estimate of the probability of flooding, the
present value of cumulated flood damage has been estimated to be £45m over
the next 100 years, the project’s time horizon (Atkins pba, 2006). The cost of
an enhancement of the flood defences is not known accurately in advance of
any tendering for this work, but it is believed to be of the order of £6m. For the
sake of this exposition only, the two scenarios of £5m and £10m for
implementing the enhanced flood defences have been assumed. The actual
cost may lie within this range or outside it, so the following analysis is
illustrative only.

A flood defence system was proposed to residents of Teignmouth, a town

directly across the Estuary from Shaldon by the Environment Agency for
England and Wales in 2004. These proposals were rejected at a late stage in
the decision-making process due to a high level of local opposition. Learning
from the experience in Teignmouth, the Environment Agency decided to take a
different approach in Shaldon by seeking to engage affected communities
early in the development of the plans, in order to gain local understanding and
agreement before final decisions were made. Currently, the engagement
process is still being undertaken. While no package of final agreements have
been announced, it is clear that much headway has been made.

10
The Engagement Process

A community development worker/consultant was employed to facilitate and

organise all of the following components. (Some of this work was carried out
using the staff of the Environment Agency, and some was carried out by a firm
specialising in community engagement.)

• A personal invitation to every household/business in the village to take part

• A staffed public exhibition and drop-in in October 2005, including café style
discussions,
• A public meeting attended by 250 people (~25% of the population). It
focused on sharing the Environment Agency’s view of flooding and
ascertaining community attitudes and preferences. 87% supported doing
more work together to reduce the flood risk (5% were against)
• Formation of a Liaison Group
ƒ This has met 12 times,
ƒ All meetings have been facilitated
ƒ Meetings last 2 hours.
ƒ Meetings have had 20+ participants
ƒ Meetings are open to all, residents working alongside 6
staff/consultants from the Environment Agency. Community
representatives have all been volunteers including
o Chair of the Parish Council
o boat users
o parents of young children
ƒ The Group has had input to all the decisions about what to do to
reduce the flood risk, has worked through all the options they
could think of, estimated costs/benefits/uncertainties and has
proposed a scheme that the whole group could support as the
best way forward
ƒ Sub groups were established to work out detail and consider
related issues on surface/sewage flooding (working with water
company and Highways Department); flooding emergency plan;
communications; design development; flood gate location and
operation.
• Public exhibition and drop in, with walking tour of proposals (June 2007).
The exhibitions were staffed jointly by members of the Environment
Agency, local authority, water company and the liaison group. About 300
people attended.
• It is expected that the engagement process will be completed by spring
2008.

Engagement after this date will relate to detailed design and the impacts of
construction.

11
Engagement Outcomes

The expected costs, from inception of the Shaldon project up to deciding

whether to proceed with a preferred scheme, are likely to lie between
£500,000 and £700,000. A little less than half of this amount represents the
cost of engagement and a similar amount represents technical work appraising
and developing the scheme. A component is also devoted to learning and
development, which would be reduced for future projects. For the sake of
exposition, we shall assume that the costs, which we shall loosely call “the
costs of engagement”, equal £0.5 million.

Engagement has caused the design of the proposals for flood defence to
evolve. We assume that these changes do not affect the benefits of the
scheme in terms of the reduction of flood damage, but that they may not be
costless to implement. The cost is not separately listed here, but is subsumed
within the range of £5m to £10m assumed as the cost of the whole project

The likely tangible outcomes of the engagement process, over and above the
traditional approach (with limited community engagement), are

o Inclusion of neighbouring Ringmore in the scheme. This was

highlighted by the community engagement and its inclusion can also be
justified technically and economically.

o Discussions towards the successful resolution of surface/sewage

flooding have started. This had not been achieved for 10 years despite
campaigning by the parish council, and would not have been tackled
without the engagement exercise.

o More appropriate scheme design

ƒ Scheme perceived more sympathetically than it would otherwise
have been (because it has utilised local knowledge/preferences it
has gained support)
ƒ Scheme more appropriate for users (e.g. recreational use)

o Reduced staff stress/time on complaints

ƒ Reduced Environment Agency staff stress (e.g. compared with a
previous project in nearby Teignmouth where no similar
engagement scheme operated)
ƒ Reduced Environment Agency staff time in dealing with individual
complaints.

12
All of these elements have some costs associated with them, but in each case,
the associated benefits have been assumed to be well above the
implementation costs. Since the assumed cost range is £5m to £10m and the
original benefits have been estimated to be £45m, there is no requirement to
quantify the additional benefits for the purpose of making a positive decision
about the project. However, the question of whether the particular community
engagement approach used is cost saving as well as being cost effective
would need this to be quantified.

On top of the tangible benefits of engagement, there are intangible benefits in

terms of the positive experiences of all who involved themselves in the
process.

ƒ Increased community resilience

o raised awareness of flood risk across the community
o involvement in community response plan
o commitment by the local population for supporting the successful
operation of the flood gates and flood defence scheme, and to
other practical actions
o improved working relationships between Environment Agency,
Parish Council, local authority and water company

ƒ Improved future working relationships

o within Environment Agency
o with partner organisations (e.g. local authority, water company,
Parish Council)

No attempt has been made to quantify these benefits.

Analysis

In the simplest case, we assume that the project whose benefits have a
present value of £45m and costs between £5m and £10m (a net benefit of
between £35m and £40m) does not go ahead without engagement but goes
ahead with engagement. The net benefit from the scheme of £35m to £40m is
released for a direct cost of engagement of £0.5 million. However, there is also
an “indirect” net cost of engagement of £1.0 to £1.3 million per year due to
postponement of construction, in that a flood might arrive in the year in which
the postponement occurs. For the sake of simplicity, assume that the indirect

13
cost of postponement is for between 12 and 18 months, and equals £1.5m.
Under these assumptions, the community engagement process is still highly
cost effective: a benefit of £35-40 million for £2 million in cost (of which £1.5m
is not an accounting cost).

However, this is a highly-simplified version of the reality. We discuss several

elements that would make the model closer to reality without being by any
means exhaustive. We have implicitly assumed that the project would not have
gone ahead without the engagement. This proposition can be represented by
Teignmouth, where limited engagement resulted in the proposed scheme
being rejected due to local opposition. Costs of this aborted scheme were at
least £179,000 (see Form G Executive Summary Teignmouth). It is interesting
to note that in Teignmouth, faced with a “straw poll”, more residents voted to
reject rather than to modify the proposals there.

However, flood defence schemes have been successfully promoted in many

places with less engagement than was undertaken at Shaldon.

A different, much more limited but cheaper form of engagement than the one
undertaken in Shaldon, of asking the residents to vote on an all-or-nothing
scheme might have given a majority in favour of the project, though it might not
have been a solution that maximised benefits. To compare the outcome from
the current engagement exercise with that of a direct vote is a different
analysis from the one undertaken. Since we do not know what the result of
such a vote would have been, it is not possible to determine whether the
eventual scheme with its somewhat higher costs would have been cost
effective compared with the vote.

More generally, there would be a probability, p1, that the scheme would be
accepted by the community without engagement of any kind, and a probability
p2 that it would be accepted after engagement. If the net benefits of the project
were £x million, then the value of the engagement would be given as
£x(p1 - p2) million. To be worthwhile, the value of the engagement must exceed
its cost, assumed to be £2m. When x = £35m, this implies that (p1 - p2) must
exceed 0.057, and when x = £40m, (p1 - p2) must exceed 0.05. That is, the

14
engagement would need to alter the probability of acceptance by between 5.7
and 7 percentage points or more to be cost effective, without considering any
of the additional benefits of the scheme. Another way of looking at this would
be to say that with the figures given, for 17 to 20 projects of this kind, if
engagement were to change the decision from NO (without engagement) to
YES (with engagement) in only one of the projects, it would be (just)
worthwhile to conduct engagement in all projects.

Furthermore, we do not know whether any opposition that there might have
been to the original scheme would be continued indefinitely in the continued
absence of a formal engagement programme. A community that opposed a
project at the outset might well, in the light of information about other flooding
in the following time periods, change its mind from opposing to favouring the
project. In other words, the comparator for the engagement exercise might not
be “no engagement” or “conduct a vote of citizens”, but might be “what would
the community prefer after a year has elapsed”. With respect to Shaldon, the
effluxion of time, in which widespread and prolonged flooding has engulfed a
number of communities in the UK, might well have changed the attitudes of a
number of members of the community without any engagement at all. Far from
being highly cost effective, that would suggest that unless the amendments
themselves were worth the cost of engagement, the engagement exercise
would not have been worth putting resources into. This puts a different
complexion on the engagement exercise, and if this scenario were to pertain,
then we would have to look carefully at the net benefits of the changes to the
scheme from the engagement, and compare them with the cost of
engagement, before we could come to a substantive view of the engagement
programme. This could be undertaken, but it is beyond the scope of this paper
to attempt this. (It also assumes that a community has the flexibility to accept
or reject a proposal from the authorities at any time, but in reality, plans are
only reviewed periodically.)

Discussion

The simplest version of a cost effectiveness analysis is quite clear. Spend £0.5
million on community engagement and be prepared to forego another £1.5

15
million in benefits due to postponement of the construction and gain £35-40
million in reduced flood damage as well as benefits to the health and well-
being of the community. Highly cost effective. But as soon as the model is
complicated by making different assumptions about the comparator (a different
form of engagement, or the possibility of community attitudes changing over
time), the possibility exists of a reversal of the cost effectiveness. In the latter
cases, it would be necessary to quantify the benefits of amending the original
scheme due to the engagement in order to determine how worthwhile the
engagement might be. In the worst-case scenario for Shaldon, the benefits of
such amendment to the plans, including the extensions of the scheme, the
incidental benefits in related areas and the likely improved community
cohesion as a result of consultation, would have to be worth about £2 million
before engagement were to be cost effective.

The analysis undertaken above could be thought of as being too inconclusive

to be worth undertaking. However, it provides the outline of a framework of
analysis, which could be carried out on a case-by-case basis for any similar
project. Such an analysis could give a reasonably good idea of community
engagement projects that would always be cost effective under any plausible
assumptions, of other such projects that would never be cost effective under
plausible assumptions, and yet other projects with cost effectiveness between
these extremes. In the last set of cases, judgement would still be required to
gauge when a project might be worthwhile and when it might not be. With
respect to Shaldon, as outlined, the case for engagement on cost
effectiveness grounds is strong.

The other aspect highlighted by this scenario is the importance of the

comparator; that is, what would have happened without the engagement. If this
cannot be gauged accurately, the value of the associated cost effectiveness
analysis will subsequently be much reduced. It is perhaps why such analyses
have not often been carried out to determine the cost effectiveness of
community engagement projects: the usefulness of such analysis will often be
limited by the lack of a clear-cut comparator.

16
For completeness we note that the perspective of many of NICE’s analyses is
that of the NHS. If viewed from that perspective, the cost of the engagement to
the NHS is zero and the health benefits are positive (in terms of lowering the
probability of water-borne disease, drownings and anxiety). As this dominates
not undertaking the project, it is infinitely cost effective.

Reference

Atkins pba (2006) Shaldon Tidal Defences: Economic Viability Report,

Revision A. (Report to the Environment Agency, currently confidential in its
details)

17