Provide a critical appraisal of a published economic

evaluation of a healthcare intervention using the

Drummond 10 question checklist (Drummond et al.,
2015). Provide evidence-based recommendations for
how the intervention could have been improved.

INTRODUCTION

This is a critical appraisal of Fehily et al (2020) on “An economic evaluation of a specialist

preventive care clinician in a community mental health service: a randomized control trial.” It

is established that the risk of engaging in chronic disease behaviours is higher in people with

mental illness (Fehily et al., 2020). Mental health services often provide preventive care

services which address the issue of chronic disease activities among its clients. This study

introduced a new intervention where specialized mental health clinicians offered preventive

care consultation in addition to the usual care. The specialist encouraged clients to accept free

telephone-based counselling and assistance over a period of six months. Conducting a

randomized control trial, Fehily et al (2020) aimed to analyze the cost, cost-effectiveness,

and budget impact of this new intervention. 

Health economics is concerned with providing evidence-based economic information that

will help decision-makers efficiently use available resources towards achieving maximum

public health benefits (Dang, Likhar and Alok, 2016). This evidence can be obtained via the

conduct of economic evaluations. Economic evaluation is defined as “the systematic

appraisal of costs and benefits of projects, normally undertaken to determine the relative

economic efficiency of programs or the comparative analysis of alternative courses of action

1
R2003D10506504
in terms of both their costs and their consequences” (Drummond et al., 2015, p. 11-12). Data

from economic evaluations can be used in negotiating intervention costs, designing cost-

effective clinical practice guidelines, comparing and adopting efficient and affordable

interventions, prioritising the allocation of available resources based on evidence and so forth

(Dang, Likhar and Alok, 2016). 

In South India for instance, a cost-effective analysis revealed that using combined inhaled

corticosteroids and long-acting bronchodilators to treat severe and very severe chronic

obstructive pulmonary disease increases the quality of life of such clients at no extra cost as

opposed to the hospital’s usual standard use of salmeterol/fluticasone, formoterol/budesonide,

and formoterol/fluticasone (Dang, Likhar and Alok, 2016). Administrators of this hospital

certainly must consider changing the clinical practice in the treatment of clients with chronic

obstructive pulmonary diseases. Another study in India revealed that mass hepatitis B

childhood immunisation is highly cost-effective for countries with an intermediate rate of

endemicity (intro 3). In their study, Aggarwal and colleagues reported a 71% reduction of

hepatitis B carrier rate, increased number of years and quality-adjusted life years (QALY)

lived by a birth cohort by 0.173 years and 0.213 years respectively (Aggarwal, 2002, quoted

in Dang, Likhar and Alok, 2016).

To optimally use health economic evidence in the delivery of public health services, the

Institute of Medicine Committee on Public Health Strategies to Improve Health Board on

Population Health and Public Health Practice made the following recommendations (Institute

of Medicine, 2013, quoted in Babarison et al., 2015).

 “Develop a model chart of accounts for use by public health agencies at all levels to

enable better tracking of funding related to program outputs and outcomes across

agencies”

2
R2003D10506504
  “Develop a robust research infrastructure for establishing the effectiveness and value

of public health and prevention strategies” 

 “Develop data systems and measures to capture research quality information on key

elements of public health delivery including program implementation costs” 

 “Develop and validate methods for comparing the benefits and costs of alternative

strategies to improve population health.”

DISCUSSION

A critical analysis of study using Drummond checklist.

Question 1: Yes

The program examined the costs of service over an appropriate period of 5 years. However,

the program effectiveness was measured only one-month post-intervention. Hence

downstream and long-term costs and benefits were not considered. Two alternatives were

compared in the study.

Analysis was based on the mental health service provider’s perspective (provider institution).

Decision making was placed in the context of cost-effectiveness analysis using ‘referral

acceptance’ as the measuring unit. From a managerial perspective, a budget impact analysis

over a period of 5 years was also performed.

The patient population was defined as all clients of the community mental health service. A

strength of this study is its consideration of all categories of mental health clients as well as

focusing on multiple chronic risk behaviours. However, no relevant subgroups were clearly

defined.

Question 2: Yes

There are diverse means a facility can adapt to provide effective mental health services to its

clients. For example, nurses can provide mental health care services.

3
R2003D10506504
People with mental illness need regular medical support lest they engage in chronic disease

risk behaviour.

Question 3: Yes

The effectiveness data came from a two-group randomised controlled trial. The protocol

reflects that in regular practice, preventive care is offered by mental health clinicians during a

routine consultation. Effectiveness was based on results from the two-group parallel

randomised control trial.

Question 4: Yes

The range of study was not wide enough. Long-term consequences of the program could not

be provided. The outcome was measured for only a one-month follow-up. Only the service

provider and managerial viewpoints were considered. Other relevant viewpoints like the

clients, third-party payers and the society were omitted.

The operating costs included wages of the specialist and receptionist, telephoning, printing,

home visit travel by specialist, and overhead costs. There are no direct patient costs. Indirect

costs, e.g., telephone costs to clients while trying to seek counsel from specialists were not

included.

Question 5: Yes

The resources were itemized and duly accounted for. The study assumed that routine mental

health consultation did not incur any additional cost in both interventions. 

Question 6: Yes

Cost of wages, printing, travel and telephoning were based on the 2018 Australian market

value. Overheads and mental health services were based on prevailing standard factors for the

health district. 

Consequences were measured as the increase in referral acceptance into the two-telephone

referral system (Get Health and Quitline) or both. If mental health services aim to reduce

4
R2003D10506504
chronic risk behaviours (like smoking, poor nutrition, harmful alcohol intake, and lack of

exercise) then a more appropriate outcome to have been measured is how the intervention has

reduced client involvement into these harmful behaviours, or how the quality of life has

improved. 

The choice of a cost-effective analysis in this study addressed technical efficiency but was

not suitable for exploring allocative efficiency.

Question 7: Not applicable

Since the intervention is not revenue-generating, the cost was estimated to increase annually.

In the budget impact analysis, these annual increases were assumed. 5% increase each in

client population, uptake of the preventive care consultation and incremental effectiveness; a

2.1% increase in wages.

Question 8: Yes

The incremental cost-effectiveness ratio (ICER) for each additional referral acceptance in the

intervention group was compared to the normal care group. Representing these findings on a

cost-effectiveness plane revealed that acceptance of the Get Healthy referral service, Quitline

referrals or both were all located on the northeast quadrant. This means that intervention was

costly and more effective compared to the usual care.

Question 9: No

Patient-level data were available for only consequences. This data was used to calculate the

proportion of participants that accepted referrals to the telephone referral (Get Health Service

and the Quitline). Referral acceptance among participants increased from 0.5%-19.2% for the

Get Healthy Services and 1.0& - 11.4% for the Quitline at the one-month follow-up.

Taking a cue from how similar effective mental health services has been provided in other

settings, Fehily et al (2020) performed sensitivity analysis by varying some key input

parameters. The analysis explored key inputs such as: when a registered nurse (with a lower

5
R2003D10506504
wage) took over the role of the specialist; when a low waged peer worker (according to the

definition of NSW Mental Health Commission) provided the preventive care support; 5-hour

additional weekly receptionist working hours against 5 hour reduced specialist weekly

working hours; and a 20% increase in total referral acceptance. 

Results were not categorized into subgroups.

Question 10: No

The conclusion was based on the incremental cost-effectiveness ratio (ICER) results. They

concluded that $68.19 per client is a relatively low annual price towards achieving effective

mental health preventive care.

Similar studies were not available to compare findings with.

The results of this study failed to consider the question of allocative efficiency. A cost-utility

analysis or cost-benefit analysis is needed to judge whether it is worthwhile implementing the

proposed intervention in a mental health facility. Although the intervention proved to be

effective, it is also very expensive. There may exist other methodologies that could yield the

same effectiveness at a more affordable cost.

Generalizability may be constrained since the study was limited to the settings state and

health district’s stringent policies.

The study suggested that implementation should decide on the specialist’s level of

engagement by considering the location and client size of the mental health facility. Since

about 60% of intervention cost was attributable to specialist wage, facilities need to decide

whether to engage the specialist full-time or part-time. 

Although the ICERs resulting from the sensitivity analysis were lower than the base case,

there exist some limitations. The analysis failed to consider their effectiveness in yielding the

desired outcomes. Future studies should explore the effectiveness of these other options. For

example, the effectiveness of a peer worker in the specialist’s role.

6
R2003D10506504
The study also suggested that future studies should also explore cost-effectiveness from the

societal perspective over a longer period and explore the option of quality-adjusted life years

in its modelling.

A comparison of costs and consequences from this study with other similar

interventions

According to the literature, this study is one of only two studies that analysed the cost-

effectiveness of a community mental health system where specialised clinicians offered

preventive care to clients. The lack of categorisation of subjects into subgroups coupled with

the unique outcome measure (acceptance into a referral preventive care) limits this study to a

category with less comparable literature.

Druss et al (2011) is a study that examined the outcome, costs, and financial sustainability of

a community mental health care led by registered nurses. Fehily et al (2020) did not provide

information on the cost of care in the usual group. It provided data on the additional cost of

implementing the intervention. Again, Fehily et al (2020) included cost from the service

provider’s perspective only while Druss et al (2011) included cost from the perspective of the

service provider, third party payer (health insurance) as well as the cost to clients. Hence,

comparing cost data of the two studies (Fehily et al.,  2020, and Druss et al.,  2011) will not be

proportionate.

Improved access to quality primary preventive care (p<0.001), cardiometabolic care

(p<0.001) and quality of life (p<0.001) was experienced from the sixth month and sustained

for the entire period (Druss et al., 2011). Fehily et al (2020) also recorded a significant

increase in referral acceptance among the intervention group over the usual care group. For

instance, acceptance to the Get Healthy service increased from 0.5% to 19.2% among the

intervention group at one-month follow-up while a decline from 1.1% to 0.6% was observed

among the usual care group. According to their consequence data, both Druss et al (2011)
7
R2003D10506504
and Fehily et al (2020) are effective. However, the different unit measures used to make the

comparison of results unproportionate. 

There needs to be some form of standardization in the conduct of economic evaluation across

every research field. For instance, all economic evaluations on mental health should measure

consequences in quality-adjusted life years (QALY). This will promote a comparative

analysis of existing economic evaluation literature. In a systematic review, Park and

colleagues examined “the cost-effectiveness of interventions to promote the physical health

of people with mental health problems” (Park et al., 2013). Unlike Fehily et al (2020) studies

reviewed had categorized subjects into smaller subgroups hence, fostering comparative

analysis. The majority of the studies also measured outcomes in QALY gained. 

IMPLICATION

To improve their cost-effective analysis study, the researchers should have measured

outcomes using the most common and acceptable units. Instead of measuring “acceptance to

referrals”, Fehily et al (2020) should have explored clinical units from standardized

laboratory testing, quality of life, years of life saved, or quality-adjusted life years (QALY)

(Tengs, 2004). The ICER per acceptance to referrals could change dramatically against

QALY or years of life saved (Tengs, 2004, p. 71).

Fehily et al (2020) recommend the intervention to policymakers as a cost-effective means of

delivering mental health preventive care following the district’s clinical guidelines. One is

justified to ask the source of funding to fully implement the proposed intervention. Will the

study continue to be cost-effective if implementation cost is borne by clients? What will be

the opportunity cost if funded by the government? Can the service provider increase the

efficiency of intervention by adjusting the mode of delivery? Is it sustainable? The study

would be more enriched if considered from a broader perspective. Evidence from Druss et

al (2011) showed that nurse-led mental health service was highly cost-effective. However,

8
R2003D10506504
the program was not sustainable from a managerial perspective. The program was terminated

after two years since a majority of clients were not on health insurance schemes, hence could

not afford the service out-of-pocket.

Lastly, the study should have concealed client and clinician allocation and increased the

follow-up period. Unconcealed allocation of clients and clinicians into the study introduces

bias and affects the study’s internal validity (Schulz, 2001). Schulz (2001) revealed that

studies with unconcealed allocation often report higher estimates of a positive outcome than

those concealed. Clients could naturally increase adherence knowing service is from a

specialist (Brooks, Chalder and Gerada, 2011). Naturally, marginal utility is inversely

proportional to time/ exposure. Hence, client satisfaction with the new intervention will

naturally decline over a longer period. The one-month follow-up of study participants did not

make room for marginal utility to take its natural course and long-term effectiveness was duly

determined. Sustained behaviour change and impact on physical health among mental health

clients requires a longer duration (Park et al (2013)).

CONCLUSION

This is a critical review of the cost-effectiveness of providing additional preventive mental

health consultation (by a specialist) in addition to the usual mental healthcare for all clients in

a community (Fehily et al., 2020). Evidence from economic evaluations is useful in

designing cost-effective clinical guidelines, efficient resource allocation of scarce resources,

selecting and implementing appropriate interventions. Using the Drummond checklist, the

economic evaluation was critically analysed.

The unit used to measure outcome coupled with failure to categorize subjects into subgroups

made it unproportionate to compare the findings of this study to other economic evaluations

in preventive mental health.

9
R2003D10506504
To improve their study, Fehily et al (2020) should have measured study outcomes in

acceptable units like QALY, quality of life or years of life saved. They should have analyzed

from a broader perspective such as client, societal, and third-party payer perspective. Again,

concealing allocation of clients and clinicians into the study and increasing follow-up period

would have eliminated biases, improved the study’s internal validity and provided evidence

on the long-term effectiveness of the intervention.

REFERENCE

1. Brooks, S.K., Chalder, T. and Gerada, C. (2011) ‘Doctors vulnerable to psychological

distress and addictions: treatment from the Practitioner Health Programme’, Journal

of Mental Health, 20(2), pp.157-164.

2. Dang, A., Likhar, N. and Alok, U. (2016) ‘Importance of economic evaluation in

health care: an Indian perspective’, Value in health regional issues, 9, pp.78-83.

3. Drummond, M.F. et al. (2015) Methods for the economic evaluation of health care

programmes. 4th edn. Oxford university press.

4. Druss, B.G et al. (2011) ‘Budget impact and sustainability of medical care

management for persons with serious mental illnesses’, American journal of

psychiatry, 168(11), pp.1171-1178.

10
R2003D10506504
 5. Fehily, C. et al.  (2020) ‘An economic evaluation of a specialist preventive care

clinician in a community mental health service: a randomised controlled trial’, BMC

health services research, 20, pp.1-16.

6. Park, A.L et al.  (2013) ‘Examining the cost effectiveness of interventions to promote

the physical health of people with mental health problems: a systematic

review’, BMC public health, 13(1), pp.1-17.

7. Rabarison, K.M. et al. (2015) ‘Economic evaluation enhances public health decision

making’, Frontiers in public health, 3, p.164.

8. Schulz, K.F. (2001) ‘Assessing allocation concealment and blinding in randomised

controlled trials: why bother?’, Evidence-based nursing, 4(1), pp.4-6.

9. Tengs, T.O. (2004) ‘Cost‐Effectiveness versus Cost–Utility Analysis of Interventions

for Cancer: Does Adjusting for Health‐Related Quality of Life Really Matter?’,  Value

in Health, 7(1), pp.70-78.

11
R2003D10506504
WORD COUNT: 2467

12
R2003D10506504