ARTICLE IN PRESS

Behaviour Research and Therapy 43 (2005) 11871201

www.elsevier.com/locate/brat

The reliability and quality of cognitive case formulation

Willem Kuykena,, Claire D. Fothergilla, Meyrem Musaa, Paul Chadwickb
b

Mood Disorders Centre, School of Psychology, University of Exeter, Exeter EX4 4QG, UK West Hampshire Trust; University of Southampton, Department of Psychiatry, Royal South Hants Hospital, Southampton, SO14 0YG, UK Received 27 December 2003; received in revised form 16 August 2004; accepted 26 August 2004

Abstract Limited research on the reliability of cognitive case formulation suggests cognitive therapists can agree about clients presenting problems but show poor agreement about the inferential aspects of formulation. There has been no research examining the quality of practitioners case formulations. This study assessed whether participants with different levels of experience could produce reliable cognitive formulations using a systematic cognitive therapy case formulation method: the J. Beck Case Conceptualization Diagram. As part of continuing education workshops on cognitive case formulation, 115 mental health practitioners were given the same case description and asked to provide case formulations. Inter-rater agreement and agreement with a benchmark formulation provided by J. Beck were measured. The results showed that participants were able to agree with each other and with the benchmark on most descriptive aspects of the formulation but rates of agreement decreased for aspects of the formulation requiring greater levels of theory-driven inference. Based on denitions and measurements of the quality of cognitive formulations derived in this study, the quality of formulations ranged from very poor to good, with only 44% rated as being at least good enough. Both reliability and quality of case formulations were associated with levels of clinical experience and accreditation status. Implications for training and supervision are discussed. r 2004 Elsevier Ltd. All rights reserved.
Keywords: Case formulation; Depression; Psychotherapy; Cognitive therapy

Corresponding author. Tel.: +44 1392 264659; fax: +44 1392 264623.

E-mail address: w.kuyken@exeter.ac.uk (W. Kuyken). 0005-7967/$ - see front matter r 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.brat.2004.08.007

ARTICLE IN PRESS
1188 W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201

1. Introduction Since the seminal publication of Cognitive Therapy and the Emotional Disorders (Beck, 1976), cognitive therapy has emerged as one of the most popular and widely taught therapeutic modalities of the last twenty years. As with all systematic models of therapy, cognitive therapy distils cognitive theories of emotional disorders to the understanding of particular cases through the case formulation method. To the scientist-practitioner cognitive therapist, individualized case formulation is the heart of evidence-based practice (Tarrier & Calam, 2002). While there can be few doubts about the efcacy and effectiveness of cognitive therapy, the outstanding challenge is to provide a scientic basis to the formulation methods that cognitive therapists use in understanding their clients presenting problems and in planning interventions. In a review of the literature on cognitive case formulation Bieling and Kuyken (2003) set out several criteria that cognitive case formulation systems must meet to be said to have scientic status: (1) Can cognitive therapists reliably formulate cases using cognitive case formulation systems (reliability)? (2) Are the key constructs in case formulations meaningfully related to the persons presenting problems (quality)? (3) Does cognitive case formulation lead to improved treatment and treatment outcomes (treatment utility)? (4) Does cognitive formulation provide a framework that practitioners can readily and usefully apply (applicability)? This paper addresses two of these criteria: reliability and quality of cognitive case formulation systems. What is case formulation? Case formulation aims to describe a persons presenting problems and use theory to make explanatory inferences about causes and maintaining factors that can inform interventions. First, there is a top-down process of cognitive-behavioral theory providing clinically useful descriptive frameworks. Second, the formulation enables practitioners and clients to make explanatory inferences about what caused and maintains the presenting issues. Third, case formulation explicitly and centrally informs intervention. Case formulation is a cornerstone of evidence-based CBT practice. For any particular case of CBT practice, formulation is the bridge between practice, theory and research. It is the crucible where the individual particularities of a given case, relevant theory and research synthesize into an understanding of the persons presenting issues in CBT terms that informs the intervention. There have been several attempts to provide individualized case formulations systems rmly based in cognitive theory that can be used by cognitive therapist in day-to-day practice and in treatment process and outcome research (Beck, 1995; Greenberger & Padesky, 1995; Persons, 1989; Muran & Segal, 1992). This study examines the J. Beck Case Conceptualization Diagram (CCD) (Beck, 1995) against the criteria of reliability and quality. That is to say, can practitioners using the same framework to formulate the same case agree about the cognitive mechanism and to what extent are their formulations good enough to be applicable in cognitive therapy best practice. The CCD offers an approach that is truly open to investigation with experimental methods because of its level of detail, specicity, and theoretical coherence. The Case Formulation Diagram uses the developmental history and several prototypical situation-thought-emotionbehavior situations to enable the therapist to infer core beliefs, dysfunctional assumptions, and maladaptive compensatory strategies. The diagram is internally consistent because developmental experiences, core beliefs, conditional assumptions, and compensatory strategies are related to each other in understandable ways (e.g., Fig. 1).

ARTICLE IN PRESS
W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201 1189

Fig. 1. Benchmark formulation of Anna (provided by J. Beck).

In contrast to the literature on brief psychodynamic psychotherapy (Luborsky & Diguer, 1998), there are only a handful of studies that have addressed the reliability of cognitive case formulation. Persons, Mooney, and Padesky (1995) asked 46 practitioners to listen to audiotapes of initial interviews for two clients using a heuristic case formulation method which involves clearly articulating the presenting problems and relating them to underlying beliefs and behaviors (Persons, 1989). The results suggested generally good agreement among judges in identifying the presenting problems, but poor agreement in identifying the hypothesized underlying cognitive

ARTICLE IN PRESS
1190 W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201

mechanisms. In a second study, Persons and Bertagnolli attempted to increase reliability by supplying practitioners with a specic set of problem domains (psychiatric symptoms, as well as interpersonal, work, nancial, health, housing, and recreational problems) and more formalized assessment of schemas (using anchor points and specic denitions of a variety of schemas). Again, however, inter-judge reliabilities in assessing schemas were adequate, but only across multiple judges, averaging .72 for ve randomly selected judges (Persons & Bertagnolli, 1999). Renements in methodology from the Persons et al. (1995) to the Persons and Bertagnolli (1999) study did not lead to improved reliability. Mumma and Smith (2001) examined the reliability of cognitive formulation using cognitive-behavioral interpersonal scenarios: narrative descriptions of prototypical sequences of behavior in response to certain types of distressing situations for a particular client (Mumma & Smith, 2001, p. 207). Two pairs of practitioners independently formulated a number of scenarios for the four cases presented on videotape. The results show that reliability of the mean ratings was good for all 15 dimensions (0.83) when aggregated across the ten clinical raters. However, as with earlier studies, reliability of a single practitioners ratings was acceptable for the descriptive elements but relatively low (from 0.33 to 0.63) for the inferential elements. The difculty in establishing reliability for the inferential aspects of cognitive case formulation may be due either to problems with study methodology or to real difculties in the formulation process. In terms of methodological issues, the nature of the case material presented for review is important. Studies have failed to use the data available to real world practitioners: a comprehensive intake interview, standardized assessment instruments, prototypical examples of situation-thought-emotion-behavior cycles, the downward arrow technique and clients emotional shifts in session (Beck, 1995). Moreover, with a few recent exceptions (Eells & Lombart, 2003), studies to date have used samples of convenience, rather than sampling a range of cognitive therapists from novice to expert. In terms of difculties in the formulation process there is extensive evidence that practitioners use a range of decision making heuristics (Kahneman, 2003; Tversky & Kahneman, 1974) that can adversely affect the quality of clinical judgment (Garb, 1998). While there is very limited direct research on cognitive case formulation it is likely that heuristics are typically used in the process of deriving a parsimonious cognitive formulation from large amounts of sometimes complex information. The problematic use of heuristics includes halo effects, illusory correlations, framing biases, recency effects, dispositionism, conrmatory biases and failure to consider normative standards (Meehl, 1954; Turk & Salovey, 1988; Wilson, 1996; Garb, 1998; Smith & Dumont, 2002; Spengler & Strohmer, 1994). It follows that more systematic and objective case formulation systems should be less prone to these inferential biases and improved rates of agreement between practitioners should be observable. Bieling and Kuyken (2003) argue that both reliability and quality criteria are important aspects of scientically based cognitive formulation. However, if practitioners can agree about the content of a case formulation, this does not mean that they are necessarily producing good formulations. That is to say, a reliable formulation is quite distinct from whether a formulation is coherently structured, whether the elements within the formulation interact with the presenting problems in the hypothesized manner across situations and time and most important whether the formulation provides a useful tool for cognitive therapy intervention. As such, establishing a way of ensuring practitioners can reliably formulate the same case

ARTICLE IN PRESS
W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201 1191

using the same cognitive therapy formulation system may be of limited value if the formulation is incoherent and unhelpful in intervention planning. For example, two highly focused and parsimonious formulations of the same case may have few elements in common (and therefore be unreliable), but enable focused and high impact therapeutic interventions. This is consistent with research showing that interventions (interpretations) based on high quality core conictual relationship theme formulations led to better treatment outcomes (Crits-Christoph, Cooper, & Luborsky, 1988). In short, it may be more important for a formulation to be a coherent and justiable account of a persons presenting problems than for it to be replicable in similar form by other cognitive therapists. We refer to this as the quality of cognitive case formulations. To our knowledge no study to date has examined the quality of cognitive case formulations. The study aims to build on previous research in several ways. We presented information about the case that as far as possible mirrors real-world cognitive therapy best practice (extensive information from an intake assessment and standardized assessments). A systematic case formulation method was selected for study (the CCD) and all participants received comparable training in using the framework. Finally, to maximize ecological validity we selected a broad range of psychological practitioners in mental health with the full range of length of experience. This study sought to address several questions. First can therapists make reliable cognitive case formulations of a complex case, using the structured and systematized CCD? Second, what is the quality of therapists case formulations benchmarked against criteria used by cognitive therapy trainers? Third, is amount of therapeutic experience and accreditation as a cognitive-behavioral therapist associated with increased reliability and quality in case formulation?

2. Method 2.1. Summary The reliability and quality of practitioners formulations using the CCD framework were assessed in three Continuing Education workshops on cognitive case formulation conducted by the rst author (WK). Practitioners were taught how to use the formulation system as part of a training workshop on cognitive case formulation. All participants were given the opportunity to practice using the model on a case during the course of the workshop. They were then presented the case of Anna (a pseudonym). Following the presentation, workshop participants completed a formulation of Anna. To address the study questions concerning reliability (inter-rater agreement) and quality, the formulations of all the participants were compared rst with one another and then with a benchmark formulation of the case provided by J. Beck. To address the question concerning quality, all formulations were assessed against the Quality of Cognitive Case Formulation Rating Scale (Fothergill & Kuyken, 2002). To address the research question concerning practitioners experience and accreditation status, all practitioners were asked to indicate their professional background and number of years since qualication (grouped as pre-qualied and qualied as

ARTICLE IN PRESS
1192 W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201

well as continuous measure of number of years post-qualication experience) and two accreditation groups (British Association for Behavioral and Cognitive Psychotherapies accreditation: no/yes). 2.2. Sample The sample comprised 115 mental health practitioners. The advertising for the workshops was through a continuing education database and through membership of the local cognitive therapy association (British Association of Behavioural and Cognitive Psychotherapy). The workshop yers suggested at least basic level knowledge and experience of cognitive therapy. The participant information (Table 1) suggests that approximately half of the participants had completed a professional training that typically includes introductory cognitive therapy training and a quarter of the participants are completing professional training. Of the qualied practitioners, they had an average 7 years of post-qualication experience working in mental health settings. One fth of the sample (20.9%) was accredited cognitive-behavioral therapy (CBT) practitioners. The participants number of years of post-qualication experience (036) did not differ across the three workshops F 2; 114o1: However, there were signicant differences in the proportion of participants who were accredited CBT practitioners (14%, 47%, and 14%) w2 14:58; DF=2,

Table 1 Participants demographic characteristics and training information Characteristics Core profession Clinical psychologist Counseling psychologist Psychiatric nurse Psychiatrist Occupational therapist Teacher Counselor Psychotherapist CBT therapist Pre-qualication students Highest degreea Ph.D./D.Clin.Psy./M.D. M.Phil./M.A./M.Sc. B.A./B.Sc. Social work diploma Nursing diploma Psychotherapy/counseling diploma Other diploma Years post-qualication clinical experience
a

Total N (%) 35 (30.4%) 1 (0.9%) 19 (16.5%) 1 (0.9%) 2 (1.7%) 1 (0.9%) 14 (12.2%) 6 (5.2%) 6 (5.2%) 29 (25.2%) 27 (21.7%) 20 (29.3%) 6 (5.2%) 2 (2.2%) 25 (2.2%)) 3 (3.3%) 9 (9.8%) X 7:31 (SD=8.58)

Workshop 1 13 (26.5%) 1 (2.0%) 10 (20.4%) 0 2 (4.1%) 0 0 1 (2.0%) 1 (2.0%) 20 (40.8%) 7 (24.1%) 8 (27.6%) 1 (3.4%) 0 12 (41.4%) 0 1 (3.4%) X 6:70 (SD=8.52)

Workshop 2 14 (46.7%) 0 5 (16.7%) 0 0 0 0 4 (13.3%) 3 (10%) 4 (13.3%) 13 (46.4%) 3 (10.7%) 2 (7.1%) 0 8 (28.6%) 1 (3.6%) 1 (3.6%) X 7:41 (SD=6.78)

Workshop 3 8 (22.2%) 0 4 (11.1%) 1 (2.8%) 0 1 (2.8%) 14 (38.9%) 1 (2.8%) 2 (5.6%) 5 (13.9%) 7 (20.0%) 9 (25.7%) 3 (8.6%) 2 (5.7%) 5 (14.3%) 2 (5.7%) 7 (20.0%) X 8:06 (SD=10.01)

Twenty three workshop participants did not complete the question about highest degree obtained.

ARTICLE IN PRESS
W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201 1193

po0:01: Across the three workshops there was a good spread of CBT knowledge, skills and experience. To maximize the range of experience and because the participants received broadly the same training, the data from the three workshops was combined for the main analyses concerned with the reliability and quality of case formulation. This provided a large sample with a broad range of professional backgrounds and experience levels (Table 1). 2.3. Materials and measures The case of Anna was used as the standardized case for all workshop participants to formulate. Anna was selected as a reasonably complex case presenting some of the issues around formulation that would be typical of clients seen in clinical practice. Anna was a client who presented with major depression and personality difculties (Kuyken, 1999b). Details about Anna were presented using a range of materials that are suggested as essential aspects of formulation (Beck, 1995). The main method of outlining the case was a detailed intake assessment, which included an account of the presenting problems, a psychosocial history, observation of the client during the assessment, standardized psychological assessments and a multi-axial diagnosis (Kuyken, 1999b). While the three workshops were almost identical in format, participants in workshop two were shown a video of the therapist using a downward arrow technique as a method for identifying Annas intermediate and deeper level cognitions. Participants in workshop three were given written copies of three prototypical thought records and shown a video of the therapist discussing the different columns of the DTR with Anna and the way she responded to her distressing thoughts.1 Each video segment was approximately ve minutes and Anna was acted by an actress with the rst author acting as the cognitive therapist. The J.Beck (1995) case formulation approach was used. In this formulation system therapists use the clients developmental history and several prototypical problematic situations to identify problematic core beliefs, dysfunctional assumptions and maladaptive compensatory strategies. Developmental experiences, core beliefs, conditional assumptions and compensatory strategies are related to each other in understandable ways. In brief, adverse developmental experiences (e.g., an intensely and enduringly critical parent) are linked to maladaptive core beliefs (e.g., I am no good), with subsidiary beliefs (e.g., If I am upbeat and bubbly at all times, no one will gure out that I am really no good) that are compensated for by a range of behavioral strategies (e.g., In all my interactions I will try to be as upbeat as possible). J. Beck supplied a benchmark cognitive formulation of the case of Anna using the intake assessment summary (Kuyken, 1999b), a description of the cognitive therapy intervention and outcome (Kuyken, 1999a) and some additional psychometric measures completed by Anna at intake (see Fig. 1). The benchmark formulation can be regarded as a good t because it is supported by ve sources of information: (i) discussions with Annas therapist; (ii) Annas intake assessment summary; (iii) written commentaries from cognitive and behavioral therapists about Annas case (Fossel & Wright, 1999; Kuyken, 1999a; Robins, 1999; Ratto & Capitano, 1999; Ramsay, 1999; Newman, 1999);
Our original study design conceptualized these different types of information as impacting on the reliability and quality of the case formulations. However, for pragmatic reasons, the workshop participants were not recruited to match on professional background and BABCP accreditation. As is clear from Table 1, the workshops participants differ in the three sites and we therefore chose to collapse the data from the three workshops.
1

ARTICLE IN PRESS
1194 W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201

(iv) samples of Dysfunctional Thought Records completed by Anna; and (v) Annas prole of scores on the Personality Beliefs Questionnaire (Beck & Beck, 1991). All workshop participants completed a demographic and training background questionnaire to gather information about professional background, amount of clinical experience and any accreditation credentials. Quality of Cognitive Case Formulation Rating Scale (Fothergill and Kuyken, 2002). This rating scale was developed for this study as a way of establishing the quality of cognitive therapy case formulations. Some elements from the Case Formulation Content Coding Method (Eells, Kendjelic, & Lucas, 1998) were used, although our scale focused on the quality of cognitive case formulations. Consistent with Becks (1995) description, the formulation diagram should make logical sense y (p. 143). Quality is dened as a parsimonious, coherent and meaningful account of a clients presenting problems in cognitive therapy terms. This refers to whether the formulation is coherently structured, with the elements within the formulation interacting with the presenting problems in a meaningful manner across situations or time. This is premised on the assumption that a parsimonious, coherent and meaningful diagram is the basis for focused and high impact interventions. Best practice in high quality cognitive therapy training programs was used to operationalize the scale. Initial phases involved establishing what made up a good enough case formulation. This process led to the scaling of the quality of case formulations from good, to good enough, to poor and to very poor. A good enough formulation includes a majority of relevant information (childhood data, core beliefs, dysfunctional assumptions and compensatory strategies) at an appropriate level of detail and links this information to prototypical problematic situations. A very poor formulation shows minimal integration of the elements, includes much irrelevant information and shows evidence of misunderstanding the CCD. The nal phase involved rening the scale through separately and then jointly reviewing 58 case formulations. Based on ten independently coded formulations an acceptable rate of agreement was observed (k 0:85). A copy of the scale is available from the rst author.

2.4. Procedure A week before the workshops, participants were sent background information about the CCD formulation method and a copy of the intake assessment summary of Anna (Kuyken, 1999b). The rst half of each workshop set out the theoretical background and main constructs of the CCD. Practitioners were taught how to complete the diagram using a demonstration by the workshop presenter of a case volunteered by a practitioner at the workshop. At the end of the rst half of the workshop, participants were asked to provide a provisional formulation on the case of Anna using the CCD. They completed the diagrams independently, using a blank case formulation diagram and the source materials. The second half of the workshops ensued when participants had completed their formulations of Anna and involved discussion of the CCD framework and its application to their clinical practice. At the end of each workshop participants completed a feedback questionnaire. Feedback on the objectives, content and presentation of the workshops was generally very positive and comparable across workshops.

ARTICLE IN PRESS
W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201 1195

3. Results 3.1. Can practitioners make reliable cognitive case formulations of a complex case? A recognized method of establishing reliability is to examine the percentage agreement amongst participants (Luborsky & Diguer, 1998; Eells et al., 1998).2 The participants formulations of the case of Anna were examined to see whether the same concepts had been formulated. This was done by rst establishing exactly what information had been formulated, then establishing how many participants had identied the information and nally, computing the agreement of the participants on a given component. Content analysis was used to derive the core concepts in mental health practitioners raw formulations. Every statement in each section of the formulation was put onto cards. The initial content analysis was completed by two judges together. This was checked by a separate experienced judge who conducted a second content analysis. The agreement of the category labels and the statements within each category was checked using Cohens Kappa: relevant childhood data k 0:91; core beliefs k 0:83; dysfunctional assumptions k 0:63; compensatory strategies k 0:84: A very small number of cards (o10) contained concepts that were largely unrelated to the case, did not clearly belong in any category and were regarded by both judges as outliers. These cards were removed from subsequent consideration. A third judge re-checked the analyses by sorting the cards having been given the category titles: relevant childhood data k 0:97; core beliefs k 0:92; dysfunctional assumptions k 0:79; compensatory strategies k 0:93: These levels of agreement suggest that the cards accurately described the categories. Table 2 shows the categories derived by workshop participants, percentage agreement on each category and whether the category had been identied in the benchmark formulation. The data suggests that practitioners can reliably formulate a complex case in terms of relevant childhood data, core beliefs and compensatory strategies. However, levels of agreement on dysfunctional assumptions were low. Furthermore, while there were subsets of data that the majority of practitioners identied as important, there were signicant categories, which only small numbers identied as important. 3.2. What is the quality of practitioners case formulations? One index of the quality of case formulations was the level of agreement with the benchmark formulation provided by J. Beck. Overall, the results show that participants agreement with the benchmark formulation of Annas case was good to high for most of the categories related to childhood data, core beliefs and compensatory strategies. Levels of agreement were particularly high for those categories that were related to Annas past relationship with both her parents: critical father, fathers history of bipolar disorder and overly involved mother. In addition the participants agreement with the benchmark formulation was particularly high for compensatory strategies related to avoidance, self-harm and control.
Consistent with our design, percent agreement enables agreement on large numbers of components within the formulation to be assessed through visual inspection. We acknowledge that percent agreement does not control for chance base rates and our interpretation of the ndings must therefore be in the context of chance base rates.
2

ARTICLE IN PRESS
1196 W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201

Table 2 Percentage agreement on each formulation aspect (overall and by professional qualication status and BABCP accreditation) General themes and specic aspects 1. Relevant childhood data Father criticala Inappropriate involvement from mothera Father controlling Fathers breakdown not discussed Bad relationship with sistersa Parental marital problemsa Father difcult to live with Overwhelmed at schoola Moved house Boyfriend cheateda Career dissatisfaction Husbands illnessearly birth of children Family history of depressiona Self-harmed when young Unhappily married 2. Core beliefs (self asy) Im incompetenta Im worthlessa Im unlovablea 3. Conditional assumptions/beliefs/rules Public fac - ade and rejectiona Personal worth linked to achievement I lack support and am isolated I must be in control at all times It is best to avoid difcult situations 4. Compensatory strategies Avoidancea Self harma Controla Acting false (disguise feelings)a
a

Overall 72 71 58 43 42 37 29 23 23 21 17 17 15 9 7 73 67 37 62 35 18 12 11 71 57 45 45

Pre-qual 66 59 38 28 38 21 28 17 17 28 21 7 14 10 10 69 66 45 45 52 10 3 10 48 45 52 52

Non-BABCP 70 73 63 45 43 42 28 27 32 15 13 27 17 8 5 70 65 35 67 27 20 13 15 83 62 38 40

BABCP 85 81 69 54 42 42 31 23 12 27 19 4 12 8 8 85 73 35 69 35 23 19 4 69 58 54 50

Categories identied in the benchmark formulation.

The second index of quality was the Quality Rating Scale. All participants formulations were coded for quality. Formulations were distributed across the range from very poor to good (very poor 22.1%; poor 33.6%; good enough 34.5%; good 9.7%). Importantly, this suggests that only 44.2% of the formulations could be said to be at least good enough. The two indices together suggest that while the content of formulations tended to be high quality on most key aspects of the cognitive case formulation, there was considerable variation in how parsimoniously, coherently and meaningfully this information tted together. It was this latter dimension of quality that was less frequently observed.

ARTICLE IN PRESS
W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201 1197

3.3. Is amount of therapeutic experience and accreditation as a cognitive-behavioral practitioner associated with increased reliability and quality in case formulation? To establish the relationship between the reliability of case formulations and practitioners prior experience, visual inspection of the percentage agreement across experience levels was undertaken (Table 2). Taking a criterion of more than ten percentage points difference across levels of experience, ve aspects of relevant childhood data, one core belief, three dysfunctional assumptions and two compensatory strategies were different across the experience levels. Generally, the pre-qualied group was least likely to identify an important aspect identied in the benchmark formulation or by qualied practitioners. In no case were the accredited cognitivebehavioral therapists less likely to identify an aspect of the formulation identied in the benchmark formulation. To establish the relationship between the quality of case formulations and practitioners prior experience, the quality of case formulations at different levels of professional training was compared. Among qualied mental health practitioners, the number of years of post-qualication experience was positively associated with the quality of formulations Spearman r 0:22; po0:05; N 113: This ts with recent ndings that practitioner orientation and experience inuence what is regarded as important in formulation and treatment planning (Eells & Lombart, 2003). To establish if level of qualication and accreditation as a CBT therapist enabled higher quality formulations, the quality of formulations for pre-qualied, non-BABCP accredited and BABCP accredited practitioners were compared. The proportion of good enough formulation generated within each of the three groups was: pre-qualied 25%; qualied but not BABCP accredited 45.9% and qualied and BABCP accredited 62.5%. A w2 analysis demonstrated that the quality of formulations differed across the three groups w2 6:05; DF=3; po0:05 and the frequencies suggest that professional qualication and BABCP accreditation appear to lead to incremental improvements in the quality of formulations.

4. Discussion We sought to examine empirically whether mental health practitioners can produce reliable and high quality formulations and whether this is affected by professional experience and accreditation. In contrast to previous research (Persons & Bertagnolli, 1999; Persons et al., 1995), we found that in the main practitioners agreed with one another and with a benchmark case formulation on the salient features of the formulation, including those where they had to make theory-driven inferences (core beliefs and compensatory strategies). However, levels of agreement declined for less salient information and when more theory-driven inference was required (e.g., inferring dysfunctional assumptions). This is consistent with research examining the reliability of psychodynamic case formulation, which suggests that following appropriate training in the use of a structured case formulation method practitioners can show high rates of agreement (Luborsky & Diguer, 1998). This study adds weight to the emerging consensus that reliability is compromised as practitioners move from the descriptive (e.g., articulating the presenting problems, summarizing childhood data) to the inferential levels (e.g., inferring dysfunctional beliefs) of formulation.

ARTICLE IN PRESS
1198 W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201

There are several possible explanations for our ndings. First, this study was based on the CCD, which provides a clear and systematic approach to formulation and contrasts with earlier studies that have used broader heuristic case formulation systems (e.g., Persons & Bertagnolli, 1999). More structured formulation systems minimize the opportunity for problematic decision making heuristics (Kahneman, 2003) that might compromise formulation. Second, as practitioners move to more inferential levels of formulation higher level cognitive therapy formulation skills are required. These skills are acquired only through extensive training and supervised experience, and a training workshop like the one used in this study would not be sufcient (J. Beck, personal communication, December 3, 2002). These two explanations are fully compatible with each other. There are several conceptual and methodological caveats in establishing the reliability of cognitive case formulation. Even though our ndings suggest that practitioners can reliably complete case formulations, this does not mean that they are valid. That is to say, it does not answer the question of whether the formulation provides a t across situations or time that the practitioner, client and a supervisor might nd compelling. Furthermore, agreement by itself does not convey anything about the quality of a formulation. We asked J. Beck to provide a benchmark cognitive therapy formulation as an index of the quality of cognitive therapists formulations. Behavioral and cognitive models of depression and personality disorders are varied and provisional and would necessarily lead to differences in formulation. A more behaviorally oriented therapist, for example, might conduct a functional analysis of the problematic behaviors. This work suggests a potential conict between two of the criteria for scientic formulation systems set out by Bieling and Kuyken (2003): namely reliability and validity. Two different formulations of the same case could both be valid. It further raises interesting questions about how these criteria might relate to outcome. Can formulations that are coherent and meaningful to the therapist and client, but which are off the mark in cognitive therapy terms, still guide cognitive therapy leading to good outcomes? One solution to this problem is the use of panels of experts working with specic therapeutic approaches. If experts cannot reach consensus it would be unrealistic to expect high rates of agreement. In both cases (expert consensus versus expert disagreement) it would be interesting to see how formulation linked to interventions and to outcome. Whether reliability between formulations is a necessary, but not sufcient, condition for the quality or validity of formulation in terms of ensuring treatment outcomes remains an empirical question. Methodologically, our research is confounded by the fact that the data were collected over three workshops and while we aimed for the workshops and workshop participants to be broadly comparable, there were differences in the professional background of participants in the three workshops and small but notable difference in the content of the workshops. We would argue, however, that the basic ndings of the research are not compromised by this methodological caveat because the pooling of the data enabled a large and more diverse sample of mental health professionals. The small differences in workshop content did not lead to discernible systematic differences in formulations supporting the rationale for pooling data and the conclusions drawn. Second, we used only a single case as the source material, which limits the generalizability of these ndings. Previous research has sampled either one or two cases (Persons et al., 1995) or related themes/scenarios from a series of cases (Mumma & Smith, 2001). Future studies should use sampled cases across the dimensions of presenting problem/diagnosis, complexity/co-morbidity and stage of assessment/therapy and we are aware of work building on our work that adopts this approach. Finally, statistically our design and data were most appropriately assessed for

ARTICLE IN PRESS
W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201 1199

reliability through the use of per cent agreement. The relatively high agreement on some inferential elements should be interpreted in the context of a likely chance base rate agreement. This study is the rst that we are aware of that explicitly examined the quality of cognitive case formulations. Strikingly, the proportion of formulations which were judged good enough was low at 44.2% and among mental health practitioners in training this fell to 24.1%. Knowledge and skill in cognitive case formulation has long been considered a fundamental skill in therapist competence and most advanced level cognitive therapy training programmes stress formulation as a central skill (Dobson & Shaw, 1993). Cognitive therapy is sometimes caricatured as a simple therapy (Holmes, 2002) which can lead practitioners to short circuit more advanced training and supervision. Our nding that accreditation (which explicitly examines training and supervision) was associated with higher quality formulations lends weight to cognitive formulation as a high level skill.3 If cognitive therapy manuals and trainers are correct in advocating the importance of a formulation in intervention planning, this has obvious implications for training mental health practitioners as cognitive-behavioral therapists. That is to say, practitioners need to learn formulation skills through training and supervised practice before they can work effectively. On the other hand, research to date fails to show any relationship between formulation-driven cognitive therapy and improved outcomes, and at least two studies suggest the supremacy of manualized over individualized approaches (Schulte, Kunzel, Pepping, & Shulte-Bahrenberg, 1992; Chadwick, Williams, & Mackenzie, 2003; Emmelkamp, Bouman, & Blaauw, 1994). Moreover two studies fail to show any impact on clients view of the therapeutic relationship and provide qualitative data showing that some clients experience case formulation negatively (Chadwick et al., 2003; Evans & Parry, 1996). Taken together these ndings suggest several intriguing possibilities: for straightforward clinical presentations and for therapists early in training better outcomes are achieved by staying close to protocols and manualized approaches; for complex presentation and advanced level practitioners high quality case formulation enhance outcomes with complex cases. These propositions require examination through cognitive therapy process-outcome research designs and until then remain as empirical questions. As recommended by cognitive therapy trainers (e.g., Beck, 1995), we included several types of information to help practitioners complete a formulation: an intake summary assessment, the downward arrow technique and dysfunctional thought records. Our nding that reliability on aspects of the diagram was acceptable suggests that it is important to use multiple and triangulated assessment methods to validate underlying beliefs and improve the accuracy of the formulation. This is likely to minimize practitioners adopting problematic heuristics in the case formulation process and maximize the chance of a formulation providing a good t as seen by both the therapist and client. We concur with recent commentators (e.g., Tarrier & Calam, 2002) that case formulation is at the heart of establishing an evidence-base for real-world cognitive therapy practice. Examining the criteria set out by Bieling and Kuyken (2003) will test the argument that demonstrations of the efcacy of cognitive therapy through randomized controlled trials are the basis for an evaluation of cognitive therapy in real world practice where more individualized approaches (i.e., case formulation approaches) are used to establish the effectiveness of cognitive therapy. This work is one step towards establishing the reliability and quality of cognitive case formulations.
3 Accreditation is used as a proxy measure of therapist quality. We acknowledge that many high quality cognitive therapists do not seek accreditation and accreditation itself does not guarantee the quality of a practitioners work.

ARTICLE IN PRESS
1200 W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201

Future research might usefully extend these ndings by examining whether practitioner experience/accreditation become important when high levels of theory-driven inferences are required. In conditions of high complexity, training and experience may enable practitioners to maintain focus and coherence in their formulations. In addition, research is needed to examine the vital question of whether formulation is linked to improved outcomes through the selection of better targeted interventions as has been shown with brief psychodynamic psychotherapy but has not yet been shown with behavioral and cognitive-behavioral psychotherapy (Chadwick et al., 2003; Jacobson et al., 1989; Schulte et al., 1992). A consensus is emerging that this criterion is most important in the development of a systematic approach to cognitive case formulation (Bieling & Kuyken, 2003). This makes the difculties to date in demonstrating any relationship to outcome notable and troubling. However, the relationship of formulation and outcome is likely to be multi-factorial, including the nature of the problem, the specicity of cognitive therapy models and manuals for the problem, the clients formulation of the problem, and the competence of the therapist. We would hypothesize that higher quality formulations would be associated with better targeted cognitive therapy interventions and improved outcomes. Acknowledgements This work is based on the M.Sc. and D.Clin.Psy. dissertations of the second and third author, respectively. We are grateful to Judith S. Beck for providing the benchmark formulation and to Rachael Carrick, Lyn Ellett and Michael Ranft for research assistance. We thank Judith S. Beck, Andy Butler, Rachel Calam, Rob Dudley, Nick Tarrier and two anonymous reviewers for their comments on an earlier draft of this manuscript.

References
Beck, A. T. (1976). Cognitive therapy and emotional disorders. New York: Meridian. Beck, A. T., & Beck, J. S. (1991). The Personality Belief Questionnaire. Unpublished Work. Beck, J. S. (1995). Cognitive therapy: basics and beyond. New York: Guilford Press. Bieling, P. J., & Kuyken, W. (2003). Is cognitive case formulation science or science ction? Clinical Psychology-Science and Practice, 10, 5269. Chadwick, P., Williams, C., & Mackenzie, J. (2003). Impact of case formulation in cognitive behavior therapy for psychosis. Behavior Research and Therapy, 41, 671680. Crits-Christoph, P., Cooper, A., & Luborsky, L. (1988). The accuracy of therapists interpretations and the outcome of dynamic psychotherapy. Journal of Consulting and Clinical Psychology, 56, 490495. Dobson, K. S., & Shaw, B. F. (1993). The training of cognitive therapistswhat have we learned from treatment manuals. Psychotherapy, 30, 573577. Eells, T. D., Kendjelic, E. M., & Lucas, C. P. (1998). Whats in a case formulation? Development and use of a content coding manual. Journal of Psychotherapy Practice and Research, 7, 144153. Eells, T. D., & Lombart, K. G. (2003). Case formulation and treatment concepts among novice, experienced, and expert cognitive-behavioral and psychodynamic therapists. Psychotherapy Research, 13, 187204. Emmelkamp, P. M. G., Bouman, T. K., & Blaauw, E. (1994). Individualized versus standardized therapy: a comparative evaluation with obsessivecompulsive patients. Clinical Psychology and Psychotherapy, 1, 95100. Evans, J., & Parry, G. (1996). The impact of reformulation in cognitive-analytic therapy with difcult-to-help clients. Clinical Psychology and Psychotherapy, 3, 109117.

ARTICLE IN PRESS
W. Kuyken et al. / Behaviour Research and Therapy 43 (2005) 11871201 1201

Fossel, R. V., & Wright, J. H. (1999). Targeting core beliefs in treating borderline personality disorder: the case of Anna. Cognitive and Behavioral Practice, 6, 5460. Fothergill, C. D., & Kuyken, W. (2002). The Quality of Cognitive Case Formulation Rating Scale. Unpublished manuscript. Garb, H. N. (1998). Studying the clinician: Judgment research and psychological assessment. Washington, DC: American Psychological Association. Greenberger, D., & Padesky, C. A. (1995). Mind over mood: change how you feel by changing the way you think. New York: Guilford. Holmes, J. (2002). All you need is cognitive behavior therapy? British Medical Journal, 324, 288294. Jacobson, N. S., Schmaling, K. B., Holtzworthmunroe, A., Katt, J. L., Wood, L. F., & Follette, V. M. (1989). Research-structured vs clinically exible versions of social learning-based marital-therapy. Behavior Research and Therapy, 27, 173180. Kahneman, D. (2003). A perspective on judgment and choiceMapping bounded rationality. American Psychologist, 58, 697720. Kuyken, W. (1999a). A developmental psychopathology approach. Cognitive and Behavioral Practice, 6, 7883. Kuyken, W. (1999b). Anna: a 26-year-old woman with major depression and borderline personality disorder. Cognitive and Behavioral Practice, 6, 5053. Luborsky, L., & Diguer, L. (1998). The reliability of the CCRT measure: results from eight samples. In Luborsky, L., & Crits-Christoph, P. (Eds.), Understanding transference: the core conictual relationship theme method, (2nd ed.) (pp. 97108). New York: Basic Books. Meehl, P. E. (1954). Clinical versus statistical prediction. Minneapolis: University of Minneapolis Press. Mumma, G. H., & Smith, J. L. (2001). Cognitive-behavioral-interpersonal scenarios: Interformulator reliability and convergent validity. Journal of Psychopathology and Behavioral Assessment, 23, 203221. Muran, J. C., & Segal, Z. V. (1992). The development of an idiographic measure of self-schemas: an illustration of the construction and the use of self-scenarios. Psychotherapy, 29, 524535. Newman, C. F. (1999). The case of Anna: addressing three special problems in treatment. Cognitive and Behavioral Practice, 6, 8491. Persons, J. B. (1989). Cognitive therapy in practice. A case formulation approach. New York: Norton. Persons, J. B., & Bertagnolli, A. (1999). Inter-rater reliability of cognitive-behavioral case formulations of depression: a replication. Cognitive Therapy and Research, 23, 271283. Persons, J. B., Mooney, K. A., & Padesky, C. A. (1995). Interrater reliability of cognitive-behavioral case formulations. Cognitive Therapy and Research, 19, 2134. Ramsay, J. R. (1999). A life of quiet desperation: The case example of Anna. Cognitive and Behavioral Practice, 6, 7377. Ratto, C. L., & Capitano, D. L. (1999). New directions for cognitive therapy: a schema-focused approach. Cognitive and Behavioral Practice, 6, 6873. Robins, C. J. (1999). A dialectical behavior therapy perspective on the case of Anna. Cognitive and Behavioral Practice, 6, 6068. Schulte, D., Kunzel, R., Pepping, G., & Shulte-Bahrenberg, T. (1992). Tailor-made versus standardized therapy of phobic patients. Advances in Behavior Research and Therapy, 14, 6792. Smith, J. D., & Dumont, F. (2002). Condence in psychodiagnosis: What makes us so sure? Clinical Psychology and Psychotherapy, 9, 298. Spengler, P. M., & Strohmer, D. C. (1994). Clinical judgment biases: the moderating roles of counselor cognitive complexity and counselor client preferences. Journal of Counseling Psychology, 1, 817. Tarrier, N., & Calam, R. (2002). New developments in cognitive-behavioural case formulation. Epidemiological, systemic and social context: an integrative approach. Behavioural and Cognitive Psychotherapy, 30, 311328. Turk, D. C., & Salovey, P. (1988). Reasoning, inference and judgment in clinical psychology. New York: Free Press. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: heuristics and biases. Science, 185, 11241131. Wilson, G. T. (1996). Manual-based treatments: the clinical application of research ndings. Behavior Research and Therapy, 34, 295314.