Knowledge Representation and Tool Support for Critiquing
Clinical Trial Protocols
Daniel L. Rubin,"2 John Gennari,3 and Mark A. Musen2
'Center for Health Care Evaluation, VA Palo Alto Health Care System, Palo Alto, CA
2Stanford Medical Informatics, Stanford University School of Medicine
3Information & Computer Science Department, University of CA, Irvine
Abstract
The increasing complexities of clinical trials have led
to increasing costs for investigators and organizations
that author and administer those trials. The process of
authoring a clinical trial protocol, the document that
specifies the details of the study, is usually a manual
task, and thus authors may introduce subtle errors in
medical and procedural content. We have created a
protocol inspection and critiquing tool (PICASSO) that
evaluates the procedural aspects of a clinical trial
protocol. To implement this tool, we developed a
knowledge base for clinical trials that contains
knowledge of the medical domain (diseases, drugs, lab
tests, etc.) and of specific requirementsforclinical trial
protocols (eligibility criteria, patient treatments, and
monitoring activities). We also developed a set of
constraints, expressed in a formal language, that
describe appropriate practices for authoring clinical
trials. If a clinical trial designed with PICASSO
violates any ofthese constraints, PICASSO generates a
message to the user and a list of inconsistencies for
each violated constraint. To test our methodology, we
encoded portions of a hypothetical protocol and
implemented designs consistent and inconsistent with
known clinical trial practice. Our hope is that this
methodology will be useful for standardizing new
protocols and improving their quality.
Background and Rationale
The progressive reliance on clinical trials for thema-
peutic advances and for developing clinical prsotice
guidelines has caused a sharp increase in the number of
protocols written each year. Developing new clinical
trial protocols is a long and expensive process. Gener-
ally, multiple layers of expert committee review are
required to finalize a protocol. Despite this oversight,
many new protocols have serious problems involving
incomplete or inconsistent specifications of treatment
procedures and patient monitoring events, as well as
errors in study methodology [1]. For instance, a proto-
col using a cardiotoxic drug may fail to require that
patients be screened for heart disease, or may not state
the need for a left ventricular function screening test if
a patient has a history of cardiac insufficiency.
Oncology trials can be particularly problematic, as
they are frequently complex and use multiple drug
treatments. Chemotherapeutic drugs often have serious
1067-5027/00/$5.00 C) 2000 AMIA, Inc.
toxicities, and multiple monitoring events that vary
with disease history must be specified. Drug contrain-
dications must be checked against all eligibility crite-
ria, and potential drug interactions with other drugs or
concurrent patient diseases must be considered. lh-
fortunately, new protocol design usually focuses on the
primary efficacy endpoint, with less attention given to
side effects relating to the primary treatment [2]. Thus,
toxicity-related omissions in protocol design are likely.
Automated methods can streamline and improve
the entire process of clinical research. The National
Cancer Institute is developing a Cancer Informatics
Infrastructure (CII) to help automate clinical research
[3]. An objective of the CII group is to create con-
puter-based tools that improve the efficacy of protocol
authoring and deployment. To support these protocol
tools, there is a need for a general-purpose knowledge
base for clinical trial protocols.
In this paper, we report on initial development of
one aspect of protocol authoring: a protocol-critiquing
tool. Tools for protocol authoring could increase the
efficiency of new trial development and reduce errors
in protocol specifications by evaluating their contents.
A critiquing tool with a knowledge base of drugs, con-
traindications, toxicities, recommended patient moni-
toring activities, and drug interactions could evaluate a
new protocol design and alert the author if inconsisten-
cies or errors are detected. If the tool had information
about possible corrections in its knowledge base, the
tool could even automatically correct problems that it
finds. In the long term, the tool could be integrated
with other protocol authoring tools, and its knowledge
base could be integrated with the knowledge base of
other clinical trial applications.
Clinical trial protocols can be critiqued on two
levels: protocol methodology and protocol structure.
Protocol methodology is the statistical and study b
sign: sample size, statistical model, definitions of end-
points, and choice of study arms. Protocol structure is
the medical and procedural content: patient treatments,
monitoring tests, and drug toxicity. Protocol critiquing
systems have been described [4, 5], but these systems
have primarily evaluated protocol methodology and
statistical design, rather than protocol structure and
treatment planning.
Errors in protocol structure may be significant be-
cause they contribute to misunderstandings and devia-
tions from the protocol [6]. To our knowledge, a sys-
724
 IPROTOCOL KNOWLED~GE]
Figure 1. Relationships in the data model for critiquing trial protocols. Circled entities are classes of knowledge in
the knowledge base. Arrows depict relationships between classes of knowledge. For example, eligibility criteria are
specified using knowledge of diseases and drug treatments.
tern that evaluates and critiques clinical trial protocols
from a structural perspective has not yet been devel-:
oped.
We hypothesize that authors will produce proto-
cols with have fewer errors if they use a tool that cri-
tiques clinical trial protocol structure. We are devel-
oping such a tool, and call it the Protocol Inspection
and Critiquing Tool of Study Structure, or PICASSO.
PICASSO automatically evaluates and critiques the
procedural and medical aspects of a clinical trial proto-
col design. It accomplishes this evaluation by using
medical knowledge, clinical trial protocol knowledge,
and formal constraints that describe the sorts of errors
we wish to find in a protocol.
Clinical trial protocols are very detailed, and the
requirements to fully model a protocol are extensive,
Therefore, we initially chose to restrict our modeling
efforts to a subset of the clinical trial protocol. Our
initial objective was to develop a knowledge base for
the critiquing tool and a prototype engine that imple-
ments critiquing with respect to diseases, drugs, and
patient monitoring. Because we are using a modular
design, our model is extensible, and can be applied to a
complete representation of protocols in the future.
A Knowledge Base for Critiquing
In our knowledge base for clinical trial protocols,
we divide knowledge into two classes: medical domain
knowledge and clinical trial domain knowledge. The
medical domain component of the knowledge base
describes information about patient conditions, diag-
nostic tests, and therapeutic procedures.< The clinical
trial domain component of the knowledge base repre-
sents issues specific to clinical trials: the specification
of the protocol, and the notions of toxicity and patient
monitoring. Both components are modeled with an
ontology: a set of related concepts that defines the rele-
vant domain of discourse. The ontology provides a
725
j|MEDICAL KNOWLEDGE I
framework within which to organize knowledge using
a hierarchy of taxonomic and other relationships. For
example, both daunorubicin and doxorubicin are sub-
classes of the "sAntitumor Antibiotic" class, which in
turn is a subclass of the cytotoxic drug class. This
knowledge organization allows us to deduce relation-
ships from the hierarchy, such as the fact that daunom-
bicin is a cytotoxic drug.
Malogolowkin analyzed the clinical trial design
process used by experts and showed that protocol con-
tent can be classified into information categories for
depicting the structure of a clinical trial [2]. We identi-
fled several of these attributes in the clinical trial and
medical domains. These attributes are fundamental to
modeling protocols for critiquing. For the clinical trial
domain, they include drugs, eligibility criteria, and
patient monitoring activities. For the medical domain,
fundamental attributes include diseases, patient treat-
ments, drug contraindications and toxicities (adverse
events), and monitoring activities. Figure 1 shows
some of these attributes, as well as some of the reb-
tionships among them in our model of clinical trial
protocols.
In our model, a clinical trial protocol consists of
eligibility criteria, study treatments, and patient moni-
toring. Eligibility criteria define the medical conditions
of the clinical trial patient population. In our knowl-
edge base, we link drug treatments in a protocol to po-
tential drug toxicities for that agent, and to specific
monitoring tests for detecting those toxicities (Figure
1).
As an example of some of the information in our
knowledge base, there is an instance for the disease
"breast cancer" that contains the knowledge that doxD-
rubicin, cyclophosphamide, and tmoxifen are indi-
cated drugs for this condition, whereas corticosteroids
and estrogen are contraindicated. Similarly, there is an
instance in the knowledge base describing daunorubi-
 Figure 2. Example output produced by PICASSO. PICASSO has evaluated Protocol 3, and critiques
this protocol with respect to drug contraindications, drug interactions, and patient monitoring.
cin, which contains information about disease indica-
tions, contraindications, and drug toxicity.
Ourmodel of clinical trial protocols is limited, and
is not sufficient to represent all aspects of a clinical
trial protocol. However, the goal for this initial stage is
to demonstrate the feasibility of critiquing protocol
structure with asubset of the domain. Because we have
used a highly organized ontology to frame our knowl-
edge representation, our model can be easily extended.
Future work will expand our knowledge base to model
clinical trial protocols more fully. An advantage of
using an extensible ontology for capturing protocol
knowledge is that the knowledge base can be easily
extended or revised to accommodate new information
without necessarily requiring changes to the critiquing
engine.
The knowledge base for the authoring tool was de-
veloped with the Protege suite of tools [7]. Our organi-
zation and understanding of the critiquing task evolved
as we built our knowledge base. Protege is designed for
rapidly evolving knowledge bases, which made man-
aging changes easier for us. The tool set also made the
knowledge base readily available to applications such
as our critiquing tool. Protege is designed as a set of
interacting "4plug-in" components. We developed
PICASSO as a plug-in, and any Protege plug-in can
access Protege knowledge bases via a Java application-
programming interface.
A Protocol Critiquiing Tool
Figure 2 shows a screen shot of our protocol cri-
tiquing tool. The tool can evaluate protocols for several
different clinical trials. The figure shows a list of pro-
tocols on the left, and a critique of the selected protocol
on the right. To use our tool, a protocol author builds a
new protocol as a collection of instances describing the
726
eligibility criteria, drug treatments, and patient moni-
toring events for the trial. Figure 3 shows an example
protocol, modeled after a clinical trial for lymphoma.
Once the user enters the protocol specification,
PICASSO can critique it.
In our approach, protocols are critiqued with pro-
tocol constraints expressed in a formal axiom lan-
guage. Thus far, we have captured three constraints
about protocols:
1) Do not gre,cribe contraindicated drugs: The
drug treatments listed in the protocol must
not be contraindicated by any of the diseases
listed in the eligibility criteria.
2) Inlude all required monitoring actions: The
monitoring actions listed in the protocol
must include all required actions listed by
each of the protocol's drug treatments.
3) Do not prescribe interacting drugs: The drug
treatments listed in the protocol must not in-
teract with each other.
The specification of these constraints is dependent
on our ontologies of medical knowledge and of proto-
cols. As these models become richer, we expect to add
new constraints that describe other aspects of protocol
authoring. To allow us to add constraints easily, we
specify our constraints in the formal axiom language
that is part of the Protege environment (PAL, or pro-
tege axiom language). Figure 4 shows the axiom for
drug-drug interactions as written in PAL.
By using this type of formal constraint specifica-
tion, we can be unambiguous, clear, and precise about
the types of protocol authoring errors we wish to find.
In addition, when we need to add new constraints, we
do not need to make any code-level modifications to
the system. We can simply add new constraints, and
the PAL general-purpose engine will check them
 Figure 3. A knowledge instance specifying a particular clinical trial protocol.
against our knowledge base. Although the current im-
plementation does not include a direct connection be-
tween the PAL engine and PICASSO, both are Protege
plug-ins, and thus, could be connected easily.
We have not yet conducted formal evaluation
studies of our tool. However, we tested the face valid-
ity of our approach by entering a few variations on a
protocol for treating lymphoma and critiquing them.
The initial draft protocol specified lymphoma and heart
failure in the eligibility criteria, five drugs (prednisone,
cyclophosphamide, vincristine, doxorubicin, and
phenytoin), and multiple monitoring events (Figure 3).
As shown in Figure 2, PICASSO reported that (1) dau-
norubicin is contraindicated because heart failure was
listed as an eligibility criterion, and that (2) three nec-
essary monitoring events were omitted. When we
changed the eligibility criteria from "heart failure" to
"rheumatic heart disease, complicated by heart failure,"
PICASSO still produced the appropriate critique (Fig-
ure 2, right top panel). Importantly, although the
knowledge base only includes daunorubicin's contrai-
dication in heart failure, PICASSO deduced that it is
plug-ins,____and_thus,___could__be__connected
eforall ?protocols (forall ?drugTreaty
forall ?drugTreat2
Figrheumati (DrugTreatients
oand(Dhe prhotocols
hAr disease,e complcae
dr(= ugTreatments?pe o ?an drugTrea
PAL theart failure,"
? drugTreat2)
(falral
ure ?not
, rihttpb?d ritria,l?drugsTperaet
pael).Imprtanly, lthogh te
(=> (and (Drug ?drugKnow
(DrugAdmwIne ?AdSrugOTrepatr))
(n is(DrugIcntracdinste ?drugnohearultlea)
no DrugAminter ed tat2)t
))) ))))
axim inPAL hat dentfiesnew knowledge often breaks the inference engine, es-
4. Asimpifie
Figure
protocols that have drug-drug interactions. The first
portion states that thls constraint only apples to proto-
cols with at least two dilstct drug treatments. The sec
ond implication searches through the drug knowledge
base and claims that there should be no drug listed un-
der the interactions for "drugTreatl" that matches
"drugTreat2".
also contraindicated in complicated rheumatic heart
disease by evaluating the latter's relationship to heart
failure in the ontology.
Dlscussi and Conclusions
Discussion and Conclusions
Many clinical trials are developed annually, yet
most are written from scratch. Errors in the protocols
for new clinical trials occur commonly [1], and include
omissions and ambiguities in protocol semantics [6].
We undertook this study to develop a tool that uncov-
ers semantic errors in clinical trial protocols by evalu-
ating their medical and procedural contents. We as-
sume that all necessary information can be modeled in
a knowledge base of the medical and clinical trial do-
mains.
In developing our critiquing model, we distinguish
medical knowledge from critiquing knowledge [8], and
model these separately. Medical knowledge includes
information about diagnostic and therapeutic proce-
dures, whereas critiquing knowledge includes the n-
formation content of clinical trial protocols and how
structural design errors may occur. This distinction is
useful because medical knowledge is independent of
clinical trial protocols, while critiquing knowledge is
more specific, and describes how to evaluate protocol
structure. Our approach to knowledge modeling for the
critiquing
moe ths taskseaaey
expert system isthat
similar that used by HyperCritic, an
toMeia'.wedeicue
advises in the management of hy-
pertensive
drues, wheeapatients
criatiqo [9]. uin
HyperCritic
knpowledgexplicitly separates
includesknowledg
critiquing knowledge from medical knowledge, which
| is advantageous for knowledge modeling and reus [9].
A challenge for designers of expert systems like
PICASSO is to deal with knowledge that changes over
time. In traditional knowledge-based systems, adding
~ ~ ~ ~ ~ ~pecial y
if that system uses declarative production
rule. on advatageoftourap c thatknowed
modeledon antextenibl tolothat e od
knowledgesi tefrso ationshipsa concets in
a hierarchy.
727
 For example, the knowledge base could have an
entry stating that all beta blocker drugs are contraindi-
cated in heart failure. As new beta blockers are devel-
oped, we could add to them knowledge base and the
critiquing tool would correctly recognize that these
drugs are also contraindicated in heart failure. Likewise
in our test case, we were able to encode high level con-
cepts in our knowledge base (daunorubicin is contrain-
dicated in heart failure), and PICASSO correctly de-
termined that daunorubicin is contraindicated in a pro-
tocol that included rheumatic heart disease,
complicated by heart failure. For these types of addi-
tions to the knowledge base, PICASSO behaves cor-
rectly without any need to change the code that imple-
ments its critiquing functionality.
Our current model and knowledge base for clinical
trial protocols are over-simplified. However, an ad-
vantage of working within Prot6g6 is that models can
be easily extended and integrated with other clinical
trial tools. For example, we have developed a tool to
assist protocol authors create eligibility criteria for new
trials [10]. This "eligibility writer" tool uses a similar
knowledge-based architecture, and could be integrated
with PICASSO to create a tool that supports both pro-
tocol authoring and critiquing.
Our team has also used Proteg6 for other sorts of
clinical trial protocol tools. For example, we have de-
veloped tools that screen patients and match them to
clinical trials, as well as tools that monitor patients
enrolled in clinical trials [11]. All of these tools can
utilize an integrated knowledge base for clinical trial
protocols. For example, the medical knowledge base
for PICASSO was created from a knowledge base de-
veloped for our patient monitoring tool. Our approach
is to model protocol knowledge from the "bottom-up":
always with a specific tool or task in mind. Then, the
Protege tool set allows us to easily combine and inte-
grate models and knowledge bases where appropriate.
We have demonstrated the feasibility of develop-
ing a tool to evaluate the structure of clinical trial pro-
tocols and critique them. The expected benefits from
our approach include: (1) a flexible critiquing model
that can be subsequently modified and extended, (2)
systematic use of standardized critiquing strategies, and
(3) improved quality and clarity of new protocols.
Although our initial prototype does not fully model
protocols, our knowledge base can be extended without
unduly affecting the critiquing engine. Future work
will extend the ontology in order to represent clinical
trial protocols more completely. A critiquing tool may
improve the quality of protocols in the future, and
could support standardization for clinical trials. Tools
like PICASSO that support the clinical trial process
could lead to faster protocol deployment and patient
accrual, and could ultimately lead to a more efficient
and successful process for clinical research.
728
Acknowledgments We wish to acknowledge the work
and assistance of the Prot6ge scientific staff, and espe-
cially William Grosso for the development of Protege
Axiom Language. Thanks to Valerie Natale for com-
ments on this manuscript.
References
1. Andersen B. Methodological Errors in Medical
Research: An Incomplete Catalogue. Boston: Black-
well Scientific Publications; 1990.
2. Malogolowkin MH, Horowitz RS, Ortega JA,
Siegel SE, Hanunond GD, Weiner JM. Tracing expert
thinking in clinical trial design. Comput Biomed Res
1989;22(2): 190-208.
3. Silva J, Wittes R. Role of clinical trials infor-
matics in the NCI's cancer informatics infrastructure.
Proc AMIA Symp 1999:950-4.
4. Wyatt JC, Altman DG, Heathfield HA, Pantin
CF. Development of Design-a-Trial, a knowledge-
based critiquing system for authors of clinical trial
protocols. Comput Methods Programs Biomed
1994;43(3-4):283-91.
5. Haag U. Knowledge representation for com-
puter-aided planning of controlled clinical trials: the
PATriCIa project. Methods Inf Med 1997;36(3): 172-8.
6. Musen MA, Rohn JA, Fagan LM, Shortliffe EH.
Knowledge engineering for a clinical trial advice sys-
tem: uncovering errors in protocol specification. Bulle-
tin du Cancer (Paris) 1987;74(3):291-6.
7. Musen MA, Fergerson RW, Grosso WE, Noy
NF, Crubezy M, Gennari JH. Component-Based Sup-
port for Building Knowledge-Acquisition Systems. In:
Proceedings of the Conference on Intelligent Inforns-
tion Processing (IIP 2000) of the International Federa-
tion for Information Processing World Computer Con-
gress (WCC 2000); 2000; Beijing; 2000.
8. van der Lei J, Musen MA. The separation of re-
viewing knowledge from medical knowledge. Methods
Inf Med 1995;34(1-2):131-9.
9. van der Lei J, Musen MA. A model for critiqu-
ing based on automated medical records. Comput Bio-
med Res 1991;24(4):344-78.
10. Rubin DL, Gennari JH, Srinivas S, Yuen A,
Kaizer H, Musen MA, et al. Tool support for authoring
eligibility criteria for cancer trials. Proc AMIA Symp
1999:369-73.
11. Tu SW, Kemper CA, Lane NM, Carlson RW,
Musen MA. A methodology for determining patients'
eligibility for clinical trials. Methods Inf Med
1993;32(4):317-25.