CE: D.C.
; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429
REVIEW PAPER
No Surgical Innovation Without Evaluation
Evolution and Further Development of the IDEAL Framework and
Recommendations
Allison Hirst, MSc,  Yiannis Philippou, MB, BChir,  Jane Blazeby, MD,y Bruce Campbell, MS,z
Marion Campbell, PhD,§ Joshua Feinberg, MD,ô Maroeska Rovers, PhD,jj Natalie Blencowe, PhD,y
Christopher Pennell, MD,ô Tom Quinn, MPhil,  Wendy Rogers, PhD,yy Jonathan Cook, PhD,zz
Angelos G. Kolias, PhD,§§ôô Riaz Agha, MBBS, MRCSEng,jjjj Philipp Dahm, MD, 
Art Sedrakyan, MD, PhD,yyy and Peter McCulloch, MD 
Objective: To update, clarify, and extend IDEAL concepts and recommen-
dations.
Background: New surgical procedures, devices, and other complex inter-
ventions need robust evaluation for safety, efficacy, and effectiveness. Unlike
new medicines, there is no internationally agreed evaluation pathway for
generating and analyzing data throughout the life cycle of surgical innova-
tions. The IDEAL Framework and Recommendations were designed to
provide this pathway and they have been used increasingly since their
introduction in 2009. Based on a Delphi survey, expert workshop and major
discussions during IDEAL conferences held in Oxford (2016) and New York
(2017), this article updates and extends the IDEAL Recommendations,
identifies areas for future research, and discusses the ethical problems faced
by investigators at each IDEAL stage.
Methods: The IDEAL Framework describes 5 stages of evolution for new
surgical therapeutic interventions—Idea, Development, Exploration, Assess-
ment, and Long-term Study. This comprehensive update proposes several
modifications. First, a ‘‘Pre-IDEAL’’ stage describing preclinical studies has
been added. Second we discuss potential adaptations to expand the scope of
IDEAL (originally designed for surgical procedures) to accommodate thera-
peutic devices, through an IDEAL-D variant. Third, we explicitly recognise
the value of comprehensive data collection through registries at all stages in
the Framework and fourth, we examine the ethical issues that arise at each
From the Nuffield Department of Surgical Sciences, University of Oxford, Oxford,
UK; yBristol Centre for Surgical Research, University of Bristol, Bristol, UK;
zUniversity of Exeter Medical School, Exeter, UK; §Health Services Research
Unit, University of Aberdeen, Aberdeen, UK; ôMaimonides Medical Center,
Brooklyn, NY; jjRadboud University Medical Center, Nijmegen, The
Netherlands; Faculty of Health, Social Care and Education, Kingston Uni-
versity and St George’s, University of London, London, UK; yyDepartment of
Philosophy and Department of Clinical Medicine, Macquarie University,
Sydney, Australia; zzNuffield Department of Orthopaedics, Rheumatology
and Musculoskeletal Sciences, University of Oxford, Oxford, UK;
§§Division of Neurosurgery, Department of Clinical Neurosciences, Adden-
brooke’s Hospital and University of Cambridge, Cambridge, UK; ôôSurgery
Theme, Cambridge Clinical Trials Unit, Cambridge, UK; jjjjDepartment of
Plastic Surgery, Guy’s and St. Thomas’ NHS Foundation Trust, London, UK;

Department of Urology, Minneapolis VAMC and University of Minnesota,
Minneapolis MN; and yyyWeill Cornell Medical College, New York, NY.
No specific funding was obtained for this update of IDEAL. The discussion meeting
was funded as part of the IDEAL International Conference in Oxford on
April 7, 2016 by several sponsors as follows: the Oxford Academic Health
Science network (Oxford AHSN), The Health Foundation, Medtronic, and
Johnson and Johnson (J&J). It was also supported by BMJ and the Lancet.
IDEAL is funded by the Oxford NIHR Biomedical Research Centre, educa-
tional grants and meetings income (PMcC and A.H.). J.M.B. is an NIHR Senior
Investigator supported in part by the Medical Research Council (MRC)
ConDuCT-II (Collaboration and innovation for Difficult and Complex ran-
domized controlled Trials in Invasive procedures) Hub (MR/K025643/1) and
Annals of Surgery  Volume XX, Number XX, Month 2018
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
stage of IDEAL and underpin the recommendations. The Recommendations
for each stage are reviewed, clarified and additional detail added.
Conclusions: The intention of this article is to widen the practical use of
IDEAL by clarifying the rationale for and practical details of the Recom-
mendations. Additional research based on the experience of implementing
these Recommendations is needed to further improve them.
Keywords: ethics, IDEAL framework and recommendations, medical
devices, randomized controlled trials, registries, research methodology,
surgical innovation
(Ann Surg 2018;xx:xxx–xxx)
S urgery is a complex intervention with properties which make it
more difficult to evaluate rigorously than drug treatments. Evalu-
ation methods that fail to address this complexity have led to much
controversy and wasted effort through poor study design, inadequate
reporting and failure to reach agreement on standards for high quality
trials. The resulting adverse consequences have included widespread
adoption of new techniques or devices which later proved to be harmful
and of refusal by healthcare funders to reimburse for innovations with
an inadequate evidence base, as well as large scale failures of surgical
research to compete successfully for public funding.
the NIHR Bristol Biomedical Research Centre at the University Hospitals
Bristol NHS Foundation Trust and the University of Bristol. The views
expressed are those of the author and not necessarily those of the UK National
Health Service, National Institute for Health Research, or Department of
Health.
This paper updates the IDEAL Framework and Recommendations following review
at international expert meetings held during IDEAL Conferences in Oxford,
UK in 2016 and New York, USA in 2017.
A.D.H. and Y.P. contributed equally as joint first authors.
A.S. and P.Mc.C. contributed equally as joint senior authors on behalf of IDEAL
Collaboration (attendees at 2016 Oxford consensus meeting listed in full in
Acknowledgements with permission).
The authors report no conflicts of interest.
Supplemental digital content is available for this article. Direct URL citations
appear in the printed text and are provided in the HTML and PDF versions of
this article on the journal’s Web site (www.annalsofsurgery.com).
Reprints: Peter McCulloch, MD, Professor of Surgical Science and Practice,
Nuffield Department of Surgical Sciences (Level 6), John Radcliffe Hospital,
University of Oxford, Oxford OX3 9DU, UK.
E-mail: peter.mcculloch@nds.ox.ac.uk; Allison Hirst, MSc, IDEAL Collabo-
ration Project Manager/Researcher, Nuffield Department of Surgical Sciences
(Level 6), John Radcliffe Hospital, University of Oxford, Oxford OX3 9DU,
UK. E-mail: allison.hirst@nds.ox.ac.uk.
Copyright ß 2018 Wolters Kluwer Health, Inc. All rights reserved.
ISSN: 0003-4932/16/XXXX-0001
DOI: 10.1097/SLA.0000000000002794
www.annalsofsurgery.com | 1
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429
Hirst et al
The IDEAL Framework and Recommendations represent a
new paradigm for the evaluation of surgical operations, invasive
medical devices and other complex therapeutic interventions.
IDEAL began with a series of meetings at Balliol College, Oxford
during 2007 to 2009 to discuss the specific challenges of evaluating
surgical innovation, recognizing, analyzing, and proposing solutions
for the challenges which arise as new procedures move from proof of
concept towards a randomized controlled trial (RCT). These dis-
cussions resulted in the publication of a five-stage Framework
describing the natural stages of surgical innovation (Idea, Develop-
ment, Exploration, Assessment, and Long-term Study), together with
recommendations for a rigorous stepwise surgical research pathway,
and suggestions for appropriate study methodology for the questions
which characterise each stage.1 –3 This was subsequently followed up
by publications offering methodological guidance.4–7
Each stage is defined by a key research question:
 IDEA—STAGE 1: What is the new treatment concept and why is
it needed?
 DEVELOPMENT—STAGE 2a: Has the new intervention
reached a state of stability sufficient to allow replication
by others?
 EXPLORATION—STAGE 2b: Have the questions that might
compromise the chance of conducting a successful RCT been
addressed?
 ASSESSMENT—STAGE 3: How does the new intervention
compare with current practice?
 LONG-TERM STUDY—STAGE 4: Are there any long-term or
rare adverse effects or changes in indications or delivery quality
over time?
Various users and funders of research have acknowledged
the utility of IDEAL (Table 1). Surgical researchers are also increas-
ingly citing and using the study designs and reporting formats
recommended by IDEAL (552 papers cited key IDEAL papers
[Web of Science searched 19 October 2017]). Despite these signs
of interest, international use of IDEAL remains limited. It is clear
that researchers need more detailed guidance about how to imple-
ment the recommendations8 –10 which were initially outlined in a
generalized way.
We revised and updated the Framework and Recommenda-
tions using a 3-step modified Delphi process comprising a 2-round
online questionnaire survey between December 2015 and April 2016
followed by an expert consensus meeting at the IDEAL Conference
TABLE 1. URL Links to Organizations Utilizing IDEAL Recommendations
Organization
National Cancer Research Institute (NCRI)
NHS Commissioners in the UK
Medical Device Epidemiology Network Initiative (MDEpiNet) partnership with the
US Food and Drug Administration (FDA)
European Network for Health Technology Assessment (EUnetHTA
National Institute of Healthcare Research in the UK (NIHR) recommends IDEAL
study designs in commissioning briefs for example, 17/17 Fibrin Glue for
Pilonidal Sinus Disease
Royal College of Surgeons/NIHR Surgical Technology Evaluation Portal
NIHR Office for Clinical Research Infrastructure (NOCRI)
2 | www.annalsofsurgery.com
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery  Volume XX, Number XX, Month 2018
at St Catherine’s College, Oxford on April 8, 2016 (Appendix 1—
link 1, http://links.lww.com/SLA/B416). The first round required
participants to address 22 questions aimed at clarifying the IDEAL
Recommendations, and the second round was focused on 11 areas
where consensus was not clear in round 1, with remaining areas of
controversy dealt with at the Oxford meeting. The international
group of experts comprised 56 participants (47 attended the final
meeting) including surgeons, methodologists, clinical trialists, eth-
icists, journal editors, Health Technology Assessment professionals,
purchasers of healthcare, and device industry representatives. Further
details of the process and findings are provided (Appendix 2, http://
links.lww.com/SLA/B417).
SCOPE
Discussions about updating IDEAL covered 4 main new areas.
First, the need for a pre-IDEAL, preclinical Stage. Second, the
application of IDEAL to other complex health interventions. Third,
a reconsideration of the place of registries in the IDEAL pathway,
and finally an explicit examination of the ethical issues that arise at
each stage of IDEAL and underpin the recommendations.
Publications reporting preclinical studies prior to ‘‘first in
human’’ studies as ‘‘IDEAL Stage 0’’ have already appeared.11
However, due to the challenge of drafting recommendations for
conduct and reporting of such a potentially broad and varied set
of different study types, we recommend the use of the term ‘‘Pre-
IDEAL’’ for these studies rather than adding a formal ‘‘Stage 0’’ to
the Framework.
IDEAL’s potential for application to therapeutic medical
device evaluation was quickly recognized (Appendix 1-link 2,
http://links.lww.com/SLA/B416), and the necessary modifications
have already been summarized in a description of an IDEAL-D
(Devices) variant12 supported by an international Delphi process.13
Other specific variants have also been proposed, such as the R-
IDEAL tool from the magnetic resonance imaging-Linear Accelera-
tor Consortium14 and IDEAL-Physio.15 However, the consensus
group agreed not to broaden the scope of the original IDEAL
Framework, but to welcome and monitor the work of groups investi-
gating potential wider uses.
With the help of academic ethicists, the scope of the IDEAL
Framework and Recommendations has been expanded to include
explicit ethical guidance at each stage. Ethical issues arising in each
of the IDEAL stages are examined in detail in an accompanying
paper.16
URL
http://www.ncri.org.uk/events/ncri-future-of-surgery-technology-
trials-in-surgery/
http://www.ideal-collaboration.net/2015/03/ideal-resources-
commissioners-nhs-right-care-website/
http://mdepinet.org/
http://www.eunethta.eu/
http://www.nihr.ac.uk/funding-and-support/documents/current-
funding-opportunities/hta/17_17cb.pdf
https://www.rcseng.ac.uk/standards-and-research/research/surgical-
trials-initiative/rcs-nihr-surgical-technology-evaluation-portal-
launched/
https://www.nihr.ac.uk/about-us/how-we-are-managed/managing-
centres/nihr-office-for-clinical-research-infrastructure/industry-
case-studies.htm
ß 2018 Wolters Kluwer Health, Inc. All rights reserved.
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429
Annals of Surgery  Volume XX, Number XX, Month 2018
UPDATE OF IDEAL: STAGE BY STAGE
The 2009 Framework comprised 5 stages: Idea (1), Develop-
ment (2a), Exploration (2b), Assessment (3), and Long-term study
(4). (3) The main purpose of each stage is summarized above.
We updated the original Framework to improve its usability
and transparency. This update includes description of each IDEAL
stage using the population, intervention, comparator, outcome frame-
work (patients/operators, interventions, comparators, outcomes),
identifying appropriate reporting guidelines and identifying when
to progress to the next stage. We also underline the key research
question for each IDEAL stage and highlight areas for future
research. The revised and original versions are shown side by side
to clarify the changes made in this update (Table 2).
Pre-IDEAL Studies
Purpose and Description
Pre-IDEAL research is essential prior to first in human trials of
an innovation.
Patients and Operators
Although patients do not receive treatments in Pre-IDEAL
studies, consultation is desirable to evaluate the societal need and
perceived value of the proposed intervention.17 Where the investi-
gators are engineers or scientists with no clinical background,
clinicians should also be consulted.
Intervention and Comparator
Preclinical studies usually do not involve a comparator. How-
ever, testing (eg, surface wear, battery drain), simulation or modeling
studies may allow comparison with the current standard.
Outcomes
Depending on the type of preclinical study, the aims and
methods may vary widely. Typically, key studies focus on demonstrat-
ing that the intervention brings about the intended physical changes.
Following this, studies to estimate reliability and safety, qualitative
studies with stakeholders to determine potential demand and accept-
ability, and modeling studies to predict overall impact on health care
costs/efficiency may be desirable. This is especially relevant for new
devices for the purpose of coverage and reimbursement.
Appropriate Study Designs
Preclinical studies include material testing, simulator,
cadaver, animal, modeling, and cost-effectiveness studies.18–20
Controlled experiments will often be appropriate and feasible
for laboratory or bench studies. Because of the wide range of study
objectives and methods, comprehensive recommendations about
study design are impractical. We therefore recommend following
the best available authoritative guidance on particular study types
(Appendix 1- link 3, http://links.lww.com/SLA/B416).
Recommended Reporting Guidelines for Protocols
and Full Reports
Explicit reporting of Pre-IDEAL findings is both practically
and ethically necessary to support optimal development of Stage 1
studies. However protection of professional or commercial advan-
tage may inhibit full reporting. Transparency around study design,
subjects, and outcomes is essential, but recommendations for report-
ing of sensitive details of mechanism or technique at this stage may
prove impractical. We recommend using the numerous high quality,
subject-specific guidelines that exist for reporting different kinds of
preclinical studies and protocols which can be found on the
ß 2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
IDEAL Framework and Recommendations Update
EQUATOR and Meridian Network databases (Appendix 1—link
4, http://links.lww.com/SLA/B416), or at the National Institutes of
Health (Appendix 1—link 5, http://links.lww.com/SLA/B416).
Ethical Aspects
Animal studies should abide by recognized ethical guidelines,
including the imperative to reduce, refine, and replace animal use
whenever possible (Appendix 1—link 6, http://links.lww.com/SLA/
B416). Animal models must be valid and the results applicable to
humans. Pre-IDEAL simulation and bench testing can raise ethical
considerations of justice where assumptions built into testing (eg,
about typical body shapes) may limit the applicability of the results,
or expose subgroups to greater risk of harm.16
Identifying Stage Endpoints
In principle, the pre-IDEAL stage should be complete before
the first in human procedure is done. Any feasible studies which can
be expected to identify avoidable and/or predictable risks of failure or
harm to the first patient should be performed.
‘‘Idea’’ Stage 1—‘‘First in Human’’ Use
Purpose and Description
IDEAL Stage 1 describes the first use of a new procedure or
device in a patient, either as part of a planned approach or in an
unplanned emergency situation.21 Where planned, the patient or
patients are usually highly selected.
Patients and Operators
Patient selection criteria should be explained in detail: if any
proposed patients refuse the procedure, or are excluded, this should
be explained with reasons. The operators are usually an individual or
small team in a single center.
Intervention
A full technical description of the new procedure or technology,
in sufficient detail to allow an equivalently skilled operator to repro-
duce it, should be provided. There is normally no explicit comparator.
Outcomes
Commonly include technical success, safety and short-term
physiological and clinical measures. Transparency demands that all
adverse events must be fully disclosed. Whenever feasible, in all
stages of IDEAL, outcomes should be described using widely
accepted standardized definitions and terms, preferably selected
from a core outcomes set (Appendix 1—link 7, http://links.
lww.com/SLA/B416).
Appropriate Study Designs
Stage 1 involves a single case or a few cases. If enough
suitable patients are available progression may occur within a short
time. It is recommended that reports explain the need for the new
treatment concept and why it might be better than currently available
treatment. Video recording and sharing is highly recommended and
can be part of on-line publication.
Recommended Reporting Guidelines for Protocols
and Full Reports
We recommend Stage 1 studies be reported as a single case
report. The SCARE Guidelines provide a useful standard reporting
structure.22 Menon et al11 demonstrate a well-reported IDEAL model
of surgical innovation in the development of robotic kidney trans-
plantation with regional hypothermia.
www.annalsofsurgery.com | 3
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
Hirst et al
TABLE 2. Summary of the IDEAL Framework, Recommendations and Proposals Including Updates
Stage of Innovation
Pre-IDEAL
Pre-clinical
Stage 1 Idea
First in human
Stage 2a Development
Single center/single
intervention; case series/
prospective cohort
Stage 2b Exploration
Bridge from observational
to comparative evaluation.
Purpose is to gain data to
decide if and how to test in
a robust RCT or other
appropriate pivotal design.
4 | www.annalsofsurgery.com
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
ANNSURG-D-17-02429
Updated IDEAL Framework
Pre-IDEAL was not described in original
IDEAL framework
Purpose: Feasibility and definition of procedure
Number and Types of Patients: preclinical
Number and Types of Surgeons: Very few;
innovators
Output: Description addressing:
 Whether intended goal of procedure is
accomplished
 Level of difficulty of performing procedure or
using device as compared with standard of
care
 Safety risks
 Desirability of intervention
Method: Various, including simulator, cadaver,
animal, modelling, and costeffectiveness studies
Stage Endpoint: Any studies that could avoid
predictable risks of failure or harm to the first
human should have been conducted.
Purpose: Proof of concept
Number and Types of Patients: Single digit;
highly selective.
Number and Types of Surgeons: Very few;
innovators
Output: Description
Intervention: Evolving; procedure inception
Methods: Structured case reports
Outcomes: Proof of concept; technical
achievement; dramatic success; adverse events,
surgeon views of the procedure
Stage Endpoint: Outcomes will determine whether
to proceed to stage 2a.
Purpose: Development of procedure
Number and Types of Patients: Few; Selected
Number and Types of Surgeons: Few; innovators
and some early adopters
Output: Technical description of procedure and its
development with reasons for changes
Intervention: Evolving; procedure development
Methods: Prospective development studies
Outcomes: Mainly safety; technical, and
procedural success
Stage Endpoint: Procedure should be refined
enough to allow replication in Stage 2b and there
should be no intent to make further major
modifications
Purpose: Achieving consensus between surgeons
and centers
Number and Types of Patients: Many; broadening
indication to include all potential beneficiaries
Number and Types of Surgeons: Many;
innovators, early adopters, early majority
Output: Effect estimate based on large sample;
Analysis of learning curves; estimate of influence
of prespecified technical variants and patient
subgroups on outcome.
Intervention: Stable; acceptable variants defined
Method: Prospective multicenter exploration cohort
study (disease or treatment based); pilot/feasibility
multicenter RCTs.
Outcomes: Safety; clinical outcomes (specific/
graded); short-term outcomes; patient centered/
reported outcomes; feasibility outcomes
Stage Endpoints: fall in to two main groups;
Demonstrate that technique can be more widely
adopted; and, Demonstrate that progression to RCT
is desirable and feasible
Annals of Surgery  Volume XX, Number XX, Month 2018
Updated IDEAL Recommendations for
Researchers
 All predictable risks to patients should
be investigated before human studies
begin
 Guidelines on best scientific practice
and ethics specific to the types of
study should be followed where
available
 A minimum dataset describing
technical consistency should be made
public before first-in-human testing.
 Provide full details of patient selection,
technique and outcomes, and patients
not selected during the time frame, and publication (eg, IJS: Case Reports and
why.
 Use standard well-defined measures for
reporting outcome and patient
characteristics
 Use structured reporting system eg,
SCARE checklist.
 Make the above information available
to peers regardless of outcome
 Make protocol for study available
 Use standard well-defined measures for Development study formats and protocols
reporting outcome and patient
characteristics
 Report and explain all exclusions
 Report all cases sequentially with
annotation and explanation of when
and why changes to indication or
procedure took place.
 Display main outcomes graphically to ethics approval required
illustrate the above.
 Make protocol for study available
 Use standard well-defined measures
for reporting outcome and patient
characteristics
 Participate in collaborative
multi-center cooperative data
collection, incorporating feasibility
issues such as:
estimating effect size,
defining intervention quality
standards,
evaluating learning curves,
exploring subgroup differences,
eliciting key stakeholder values and
preferences,
analysis of adverse events:
 Preplanned consensus meeting prior to
progressing to an RCT to identify
feasibility and ability to recruit,
intervention and comparator
definitions, appropriate patient
selection criteria, primary endpoint.
ß 2018 Wolters Kluwer Health, Inc. All rights reserved.
Updated IDEAL Proposals for Improving
the Surgical Research Environment
Journals:
Editors should require publication of the Pre-
IDEAL minimum dataset before or together
with first-in-human reports
Regulatory/legal: Regulators should develop
a definition of the minimum publically
available dataset required to allow First-in
Human studies of new devices to proceed.
Ethical aspects: general standards of
research integrity apply
Journals: Encourage or require registration
of the innovation when considering for
www.researchregistry.com)
Regulatory/legal: Provide public interest
defence from legal discovery for registries
specifically for first-in-human studies.
Ensure local hospital policy on innovative
procedures groups foster innovation (ie, IRB
or new procedure committee)
Ethical aspects: multiple strategies required
to minimize harms to patients, including
formal human research ethics approval for
selected planned interventions
Journals: Support for publication of
Regulatory/legal: Ensure that patient
consent includes information about known
outcomes from Stage 1 , about unknown
risks and—inform the patient that the
surgeon has carried out few of the
procedures previously
Ethical aspects: formal human research
Funders: Support Stage 2b Exploratory
cohort studies as preliminary ‘‘pilot/
feasibility’’ phases for RCT proposals.
Journals: Support publication of IDEAL
Exploration studies and protocols
Ethical aspects: formal human research
ethics approval required Ensure that potential
harms from the learning curve are minimized
by training and mentoring prior to
progressing to Stage 3
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429

Annals of Surgery  Volume XX, Number XX, Month 2018 IDEAL Framework and Recommendations Update

TABLE 2. (Continued)
Stage of Innovation Updated IDEAL Framework
Stage 3 Assessment Purpose: Comparative effectiveness testing
Definitive comparative Number and Types of Patients: Many; expanded
evaluation of main efficacy indications (well-defined)
and safety aspects of new Number and Types of Surgeons: Many; early
technique against current majority
best treatment. Output: Comparison with current standard therapy
Intervention: Stable
Method: RCT with or without additions/
modifications; alternative designs (cluster,
preference RCTs, stepped wedge, adaptive designs)
Outcomes: Clinical outcomes (specific and
graded); potentially Patient Reported outcomes,
Health Economic outcomes
Stage Endpoints: two main endpoints; Clear valid
evidence on relative effectiveness of innovation;
and, Identification of issues requiring long-term
monitoring.
Stage 4 Long-term monitoring Purpose: Surveillance
Number and Types of Patients: All eligible
Number and Types of Surgeons: All eligible
Output: Description; audit; regional variation;
quality assurance; risk adjustment
Intervention: Stable
Method: Registry; routine database; rare-case
reports
Outcomes: Rare events; long-term outcomes;
quality assurance
Additions: Registries for devices—IDEAL-D
Registries at earlier stages of IDEAL

Registries should be organized according to the IDEAL recommendations and should be available for enrolment at any Stage.
Patient consent should always include outcomes from previous IDEAL Stage.
Actions for Professional societies:
 Ensure guidelines explicitly support IDEAL model of technical development and evaluation.
 Require members to use appropriate registers for the various stages of innovation as a condition of specialist recognition.
RCTs ¼ randomized controlled trials.
Updated IDEAL Recommendations for Updated IDEAL Proposals for Improving
Researchers the Surgical Research Environment
 Register on an appropriate Funders: Support trial proposals
international register (eg, incorporating preparatory Stage 2b work
clinicaltrials.gov) Journals: Encourage authors to refer to work
 Use standard well-defined measures on innovation in prior IDEAL stages
for reporting outcome and patient preceding RCT.
characteristics Support use of appropriate reporting
 Incorporate information about patient guidelines.
and clinician values and preferences in Mandate registration of RCT in trials register
consent information and outcome prior to publication.
measure design Ethical aspects: formal human research
 Reporting guidelines: ethics approval required
CONSORT update of 2010 with
extension for nonpharmacological
treatments
COMET
TIDieR
SPIRIT (for RCT protocol design)
 Registries may begin from the earliest Funders: Link funding for purchasing
stages of human use treatment to delivery of adequate long-term
 Registry datasets should be defined by follow-up
the clinical community with patient Ethical aspects: resolve issues of consent for
input data use and especially for nested RCTs
 Datasets should be simple, cheap, and
easy to collect
 Curation of registries by clinical
community is desirable
 Funding of registries should be agreed
between government and commercial
interests but kept separate from
curation
 Consent for use of registry data in
research should be broad and where
possible automatic

Ethical Aspects interventions is critical to prevent repetition of harmful errors.

Significant ethical issues arising in Stage 1 include: minimiz-
ing patient harm; ensuring adequately informed patient consent;
optimizing communication about the innovation in the surgical team;
identifying and managing conflicts of interest; and obligatory full
and accurate reporting of outcomes to prevent avoidable harms to
future patients. Reflection is important to confirm the potential for
the new innovation to solve a real clinical problem. Planned inno-
vations should be conducted in compliance with local hospital
and research ethics frameworks such as IRBs. Local organizations
should be responsible for ensuring that review supports rather than
discourages innovation.16
Identifying Stage Endpoints
Once Stage 1 is completed, deciding whether to progress to
Stage 2a depends on proof of concept, technical achievement,
apparent safety, and potential efficacy.
Areas for Future Work and Research
The IDEAL Collaboration strongly recommends registration
of all first-in-human procedures. There is an ethical obligation
on investigators to make their research available to others
contemplating similar interventions. Registration of unsuccessful
However, investigators may be discouraged if transparency
exposes them to legal and other challenges. Therefore, regulatory
and governance frameworks need to evolve to accommodate
these tensions. Public interest protection rules similar to those
shielding accident investigations in the transport industries
could achieve this. The recently enacted UK ‘‘Access to Medical
Treatments (Innovations) Bill’’ allows the creation of a ‘‘medical
innovations register’’ which, with appropriate protection could
represent an IDEAL Stage 1 register (Appendix 1—link 8, http://
links.lww.com/SLA/B416).
Incentives may be necessary to ensure that innovations with
adverse outcomes are registered. Potential mechanisms to achieve
this include making reporting a professional or legal obligation or
requiring registration as a prerequisite to publishing first-in-human
case reports.
‘‘Development’’ (Stage 2a)—Toward Stabilization
of the Technique
Purpose and Description
In IDEAL Stage 2a, procedures are typically undergoing
iterative modification toward a final, stable version.

ß 2018 Wolters Kluwer Health, Inc. All rights reserved. www.annalsofsurgery.com | 5

Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429
Hirst et al
Patients and Operators
Usually, only a few patients and a few operators are involved,
within a single institution for technique-based innovations. For new
devices the number of operators and centers may be larger.
Intervention
A technical description of the initial version of the interven-
tion is needed and an account of when, why and how modifications to
either technique or selection criteria were made.23 No comparator is
involved this stage. For example, Diez del Val et al23 report the
development of robotically assisted oesophagectomy by a two-
surgeon team from the first robotic case onwards, presenting the
cases sequentially and showing key outcomes (blood loss, robotic
operating time, lymph node yield, length of stay, and complications)
for each patient. The reasons for rejecting cases for robotic surgery
are explained, and all changes to technique or indication are
highlighted, showing when they occurred and explaining why they
were instituted.
Outcomes
Relevant outcomes include short-term clinical, technical, and
safety outcomes.
Appropriate Study Designs
Normally a small single center prospective cohort. A typology
which deconstructs interventions into their component parts may
help with precise definition of procedures, and clarify description of
which parts of the procedure change as it is modified and updated.24
Recommended Reporting Guidelines for Protocols
and Full Reports
IDEAL advocates that Stage 2a studies should report:
 Patient inclusion and exclusion criteria, how many patients were
assessed for treatment, which candidates were excluded, and the
reasons for their exclusion. Patients considered for, but ultimately
not offered the intervention may also be described, together with
their outcomes.
 Consecutive presentation of case-specific outcomes for all cases
treated. May involve use of statistical monitoring techniques, for
example, CUSUM.25
 When and why modifications to the technique or indications
occurred, to reduce the risk of avoidable harm by preventing
repetition of unsuccessful modifications. A graphic representation
showing when technical modifications occurred during the series
is useful.
Ethical Aspects
Stage 2a studies are planned, and are therefore subject to
appropriate institutional research ethics review. Issues specific to
Stage 2a are minimizing harm during development of the new
procedure, informed consent, and transparent publication of out-
comes. Consent should include information about Stage 1 outcomes,
acknowledging that risks cannot be reliably predicted or quantified at
this stage. Collecting and reporting outcomes in accessible ways
minimizes avoidable harm to future patients and guides the fair and
equitable uptake of innovation. In this and subsequent stages an
independent oversight group to monitor outcomes during the study is
highly desirable particularly for high risk procedures.16
Identifying Stage Endpoints
By the end of this stage, the procedure and indications should
be stable enough to permit multicenter replication during Stage 2b.
6 | www.annalsofsurgery.com
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery  Volume XX, Number XX, Month 2018
The phase of intentional iterative change should be complete,
although limited further modifications may still occur in Stage 2b.
Areas for Future Work and Research
Further work is needed to guide development of protocols to
plan Stage 2a studies, and to develop methods to evaluate when Stage
2a endpoints have been reached. (Appendix 1—link 9, http://links.
lww.com/SLA/B416).
‘‘Exploration’’ (Stage 2b)—Bridge to a Pivotal Trial
Purpose and Description
In the ‘‘Exploration’’ phase a greater experience of the new
intervention is gained in a wider group of surgeons and patients to
collect information which will determine whether and how to
progress to a definitive comparison against current best treatment.
The data should be used to promote consensus on the design and
conduct of a future RCT, and to improve its feasibility by minimizing
potential barriers to delivery.26,27 Chen et al27 conducted a 20-center
nonrandomized prospective cohort study of treatments for uterine
fibroids, in which patients chose to receive hysterectomy, myomec-
tomy or high-intensity focused ultrasound therapy. The very large
differences reported in complications and short-term recovery ruled
these out as primary outcomes for a future RCT, as equipoise
appeared impossible. Using quality of life measures at 6 months
was also infeasible, as the very similar results found implied an
enormous trial population requirement. There remained, however, an
answerable question about recurrence after treatment.
Patients and Operators
The patient group is less selective than for 2a studies, involv-
ing more patients in more centers; case numbers will commonly be
over 100. More surgeons are learning and undertaking the new
technique, and may disagree on inclusion criteria.
Intervention and Comparator
The intervention will now be relatively well defined, but minor
differences in technique are still common and can be explored
through preplanned subgroup analyses. A comparator intervention
may or may not exist.
Outcomes
These include safety, a precise effect estimate of short-term
clinical outcome useful for trial size calculations, identification of
possible subgroup outcome differences, assessment of surgical qual-
ity and learning curves, qualitative evaluation of trial feasibility, and
definition of core outcomes measures for the future trial. Validated
core outcome measures should be used if available (Appendix 1—
link 7, http://links.lww.com/SLA/B416).
Appropriate Study Designs
IDEAL Exploration studies are typically collaborative multi-
center prospective cohort studies and feasibility RCTs designed to
enhance investigator consensus on key issues. The goal is to resolve
the problems which most commonly prevent surgical RCTs: in doing
so, however, Exploration studies may sometimes make it clear that an
RCT is inappropriate or infeasible. In such cases Exploration studies
may facilitate ‘‘next best’’ approaches such as propensity scoring of
the observational data collected, or point to the need for a registry
approach. We therefore recommend an early preplanned consensus
meeting to evaluate short-term results and agree on whether an RCT
can be done, and where feasible, to develop its design. Statistical
analysis of learning curves may be useful.28
ß 2018 Wolters Kluwer Health, Inc. All rights reserved.
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429
Annals of Surgery  Volume XX, Number XX, Month 2018
Recommended Reporting Guidelines for Protocols
and Full Reports
Guidance to assist design and reporting of Exploration Stage
studies includes the recent CONSORT extension for pilot and
feasibility RCTs29 and the updated STROBE guideline for observa-
tional studies.30 IDEAL’s development of a reporting checklist for
this stage is underway.
Ethical Aspects
Stage 2b studies require institutional research ethics review.
Potential harms from the learning curve should be minimized by
training and mentoring. Informed consent should be based upon
information from Stage 2a. For reasons of justice and equity patient
selection should reflect the population in which the innovation can be
expected to be effective; and the data set should include outcomes of
importance to patients.16 An independent study oversight group to
monitor outcomes is recommended.
Identifying Stage Endpoints
By the end of Stage 2b, all the requirements needed to progress
to a pivotal RCT should be complete. These fall into 2 groups;
Endpoints That Demonstrate That the Technique
Can Be Widely Adopted by Surgeons
1. Agreement on the definition of the intervention (and acceptable
variants) for the purposes of an RCT;
2. Agreement on quality standards for delivery of the intervention;
and
3. Assessment of learning curves to allow decisions on admission of
clinicians into an RCT and how to evaluate their outcomes to
avoid bias in a comparison versus standard treatment.
Endpoints That Demonstrate That Progression to a
Formal Randomized Controlled Trial Is Feasible
1. Confirmation of the appropriate target patient population. Dis-
agreements on details of patient selection are a common cause of
failure to achieve consensus for an RCT;
2. Confirmation of the appropriate comparator treatment;
3. Confirmation of the appropriate primary endpoint for outcome
assessment in the RCT. The DELTA guidance on specifying the
target difference for a RCT is recommended (Table 2- link 10).
4. Evidence of consensus amongst surgeons and patients that they
are willing to accept randomization between the proposed
treatment options.
We recommend the use of nested qualitative studies to explore
the attitudes and values of participants.
Areas for Future Research
Successful Stage 2b Exploration studies require the collabo-
ration of multiple researchers and institutions. These types of studies
may therefore be especially suitable to cooperative groups such as the
UK Surgical Trainee collaborative.31 Methods for establishing that
the intervention is sufficiently defined and stable for RCT evaluation
need further work. Empirical evaluation of the impact of 2b studies
on the probability of developing a successful RCT is needed. If large
scale procedure registry systems or permanent audits exist, methods
for nesting 2b studies within the system are needed.
‘‘Assessment’’ (Stage 3)—Pivotal Study/RCT
Purpose and Description
In the ‘‘Assessment’’ Stage a pivotal comparative evaluation
occurs, usually against current standard treatment. This should take
ß 2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
IDEAL Framework and Recommendations Update
place soon after the new procedure is stable, and prior to its extensive
use, to avoid loss of equipoise among clinicians and patients.
Patients and Operators
This will typically involve a substantial number of patients and
operators at multiple centers. Clear patient and operator selection
criteria should be based on data gathered in previous stages.26,27
Intervention and Comparator
The new intervention is compared with a clinically relevant
comparator. Typically, this will be ‘‘best usual care’’ but on occa-
sions, a placebo or sham surgery control might be appropriate.32
Outcomes
Outcomes reflecting the values of patients and operators
should be identified during the Stage 2b collaborative study
and consideration given to developing a new core outcomes set
if one does not already exist.33 IDEAL 2b studies provide oppor-
tunities to identify and pilot test the primary outcome for a definitive
trial.
Appropriate Study Designs
Where feasible a multisurgeon, multicenter randomized
trial6,34 should be performed. Variants, including cluster-randomized
or expertise-based RCTs35,36 or stepped wedge designs, may be
appropriate. Where circumstances preclude randomization, accept-
able alternatives include controlled interrupted time series or obser-
vational designs using efficient post-hoc techniques (eg, propensity
scoring) to minimize known sources of bias. Nesting medical device
based trials within national population/disease based registries is
recommended to facilitate larger scale pragmatic trials12
Recommended Reporting Guidelines for Protocols
and Full Reports
Several robust guidelines for reporting surgical RCTs are
recommended including the updated 2010 CONSORT Statement,37
its extension for NPT (non-pharmacological treatments)38 and a
template for intervention description and replication (TIDieR State-
ment).39
The SPIRIT statement provides excellent guidance for writing
protocols for RCTs.40 These and other more specific reporting
guidance for surgery can be found at the EQUATOR website
(Appendix 1— link 4, http://links.lww.com/SLA/B416).
Ethical Aspects
Stage 3 trials require institutional research ethics review and
independent oversight. Relevant ethical issues concern the genera-
tion of valid data; fair inclusion and exclusion criteria; access and
equity in research participation; use of outcomes relevant to patients;
measures to minimize surgeon bias; efforts to minimize patient harm
due to the learning curve; fair treatment of the data and prompt
publication of all relevant results. Placebo or sham surgery controls
may be justifiable where they offer the best chance to resolve
uncertainty, do not involve unacceptable risks to patients and are
acceptable to key stakeholders.16 All RCTs should be registered on
an appropriate international register, for example https://clinical-
trials.gov/.
Identifying Endpoints of Stage
The 2 main endpoints of Stage 3 are 1) valid evidence on the
intervention’s relative effectiveness; and 2) identification of aspects
which require long-term monitoring (typically late and rare out-
comes).
www.annalsofsurgery.com | 7
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429
Hirst et al
Areas for Future Work and Research
Many current initiatives will contribute to improvements in
surgical trial design, conduct and reporting. These include Trial
Forge (Appendix 1—link 11, http://links.lww.com/SLA/B416) to
improve trial efficiency,41 and PRECIS-2, a tool to improve trial
design.42 Research to establish better methods for early phase
IDEAL studies is also being undertaken at the Bristol Centre for
Surgical Research in the UK (Appendix 1— link 12, http://links.
lww.com/SLA/B416).
‘‘Long-term Study’’ (Stage 4)—Identifying Rare and
Later Outcomes
Purpose and Description
The IDEAL Framework proposes registries for data collection
in Stage 4 (Long-term study). Their strength lies in recognizing late
or uncommon safety outcomes and identifying changes in the use of
procedures—so called ‘‘indication creep,’’ and trends in outcomes
which may reveal variations in the quality of surgery. Registries
allow evaluation of ‘‘real world’’ outcomes and, very importantly,
can allow ongoing feedback to clinicians and manufacturer.43 This
update of IDEAL introduces the recommendation for the use of
registries at a much earlier stage in the framework.
Registries for Device Surveillance and Life Cycle
Assessments
Registries or ‘‘registry like’’ systems can detect long-term or
rare safety problems with devices such as the failures of metal-on-
metal hip implants,44,45 problems with vascular closure devices,46
and surgical meshes.47,48 Comprehensive coverage requires commit-
ment from users and sustainable partnership with stakeholders.
Registries should have independent supervision, use standard uni-
versally applicable definitions of outcomes and relevant confound-
ers, and cover equivalent devices from all relevant manufacturers.
However, manufacturers are often involved in registry funding, and
are naturally wary of exposure of their data to competitors. IDEAL
promotes comprehensive, high quality registry/big data development
while recognizing the conflicts and the need to address stakeholder
concerns.49
Introducing Registries at an Early Stage
Whereas IDEAL originally envisaged registries as confined to
Stage 4, registry-type data collection can usefully begin from IDEAL
Stage 1. Device manufacturers typically develop competitor products
more or less simultaneously, and evaluate them through exclusive
arrangements with a relatively small pool of clinicians. The evident
scientific and public interest in pooling of these datasets is in tension
with competition between manufacturers. Starting a registry at an
early stage can allow data harmonization and pooling of resources
and data if agreements can be reached (50 & Appendix 1— link 13,
http://links.lww.com/SLA/B416).
Developing Trials Within Registries
IDEAL Stage 3 RCTs can be ‘‘nested’’ within registries or
cohort studies (51,52 & Appendix 1—link 14, http://links.lww.com/
SLA/B416), potentially enhancing trial recruitment, but informed
consent processes require careful consideration (see below).
Patients and Operators
The patient population is dictated by the stage at which the
registry is introduced. Registries introduced at an early stage are
mostly procedure or device driven with clinicians entering all
patients treated with the innovation. Disease-driven registries,
8 | www.annalsofsurgery.com
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery  Volume XX, Number XX, Month 2018
consisting of all known patients with a specific medical condition,
are preferable scientifically but are not easy to implement and rarely
sustainable. A feasible compromise approach which facilitates
nested trials and other secondary data usage is the ‘‘all comers’’
model which collects data on available treatments in a specific
condition or area of practice.53 Advances in electronic records
systems’ capacity to code accurately for both diagnosis and treatment
will rapidly improve our ability to develop such registries in
the future.
Intervention and Comparator
Registries may be procedure/device-driven or disease-driven.
Which design is optimal depends on the objectives of the registry and
should be clinician- and patient driven. Registries can be utilized
within specific trials where appropriate to increase their efficiency,
for example the UK REBOA (Resuscitative Endovascular Balloon
Occlusion of the Aorta) RCT will be using data collected via the UK
trauma registry (Appendix 1—link 15, http://links.lww.com/SLA/
B416).
Outcomes
Registries allow for analysis of long-term outcomes that may
not be captured within the lifetime of an RCT. Early stage registries
may be particularly useful in allowing cooperating innovators to pool
learning on procedural modifications to arrive at the optimal tech-
nique more rapidly.
Appropriate Study Designs
Key design issues for registries center on the dataset and on
fostering engagement. Datasets should be as small and cheap to
collect as possible, while reliably capturing patient and device/
procedure identity, diagnosis, and the key influences on outcome.
Standardized terminology should be used for all data items. Careful
design of contributor recognition, data entry systems, and feedback
methods should maximize incentives for and minimize barriers to
full data submission.
Recommended Reporting Guidelines for Protocols
and Full Reports
Clear plans for analysis and reporting of data should be
developed at the outset, specifying time intervals, numbers of
patients registered, or prespecified safety signals which will trigger
analysis. The form, authorship, and means of distribution of reports
needs to be agreed in advance, giving all contributors appropriate
recognition, notice of reports, and access to the data.
Ethical Aspects
In Stage 4, interventions are no longer within a research ethics
framework, but part of routine clinical practice. However, as registry
data is collected this raises issues about patient consent for future use
of data including the development of pragmatic nested RCTs.
Integrated verbal consent can be an alternative to written consent
in some contexts.54 Issues of access and equity may arise in Stage 4 if
the innovation is more expensive than alternatives, or concentrated in
specialist centers. Conflicts of interest created by financial or repu-
tational rewards and/or aggressive marketing may bias practice
toward the innovation even for indications where there is little
evidence.16
Areas for Further Research
Research is needed on the specific value of different features
of registries, and on the factors which facilitate or prevent their
introduction. Further analysis of real-world application of trials
ß 2018 Wolters Kluwer Health, Inc. All rights reserved.
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429
Annals of Surgery  Volume XX, Number XX, Month 2018
within registries is required to inform practical guidance and future
IDEAL recommendations.
Significant variation exists in the outcome measures used in
international patient registries within the same clinical area, reducing
their utility by making comparison, linkage, and aggregation of data
more difficult. International collaborations are emerging to advance
these issues (Appendix 1 links 16 and 17, http://links.lww.com/SLA/
B416).
LIMITATIONS
The recommendations in this article arose from a relatively
small selected group of experts. Different perspectives may have
emerged with a wider range of participants. The use and uptake of
IDEAL is continuously evolving, and this paper presents an update at
a specific point in time. It will undoubtedly be refined further in
the future.
CONCLUSIONS
The idea of an integrated stepwise evaluation pathway for
complex interventions is generally accepted as desirable. The IDEAL
approach meets this need in a logical and widely accepted manner.
However, IDEAL requires on-going review and updates. Experience
in the planned use of the IDEAL Framework and Recommendations
is currently limited. Empirical analysis of the outcomes and impact
of using IDEAL will be an important driver for future incremental
and evidence-based modifications.
REFERENCES
1. Barkun JS, Aronson JK, Feldman LS, et al., Balliol Collaboration. Evaluation
and stages of surgical innovations. Lancet. 2009;374:1089–1096.
2. Ergina PL, Cook JA, Blazeby JM, et al., Balliol Collaboration. Challenges in
evaluating surgical innovation. Lancet. 2009;374:1097–1104.
3. McCulloch P, Altman DG, Campbell W, et al., Balliol Collaboration. No
surgical innovation without evaluation: the IDEAL recommendations. Lancet.
2009;374:1105–1112.
4. McCulloch P, Cook JA, Altman DG, et al., IDEAL Group. IDEAL Framework
for surgical innovation 1: the idea and development stages. BMJ.
2013;346:f3012.
5. Ergina PL, Barkun JS, McCulloch P, et al., IDEAL Group. IDEAL Framework
for surgical innovation 2: observational studies in the exploration and assess-
ment stages. BMJ. 2013;346:f3011.
6. Cook JA, McCulloch P, Blazeby JM, et al., IDEAL Group. IDEAL Framework
for surgical innovation 3: randomised controlled trials in the assessment stage
and evaluations in the long term study stage. BMJ. 2013;346:f2820.
7. Pennell CP, Hirst AD, Campbell WB, et al. Practical guide to the Idea,
Development and Exploration stages of the IDEAL Framework and Recom-
mendations. Br J Surg. 2016;103:607–615.
8. Gudeloglu A, Fleischman SR, Dahm P. Awareness of the IDEAL recommen-
dations among the members of the Robotic-Assisted Microsurgery and
Endoscopic Society (RAMSES) Abstracts from the IDEAL Collaboration
Conference 2016, 7 April 2016, Oxford, UK. Int J Surg. 2016;36(suppl
2):S143.
9. Blazeby JM, Blencowe NS, Titcomb DR, et al. Demonstration of the IDEAL
recommendations for evaluating and reporting surgical innovation in mini-
mally invasive oesophagectomy. Br J Surg. 2011;98:544–551.
10. Currie A, Brigic A, Blencowe NS, et al. Systematic review of surgical
innovation reporting in laparoendoscopic colonic polyp resection. Br J Surg.
2015;102:e108–116.
11. Menon M, Abaza R, Sood A, et al. Robotic kidney transplantation with
regional hypothermia: evolution of a novel procedure utilizing the IDEAL
guidelines (IDEAL Phase 0 and 1). Eur Urol. 2014;65:1001–1009.
12. Sedrakyan A, Campbell B, Merino JG, et al. IDEAL-D: a rational framework
for evaluating and regulating the use of medical devices. BMJ. 2016;353:
i2372.
13. Pennell CP, Hirst A, Sedrakyan A, et al. Adapting the IDEAL Framework and
Recommendations for medical device evaluation: a modified Delphi survey.
Int J Surg. 2016;28:141–148.
ß 2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
IDEAL Framework and Recommendations Update
14. Verkooijen HM, Kerkmeijer LGW, Fuller CD, et al. R-IDEAL: a framework
for systematic clinical evaluation of technical innovations in radiation oncol-
ogy. Front Oncol. 2017;7:59.
15. Beard D, Hamilton D, Davies L, et al. Evidence-based evaluation of practice
and innovation in physical therapy using the IDEAL-Physio Framework. Phys
Ther. 2018;98:108–121.
16. Rogers WA, Hutchison K and McNair A. Ethical issues across the IDEAL
stages of surgical innovation (manuscript under consideration).
17. Pound P, Blaug R. Transparency and public involvement in animal research.
Altern Lab Anim. 2016;44:167–173.
18. Zendejas B, Brydges R, Hamstra SJ, et al. State of the evidence on simulation-
based training for laparoscopic surgery: a systematic review. Ann Surg.
2013;257:586–593.
19. van der Worp HB, Howells DW, Sena ES, et al. Can animal models of disease
reliably inform human studies? PLoS Med. 2010;7:e1000245.
20. McAteer H, Cosh E, Freeman G, et al. Cost-effectiveness analysis at the
development phase of a potential health technology: examples based on tissue
engineering of bladder and urethra. J Tissue Eng Regen Med. 2007;1:343–349.
21. Stannard A, Eliason J, Rasmussen T. Resuscitative endovascular balloon
occlusion of the aorta (REBOA) as an adjunct for hemorrhagic shock. J
Trauma. 2011;71:1869–1872.
22. Agha RA, Fowler AJ, Saeta A, et al., SCARE Group. The SCARE statement:
consensus-based surgical case report guidelines. Int J Surg. 2016;34:180–186.
23. Diez del Val I, Loureiro C, McCulloch P. The IDEAL prospective development
study format for reporting surgical innovations. An illustrative case study of
robotic oesophagectomy. Int J Surg. 2015;19:104–111.
24. Blencowe NS, Mills N, Cook JA, et al. Standardizing and monitoring the
delivery of surgical interventions in randomized clinical trials. Br J Surg.
2016;103:1377–1384.
25. Biau DJ, Porcher R. A method for monitoring a process from an out of control
to an in control state: application to the learning curve. Stat Med.
2010;29:1900–1909.
26. Penna M, Hompes R, Arnold S, et al., TaTME Registry Collaborative.
Transanal total mesorectal excision: international registry results of the first
720 cases. Ann Surg. 2017;266:111–117.
27. Chen J, Li Y, Wang Z, et al. Committee of the Clinical Trial of HIFU versus
Surgical Treatment for Fibroids. Evaluation of HIFU Ablation for
Uterine Fibroids: an IDEAL Prospective Exploration Study. BJOG. 2018;125:
354–364.
28. Cook JA, Ramsay CR, Fayers P. Statistical evaluation of learning curve effects
in surgical trials. Clin Trials. 2004;1:421–427.
29. Eldridge SM, Chan CL, Campbell MJ, et al., PAFS consensus group. CON-
SORT 2010 statement: extension to randomised pilot and feasibility trials.
BMJ. 2016;355:i523930.
30. The PLOS Medicine Editors. Observational studies: getting clear about
transparency. PLoS Med. 2014;11:e1001711.
31. Jamjoom AA, Phan PN, Hutchinson PJ, et al. Surgical trainee research
collaboratives in the UK: an observational study of research activity and
publication productivity. BMJ Open. 2016;6:e010374.
32. Wartolowska K, Judge A, Hopewell S, et al. Use of placebo controls in the
evaluation of surgery: systematic review. BMJ. 2014;348:g3253.
33. Williamson PR, Altman DG, Bagley H, et al. The COMET Handbook: version
10. Trials. 2017;18(suppl 3):280.
34. Blencowe NS, Cook JA, Pinkney T, et al. Delivering successful randomized
controlled trials in surgery: methods to optimize collaboration and study
design. Clin Trials. 2017;14:211–218.
35. Devereaux PJ, Bhandari M, Clarke M, et al. Need for expertise based
randomised controlled trials. BMJ. 2005;330:88.
36. Cook JA, Elders A, Boachie C, et al. A systematic review of the use of an
expertise-based randomised controlled trial design. Trials. 2015;16:241.
37. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010
statement: updated guidelines for reporting parallel group randomised trials.
BMJ. 2010;340:c332.
38. Boutron I, Moher D, Altman DG, et al., CONSORT Group. Methods and
processes of the CONSORT Group: example of an extension for
trials assessing nonpharmacologic treatments. Ann Intern Med. 2008;148:
W60–W66.
39. Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions:
template for intervention description and replication (TIDieR) checklist and
guide. BMJ. 2014;348:g1687.
40. Chan A-W, Tetzlaff JM, Altman DG, et al. SPIRIT 2013 statement: defining
standard protocol items for clinical trials. Ann Intern Med. 2013;158:200–207.
www.annalsofsurgery.com | 9
CE: D.C.; ANNSURG-D-17-02429; Total nos of Pages: 10;
ANNSURG-D-17-02429
Hirst et al
41. Treweek S, Altman DG, Bower P, et al. Making randomised trials more
efficient: report of the first meeting to discuss the Trial Forge platform. Trials.
2015;16:261.
42. Loudon K, Treweek S, Sullivan F, et al. The PRECIS-2 tool: designing trials
that are fit for purpose. BMJ. 2015;350:h2147.
43. Sedrakyan A, Campbell B, Graves S, et al. Surgical registries for advancing
quality and device surveillance. Lancet. 2016;388:1358–1360.
44. Smith AJ, Dieppe P, Vernon K, et al. National Joint Registry of England and
Wales. Failure rates of stemmed metal-on-metal hip replacements: analysis of
data from the National Joint Registry of England and Wales. Lancet. 2012;379:
1199–1204.
45. Smith AJ, Dieppe P, Howard PW, et al. National Joint Registry for England
and Wales. Failure rates of metal-on-metal hip resurfacings: analysis of data
from the National Joint Registry for England and Wales. Lancet. 2012;380:
1759–1766.
46. Resnic FS, Majithia A, Marinac-Dabic D, et al. Registry-based prospective,
active surveillance of medical-device safety. N Engl J Med. 2017;376:
526–535.
47. Chughtai B, Mao J, Buck J, et al. Use and risks of surgical mesh for pelvic
organ prolapse surgery in women in New York state: population based cohort
study. BMJ. 2015;350:h2685.
10 | www.annalsofsurgery.com
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Annals of Surgery  Volume XX, Number XX, Month 2018
48. Chughtai B, Barber MD, Mao J, et al. Association between the amount of
vaginal mesh used with mesh erosions and repeated surgery after repairing
pelvic organ prolapse and stress urinary incontinence. JAMA Surg. 2017;152:
257–263.
49. Krucoff MW, Sedrakyan A, Normand SL. Bridging unmet medical device
ecosystem needs with strategically coordinated registries networks. JAMA.
2015;314:1691–1692.
50. Campbell WB, Poole RL, Ray AF, et al. Collecting data on treatments for
varicose veins: getting the focus right. Phlebology. 2016;32:224–226.
51. van der Velden JM, Verkooijen HM, Young-Afat DA, et al. The cohort
multiple randomized controlled trial design: a valid and efficient alternative
to pragmatic trials? Int J Epidemiol. 2017;46:96–102.
52. Li G, Sajobi TT, Menon BK, et al. 2016 Symposium on Registry-Based
Randomized Controlled Trials in Calgary. Registry-based randomized con-
trolled trials- what are the advantages, challenges, and areas for future
research? J Clin Epidemiol. 2016;80:16–24.
53. Institute of Medicine. 2001. Crossing the Quality Chasm: A New Health
System for the 21st Century. Washington, DC: The National Academies Press.
Available at: https://doi.org/10.17226/10027. Accessed April 17, 2018.
54. Kim SY, Miller FG. Informed consent for pragmatic trials—the integrated
consent model. N Engl J Med. 2014;370:769–772.
ß 2018 Wolters Kluwer Health, Inc. All rights reserved.