Design Research Society

DRS Digital Library

DRS Biennial Conference Series DRS2022: Bilbao

Jun 25th, 9:00 AM

How might we evaluate co-design? A literature review on existing

practices
Zixuan Wang
College of Design and Innovation, Tongji University, Shanghai, China

Tingting Jiang
College of Design and Innovation, Tongji University, Shanghai, China

Jiawen Huang
College of Design and Innovation, Tongji University, Shanghai, China

Yixuan Tai
College of Design and Innovation, Tongji University, Shanghai, China

Paola Maria Trapani

College of Design and Innovation, Tongji University, Shanghai, China

Follow this and additional works at: https://dl.designresearchsociety.org/drs-conference-papers

Part of the Art and Design Commons

Citation
Wang, Z., Jiang, T., Huang, J., Tai, Y., and Trapani, P.M. (2022) How might we evaluate co-design? A
literature review on existing practices, in Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P.
(eds.), DRS2022: Bilbao, 25 June - 3 July, Bilbao, Spain. https://doi.org/10.21606/drs.2022.774

This Research Paper is brought to you for free and open access by the DRS Conference Proceedings at DRS Digital
Library. It has been accepted for inclusion in DRS Biennial Conference Series by an authorized administrator of DRS
Digital Library. For more information, please contact dl@designresearchsociety.org.
How might we evaluate co-design? A literature re-
view on existing practices
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani*
College of Design and Innovation, Tongji University, China
*corresponding e-mail: paola@tongji.edu.cn
doi.org/10.21606/drs.2022.774

Abstract: In the last 30 years, co-design has seen exponential adoption in response to
complex socio-technical transitions. However, the appraisal around co-design still lacks
development, making its quality, effect, and value ambiguous. This study aims to en-
courage more research and practice in co-design evaluation by summarizing existing
attempts. We first conduct an integrative literature review in Scopus by analyzing and
comparing existing methods used for assessing co-design. Then we iteratively code the
reviewed articles in ATLAS.ti following grounded theory. Through critical reflection and
synthesis of the codes, we form higher-level themes describing different aspects of
assessment. Based on these, we propose a co-design evaluation framework with five
steps guiding the practice. An overview of the methods, values, challenges, and sug-
gestions for co-design assessment has been presented, and an assessment framework
is proposed to generate more interest and practice in this area.

Keywords: co-design, evaluation, tools and methods, literature review

1. Introduction
Our society is confronted with complex socio-technical challenges that are often cross-disci-
plinary. Transdisciplinary collaborations like co-design have been considered a promising ap-
proach to solving open, complex, dynamic, and networked problems (Dorst, 2015) through
the participation of people from various domains (Manzini, 2015; Meroni et al., 2018).
While many claims are made about the value of co-design, it has rarely been rigorously eval-
uated to prove that it achieves what it promises (Blomkamp, 2018). Typically, co-design
tends to be more time-consuming than traditional design by professionals, lasting from a
few hours to months or years, yet does not guarantee successful results. There is a growing
need to demonstrate better the return on investment of co-design approaches (Forss et al.,
2011; Manohar et al., 2016). Previous reviews on co-design have mainly focused on tools
and methods for conducting co-design (Brandt et al., 2013; Peters et al., 2020) and co-design
applications in specific settings like healthcare (Eyles et al., 2016; Greenhalgh et al., 2019;
Noorbergen et al., 2021) or communities (David et al., 2013). More relevant to us, Steen et

This work is licensed under a Creative Commons Attribution-NonCommercial

4.0 International Licence.
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

al. (2011) and Blomkamp (2018) have discussed the promise and benefits of co-design in
public policy and service design projects, respectively. However, to the best of our
knowledge, no literature review has been conducted on methods for evaluating co-design in
general.
This study aims to fill the gap by providing an overview of the co-design appraisal methods
based on a literature review and proposing an evaluation framework to guide practitioners.
First, we summarized the usage of each assessment method in existing publications based
on the aspects it evaluated (i.e., approach, process, and outcome). Second, we critically read
and coded the authors’ reflections on co-design evaluation. The codes are then restructured
to form themes about the values, challenges, and suggestions around assessment. Third, we
proposed an evaluation framework and outlined the considerations needed at each stage
when preparing and conducting a co-design appraisal.

2. Method

Figure 1. Overview of the literature review process and methods.

2.1 Data collection method

The authors chose Scopus1 as the database for literature review since it is the largest ab-
stract and citation database of peer-reviewed literature2. To understand existing assessment
efforts, the authors used “evaluat* AND co-design” as the search term after several trials.
The search range is within the article title, abstract, and keywords. However, the initial 1852
results contained a lot of unrelated publications, such as the realization and evaluation of a
computer system that supports remote co-design (Li et al., 2004) and algorithms for hard-
ware or software co-design (Wang & Xu, 2019; Hou et al., 2020) from computer science, en-
gineering, etc. Therefore, the authors further filtered the results by excluding literature in
adjacent disciplines, leaving only Arts and Humanities; Business, Management, and Account-
ing; Decision Science; Economics, Econometrics, and Finance; Environmental Science; Psy-
chology; Multidisciplinary; Social Sciences. The authors then individually screened the final
95 search results by reading the titles and abstracts to determine whether it was relevant

1 www.scopus.com
2 https://service.elsevier.com/app/answers/detail/a_id/15100/supporthub/scopus/related/1/

2
 How might we evaluate co-design?

and wrote down the reasons for the judgment. In case of uncertainty, the full text was
checked regarding the eligibility. We distributed the articles among the authors to ensure
that at least two authors reviewed each paper, and the disputes were discussed with a third
author until a consensus was reached.

2.2 Key term definition & exclusion criteria

The screening process is not straightforward due to the ambiguous usage of the term "co-
design". Despite the 50-year history of collective creativity in design and the wide applica-
tion of co-design, its definition is continuously evolving, and there is no consensus. Thus, it is
necessary to clarify this paper's definition and scope of co-design.
A simple way to understand the term “co-design” is by breaking it down into its constitutive
parts. The “co” is typically considered an abbreviation for ‘cooperative’ or ‘collaborative’,
which Sanders & Stappers (2008) described as the process of “designers” and “people not
trained in design” working together. Thus, co-design highlights the designer’s role as a
toolmaker or facilitator, which means it is an intentionally designed collaborative process ra-
ther than a spontaneous one. The term “design” is defined by Simon (1988) as “to devise
courses of action aimed at changing existing situations into preferred ones”. Considering this
definition, the outcome of co-design could be a product, service, program, or mindset (e.g.,
a change in behaviour). In addition, the application of co-design is now covering the entire
span of the creative process, from opportunity finding to idea generation to prototyping
(Meroni et al., 2018). For this research, we sum up three types of co-design according to its
aim: generating ideas, empowering participants, and validating concepts.

Moreover, this review focuses specifically on evaluating "a co-design session". In the discus-
sion about experience, Dewey distinguished between the general experience that happens
continuously and "an experience", which has a clear moment of inception, then develops un-
til a moment of fulfilment (Dewey, 2005). A similar distinction occurs between co-design in
the long-term and specific co-design sessions. Although we acknowledge that meaningful co-
design is rarely a one-off event, the ambiguous interactions, and complex social influences
between co-design sessions during a long-term co-design make its quality and effect almost
impossible to evaluate. Therefore, long-term co-design is only analyzed in this review as a
series of individual co-design sessions.
Based on these reasons, the final exclusion criteria are as below:
• Articles describing a co-design that is out of our scope (e.g., long-term or spon-
taneous cooperation without an intentionally designed collaborative session)
are excluded.
• Articles without an evaluation on co-design (e.g., assessing other things like
products or services through co-design) are excluded.

3
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

• Articles evaluating a co-design tool or output independently without consider-

ing its relation to co-design are excluded because they are more related to
product evaluation than the unique characteristic of co-design.
In addition, three articles without access and one without English version are excluded,
which results in the final 31 articles3 meeting the criteria.

2.3 Data analysis method

Considering the limited quantity of the included articles, we adopted an integrative litera-
ture review approach that combines the systematic and narrative review methodologies
(Onwuegbuzie & Frels, 2016). There are two main focuses during the analysis: the evaluation
methods used and the researchers' reflections. We divided the articles among authors and
conducted three reading and data checking rounds to ensure reliability.
The method-related data were first identified and coded in ATLAS.ti4 and then extracted into
a standardized spreadsheet containing: the articles’ publishing year, the co-design applied
area (according to the articles’ keywords and ASJC scheme5), the aim of co-design (generat-
ing ideas, empowering participants, validating concepts), the mode of communication
(online, offline), the aspects evaluated (approach, process, outcome), and the evaluation
methods used. We first wrote down the information from each article and then abstracted
and merged similar content into a unified tag. For example, questionnaires and scales are all
labeled as surveys in the final table. A screenshot of the method collecting table is shown in
Figure 2.

Figure 2. A screenshot of the method collecting table.

The reflection-related content was identified through iteratively coding in ATLAS.ti following
grounded theory as it enables the researchers to come up with a theory-based or concept-
centric yet accurate review (Wolfswinkel et al., 2013; Charmaz & Belgrave, 2015). The au-
thors first read and marked anything in each paper that seemed relevant to the review’s

3 One article (Farr, 2018) describes two different co-designs, which were later split into two cases for data analysis.
4 https://web.atlasti.com/
5 https://service.elsevier.com/app/answers/detail/a_id/15181/c/10547/supporthub/scopus/

4
 How might we evaluate co-design?

scope and research question individually. Then, the authors re-read these “excerpts” and
open-coded any “concepts” (categories or insights represented by a set of “excerpts”) that
emerged during the process. After that, axial coding and selective coding are conducted
through continuous comparative analysis to identify the interrelations between categories
and sub-categories before merging and refining them into the main “themes”. The analysis
steps (open, axial, and selective coding) were performed in an intertwined fashion, going
back and forth between papers, excerpts, concepts, categories, and sub-categories until the-
oretical saturation had occurred. The authors continuously discussed and negotiated any un-
certain codes during the process until reaching an agreement.
The method-related data analysis results are presented in Section 3 and the reflection-re-
lated content analysis results are described in Section 4. The article screening list, method
collecting table, reflection-related codes are available in the supplementary material link6.

3. Result: Co-design evaluation methods

3.1 Overview

Figure 3. Data visualization of reviewed articles.

As shown in Figure 3, in our review, articles mentioning co-design evaluation were published
from 2011 to 2021, covering various application areas and aims. Among them, the most pre-
dominant fields are education (N=12), health & nursing (N=6), and management & policy-
making (N=4). It is noteworthy that most of the articles were published in the last two years.
All the included articles investigated offline co-design evaluation, while none of the evalua-

6 https://github.com/MelodyOrz/DRS22_EvaluatingCo-design_SupplementaryMaterials

5
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

tion methods was conducted for online co-design. Presumably, co-design evaluation re-
search is an emerging area in academia, and evaluation studies specifically on online co-de-
sign are relatively lacking.
We grouped the results into three categories based on the evaluated aspects: approach,
process, and outcome. The evaluation of the co-design approach focuses on the materials
designed, produced, and executed by the design team before the co-design session, such as
tools, activities, and methods. The evaluation of the co-design process focuses on events
during the co-design session, such as the experience, the engagement, and the collaboration
of participants. The assessment of the co-design outcome focuses on the results of the co-
design sessions, such as the direct output, its short-term implementation, and the long-term
impact. In our analysis, articles concerning the evaluation of co-design outcomes dominated,
accounting for 24 among the 31 papers included, while only 15 and 11 articles assessed the
process and the approach, respectively. The methods used for evaluation are diverse. Sur-
veys, interviews, and observation field notes are the top three data collection methods.

3.2 Methods evaluating the approach of co-design

Fourteen articles describe the evaluation methods of the co-design approach, covering the
assessment of co-design methods, tools, and activities designated before the workshop.
Most of them (N=10) are studies assessing a generic co-design methodology or tool. Only
four papers were targeted at the activities and tools used in a particular co-design workshop.
When evaluating the approach, the most frequently used method is the designer’s reflection
(4 out of 9). However, most of these assessments are based on the opinions of the research-
ers who designed and facilitated the co-design sessions, so the evaluation is mainly subjec-
tive and could also be biased. Only Koens et al. (2020) commissioned an observation to re-
searchers who were not involved in the facilitation, and Lee et al. (2011) evaluated the ap-
proach by senior designers based on junior designers’ weekly diaries when learning and con-
ducting co-design. Some authors developed evaluation criteria according to existing theories
in the related literature (Dietrich et al., 2017) to improve the validity, e.g., the framework of
purposes and activities supported by co-design tools (Sanders et al., 2010). Some compare
the data against quotes from field notes (Ey & Spears, 2020; Löfström et al., 2020), record-
ings (Löfström et al., 2020), interviews, or post-session surveys (Halar et al., 2020) to give
more evidence to the claimed findings. Ey and Spears (2020) even read back key findings to
participants for memory checking to ensure the trustworthiness and legitimacy of the re-
sults.
Surveys also proved to be a convenient and easy-to-use method when assessing the perfor-
mance and preference of the co-design activities and tools. The most common format is
point scales with open responses to obtain a quantitative comparison while leaving space for
qualitative comments (van Beusekom et al., 2021; Woltersdorf et al., 2019). Surveys some-
times accompany interviews to gain more intuitive aspects about participants' opinions on
the approach (Halar et al., 2020; Quinn, 2015). Typical assessment items include a “5-point

6
 How might we evaluate co-design?

scale from excellent to very poor, about each co-design activity” (van Beusekom et al., 2021)
and “activities that worked particularly well or did not work well” (Ey & Spears, 2020; Halar
et al., 2020). Specifically, Koens et al. (2020) developed a statistical measure to analyze the
survey responses using the Chi-square test to indicate which co-design activity contributes
to the experience and outcome. It is worth noting that most of these methods are con-
ducted with co-design participants. Only Whicher & Crick (2019) interviewed the stakehold-
ers who organized and hosted the co-design workshops to investigate the main determi-
nants of effective co-design.
Besides, participants’ reflection appears also in two cases. Gottwald et al. (2021) designed
moderation cards that allow the participants to evaluate the workshop by writing down
what they were taking away positively from the activity (“the suitcase”), the remaining ques-
tions (“the question mark”), and the unnecessary or negatively perceived elements of the
workshop (“the garbage bin”). McEwan (2017) asked participants to share their reflections
on the day’s activities at the end of each session, emphasizing the beneficial aspects, the ele-
ments to improve, and whether the workshop framework was optimal.

3.3 Methods evaluating the process of co-design

Sixteen articles describe the methods used to evaluate the co-design process, covering the
assessment of the experience, engagement, and collaboration of participants during the
workshop. Since the process of co-design is often hard to measure quantitatively, ways to
assess it are often qualitative, such as surveys with open-ended questions (N=7), interviews
(N=6), and observation field notes (N=5). Although qualitative methods proved helpful in
evaluating how participants worked together, challenges remained in gaining insights into
the group dynamics or changes in perspectives within individuals (Koens et al., 2020).
When evaluating the process, the most frequently used methods are participants’ subjective
expressions through surveys, interviews, or focus groups. Typical assessment elements of
concern are participants' “views on the co-design experience” (Koens et al., 2020; Mumaw &
Raymond, 2021), “engagement with the program” (Ey & Spears, 2020), "perceptions of the
interaction" (Senabre et al., 2018). Semi-structured interviews prove to be effective in pro-
ducing perspectives of people's understandings, motivations, and behaviours over time (Bry-
man, 2016). Nevertheless, there are some attempts to use quantitative scales measuring the
qualitative elements (Roemer et al., 2020; van Beusekom et al., 2021; Woltersdorf et al.,
2019), such as “participants' satisfaction levels with the workshops”.
Another constructive way of evaluating the co-design process is by analyzing the observation
field notes of an outsider, usually a researcher or designer. Through observing, more atten-
tion goes to the social elements like group collaboration and interaction. Since helpful infor-
mation could be easily missed if there are situations like language barriers (Koens et al.,
2020) or complex group dynamics (Hurley et al., 2018), process recording (including video,
audio, and photo records) becomes an integral approach to support the observation. Besides

7
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

the direct examination of the co-design process, Janigo & Wu (2015) adopted a “collabora-
tive process” to analyze observation and recording data of the focus group around partici-
pants' experiences and claimed that the combination of these methods helps capture the es-
sence of co-design experience.

3.4 Methods evaluating the outcome of co-design

Twenty-four articles describe the methods used to evaluate the co-design outcomes, cover-
ing the assessment of the design outputs, implementation, and impact after the co-design
session. From the papers we reviewed, most of the co-design outcomes are non-physical
items such as services, frameworks, strategies, and programs. They are multi-dimensional
and with mostly long-term impact, which leads to the difficulty of assessment.
When evaluating the outcomes, the most distinctive method used is comparison experi-
ments that compare the co-designed outputs' quality and effectiveness with existing or ex-
pert-driven ones. Hurley et al. (2018) and Dietrich et al. (2016) adopted a similar comparison
approach and found that the co-designed alcohol education programs differed substantially
from the original expert-driven ones. Rutter, Stones & Macduff (2020) analyzed and com-
pared factors in the professional posters targeted at children from Google Image and the
posters co-designed by children. Ey and Spears (2020) assessed the co-designed educational
programs by comparing the learning effects of two groups of children. Among the teachers
of these two groups of children, one was involved in the co-design and implementation of
the program, while the other was not, i.e., they kept using the previous educational pro-
gram.
Surveys and interviews are also frequently used to indicate the impact of co-design by com-
paring the participants’ answers to the same questions before and after the co-design. Ward
et al. (2019) found a significant change in participants’ reporting behavioural after the edu-
cational co-design. Others compare the answers of participants in the “self-assessment of
innovative thinking” (Laptev & Shaytan, 2021), “awareness and satisfaction with the pro-
gram” (Bogomolova et al., 2021), and “learning and understanding” about the co-design
topic (Ey & Spears, 2020; Koens et al. 2020). Data collected from observation field notes and
designers’ reflections could also be valuable sources. The field notes can record non-verbal
behaviours to assist the evaluation (Scott et al., 2020). The researcher’s notes on essential
outputs or discussions could become qualitative data for outcome evaluation (Bremmer &
Hoek, 2019; Rodriguez et al., 2019; Ward et al., 2019). As a critical role in organizing co-de-
sign sessions, designers could synthesize various perspectives to reflect on the co-design
outcomes (Halar et al., 2020; Mäkelä et al., 2018; Quinn, 2015).
Many articles highlighted the importance of setting up long-term success indicators and reg-
ular review systems. For example, Koens et al. (2020) interviewed the stakeholders six
months after the co-design to understand the longstanding effects. Diaries and online sur-

8
 How might we evaluate co-design?

veys can be provided to the participants for continuous assessment. Data from outer plat-
forms like social media and service data could also be useful to demonstrate the impact (Bo-
gomolova et al., 2021).

4. Result: Reflections on co-design evaluation

The themes generated from iteratively reading, coding, and analyzing the literature’s reflec-
tion part about co-design assessment are further grouped into three meta themes: the per-
ceived value, existing challenges, and future suggestions on co-design evaluation.

4.1 Perceived value of evaluation

Manage Expectations: Although co-design is well known in the design community, it may
feel like "buzzword bingo" to clients unfamiliar with design. Whicher and Crick (2019) men-
tioned the difficulty for civil servants to grasp the concept of co-design because it is intangi-
ble. Clients who choose to adopt co-design may not fully understand what the process and
the output could be (Whicher & Crick, 2019). Some may even form unrealistic expectations
for co-design “to provide them with the ‘best’ solutions, which could then simply be imple-
mented afterwards” (Koens et al., 2020). These expectations, if not well-managed, will likely
lead to the disillusion of the clients on co-design. Some articles argue that through co-defin-
ing the evaluation criteria for co-design, we can help clients set the right expectations.
Whicher and Crick (2019) advised allocating resources to identify ex-post evaluation metrics,
ongoing benchmarking and monitoring methods during the co-design preparation phase.
Amplify Outcomes: Several articles we reviewed mentioned dissatisfaction with the co-de-
signed outputs. Lee et al. (2011) described the disappointment of student designers due to
the number and quality of the ideas generated from participants. van Beusekom et al.
(2021), Pau and Hall (2021), and Koens et al. (2020) were all concerned about personal-spe-
cific and off-topic ideations during co-design. Specifically, Pau and Hall (2021) critiqued the
simplicity and opaqueness of the co-designed output that makes it challenging to interpret
without the context and participants.
Evaluation methods like designer's reflection can help interpret the outputs. Lee et al. (2011)
argued that the results of co-design are not only meaningful when scientifically analyzed.
They believe that empathic understanding with users and continuous reflection is crucial.
Researchers could learn a lot from translating a project into a case study, even if it was not
classified as successful (Whicher & Crick, 2019). The evaluation also claimed to have the po-
tential to amplify the co-design outcome. Bogomolova et al. (2021) highlighted that the par-
ticipants' reflection during co-design helps ensure their understanding of the aim of co-de-
sign, and constant reflective practice is essential to institute changes (Farr, 2018). Sharing
the co-design outputs with global stakeholders, enlarged the validity of a single site co-de-
sign output (van Beusekom et al., 2021).
Prompt Implementation: Several articles mentioned the frustration related to the imple-
mentation phase of co-design (Ey & Spears, 2020; Whicher & Crick, 2019), even though the

9
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

participants enjoyed the co-design process (Koens et al., 2020). One reason behind this
might be that participants could develop ideas with no bounds in reality if evaluation criteria
were not conveyed (Whicher & Crick, 2019). Another reason is the insufficient resources al-
located around operation and the disapproval of the pivotal collaborators (Whicher & Crick,
2019). Whicher and Crick (2019) noticed that a clients’ active engagement and participation
in co-design is the most significant precondition for the execution of the concepts. Halar et
al. (2020) suggested inviting vital stakeholders to the validating workshop to help the opera-
tion team efficiently and effectively distribute resources. Through co-prototyping, co-testing,
and co-iteration with stakeholders, the co-designed concept is more likely to be carried out
and upscaled (Whicher & Crick, 2019).
Motivate Participation: One common challenge many articles discussed is the low motiva-
tion of participating in co-design. One on hand, there are limitations of time and resources
(Ey & Spears, 2020). On the other hand, it might stem from the fact that some people do not
believe their voices will be taken into consideration (Mäkelä et al., 2018; van Beusekom et
al., 2021), and some co-design topics are too broad to be connected to their daily life (Löf-
ström et al., 2020). Existing co-design assessments could help researchers convey to possible
participants their potential impact and gains from the workshop (Whicher & Crick, 2019). It
is essential for vulnerable users who usually don’t have the motivation to join co-design (Die-
trich et al., 2017).
After the participation, regularly following up with the participants is also significant.
Whicher and Crick (2019) mentioned the disillusion of participants when their recommenda-
tions were not followed through. The critical issue here is not to ensure all the ideas of par-
ticipants will be carried out, but rather to communicate to the participants how their views
were taken into account and why some of their ideas were not implemented (Löfström et
al., 2020). Justifying design choices based on clear criteria and ensuring a transparent deci-
sion process (Koens et al., 2020) is likely to decrease the experience of disappointment
(Mäkelä et al., 2018). Besides the participants, Löfström et al. (2020) and Whicher and Crick
(2019) both suggested the continuous promotion of the evaluation results to the general
public via channels like social media. This can help secure additional support and interest
and build legitimacy and awareness of co-design (Whicher & Crick, 2019).

4.2 Existing challenges of co-design evaluation

Lack of Enough Data: Many articles are concerned about lacking sufficient data support for
the co-design appraisal. The small sample size of the evaluation data could reduce the relia-
bility of the evaluation and limit the possibilities for statistical testing (Koens et al., 2020;
Ward et al., 2019). On the one hand, co-design is often composed of a small number of par-
ticipants due to the time cost, the intensive facilitation and guidance needed (van Beusekom
et al., 2021), or the difficulties of enrolling certain types of participants, such as "senior clini-
cians" (Ward et al., 2019). On the other hand, the low willingness to spare extra time for the
evaluation leads to an even lower response rate for the assessment. For example, Ward et

10
 How might we evaluate co-design?

al. (2019) spoke of the limitation of more inadequate post-intervention responses due to
some participants dropping out. The limited attention and energy of participants when filling
the assessment after a long, intense session could further diminish the validity of the results.
Koens et al. (2020) argued that the in-depth contributions or comprehensive collaborations
during co-design workshops could make participants too exhausted to complete the evalua-
tion thoughtfully.
Hard to Evaluate Impact: Since the outcome of co-design needs time to cultivate, long-term
evaluation is essential to many projects. However, how to evaluate the long-term impact of
co-design remains challenging. Many researchers comment on the difficulty of measuring
the longer-term change due to the limited project time-span of the (Mäkelä et al., 2018; Ro-
driguez et al., 2019; Roemer et al., 2020; Scott et al., 2020). The evaluation complexity in-
creases as the lead time for co-design implementation extends, such as projects in policy
contexts that could last over a year (Whicher & Crick, 2019). Dietrich et al. (2017) and Robin-
son, Moss & Jones (2021) emphasized the necessity to follow up the evaluation several
months after co-design to ensure that changes in knowledge and practice are sustained.
Potential Bias: Researchers reflected on three kinds of potential bias in their studies: partici-
pant selection bias, position bias of researchers, and evaluation approach bias. First, co-de-
sign teams need to be aware of the potential bias generated from the sample of partici-
pants. Researchers reported the influence of non-inclusive participant selection on the out-
come of co-design, such as sample homogeneity (Janigo & Wu, 2015; van Beusekom et al.,
2021) and the exclusion of vulnerable groups (Dietrich et al., 2017). A convenience sample
may cause a friendliness bias in participants as the group will tend to avoid negative evalua-
tions (Laptev & Shaytan, 2021; Koens et al., 2020; Woltersdorf et al., 2019). Roemer et al.
(2020) suggested future studies to avoid this shortcoming by randomly selecting and incen-
tivizing participants. Secondly, bias could also come from the researchers' position. As Ey
and Spears (2020) stated in their study, researchers need to be aware and willing to relin-
quish their control and ownership to enable participants to take the lead in their areas of ex-
pertise. Thirdly, the approaches of the co-design assessment could also cause bias. Laptev
and Shaytan (2021) described the habituation bias in participants caused by the repetitive-
ness and the order of the questions. Rutter et al. (2020) found that the selected messages of
researchers in the first activity influenced the creativity of children in the following exercise.
Some of these biases can be reduced by choosing a proper time and environment for the as-
sessment, rephrasing and reordering the questions. However, the subjective nature of self-
assessment methods remains unavoidable (Laptev and Shaytan, 2021).

4.3 Suggestions for future method development

Need for Mixed Methods: Further research on methods could combine qualitative and
quantitative assessment data. On the one hand, the subjectivity of many qualitative ap-
proaches limits the reliability of the result. On the other hand, many co-design workshops
are either tricky to test with quantitative tools or lack a sufficient sample size for a validated

11
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

result, as described in Section 4.2. Koens et al. (2020) recommended further exploring quali-
tative methodologies and highlighted the need for quantitative methods that demand fewer
participants (e.g., q-sort studies). Such approaches could keep advantages from the two per-
spectives and complement the limitations of each other (Koens et al., 2020; Whicher & Crick,
2019).
Methods That are More Creative: Future studies should develop methods that are more
creative and can capture more rich data. Koens et al. (2020) argued for tools that foster a
natural and uninterrupted flow of collaboration and provide good appraisal and feedback in-
formation. The research of Bremmer and Hoek (2019) showed that, in the co-debriefing
phase, students gave more positive feedback using a more flexible video reflection format
than a written one. In addition, as co-design workshops usually have a group of people work-
ing on the activity simultaneously, the production phases have high bandwidth — partici-
pants express a lot of information at once. Pau and Hall (2021) suggested exploring methods
that could better record the entire co-design bandwidth, such as auto-documentation.
Systematic Methodology: As described in the previous section 4.2, the challenges of evalu-
ating co-design have shown the need for a systematic methodology guiding practitioners on
when and how to assess and prevent potential bias. Whicher & Crick (2019) mentioned the
condition where co-design effects cannot be evaluated because baseline data have not been
collected and suggested building an assessment plan from the outset. Bremmer & Hoek
(2019) also reported that the evaluation should happen earlier to allow the feedback appli-
cation. Robinson, Moss & Jones (2021) suggested setting regular follow up mechanisms for
the co-design impact several months after the implementation. In summary, it appears from
this review that the urgency of a more systematic evaluation methodology constitutes the
opportunity for this paper’s contribution.

5. Proposed co-design evaluation framework

Figure 4. The evaluation framework.

12
 How might we evaluate co-design?

We propose a five-step evaluation framework based on the literature review and analysis.
The scheme could offer guidance to other researchers and help them construct appropriate
assessments for future projects. The abstraction level makes the framework suitable for all
co-design scenarios. However, we must acknowledge that different co-designs will involve
various activities, settings, and participants. Practitioners should use and adjust according to
the specific contexts.
Step 1, consider evaluation from the beginning of co-design. This step aims to develop con-
sensus within the design team on the value of co-design assessment, such as managing ex-
pectations, amplifying outcomes, promoting implementation, and motivating future partici-
pation (as explained in Section 4.1). Hopefully, this will help the design team examine their
evaluation needs and incorporate the assessment from the outset of co-design.
Step 2, align expectations by setting evaluation criteria. We suggest the design team collab-
orate with key stakeholders to define the co-design evaluation criteria. Negotiating what
success would look like and the key performance indicators within the operation group can
help stakeholders unfamiliar with the co-design methodology hold realistic expectations for
the results. Communicating the criteria to the co-design participants could also avoid off-
topic ideation and generate more goal-oriented concepts.
Step 3 & 4, choose the appropriate evaluation methods from the Method Toolbox and crit-
ically interpret the results. The detailed examination of existing methods assessing co-de-
sign is described in Section 3. We suggest clarifying the aim and other evaluation aspects
(e.g., approach, process, and outcome) before defining the strategy. During the assessment
design, execution, and interpretation, it is necessary to keep in mind the challenges of valid-
ity, long-term evaluation, and potential bias, as detailed in Section 4.2. It is also recom-
mended to check the cases and references of each method when applying.
Step 5, communicate the evaluation results. To gain more feedback and increase the valid-
ity, the appraisal of the co-design results can be shared with other remote partners. Co-de-
sign participants and the wider public inclusion could motivate future contributions. The as-
sessment can also serve as a tool to communicate with the clients or sponsors to gain addi-
tional resources and support. Lastly, we encourage co-design practitioners to continuously
evaluate and iterate the co-design and assessment methodology for a long-term project to
achieve a lasting impact.

6. Conclusion
Most designers, as facilitators, lack education and guidance in adequately evaluating and
communicating the value of co-design. Despite the widespread use of co-design in many
fields, its effectiveness and impact are often not evaluated or disseminated. The lack of as-
sessment weakens the efforts and the outputs persuasiveness. This paper reviews existing

13
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

co-design evaluation methods in Scopus and summarizes the values, challenges, and sugges-
tions of co-design assessments. Finally, this study presents a step-by-step evaluation frame-
work to help practitioners better appraise their future projects.
The limitations and future work of our research are as below:
• The accuracy of the research results may suffer from lacking first-hand data. As
the authors obtained data through their understanding and analysis of the lit-
erature and could not validate them with the actual co-design scenarios, we
welcome the authors of the reviewed articles to check and correct the relevant
contents.
• The range of selected articles suffers from a two-fold restriction. On the one
hand, the article's retrieval is limited to Scopus. It is possible to expand the lit-
erature data, for example, by screening the backward and forward citations of
the included articles (Wolfswinkel et al., 2013). On the other hand, the search
term is limited to “co-design” and does not include variants like “participatory
design”. We suggest future work on these two perspectives to verify the relia-
bility of our research results.
• The research is focused only on co-design evaluation practices in academia. Fu-
ture studies can investigate practices in the industry to enrich the proposed
framework with a further contribution.
• The evaluation framework is proposed through theoretical analysis. Future re-
search is recommended to apply and test it in real co-design projects.
Through this work, we contribute to “evaluating co-design” and fill the gap in the systematic
understanding of co-design assessment methods, which is crucial for co-design develop-
ment. We envision inspiring more exploration of co-design evaluation in the future through
a detailed review of existing evaluation practices and a practical framework that co-design
practitioners can easily apply.

Acknowledgements: We especially thank our friend Shengchen Zhang for her support
during the process and for proofreading the final article. We thank Xinyang Shan, who
joined and contributed initially and left due to personal matters.

7. References
Blomkamp, E. (2018). The Promise of Co-Design for Public Policy. Australian Journal of Public Admin-
istration, 77(4), 729–743. https://doi.org/10.1111/1467-8500.12310
Bogomolova, S., Carins, J., Dietrich, T., Bogomolov, T., & Dollman, J. (2021). Encouraging healthier
choices in supermarkets: A co-design approach. European Journal of Marketing, 55(9), 2439–
2463. https://doi.org/10.1108/EJM-02-2020-0143
Brandt, E., Binder, T., & Sanders, E. B.-N. (2013). Tools and techniques: Ways to engage telling, ma-
king and enacting. In Routledge International Handbook of Participatory Design (1st Edition).
Routledge.

14
 How might we evaluate co-design?

Bremmer, M., & Hoek, E. (2019). Teamplayers: CO4-teaching in Arts Education in Primary Education.
International Journal of Education & the Arts, 21(9). https://doi.org/10.26209/ijea21n9
Bryman, A. (2016). Social Research Methods. Oxford University Press.
Charmaz, K., & Belgrave, L. L. (2015). Grounded Theory. In G. Ritzer (Ed.), The Blackwell Encyclopedia
of Sociology (p. wbeosg070.pub2). John Wiley & Sons, Ltd.
https://doi.org/10.1002/9781405165518.wbeosg070.pub2
David, S., Sabiescu, A. G., & Cantoni, L. (2013). Co-design with communities. A reflection on the litera-
ture. Proceedings of the 7th International Development Informatics Association Conference,
152–166.
Dewey, J. (2005). Art as Experience. Penguin.
Dietrich, T., Rundle-Thiele, S., Schuster, L., & Connor, J. (2016). Co-designing social marketing pro-
grams. Journal of Social Marketing, 6(1), 41–61. https://doi.org/10.1108/JSOCM-01-2015-0004
Dietrich, T., Trischler, J., Schuster, L., & Rundle-Thiele, S. (2017). Co-designing services with vul-
nerable consumers. Journal of Service Theory and Practice, 27(3), 663–688.
https://doi.org/10.1108/JSTP-02-2016-0036
Dorst, K. (2015). Frame Innovation: Create New Thinking by Design (Illustrated edition). The MIT
Press.
Ey, L.-A., & Spears, B. (2020). Engaging early childhood teachers in participatory co-design workshops
to educate young children about bullying. Pastoral Care in Education, 38(3), 230–253.
https://doi.org/10.1080/02643944.2020.1788129
Eyles, H., Jull, A., Dobson, R., Firestone, R., Whittaker, R., Te Morenga, L., Goodwin, D., & Mhurchu, C.
N. (2016). Co-design of mHealth delivered interventions: A systematic review to assess key me-
thods and processes. Current Nutrition Reports, 5(3), 160–167.
Farr, M. (2018). Power dynamics and collaborative mechanisms in co-production and co-design pro-
cesses. Critical Social Policy, 38(4), 623–644. https://doi.org/10.1177/0261018317747444
Forss, K., Marra, M., & Schwartz, R. (Eds.). (2011). Evaluating the Complex: Attribution, Contribution
and Beyond (1st edition). Routledge.
Gottwald, S., Brenner, J., Janssen, R., & Albert, C. (2021). Using Geodesign as a boundary manage-
ment process for planning nature-based solutions in river landscapes. Ambio, 50(8), 1477–1496.
https://doi.org/10.1007/s13280-020-01435-4
Greenhalgh, T., Hinton, L., Finlay, T., Macfarlane, A., Fahy, N., Clyde, B., & Chant, A. (2019). Frame-
works for supporting patient and public involvement in research: Systematic review and co-de-
sign pilot. Health Expectations, 22(4), 785–801.
Halar, J., Amare, E., Labrecque, L., Didic, S., & Rowan, M. (2020). The Use and Benefits of Evaluation
Framework Modules at the Canadian Foundation for Healthcare Improvement: Engaged Capacity
Building and Collaborative Program Evaluation Planning. Canadian Journal of Program Evaluation,
35(2), Article 2. https://doi.org/10.3138/cjpe.69230
Hou, N., He, F., Zhou, Y., & Chen, Y. (2020). An efficient GPU-based parallel tabu search algorithm for
hardware/software co-design. Frontiers of Computer Science, 14(5). Scopus.
https://doi.org/10.1007/s11704-019-8184-3
Hurley, E., Trischler, J., & Dietrich, T. (2018). Exploring the application of co-design to transformative
service research. Journal of Services Marketing, 32(6), 715–727. https://doi.org/10.1108/JSM-09-
2017-0321
Janigo, K. A., & Wu, J. (2015). Collaborative Redesign of Used Clothes as a Sustainable Fashion Solu-
tion and Potential Business Opportunity. The Journal of Design, Creative Process & the Fashion
Industry, 7, 24.

15
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

Koens, K., Klijs, J., Weber-Sabil, J., Melissen, F., Lalicic, L., Mayer, I., Önder, I., & Aall, C. (2020). Seri-
ous gaming to stimulate participatory urban tourism planning. Journal of Sustainable Tourism, 1–
20. https://doi.org/10.1080/09669582.2020.1819301
Laptev, G., & Shaytan, D. (2021). Co-design-based learning for entrepreneurs in the digital age. Mea-
suring Business Excellence, ahead-of-print(ahead-of-print), Article ahead-of-print.
https://doi.org/10.1108/MBE-11-2020-0158
Lee, J.-J., Vaajakallio, K., & Mattelmäki, T. (2011). Tracing situated effects of innovative design me-
thods: Inexperienced designers’ practices. Proceedings of the Second Conference on Creativity
and Innovation in Design - DESIRE ’11, 103. https://doi.org/10.1145/2079216.2079231
Li, W. D., Fuh, J. Y. H., & Wong, Y. S. (2004). An Internet-enabled integrated system for co-design and
concurrent engineering. Computers in Industry, 55(1), 87–103. Scopus.
https://doi.org/10.1016/j.compind.2003.10.010
Löfström, E., Klöckner, C. A., & Nesvold, I. H. (2020). Nature in Your Face – Disruptive Climate Change
Communication and Eco-Visualization as Part of a Garden-Based Learning Approach Involving Pri-
mary School Children and Teachers in Co-creating the Future. Frontiers in Psychology, 11,
568068. https://doi.org/10.3389/fpsyg.2020.568068
Mäkelä, T., Helfenstein, S., Lerkkanen, M.-K., & Poikkeus, A.-M. (2018). Student participation in learn-
ing environment improvement: Analysis of a co-design project in a Finnish upper secondary
school. Learning Environments Research, 21(1), 19–41. https://doi.org/10.1007/s10984-017-
9242-0
Manohar, A. K., Smith, M., & Calvo, M. (2016, June 25). Capturing the “How”: Showing the value of
codesign through creative evaluation. Design Research Society Conference 2016.
https://doi.org/10.21606/drs.2016.469
Manzini, E. (2015). Design, When Everybody Designs: An Introduction to Design for Social Innovation
(R. Coad, Trans.). The MIT Press.
McEwan, L. (2017). Investment piece: Co-design strategies to support the emerging designer sector
of the New Zealand fashion industry. Scope, 2017(15), 71–80. Scopus.
Meroni, A., Selloni, D., & Rossi, M. (2018). MASSIVE CODESIGN A Proposal for a Collaborative Design
Framework.
Mumaw, L. M., & Raymond, C. M. (2021). A framework for catalysing the rapid scaling of urban bio-
diversity stewardship programs. Journal of Environmental Management, 292, 112745.
https://doi.org/10.1016/j.jenvman.2021.112745
Noorbergen, T. J., Adam, M. T., Roxburgh, M., & Teubner, T. (2021). Co-design in mHealth systems
development: Insights from a systematic literature review. AIS Transactions on Human-Computer
Interaction, 13(2), 175–205.
Onwuegbuzie, A. J., & Frels, R. (2016). Seven Steps to a Comprehensive Literature Review: A Multi-
modal and Cultural Approach. SAGE.
Pau, S. S. T., & Hall, A. (2021). New spaces for healthcare futures studies: Connecting existing theory
to deeper participatory practice. Futures, 126, 102689. https://doi.org/10.1016/j.fu-
tures.2020.102689
Peters, D., Loke, L., & Ahmadpour, N. (2020). Toolkits, cards and games – a review of analogue tools
for collaborative ideation. CoDesign, 0(0), 1–25.
https://doi.org/10.1080/15710882.2020.1715444
Quinn, S. (2015). Collaborative Learning Design in the Middle School: Sculpting 21st Century
Learners. International Journal Of Learning: Annual Review, 21.

16
 How might we evaluate co-design?

Rodriguez, A., Beaton, L., & Freeman, R. (2019). Strengthening Social Interactions and Constructing
New Oral Health and Health Knowledge: The Co-design, Implementation and Evaluation of A Pe-
dagogical Workshop Program with and for Homeless Young People. Dentistry Journal, 7(1), 11.
https://doi.org/10.3390/dj7010011
Roemer, C., Rundle-Thiele, S., Pang, B., David, P., Kim, J., Durl, J., Dietrich, T., & Carins, J. (2020). Re-
wiring the STEM pipeline—A C-B-E framework to female retention. Journal of Social Marketing,
10(4), 427–446. https://doi.org/10.1108/JSOCM-10-2019-0152
Rutter, S., Stones, C., & Macduff, C. (2020). Communicating Handwashing to Children, as Told by
Children. Health Communication, 35(9), 1091–1100.
https://doi.org/10.1080/10410236.2019.1613478
Scott, D. A. H., Currie, C., Stones, T., Scott, C., John, J., & Wanyonyi, K. (2020). Co-design of an oral
health promotion animated film with families in the South of England. British Dental Journal,
228(3), 164–170. https://doi.org/10.1038/s41415-020-1208-4
Senabre, E., Ferran-Ferrer, N., & Perelló, J. (2018). Participatory design of citizen science experi-
ments. Comunicar, 26. https://doi.org/10.3916/C54-2018-03
Simon, H. A. (1988). The Science of Design: Creating the Artificial. Design Issues, 4(1/2), 67–82.
https://doi.org/10.2307/1511391
Steen, M., Manschot, M., & Koning, N. D. (2011). Benefits of Co-design in Service Design Projects. 8.
van Beusekom, M., Cameron, J., Bedi, C., Banks, E., Harris, R., & Humphris, G. (2021). Using Co-design
With Breast Cancer Patients and Radiographers to Develop “KEW” Communication Skills Training.
Frontiers in Psychology, 12, 629122. https://doi.org/10.3389/fpsyg.2021.629122
Wang, X., & Xu, J. (2019). Co-design of joint radar and communications systems utilizing frequency
hopping code diversity. 2019 IEEE Radar Conference, RadarConf 2019. Scopus.
https://doi.org/10.1109/RADAR.2019.8835576
Ward, M., Ní Shé, É., De Brún, A., Korpos, C., Hamza, M., Burke, E., Duffy, A., Egan, K., Geary, U., Hol-
land, C., O’Grady, J., Robinson, K., Smith, A., Watson, A., & McAuliffe, E. (2019). The co-design,
implementation and evaluation of a serious board game ‘PlayDecide patient safety’ to educate
junior doctors about patient safety and the importance of reporting safety concerns. BMC Medi-
cal Education, 19(1), 232. https://doi.org/10.1186/s12909-019-1655-2
Whicher, A., & Crick, T. (2019). Co-design, evaluation and the Northern Ireland Innovation Lab. Public
Money & Management, 39(4), 290–299. https://doi.org/10.1080/09540962.2019.1592920
Wolfswinkel, J. F., Furtmueller, E., & Wilderom, C. P. M. (2013). Using grounded theory as a method
for rigorously reviewing literature. European Journal of Information Systems, 22(1), 45–55.
https://doi.org/10.1057/ejis.2011.51
Woltersdorf, L., Lang, P., & Döll, P. (2019). How to set up a transdisciplinary research project in Cent-
ral Asia: Description and evaluation. Sustainability Science, 14(3), 697–711.
https://doi.org/10.1007/s11625-018-0625-7

17
Zixuan Wang, Tingting Jiang, Jiawen Huang, Yixuan Tai, Paola Maria Trapani

About the Authors:

Zixuan Wang is a double degree graduate student majoring in Product

Service System Design at Tongji University and Politecnico di Milano.
Her research interests lie in co-design methodology, human-robot in-
teraction, and design ethics.

Tingting Jiang is a graduate student majoring in Interaction Design at

Tongji University. Her research interests are co-design methodology
and digital design, and she is also highly motivated in design anthro-
pology.

Jiawen Huang is a double degree graduate student majoring in Prod-

uct Service System Design at Tongji University and Politecnico di Mi-
lano. Her research interests include co-design methodology, social in-
novation, human-robot interaction, and human-machine interaction.

Yixuan Tai is a double degree graduate student majoring in Communi-

cation Design at Tongji University and Politecnico di Milano. Her re-
search interests are Integrated Media Design, Data Visualization, and
Media History and Theory.

Paola Maria Trapani recently joined AHO in Oslo as an Associate Pro-

fessor of Service Design. Previously, she has served as Director of the
Master in Product-Service System Design at the College of Design &
Innovation at Tongji University in Shanghai.

18