Agroecology and Sustainable Food Systems

ISSN: 2168-3565 (Print) 2168-3573 (Online) Journal homepage: https://www.tandfonline.com/loi/wjsa21

DELTA: An Integrated Indicator-Based Self-

Assessment Tool for the Evaluation of Dairy Farms
Sustainability in Quebec, Canada

Valérie Bélanger, Anne Vanasse, Diane Parent, Guy Allard & Doris Pellerin

To cite this article: Valérie Bélanger, Anne Vanasse, Diane Parent, Guy Allard & Doris Pellerin
(2015) DELTA: An Integrated Indicator-Based Self-Assessment Tool for the Evaluation of Dairy
Farms Sustainability in Quebec, Canada, Agroecology and Sustainable Food Systems, 39:9,
1022-1046, DOI: 10.1080/21683565.2015.1069775

To link to this article: https://doi.org/10.1080/21683565.2015.1069775

Accepted author version posted online: 17

Jul 2015.
Published online: 02 Sep 2015.

Submit your article to this journal

Article views: 208

View Crossmark data

Citing articles: 2 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=wjsa21
Agroecology and Sustainable Food Systems, 39:1022–1046, 2015
Copyright © Taylor & Francis Group, LLC
ISSN: 2168-3565 print/2168-3573 online
DOI: 10.1080/21683565.2015.1069775

DELTA: An Integrated Indicator-Based

Self-Assessment Tool for the Evaluation of Dairy
Farms Sustainability in Quebec, Canada

VALÉRIE BÉLANGER, ANNE VANASSE, DIANE PARENT,

GUY ALLARD, and DORIS PELLERIN
Faculté des Sciences de l’Agriculture et de l’Alimentation, Université Laval, Québec, Canada

A self-assessment tool was developed for use by dairy farmers in

Quebec and aims to measure the agri-environmental,
techno-economic, and social dimensions of sustainability.
Some of the methodological choices made in the development
of the tool and its validation by the end users are presented.
Validation of the tool was conducted with 17 farmers using a
questionnaire divided into three sections regarding their
perception of the application procedure, the consistency of the
results in regards to the reality of their farm, and the utility of
the tool. Analysis of the questionnaires confirms the relevance
of the tool. Four case studies were used to examine the results
in depth and clarify the relationships between the three compo-
nents of sustainability of dairy farms and between these com-
ponents and the indicators. Case studies results demonstrate
that the aggregation of indicators into a single score could
remove information necessary for the comprehension. The
visual integration through radar diagrams leaves the flexibility
for the farmer to understand his scores and compare himself to
peers. Based on DELTA, a new schematization of a sustainable
farm has emerged where a farmer makes compromises to
manage sustainability on his farm and maintain the balance
of the farm system.

KEYWORDS aggregation, case studies, end-use validation,

indicators, sustainability assessment

Address correspondence to Anne Vanasse, Faculté des Sciences de l’Agricutlure et de l’alimen-

tation, département de phytologie, Université Laval, 2425 Rue de l’Agriculture, Quebec; QC G1V
0A6, Canada. E-mail: anne.vanasse@fsaa.ulaval.ca

1022
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1023

1. INTRODUCTION

In the quest for agricultural sustainability, worldviews have evolved. Beyond

farm productivity assessment and their technical aspects, natural resource
management and social aspects are now included in sustainable agriculture.
European countries have developed an expertise to assess sustainability at the
farm level up to the multi-regional level with the objective of rendering these
tools usable for policymakers (Bezlepkina et al. 2011). Indicators are becom-
ing a widespread tool to evaluate sustainability (Zhen and Routray 2003).
While the use of indicators providing a tripled-bottom line evaluation is now
accepted, some questions remain. Can we aggregate the three dimensions of
sustainability at the farm level and if so how should it be done—or do we
need to integrate them?
A distinction has to be made between integration and aggregation.
Aggregation is the result of combining different components or indicators
into one single unit also called a composite indicator (Van Passel and Meul
2012). As for integration, it is basically a means by which individual and
different indicators can be linked to provide a holistic view of sustainability
(Bell and Morse 2003). Some kind of integration is possible without
aggregation.
The first type of integration is called the numerical integration
approach using aggregation (Van Passel and Meul 2012). There are many
ways to aggregate the attributes and dimensions of agricultural sustainabil-
ity. The use of a multi-criteria method is probably the most common. In
agriculture, many researchers have used different forms of multi-criteria
methods: the fuzzy logic (van der Werf and Zimmer 1998), the qualitative
multi-attribute model (Sadok et al. 2009), and the multi-attribute value
theory (MAVT) (Dantsis et al. 2010). Other authors have written reviews
on the potential methods to assess agricultural sustainability using indica-
tors (Bockstaller et al. 2008; Sadok et al. 2008). However, these methods
generally provide models and simulations and are often used ex ante
(before any chance to analyze the impact of possible scenarios). These
models and simulations have a predictive function (Bockstaller et al. 2008).
Furthermore, aggregation requires transforming indicators into nondimen-
sional variables to be able to sum or to weigh different variables that have
multiple units. Information is, however, lost by using such a procedure
(Bockstaller et al. 2008).
The second type of integration is named the visual integration approach
and generally uses radar diagrams to assess farm-level sustainability in the
three dimensions (Häni et al. 2003; Meul et al. 2008; Vilain 2008). Although
using such radar diagrams provides a visual basis for integration, individual
indicators are still presented separately.
1024 V. Bélanger et al.

Level of analysis Condensation

of data

Regional or national
level
Public Composite indicator,
indice or single index
Farm to local and
regional level
Policy-maker

Indicator
Field or farm level Farmers

Field or specific aspect level Scientists Data

Total quantity of information

FIGURE 1 Relationships between indicators, users and the level of analysis (adapted from
Braat 1991).

Linked to these two integration approaches, Van Passel and Meul

(2012) have remarked that the tools presenting a visual integration were
directed to farmer end users while all the tools directed to policymakers
used a numerical integration. Figure 1 presents the relationship between
indicators, intended users, and the level of analysis. Scientists and techni-
cians want raw data to be able to explain every single relation and want to
be able to do statistical analysis (Pacini et al. 2003). At the opposite end,
policymakers and the general public that work at local, regional, or upper
levels need aggregated data to classify and rank different systems in order
to make decisions (Pacini et al. 2003). Decisions are usually more difficult
to make when a plethora of indicator results are available. However, some
critics are against highly aggregated indices as they present a lack of
transparency when communicating (Bell and Morse 2003). Also, this
could leave aside dimensions that are more difficult to assess leading to
inappropriate policies if such dimensions are ignored (Gómez-Limón and
Sanchez-Fernandez 2010). Many tools are developed at upper levels for
policymakers but the relevance of individual indicators is still necessary.
In Europe, assessment tools for use by farmers have been created in
the last decade. One of them used in France is the IDEA (Indicateurs de
Durabilité des Exploitations Agricoles) method. IDEA assesses whole-farm
sustainability with agri-ecological (18 indicators), socio-territorial (18 indi-
cators), and economic (six indicators) dimensions (Vilain 2008). Within
each dimension, components are grouped within indicators, with all
indicators responding to at least one of the 16 objectives for sustainability
defined by the IDEA research team. The IDEA indicators were developed
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1025

by a multidisciplinary team of researchers and stakeholders and the

method can assess multiple agricultural production system. Radar dia-
grams are used to illustrate the results to the farmers. Another integrated
tool, MOTIFS (MOnitoring Tool for Integrated Farm Sustainability, devel-
oped and tested on Flemish dairy farms (Meul et al. 2008), focuses on
communication and user-friendly aspects. Experts and stakeholders
develop the indicator methodology, and the results are also presented
to farmers using radar diagrams. Using this tool, indicators are grouped
into themes. There are 21 environmental indicators, 8 economic indica-
tors, and 18 social indicators divided into 10 themes.
In North America, there are relatively few tools that have been designed
to provide sustainability self-assessment to farmers and even less in a specific
agricultural production system. One of them is the Dairy Stewardship Alliance
initiative, developed by an agricultural extension group, at the University of
Vermont (USA; Matthews, 2010). Farmers complete a self-assessment; 10
modules of sustainable practices are translated into indicators. In each mod-
ule, there are between 6 and 10 indicators. Bar graphs are used to illustrate
the results to farmers. In Canada, development of an integrated indicator set
including the three dimensions of sustainability is needed for dairy produc-
tion. Quebec is the leading province for milk production in Canada, with 48%
of all dairy farms (Statistics Canada 2012). The specific context under supply
management brings different aspects from other regions or countries to take
into account in terms of sustainability.
The general aim of this article is to present the methodological approach
for the construction of DELTA, a self-assessment tool for use by dairy farmers in
Quebec, Canada regarding the integration of the three dimensions of sustain-
ability. The first objective is to present an end-user validation as a way to
demonstrate the adequacy of our methodological choices and further deter-
mine if indicators are a reliable representation of the farm realities. The second
objective is to use the indicators in four case studies to evaluate if the developed
tool can be operationalized and if a visual integration could provide a way to
enhance, at first attempt, the comprehension of farm sustainability.

2. METHODOLOGY
2.1. Premises to the Development of DELTA, A Self-Assessment Tool
for Dairy Farms
Methodological aspects that are inherent to the indicators must be taken into
consideration for the construction of a sustainability integrated self-assessment
tool for use by farmers. For Binder, Feola, and Steinberger (2010), it is
necessary to emphasize what such assessment is meant to achieve. Then,
several premises were considered in order to develop the farm-level
1026 V. Bélanger et al.

self-assessment tool named DELTA for dairy farms in Quebec. The premise of
each are discussed herein.

● Based on relevant indicators in the three dimensions of

sustainability. Sustainability could be evaluated by the classical three
dimensions of sustainability (environment, economic, and social) or by
using a conceptual approach based on the attributes of sustainability to
define the system as for example: adaptability, productivity, and resilience
(Conway 1994; López-Ridaura, Masera, and Astier 2002). Some authors
classify the evaluation with the three dimensions as content based or goal
oriented while the evaluation with the attributes is system based or property
oriented (von wirén-Lehr 2001; Van Cauwenbergh et al. 2007). While the
linkage of indicators with the attributes of the system enables easier com-
prehension of interactions among indicators, it also exposes a concept of
systemic property often considered too abstract for non-initiated users
(Bockstaller et al. 2008). The self-assessment presented here is based on
the three dimensions of sustainability.
● A sustainable farm is reproducible, viable, liveable, and
transmissible. A definition is primordial when an objective of self-assess-
ment is desired at the farm level. A sustainable farm needs to be reprodu-
cible, viable, livable, and transmissible (Landais 1998; Parent 2001).
Reproducibility is defined by the rational use of natural resources (water,
soil, air) through good agricultural practices and the reproducible potential
of those resources. Viability depends on the techno-economic performance
of the farm and implies that a farm must be able to generate a secure long-
term income. Livability reflects the quality of life of farmers and their
families, both on the farm and in the community. Finally, transmissibility
expresses the potential of a farm to be taken over through succession by the
next generation as well as the role of agriculture in the dynamics of local
development. To develop the indicator set in the three dimensions of
sustainability, this definition of a sustainable farm has been taken into
consideration.
● Site-specific and precise temporal and spatial scale. The concept of
sustainability is a generic one and needs a precise definition each time it is
being utilized. Some authors depict the fact that sustainability is a spatial
and temporal concept and those boundaries have to be defined before the
construction of an assessment tool (Van Cauwenbergh et al. 2007; Bock-
staller et al. 2008). While many tools are developed to assess all types of
agricultural production, some authors preferred to narrow them down to
one specific context and one type of management system (López-Ridaura,
Masera, and Astier 2002; Freebairn and King 2003). The self-assessment
described here is a snapshot of the farm for one year. Indicators are mostly
measured at the farm level but some agri-environmental indicators data are
determined at the field-level and summed up to the farm level. Adding to
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1027

the temporal scale, sustainability includes a notion of time. How long a farm
should be sustainable? It was necessary to define the duration of our
assessment to bring more sense in the process of indicator selection. For
us, a sustainable farm should, at least, still be in activity over the next two
generations. Such a context was described to the experts that participated in
the selection of indicators through the use of participatory processes.
● Target the end-user group: farmers. The identification of end users and
the definition of practical objectives for the indicators were identified as an
essential step by several authors (Mitchell, May, and McDonald 1995; Gir-
ardin, Bockstaller, and van der Werf 1999). As mentioned in the first section
of this article, different end-user groups require different types of indicators
(Figure 1). Here, as this tool is developed for self-assessment, farmers are
first in attempting to use it.
● Use of participatory processes. Since sustainability could be seen as a
social construct, the participation of different actors is essential through a
mix of bottom-up and top-down approaches. If only bottom-up
approaches are used, it can lead to an unequal representation of the
three dimensions of sustainability (Ripoll-Bosch et al. 2012). As such,
several authors agree that the participation and consultation of farmers is
a key element in building and developing indicators (King et al. 2000;
Rigby et al. 2001). For this project, different sets of experts were chosen
for each dimension of sustainability during the selection of indicators.
Expert groups were made of farmers, researchers, and advisors or stake-
holders from government.
Those premises and boundaries are necessary for choosing the most
preferred indicators and their characteristics in a specific context. The
complete methodology to construct and select an indicator set using a
participatory approach is presented in Bélanger et al. (2012). In this
article, the use of participatory processes is fully detailed as it is a central
point to the construction of our tool. A total of 13 indicators were chosen
for the agri-environmental dimension, 8 indicators for the techno-eco-
nomic dimension, (Larochelle 2011) and 20 indicators for the social
dimension. In each dimension of sustainability, indicators are grouped
into components to facilitate comprehension and presentation. Each
dimension is of equal significance, but within it, relative weights have
been given to components and, within the components, to individual
indicators. The choice of reference values for indicators is also explained.
Social issues are central to explain farm sustainability (Ripoll-Bosch et al.
2012). Social aspects have been put aside in many attempts to assess
sustainability (Binder, Feola, and Steinberger 2010). Adding to that, as
mentioned by Gomez-Limòn and Sanchez-Fernandez (2010), the sustain-
ability concept needs to be understood as a social construct, which
changes as a function of society and, thus, needs to be specifically
formulated for any given set of geographical and temporal conditions.
1028 V. Bélanger et al.

The selection of social indicators in our tool and the explanations con-
cerning their construction are forthcoming.

2.2. End-User Validation

After testing the indicators for their operationalization on 40 farms in two
different Quebec regions contrasted for their pedo-climatic conditions and
their proximity to urban centers (for a complete description of both regions,
refer to Bélanger et al. 2012), an additional methodological step was
included to do an end-user validation (Bockstaller and Girardin 2003). A
questionnaire was sent to the 34 farms that completed the three question-
naires (one for each of the three dimensions of farm sustainability). The
questionnaire related to the end-user validation contained three sections: 1)
procedure perception, 2) results consistency, and 3) tool utility. The first
section verified if data collection on the farm was easy and if the radar
diagrams were well understood by the farmers. It also inquired about the
general importance given to each dimension of sustainability by the farmer.
The second section was used to validate if, according to the farmer, the
scores from the developed indicators in each dimension corresponded to the
reality of his farm. The last section questioned farmers about some ways to
present their results and also aimed at knowing if they would like to use the
tool regularly (every year or every other year) to follow their farm improve-
ments. Moreover, it questioned farmers about their perception on how this
tool could enhance the sustainability of their practices. The questionnaire
was formatted to be user friendly and the Likert scale was chosen to
measure the perception of farmers through the self-assessment tool. This
step also allowed us to determine if our tool responds to a certain commu-
nication objective. As Meul, Nevens, and Reheul (2009) mentioned, the
credibility of an indicator relates to the degree of confidence that the
potential end users have in it and their willingness to effectively use it in
practice. This end-user validation should enable us to improve our tool.

2.3. Case Studies

While it is not feasible to describe and analyze the results for each of the 17
farms that completed the end-user validation, we opted to use a case study
approach. A case study is defined as a detailed examination of a single unit
(Flyvbjerg 2006). A case study may be descriptive, exploratory or explanatory
(Yin, 2009). It allows to better understand farm-level sustainability using a
reduced sample of farms. Also, a case study facilitates learning through the
study of the particularities and complexities of a bounded system (Stake
1995). Four farms were chosen and analyzed for their results to determine
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1029

possible explanations and the variability that those indicators can provide in
terms of results. The choice of the four farms was based on their results from
the section on results consistency in the end-user validation. For Flyvbjerg
(2006), one possible strategy for the selection of cases is a selection based on
information-oriented selection. This strategy maximizes the utility of informa-
tion from a small sample. Cases are selected on the basis of expectations
about their information content. Two farms were selected in each of both
regions. Similar methodology using a case study investigation approach has
been employed by Ripoll-Bosch et al. (2012) to achieve a comparative eva-
luation of sustainability in four different sheep farming systems.

3. RESULTS AND DISCUSSION

3.1. End-User Validation
A total of 17 completed questionnaires out of 34 were received. The answer rate
was 50%. This low rate could be explained by the length of the overall research
project of four years. Some farmers were disappointed by the time it took to
receive their results. Some of them mentioned that their results were not up to
date anymore. However, the 17 farmers who completed the questionnaire
brought new insights to the operationalization of the tool. In Table 1, some
results from the three sections of the questionnaire are presented using the mean
and the standard deviation of the measurement. Moreover, for each question
asked, we put the percentage of farmers that agree (3 on the Likert scale) or
strongly agree (4 on the Likert scale) and the total of these two answers.

3.1.1. PROCEDURE PERCEPTION

A total of 98% of farmers (20% agreed and 78% strongly agreed) agreed
that it was easy to collect the main data on their farms (Table 1). These data
included their agri-environmental fertilization plan and their financial state-
ments. These results corresponded to our objective to develop indicators with
data easily accessible on the farm. It must be noted that in Quebec the agri-
environmental fertilization plan is compulsory for each farm, so it was rela-
tively easy to collect data for the agri-environmental dimension. While the
social data collection was not part of this question, a pretest was used for the
social questionnaire that confirmed its easiness to fill out while not time-
consuming.
The choice of presenting the results through a visual integration using the
radar diagrams seemed to be appreciated by the farmers while 38% agreed
and 56% strongly agreed (Table 1). This farmer easiness to understand radar
diagrams could come from their acquaintance with such tools that are also
used by many dairy farmer groups in Quebec for their management practices.
 TABLE 1 Results from the end-user validation

Section of the % farmers choosing 3 % farmers choosing 4 Total % farmers

questionnaire Questions asked n Mean* SD (on Likert scale) (on Likert scale) choosing 3 and 4

Procedure perception Information and data easily 17 3.7 0.54 20 78 98

accessible
Easy interpretation of results 16 3.4 0.66 38 56 94
presented with radar
graphs
Importance devoted to agri- 17 3.2 0.56 65 29 94
environmental dimension
Importance devoted to 17 3.6 0.51 41 59 100
techno-economic
dimension
Importance devoted to 17 3.0 0.61 65 18 83
social dimension

1030
Scores consistency Agri-environmental scores 17 2.8 0.90 53 18 71
Techno-economic scores 17 2.7 0.92 47 18 65
Social results 17 2.8 0.64 59 12 71
Tool utility Presentation of scores using 17 3.4 0.49 65 35 100
tables
Presentation of the regional 17 3.5 0.51 53 47 100
average
Presentation of the leading 17 3.3 0.69 47 41 88
group average
Meeting with advisor to 17 2.7 0.69 47 12 59
interpret results
Follow-up every year or two 16 2.6 0.70 50 6 56
years using an online tool
How the tool can enhance 17 2.9 0.56 71 12 83
the farmer practices
*1 = not important, not satisfied or not at all and 4 = very important, very satisfied or totally.
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1031

It was then logical to keep this form of diagram to present the data. These
results provide evidence for visual representation rather than aggregated
measures of sustainability.
The last constituent analyzed from the first section of the questionnaire
was the overall importance for farmers regarding the three sustainability
dimensions. Farmers gave more importance to the techno-economic dimen-
sion, followed by the agri-environmental dimension, and to a lower degree
the social dimension. This is different than what Ripoll-Bosch et al. (2012)
reported at the local and the farm level: higher importance was given to the
economic and social dimensions compared to the environmental dimension.
The techno-economic dimension was rated very important by 59% of farmers.
Combining the ranks important and very important, 100% was reached for the
techno-economic dimension, 94% for the agri-environment dimension and
83% for the social dimension. The agri-environmental dimension rated in
second place by the farmers could be explained by the implementation in
2002 of the Agricultural Operations Regulation (REA) in Quebec aiming,
among other things, to strike a better balance between fertilizer applications
and plant requirements. It also aims at ensuring adequate management of
livestock manure while reducing the environmental impact of its use as
fertilizer. The introduction of this regulation was perceived by farmers as an
increased consideration for the environment in agricultural activity. Contex-
tual events explaining farmer reactions have also been considered by Meul,
Nevens, and Reheul (2009). They mentioned that sustainability had a negative
connotation with farmers because this concept was first introduced when
agricultural activity was considered to be a major source of environmental
problems in the Flanders region.

3.1.2. RESULTS CONSISTENCY

The second section of the questionnaire allowed us to determine if the

measurements of sustainability using our indicators were reliable or not.
We believe that farmers were in the best position to judge their farm and
thus to evaluate if the estimated results represented their farm reality. For
the agri-environmental scores, a mean of 2.8 over 4.0 was obtained, and
the same scores with the social dimension. For both dimensions, a total of
71% of farmers reported that the overall scores for their farm represented
fairly or entirely their reality. In our view on a 1- to 4-point scale using
classes (not at all, a little, fairly, and entirely), a ranking of 3 or 4 can be
considered as a good score. For the techno-economic results, 65% were
fairly or entirely in accord with their scores. Following the question about
the score accuracy, farmers had the possibility to indicate which indicators
seemed less coherent with their view of their farm. As it is impossible here
to report the individual responses, we can pay special attention to the few
1032 V. Bélanger et al.

indicators that were reported to incorrectly estimate sustainability on more

than one farm.
For the agri-environmental dimension, the integrated pest manage-
ment (IPM) indicator was identified three times as mistakenly representing
the farm. Two explanations are possible. First, farmers do not fully accept
the given ranking for IPM use or they do not fully understand what is
required to be ranked as a farm using IPM. The three levels of IPM in the
agricultural extension literature in Quebec were minimum, intermediate,
and advanced. The minimum level refers to the use of pesticides as the
only means of crop protection when pests are first identified visually in the
field. The intermediate level includes alternative ways of reducing pesticide
use (e.g., mechanical weed control). The advanced level includes alterna-
tive options (e.g., intercrops) for all crops in production. Some farmers may
have a hard time accepting a weak ranking. Second, the indicator could be
better estimated. An early decision with the focus group was to determine
this indicator using a qualitative scale as the level of adoption of IPM
instead of measuring the active ingredient for each pesticide treatment
which should be time consuming. The different levels could be not dis-
criminating enough. A solution to this issue could be to do a quantitative
estimation using the ratio of total land area with the different level of
adoption (Thivierge 2011). This could be a compromise to obtain more
accurate results.
In the techno-economic dimension, two indicators were mentioned a
few times by the farmers as incorrectly representing their farm situation.
These two indicators are milk from forage and forage self-sufficiency. Milk
from forage is an estimation of the milk produced by the forages when one
deducts from the total milk, the milk produced by concentrates (Charbon-
neau et al. 2006). Without going into the details of calculation, the fact that
this is an estimation could be a factor explaining disagreement with the
farmers. Also, it was possible that farmers did not pay enough attention to
the forage quality and productivity on their farm leading them to serve large
amounts of concentrates to maintain high milk production. Forage self-
sufficiency is based on the proportion of forage requirements for the dairy
herd filled by forages produced on the owned or rented lands (lease of
more than seven years). Long-term renting could be the concept that
annoyed farmers. It is difficult to sign up a long term lease because it is
difficult to foresee what is going to happen in the long term. However, the
sustainability concept includes a notion of long term that is translated into
this indicator. Another explanation could be that the threshold values for
these two indicators are too difficult to reach. These indicators have been
developed in a way to enhance the valorization of forages. Every system is
unique and specific indicators may or may not be relevant for all cases
(López-Ridaura et al. 2005). Therefore, creating an indicator set that will
satisfy all dairy farmers is a difficult task to uphold.
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1033

Within the social dimension, some indicators have been mentioned once
or twice by different farmers and because there are many indicators in the
social dimension, it was difficult to identify precise ones. However, this could
underline the fact that for many farmers, it was the first time that they looked
upon the social aspects of their farm sustainability, as was reported by some
farmers in the questionnaire.

3.1.3. TOOL UTILITY

The last section of the end-user validation questionnaire referred to the

usefulness of the tool. Questions related to the presentation of the results and
their future utilization. The presentation of individual scores for each indicator
in tables was important or very important for 100% of the farmers. It enables
them to see the overall scores of the farm and it was another argument to
keep separated scores instead of aggregating them into one single index. This
observation is consistent with Ripoll-Bosch et al. (2012) who stated that the
analysis of individual indicators is necessary to describe farming systems in
details, and tables were presented for each sustainability dimension. Regard-
ing the presentation for the regional average, again, 100% of the farmers
wanted to have access to this information. However, since the tool is con-
structed as a self-assessment one, further validation using a large number of
dairy farms will be necessary to create those regional averages. In our view,
the presentation of the leading group average could be more meaningful in a
sustainable context even if fewer farmers agreed (88% compare to 100% for
the regional average). The leading group includes the 25% of farmers with the
highest scores. Questions regarding comparisons with the provincial average
and the provincial leading group were also asked (data not shown) but lower
numbers were obtained, confirming that comparisons with local and regional
scores are more suitable. Provincial scores would encompass too much
variability due to different production’s context. According to Halberg,
Verschuur, and Goodlass (2005), there are two empirical methods to compare
scores and assess relations among indicators. The first one is to look at the
differences among similar farms and the second one is to look at the differ-
ence among assessments of the same farm year after year; the latter was what
we intended.
Farmers agreed at 50% to do a self-assessment every year or every two
years whereas 6% strongly agreed. Whereas the evaluation of sustainability
includes a notion of long term, it could be more appropriate to do the
assessment every five years. Despite the low score to redo the self-assess-
ment, a total of 83% of the farmers agreed or strongly agreed that it can
enhance their practices to be more sustainable. Adding to that, 59% of farmers
agreed that a meeting with an advisor could help to interpret the scores.
1034 V. Bélanger et al.

Through this end-user validation, we found out that the tool was correctly
designed and that our set of indicators fairly represents the dairy farms reality.

3.2. Case Studies of Four Farms

The case studies enable us to better understand the possible relationships
among the different sustainability dimensions of dairy farms. Table 2 presents
the components scores of four farms and their overall score for each dimen-
sion. Some general characteristics of the four farms are shown in the first
section of the table to help in contextualizing the information. Potential
relations can be drawn from the results to discuss what the farmer could do
with his self-diagnosis.
Both farms (A and B) in region one gave a very high importance to the
techno-economic dimension with high overall scores (92.0% and 95.3%)
(Table 2). Farms A and B presented low agri-environmental scores. A trade
off exists between agri-environmental and techno-economic dimensions. In

TABLE 2 Scores for each component of the farm case studies with main characteristics for
each farm

Characteristics/components Score Farm A Farm B Farm C Farm D

Region* 1 or 2 1 1 2 2
Number of cows Nb 63 43 48 51
Land area Ha 157.7 226.4 127.5 135.4
Importance devoted to agri- 1 to 4 3 4 3 3
environmental dimension**
Importance devoted to techno- 1 to 4 4 4 3 3
economic dimension**
Importance devoted to social 1 to 4 3 3 3 2
dimension**
Agri-environment 100 65.2 78.1 90.1 95.6
Soil quality 20 17.9 16.8 18.4 20.0
Cropping practices 30 14.9 19.1 30.0 30.0
Fertilization management 30 19.5 22.3 28.7 26.6
Farmland management 20 13.0 20.0 13.0 19.0
Techno-economic 100 92.0 95.3 60.4 66.1
Technical management 20 20.0 15.3 11.2 16.5
Economic viability 25 20.8 25.0 23.0 11.0
Expense control 30 26.2 30.0 14.2 16.7
Labor efficiency 15 15.0 15.0 1.9 12.4
Forage self-sufficiency 10 10.0 10.0 10.0 9.5
Social 100 74.0 65.0 56.0 67.0
Quality of life 25 13.0 19.0 15.0 16.0
Social integration 15 9.0 9.0 8.0 11.0
Farm succession 30 25.0 14.0 17.0 23.0
Entrepreneurship 30 27.0 23.0 16.0 17.0
*1 = Monteregie region and 2 = Bas-St-Laurent region.
**1 = not important and 4 = very important.
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1035

farms A and B, a larger gap was observed in cropping practices and fertiliza-
tion management. This could be explained by the context of the regions.
Monteregie region (1) has a longer growing season than Bas St-Laurent region
(2) and is situated near the urban center of Montreal. With the longer growing
season than farms C and D, farms A and B have the possibility to grow corn
and by extension, grow less perennial forage species. This difference could
explain the lower scores obtained for cropping practices and fertilization
management components. For the first three indicators in the cropping prac-
tice components, scores were lower for farms A and B (Table 3). If farms A
and B are growing corn on a large area, it carries at the same time smaller
areas for perennial crops that experts have classified as essential on a dairy
farm for sustainability. Also, corn production limits the choice for different soil
tillage practices under our climatic conditions where corn harvesting is done
in late autumn making it impossible to seed green manures into those fields.
In the fertilization management component, the manure management indica-
tor can be understood in the same way since late harvest does not allow
spreading manure in those fields without environmental risks. Another obser-
vation was the possible synergy existing between the techno-economic
dimension and the component of entrepreneurship in the social dimension.
Both farms A and B had the highest score for this component. It seemed to be
profitable to have some characteristics of an entrepreneur such as a good
education, human resource management on the farm and entrepreneurial
abilities.
The reverse relation was observed for farms C and D in region 2, the Bas-
St-Laurent region, East of Quebec city and where fewer crops are possible due
to climatic conditions. For farms C and D, lower scores were obtained for the
techno-economic dimension instead of the agri-environmental dimension.
The larger gap was found in the expense control component. Despite that
the scores for this component were similar in farms C and D (Table 2), large
differences were observed in the individual indicators for this component
(Table 3) and it is in accord with Ripoll-Bosch et al. (2012) who mentioned
that similar average scores for sustainability attributes can also hide large
differences within individual indicators. Farm C had a low score for the
operational expense on gross income and this, in turn, could be related to
higher purchases of concentrates as expected from a low milk from forage
indicator. For farm D, machinery expenses per hectoliter was one indicator
with a low score and could probably be linked with the low security margin
indicator as the security margin is calculated using all expenses and that the
machinery expenses are part of it.
For the social dimension, results of the four farms were different. Even if
it was not the goal of these case studies, comparisons helped to understand
the tool and how it could be useful for farmers to see their farm through the
scores. The social integration component score was similar for the four farms.
There was also room for improvement in the quality of life component for the
1036 V. Bélanger et al.

TABLE 3 Scores for each indicator in the farm case study

Farm Farm Farm Farm

Components Indicators Score* A B C D

Soil quality Organic matter content 12.0 10.4 9.4 11.0 12.0
Phosphorus soil saturation 8.0 7.4 7.3 7.4 8.0
................................................................................................................................................................
Cropping practices Perennial crops 16.0 9.0 9.8 16.0 16.0
Annual crops: soil tillage practices 4.0 0.7 2.2 4.0 4.0
Annual crops: green manure 3.0 1.1 0.1 3.0 3.0
Annual crops: crop rotations 4.0 3.0 4.0 4.0 4.0
Annual crops: integrated pest 3.0 1.0 3.0 3.0 3.0
management
................................................................................................................................................................
Fertilization Manure storage structure 3.0 0.0 3.0 3.0 2.0
management Manure management 9.0 3.4 5.3 8.6 7.6
Phosphorus balance 9.0 7.8 6.7 8.1 9.0
Nitrogen balance 9.0 8.3 7.3 9.0 7.9
................................................................................................................................................................
Farmland Watercourse protection 6.0 6.0 6.0 6.0 6.0
management Land drainage 4.0 4.0 4.0 4.0 4.0
Windbreaks 4.0 0.0 4.0 0.0 3.0
Field slope 3.0 3.0 3.0 3.0 3.0
On-farm woodlot 3.0 0.0 3.0 0.0 3.0

Technical Production per cow 8.0 8.0 8.0 7.7 4.5

management Milk from forages 12.0 12.0 7.3 3.6 12.0
................................................................................................................................................................
Economic viability Security margin 15.0 15.0 15.0 15.0 1.0
Debt per hL 10.0 5.8 10.0 8.0 10.0
................................................................................................................................................................
Expense control Operational expense on gross 20.0 18.5 20.0 4.2 12.3
income
Machinery expense per hL 10.0 7.7 10.0 10.0 4.4
................................................................................................................................................................
Labor efficiency Milk per worker (labor unit) 15.0 15.0 15.0 1.9 12.4
................................................................................................................................................................
Forage self- Forage self-sufficiency 10.0 10.0 10.0 10.0 9.5
sufficiency

Quality of life Work and workload 2.0 0.0 2.0 0.0 0.0
Holidays 2.0 0.0 1.0 0.0 2.0
Satisfaction 7.0 3.0 4.0 5.0 5.0
Social support 4.0 3.0 3.0 3.0 4.0
Health and stress 5.0 2.0 5.0 2.0 2.0
Social and professional 5.0 5.0 4.0 5.0 3.0
relationships
................................................................................................................................................................
Social integration Contribution in local services 2.0 0.0 0.0 1.0 0.0
Agricultural neighborhood 4.0 4.0 4.0 2.0 4.0
Cohabitation 6.0 4.0 4.0 4.0 6.0
Social contribution 1.0 0.0 1.0 1.0 1.0
Regional presence of agriculture 2.0 1.0 0.0 0.0 0.0

(Continued )
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1037

TABLE 3 (Continued)

Farm Farm Farm Farm

Components Indicators Score* A B C D

Farm succession Continuity value 9.0 9.0 9.0 9.0 9.0

Presence of farm succession 8.0 8.0 5.0 5.0 8.0
Preparation for retirement 7.0 7.0 0.0 0.0 6.0
Farm succession integration 6.0 1.0 0.0 3.0 0.0
................................................................................................................................................................
Entrepreneurship Formation 6.0 5.0 5.0 4.0 4.0
Use of advisory services 6.0 6.0 4.0 4.0 5.0
Vision 6.0 6.0 4.0 0.0 4.0
Human resources management 6.0 4.0 4.0 3.0 2.0
Entrepreneurial abilities 6.0 6.0 6.0 5.0 2.0
Note: The dotted lines are used in this table to separate the components into one dimension and solid line
are used to separate the dimensions of sustainabiliy.
*This column present the maximum score for each indicator.

four farms (Table 2). When we looked at the individual indicators level, only
one farm received points for the work and workload indicator. For the social
support indicator and the social and professional relationships indicator, all
farms obtained an acceptable score.
From the examination of these case studies, another important point was
revealed for the farm succession component. Farms A and D had higher overall
general scores (Table 2) for the farm succession component. At the indicator
level (Table 3), we observed that those higher scores were linked to a better
score for the preparation for retirement indicator. For the four indicators in this
component (Table 3), there were three of them that must have both generations
on a farm to have the full score for the indicator. Calculations of these three
indicators at the moment cannot correctly evaluate this consideration equally
between farms with both generation and farms without having already a
succession plan. Although we expect an older farmer who does not know if
he will have someone to take over the farm scores low, we cannot penalize a
young farmer starting up for not having a lot of preparation for his retirement.
So calculations of these three indicators must be reviewed. As another example,
one variable in the calculation of the farm succession integration indicator is the
percentage of assets of the farm that is transmitted today. It would be better to
calculate the percentage of the farm assets owned by each generation: the
parents, the current generation, the children and others if applicable. It is
noticeable as well that such information would not be available if aggregated
measures were used leaving out the possibility for improvement of the tool. As
Ikerd (2006) mentioned, better indicators and measures of sustainable farming
should at least improve the process of exploration.
These case studies brought the notion of individual results versus aggre-
gated results in regard of farmers’ needs to better understand the sustainability of
their farm. The four case studies and their scores by components and by
1038 V. Bélanger et al.

indicators enabled us to understand that in an aggregation through one compo-

nent, some information will be lost for farmer understanding. This loss of infor-
mation may be acceptable when decisions are made over certain criteria and at a
policy level but, at the farm level, a farmer needs to have all the information in
hand to better understand the changes that are occurring. Also, our main objec-
tive was to create a self-assessment tool and therefore, there was no reason to
aggregate data into a single measure of farm sustainability. The one size fits all
equation does not exist in sustainability. Two farms with the same overall score
are not equally sustainable. The use of multiple indicators also enables the
identification of tradeoffs and synergies (National Research Council 2010).
Many ways can bring a system to be more sustainable and those different paths
are available when seeing individual indicator scores. The tradeoffs, without
being quantitative, can be viewed through these results by the farmers. As Con-
way (1994) mentioned, tradeoffs are most of the time qualitative. At a lower level,
the main objectives of evaluation are to assess the feasibility and impact of
alternative management practices (López-Ridaura et al. 2005). This self-assess-
ment attempted to identify where the challenges into the management practices
are. It could be seen as a managerial toolbox for the farmer.

3.3. Visual Integration for the Self-Assessment Tool

While tables give an overview of all components and indicator scores, the
radar diagrams present the same results but in two different ways. First, scores
by components (all components of the three dimensions of sustainability) are
presented against the leading group scores in the first radar diagram (Figure 2).
Figure 2). Then, if a lower score is identified through one component, farmers
can go to one of the three radar diagrams (one for each dimension of
sustainability) and see the detailed results for a specific dimension
(Figure 2b). Radars as visual integration have been seen more pragmatic to
farmers and useful; in one rapid overview the strengths and weaknesses of
the farm can be visualized. The relative weight of each component or indi-
cator on the radar facilitates identification of major challenges of sustainability
according to the experts who determined the weighting. The radar diagrams
enabled farmers to compare their scores with their region leading group
scores and can help them to understand the possible tradeoffs and synergies
between components or indicators. In Figure 2a, the components scores of
farm A for the three dimensions of sustainability were compared with the
leading group scores of region 1. Looking at the radar enables the farmer to
detect good performance for certain components that are beyond the score of
the leading group. The comparison with the leading group showed a ten-
dency of the curve for the overall radar. For example, while farmer A had a
low score for two components in the social dimension (quality of life and
social integration), the leading group also had a similar weaker curve without
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1039

a) Soil quality (20)

100,0
Entrepreneurship (30) Cropping practices (30)
80,0
Fertiliza tion management
Fa rm succession (30) 60,0
(30)
40,0

20,0 Fa rmla nd management

Social integration (15)
(20)
0,0

Technica l Mana gement

Quality of life (25)
(20)

Forage self-sufficiency
Economic viability (25)
(10)
Labor efficiency (15) Expense control (30)

Hea ding group results (25% top) Results from Farm A

b)
Production per cow
(8)
100
Forage self- 80 Milk from forages
sufficiency (10) (12)
60
40
20
Milk per worker (15) 0 Security margin (15)

Machinery expenses
Debt per hL (10)
per hL (10)

Operational expense
on gross income (20)

Heading group results (25% top) Results from Farm A

FIGURE 2 Radar diagrams presenting the results of the Farm A and the results of the leading
group for a) the component scores of the three dimensions of sustainability and b) the indicator
scores for the techno-economic dimension. Numbers in parentheses in the diagrams are the
maximum possible scores for the component (a) or for the indicator (b).

having the same low scores as of farm A. It can be a general tendency and
farmers have to work on these components. Again in Figure 2a, farmer A can
see good scores in the techno-economic dimension within the components:
technical management, expense control, labor efficiency and forage self-
sufficiency but had a lower score than the leading group in only one compo-
nent: economic viability. In Figure 2b, the radar presents the results from farm
1040 V. Bélanger et al.

A but for all individual indicators of the techno-economic dimension. Then the
economic viability component is viewed through the two indicators compris-
ing it: debt per hectoliter and operational expense on gross income. Farmer A
can see which one had the lowest score and in this case, it was the indicator
debt per hectolitre (Figure 2b). Even if radar diagrams do not reflect all
interrelations between the subsystems, they allow for the identification of
hotspots at the farm level (von wirén-Lehr 2001).
It is also possible to see that maximum score has been reached for some
components (Figure 2a) and for some indicators (Figure 2b). Achieving max-
imum score can be very attractive for farmers but it could also raise some
questions. For example, in Figure 2a, the maximum score for technical man-
agement was attained but the score for its cropping practices was lower than
the one for the leading group. Since cropping practices had a higher relative
weight than technical management, this could mean that farmer A was putting
more effort on the dairy herd than in the fields. Such examples should be
viewed as eye opening for farmers.
Another positive point of a visual integration is the shape of the area under
the curve. With a rapid overview, if the curve is shaped like a star on the radar, it
means that the farm is not in equilibrium. In agricultural management, to get
good economic results, it is better to be average in all subsystems than to be
high in one and low elsewhere (Levallois and Pellerin 2008). Maximizing one
component can come at the expense of the resilience for the overall system,
which in turn reduces overall sustainability (National Research Council 2010).
The resilience is defined by the capacity of a system to buffer shocks and
stresses (Pretty 2008). Häni (2006) mentioned that an optimal farm sustainabil-
ity situation will not be achieved by one individual maximum indicator but
more by a balance between the scores for all indicators. If we look back at
Figure 2a, farm A had a radar diagram with an acceptable shape (not a sharp star
shape) and moreover, the shape followed the one for the leading group mean-
ing that farm A may be on the right path to achieve more sustainable practices.
But it also enabled farm A to identify which components have a lower score and
the need to start working on them to improve farm sustainability. As Ikerd
(2006) mentioned, no objective quantifiable means exist for ensuring balance
and harmony among the economic, environmental and social dimensions of
sustainable farming and the three dimensions are inherently interrelated;
increasing one while the two others decrease is not good.

3.4. A Self-Assessment Tool for a Sustainable Dairy Farm

We demonstrated that our self-assessment tool can be a practical step for a
farmer to comprehend the sustainability of the farm. As Ripoll-Bosch et al. (2012)
mentioned, the classification of indicators in the traditional sustainability dimen-
sions could be more straightforward and understandable. Simplicity and user
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1041

friendliness have been central throughout the development of our tool. The
visual integration leaves freedom to farmers to view and understand the tradeoffs
between subsystems and decide what could be done to attain a more sustainable
farm. Without aggregating indicators into one single index, choices are clearer
since they have all the results in hand. Reflections can be made to set the main
priority between the three dimensions of sustainability. According to the results
from the end-user validation, the techno-economic dimension was the most
important for dairy farmers. It is also reported in the literature that economic
viability is often a precondition for several aspects of the social dimension as well
(Van Cauwerbergh, et al. 2007). In our view, in agreement with Bennett, Pang-
born, and Bywater (2010), the priority will refer to the internal farm context.
Farmers perceive sustainability as a division between internal factors they can
control and external factors that are difficult to control. As one farmer mentioned
during his interview, some choices can be made to enhance life quality without
being optimal economically. This demonstrates the intrinsic tradeoffs that occur
on each farm when decision-making has to be done to enhance farm resilience.
Sustainability in agricultural systems incorporates concepts of both resilience and
persistence (Pretty 2008). Bruges and Smith (2008) mentioned that the transition
towards a more sustainable agriculture is seen as a function of learning about
economic, social, and ecological complexities and uncertainties.

3.5. Toward a New Schematization of the Sustainable Farm

With self-assessment, a developing interest was to provide a new heuristic sche-
matization of a sustainable farm (Figure 3). Based on the definition and schema-
tization of Landais (1998) and Parent (2001), Figure 3 shows the four characteristics
(described in the second premise in section 2.1) but now in a different view.
Economic, social, and environmental dimensions are seen equally on a line but,
economic dimension is first (at the left of the figure) according to the results of our
sample. Secondly, in contrast to farmers’ response (as measured in this study), the
social dimension is central to farm sustainability. In fact, we need to separate the
score given by the farmers for the social dimension from its real importance
because this dimension is not well defined yet in all communities. The farmer is
the pillar of farming activities and if the farm succession is not present, there will no
longer be a farm. Moreover, the livability of a farm depends on the people on the
farm and their relation with the community. Even if the three dimensions are
weighted equal at the farm level, the impact of sustainability is on the short term
for the economic dimension, mid-term for the social dimension and at longer term
for the environmental dimension. The dash arrows represent possible tradeoffs
made by farmers to maintain a balance on their farms. In fact, tradeoffs are the
means by which farmers express their farm resilience in face of change. As Filson
and Pfeiffer (2004) mentioned, a sustainable system may be one that allows people
to choose their management style. A multicriteria analysis that could rank, select,
1042 V. Bélanger et al.

A sustainable farm

Economic Social Environment

Viable Transmissible Reproducible

Liveable

Community

FIGURE 3 New schematization of the sustainable farm (adapted from Landais 1998; Parent
2001).

and classify indicators and suggest to farmers the most important challenges to face
cannot replace the knowledge of the farmers of their own farm as well as their
awareness of what they will be able to do to recover from a difficult situation. By
the observation of their own scores in tables and radar diagrams, farmers will know
and set their priorities to enhance their farms. Some researchers work on resilience
to understand the dynamics of a farming system and to manage farm sustainability
(Darnhofer, Fairweather, and Moller 2010). This work must continue to better
integrate the concept of resilience in the sustainability assessment at the farm level.

4. CONCLUSION

This article outlines the construction and the application of DELTA, an integrated
indicator-based self-assessment tool for the evaluation of dairy farm sustainabil-
ity in Quebec, Canada. While several premises were presented in order to justify
our methodological decisions made for DELTA, the end-user validation results
enabled us to conclude that our indicators were consistent with the dairy farm
reality. Moreover, it confirmed that the presentation of all individual indicator
scores was necessary for the farmer’s understanding. The visual integration
using the radar diagrams was also part of this understanding. According to our
results from the end-user validation, the techno-economic dimension was the
most important for dairy farmers. Reflections have to continue on the main
priority among the three dimensions of sustainability in order to better define
the place of the social dimension. The presentation of four case studies facili-
tated the identification of major tradeoffs occurring in farm management to
further enhance farm resilience. However, the work on the link between resi-
lience and sustainability is still challenging for the operationalization of resilience
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1043

at the farm level. Therefore, further research on indicators to measure the

resilience of farmers and their farms’ system is needed to complete an indicator
set as a self-assessment or monitoring tool. Sustainability cannot be solved but
only managed. The self-diagnostic tool DELTA brought new insights into this
sustainability management at the farm level.

ACKNOWLEDGMENTS

The authors especially thank the farmers who decided to participate in the
follow up of this research project by filling out the questionnaire used for the
validation of the tool. This work has not been published elsewhere and has
not been submitted for publication elsewhere.

FUNDING

This work was supported by the Fonds québécois de la recherche sur la

nature et les technologies under Grant [2006-PR-105650] and by the Canadian
Dairy Commission and Novalait Inc. for the scholarship awarded to the Ph.D.
candidate.

REFERENCES

Bélanger, V., A. Vanasse, D. Parent, G. Allard, and D. Pellerin. 2012. Develop-

ment of agri-environmental indicators to assess dairy farm sustainability in
Quebec, Eastern Canada. Ecological Indicators 23:421–30. doi:10.1016/j.
ecolind.2012.04.027
Bell, S., and S. Morse. 2003. Measuring sustainability: Learning from doing. London,
UK: Earthscan.
Bennett, M. R., M. C. Pangborn, and A. C. Bywater. 2010. Perceptions of sustainability
of farmers with dairy support land. In Proceedings of the 4th Australasian Dairy
Science Symposium, 107–11. August 31–September 2. Christchurch, New Zeal-
and: Caxton Press.
Bezlepkina, I., P. Reidsma, S. Sieber, and K. Helming. 2011. Integrated assessment of
sustainability of agricultural systems and land use: Methods, tools and applica-
tions. Agricultural Systems 104(2):105–09. doi:10.1016/j.agsy.2010.11.002
Binder, C. R., G. Feola, and J. K. Steinberger. 2010. Considering the normative,
systemic and procedural dimensions in indicator-based sustainability assess-
ments in agriculture. Environmental Impact Assessment Review 30(2):71–81.
doi:10.1016/j.eiar.2009.06.002
Bockstaller, C., and P. Girardin. 2003. How to validate environmental indicators.
Agricultural Systems 76(2):639–53. doi:10.1016/S0308-521X(02)00053-7
1044 V. Bélanger et al.

Bockstaller, C., L. Guichard, D. Makowski, A. Aveline, P. Girardin, and S. Plantureux. 2008.

Agri-environmental indicators to assess cropping and farming systems. A review.
Agronomy for Sustainable Development 28(1):139–49. doi:10.1051/agro:2007052
Braat, L. 1991. The predictive meaning of sustainability indicators. In Search of
indicators of sustainable development, Eds. O. Kuilk and H. Verbruggen,
57–70. Dordrecht: Kluwer Academic.
Bruges, M., and W. Smith. 2008. Participatory approaches for sustainable agriculture:
A contradiction in terms? Agriculture and Human Values 25(1):13–23.
doi:10.1007/s10460-007-9058-0
Canada, S. 2012. Census of agriculture 2011. http://www29.statcan.gc.ca/ceag-
web/fra/transpose-var-transposer.action?geoId=0&selectedVarIds=11
(accessed November 8, 2013).
Charbonneau, E., P. Y. Chouinard, G. Allard, H. Lapierre, and D. Pellerin. 2006. Milk
from forage as affected by carbohydrate source and degradability with alfalfa
silage-based diets. Journal of Dairy Science 89:283–93. doi:10.3168/jds.S0022-
0302(06)72093-8
Conway, G. 1994. Sustainability in agricultural development: Trade-offs between
productivity, stability and equitability. Journal for Farming System Research-
Extension 4(2):1–14.
Dantsis, T., C. Douma, C. Giourga, A. Loumou, and E. A. Polychronaki. 2010. A
methodological approach to assess and compare the sustainability level of
agricultural plant production systems. Ecological Indicators 10(2):256–63.
doi:10.1016/j.ecolind.2009.05.007
Darnhofer, I., J. Fairweather, and H. Moller. 2010. Assessing a farm’s sustainability:
Insights from resilience thinking. International Journal of Agricultural Sustain-
ability 8(3):186–98. doi:10.3763/ijas.2010.0480
Filson, G. C., and W. C. Pfeiffer. 2004. Challenges awaiting the dairy industry as a
result of its management decision environment. In Intensive agriculture and
sustainability: A farming systems analysis, ed. G. C. Filson, 126–44. Vancouver,
BC, Canada: UBC Press.
Flyvbjerg, B. 2006. Five misunderstandings about case-study research. Qualitative
Inquiry 12(2):219–45. doi:10.1177/1077800405284363
Freebairn, D. M., and C. A. King. 2003. Reflections on collectively working toward
sustainability: Indicators for indicators! Australian Journal of Experimental Agri-
culture 43(3):223–38. doi:10.1071/EA00195
Girardin, P., C. Bockstaller, and H. M. G. van der Werf. 1999. Indicators: Tools to
evaluate the environmental impacts of farming systems. Journal of Sustainable
Agriculture 13(4):5–21. doi:10.1300/J064v13n04_03
Gómez-Limón, J., and G. Sanchez-Fernandez. 2010. Empirical evaluation of agricul-
tural sustainability using composite indicators. Ecological Economics 69
(5):1062–75. doi:10.1016/j.ecolecon.2009.11.027
Halberg, N., G. Verschuur, and G. Goodlass. 2005. Farm level environmental
indicators; are they useful?: An overview of green accounting systems for
European farms. Agriculture, Ecosystems & Environment 105(1–2):195–212.
doi:10.1016/j.agee.2004.04.003
Häni, F. 2006. Global agriculture in need of sustainability assessment. In Proceedings
and outputs of the first symposium of the International Forum on Assessing
 DELTA: Self-Assessment Tool for Dairy Farm Sustainability 1045

Sustainability in Agriculture (INFASA), eds. F. J. Häni, L. Pinter, and H. R.

Herren, 3–17. March 16, 2006. Winnipeg, MB, Canada: International Institute
for Sustainable Development.
Häni, F., F. Braga, A. Stämpfli, T. Keller, M. Fischer, and H. Porsche. 2003. RISE, a tool
for holistic sustainability assessment at the farm level. International Food and
Agribusiness Management Review 6(4):78–90.
Ikerd, J. 2006. Economic analysis and multiple impact valuation strategies. In
Developing and extending sustainable agriculture a new social contract, eds.
C. A. Francis, R. P. Poincelot, and G. W. Bird, 109–39. New York: Haworth
Press.
King, C., J. Gunton, D. Freebairn, J. Coutts, and I. Webb. 2000. The sustainability
indicator industry: Where to from here? A focus group study to explore the
potential of farmer participation in the development of indicators. Australian
Journal of Experimental Agriculture 40(4):631–42. doi:10.1071/EA99148
Landais, E. 1998. Agriculture durable: Les fondements d’un nouveau contrat social.
Courrier De L’environnement De L’ INRA 33:5–22.
Larochelle, D., 2011. Méthode d’évaluation de la durabilité technico-économique des
fermes laitières québécoises. (Thesis (MSc.)), Université Laval: Québec.
Levallois, R., and D. Pellerin. 2008. L’évaluation de la performance technico-
économique d’une entreprise laitière. In Les bovins laitiers, ed. collectif
d’auteurs, 7–16. Québec, QC, CA: CRAAQ.
López-Ridaura, S., H. V. Keulen, M. K. V. Ittersum, and P. A. Leffelaar. 2005. Multiscale
methodological framework to derive criteria and indicators for sustainability
evaluation of peasant natural resource management systems. Environment,
Development and Sustainability 7(1):51–69. doi:10.1007/s10668-003-6976-x
López-Ridaura, S., O. Masera, and M. Astier. 2002. Evaluating the sustainability of
complex socio-environmental systems. The MESMIS framework. Ecological Indi-
cators 2(1–2):135–48. doi:10.1016/S1470-160X(02)00043-2
Matthews, A., 2010. Dairy Stewardship Alliance. Sustainability indicators for dairy
farms-Final report. University of Vermont-Center for Sustainable Agriculture,
Burlington, VT.
Meul, M., F. Nevens, and D. Reheul. 2009. Validating sustainability indicators: Focus
on ecological aspects of flemish dairy farms. Ecological Indicators 9(2):284–95.
doi:10.1016/j.ecolind.2008.05.007
Meul, M., S. Van Passel, F. Nevens, J. Dessein, E. Rogge, A. Mulier, and A. Van Hauwer-
meiren. 2008. MOTIFS: A monitoring tool for integrated farm sustainability. Agron-
omy for Sustainable Development 28(2):321–32. doi:10.1051/agro:2008001
Mitchell, G., A. May, and A. McDonald. 1995. PICABUE: A methodological framework
for the development of indicators of sustainable development. International
Journal of Sustainable Development & World Ecology 2(2):104–23. doi:10.1080/
13504509509469893
National Research Council. 2010. Toward sustainable agricultural systems in the 21st
century. Washington, DC: The National Academies Press.
Pacini, C., A. Wossink, G. Giesen, C. Vazzana, and R. Huirne. 2003. Evaluation of
sustainability of organic, integrated and conventional farming systems: A farm
and field-scale analysis. Agriculture, Ecosystems & Environment 95(1):273–88.
doi:10.1016/S0167-8809(02)00091-9
1046 V. Bélanger et al.

Parent, D. 2001. D’une agriculture productiviste en rupture avec le territoire à une

agriculture durable complice du milieu rural. Teoros 20(2):22–25.
Pretty, J. 2008. Agricultural sustainability: Concepts, principles and evidence. Philoso-
phical Transactions of the Royal Society B: Biological Sciences 363(1491):447–65.
Rigby, D., P. Woodhouse, T. Young, and M. Burton. 2001. Constructing a farm level
indicator of sustainable agricultural practice. Ecological Economics 39(3):463–78.
doi:10.1016/S0921-8009(01)00245-2
Ripoll-Bosch, R., B. Díez-Unquera, R. Ruiz, D. Villalba, E. Molina, M. Joy, A. Olaizola,
and A. Bernués. 2012. An integrated sustainability assessment of mediterranean
sheep farms with different degrees of intensification. Agricultural Systems 105
(1):46–56. doi:10.1016/j.agsy.2011.10.003
Sadok, W., F. Angevin, J.-É. Bergez, C. Bockstaller, B. Colomb, L. Guichard, R. Reau,
and T. Doré. 2008. Ex ante assessment of the sustainability of alternative
cropping systems: Implications for using multi-criteria decision-aid methods. A
review. Agronomy for Sustainable Development 28(1):163–74. doi:10.1051/
agro:2007043
Sadok, W., F. Angevin, J.-E. Bergez, C. Bockstaller, B. Colomb, L. Guichard, R. Reau,
A. Messéan, and T. Doré. 2009. MASC, a qualitative multi-attribute decision
model for ex ante assessment of the sustainability of cropping systems. Agron-
omy for Sustainable Development 29(3):447–61. doi:10.1051/agro/2009006
Stake, R. E. 1995. The art of case study research. Thousand Oaks, CA: Sage.
Thivierge, M.-N., 2011. Développement d’un ensemble d’indicateurs pour l’évalua-
tion de la durabilité environnementale des fermes québécoises en grandes
cultures Master thesis, Université Laval, Québec.
Van Cauwenbergh, N., K. Biala, C. Bielders, V. Brouckaert, L. Franchois, V. G. Cidad,
M. Hermy, E. Mathijs, B. Muys, J. Reijnders, X. Sauvenier, J. Valckx, M. Vancloos-
ter, B. Van Der Veken, E. Wauters, and A. Peeters. 2007. SAFE - A hierarchical
framework for assessing the sustainability of agricultural systems. Agriculture,
Ecosystems & Environment 120(2–4):229–42. doi:10.1016/j.agee.2006.09.006
van der Werf, H. M. G., and C. Zimmer. 1998. An indicator of pesticide environmental
impact based on a fuzzy expert system. Chemosphere 36(10):2225–49.
doi:10.1016/S0045-6535(97)10194-1
Van Passel, S., and M. Meul. 2012. Multilevel and multi-user sustainability assessment
of farming systems. Environmental Impact Assessment Review 32(1):170–80.
doi:10.1016/j.eiar.2011.08.005
Vilain, L. 2008. La méthode IDEA, indicateurs de durabilité des exploitations agri-
coles, 3e édition, guide d’utilisation. Dijon, France: Educagri Éditions.
von wirén-Lehr, S. 2001. Sustainability in agriculture. An evaluation of principal
goal-oriented concepts to close the gap between theory and practice. Agricul-
ture, Ecosystems & Environment 84(2):115–29. doi:10.1016/S0167-8809(00)
00197-3
Yin, R. K. 2009. Case study research: Design and methods, 4th ed. Thousand Oaks,
CA: Sage.
Zhen, L., and J. K. Routray. 2003. Operational indicators for measuring agricultural
sustainability in developing countries. Environmental Management 32(1):34–46.
doi:10.1007/s00267-003-2881-1