Professional Documents
Culture Documents
Ejcn 2009126 A
Ejcn 2009126 A
Ejcn 2009126 A
& 2010 Macmillan Publishers Limited All rights reserved 0954-3007/10 $32.00
www.nature.com/ejcn
ORIGINAL ARTICLE
Validity and reliability testing of a short
questionnaire developed to assess consumers’ use,
understanding and perception of food labels
D Mackison1, WL Wrieden2 and AS Anderson1
1
Centre for Public Health Nutrition Research, Division of Clinical and Population Sciences and Education (CPSE), University of
Dundee, Ninewells Hospital and Medical School, Dundee, UK and 2School of Pharmacy and Life Sciences, The Robert Gordon
University, Aberdeen, UK
Background: Food labels may have an important function in communicating nutrition information and have considerable
potential to influence food choice and dietary behaviour.
Objectives: To assess the validity and reliability of a short (self-complete) questionnaire designed to measure consumers’ use
and understanding of food labels.
Methods: Questionnaire content was determined by gaps highlighted in a literature review of food labelling. Nineteen
questions (49 items) assessing frequency of label reading perceived importance of food labels, regularity of dining out, desire to
have nutrition information at specific catering outlets and ability to perform nutrition information tasks were formulated and
presented on four pages of A4. With the exception of two open-ended questions, all items were presented as closed (field box)
structures. Content validity, face validity, item analysis, repeat and internal reliability were assessed.
Results: Nutrition experts (26) completed detailed content validity assessment, resulting in high scores for appropriateness,
importance and phrasing of questions, although grammar and terminology changes were required. Face validity indicated that
the questionnaire was quick to complete (o15 min), easy to follow and comprehensible. Cronbach’s alpha scores (internal
reliability) for questions with multiple sections ranged from 0.72 to 0.91, indicating good internal consistency. Repeat reliability
testing showed Spearman’s correlation coefficients ranging from 0.51 to 0.97 (all Po0.001) showing high temporal stability.
Item Difficulty analysis indicated that questions (Section C only) were at an appropriate level (with P between 20 and 80% for all
items). Item discrimination analysis ranged from r ¼ 0.43 to 0.70, highlighting that items were suitable for inclusion.
Conclusions: This questionnaire is a suitable tool for assessing consumers’ use, understanding and perception of food labels.
European Journal of Clinical Nutrition (2010) 64, 210–217; doi:10.1038/ejcn.2009.126; published online 11 November 2009
Reliability
Results Ninety-seven participants, 61 females and 36 males, com-
pleted the questionnaire at time 1. Eighty-one participants,
Content validity 50 females and 31 males, completed the questionnaire at
Twenty-six nutritionists/dietitians responded to the e-mail time 2. The mean time in days between questionnaire
request for comments. Of the 26 nutrition professionals, completions was 7 days, with a minimum interval period
84.6% had experience of teaching or advising on food labels. of 6 days and a maximum interval period of 9 days.
With the exception of question 2 (‘Where do you read food Internal consistency of frequency of using specific label
labels?’), all questions scored highly for importance, appro- attributes (question 1), frequency of dining at specific
priateness and phrasing with mean scores above 8.0 for each catering outlets (question 17) and desire to obtain nutri-
question. Overall means for importance, appropriateness tional and additional information at catering outlets (ques-
and phrasing were 8.77 s.d.±1.41, 8.79 s.d.±1.34 and 8.84 tions 18 and 19) were assessed using Cronbach’s alpha
s.d.±1.20, respectively. coefficient. Given that the largest response was obtained at
Amendments to the questionnaire included grammatical time 1 (n ¼ 97) and participants had no earlier exposure to
changes and the addition of explanatory notes in question 1 the questionnaire at this point, time 1 data was used to assess
(with regards to allergen, nutrition and producer informa- Cronbach’s alpha values. Cronbach’s alpha coefficients were
tion), emphasis on the terms ‘read’ and ‘use’ in questions 1, 2 0.85, 0.75 and 0.91 for frequency of using specific label
and 4, and the addition of ‘workplace canteen’ to questions attributes, frequency of dining at specific catering outlets
15–17. Despite scoring poorly, question 2 (‘Where do you and desire to obtain nutritional and additional information
read food labels?’) was not removed as it was felt that this at catering outlets, respectively, showing good internal
question provided a valuable insight into label reading consistency.
behaviour (pre- and post-purchase). Repeat reliability testing was carried out on the data
obtained from the time 2 sample (n ¼ 81). Good temporal
stability was shown with Spearman’s correlation coefficients
Face validity ranging from 0.51 to 0.97, all significant at Po0.001.
Twenty adults from various socio-demographic and educa- Item Difficulty Index analysis was carried out using
tional backgrounds completed a short interview after questionnaires completed at time 1 (n ¼ 97). Five out of the
eight questions in Section C were answered correctly by Table 2 Spearman’s correlation coefficient for the final version of
480% of the respondents, indicating that they were too easy Section C
for inclusion in the final questionnaire (Table 1). On the Question N Test–retest
basis of this analysis, this section was revised and indepen- reliabilitya
dently underwent reliability testing.
Nutrition colleagues working within the research centre C7. Sugar in two servings 51 0.728
C8. Grams of fat in half the pack 51 0.779
were consulted during the revision of Section C. Reliability
C9. Servings in the product 51 0.936
testing of the revised Section C was conducted. Participants C10. Fat content in Product 1 51 0.570
received only Section C of the questionnaire and submitted C11. Fat content in Product 2 51 0.772
their post-code for matching purposes; no other demo- C12. Sugar content in Product 3 51 0.891
C13. Comparison between Products 2 and 3 51 0.790
graphic data was collected. Seventy participants completed
C14. Lowest saturated fat content 51 0.795
Section C at time 1 and 51 participants completed the C15. Losing weight 51 0.814
questionnaire at time 2 (with the average completion time C16. Reducing salt intake 51 0.616
between questionnaires, 7 days). a
Significant at Po0.001.
Internal consistency was assessed using Cronbach’s alpha
on data obtained during time 1 (n ¼ 70). For ability to make
judgements on nutrition content from available nutrition
information (questions 10–16), the Cronbach’s alpha was altered (from 16–25 to 18–25) to remain consistent with the
0.72, indicating good internal consistency. sample group used.
Repeat reliability of Section C was carried out using data Two additional questions were added to the questionnaire.
obtained for the 51 participants who completed Section C at Both questions were included in Section E of the question-
both time 1 and 2. Spearman’s correlation coefficients naire and assessed the participant’s motivation to eat a
ranged from 0.57 to 0.94 and were statistically significant healthy diet (‘How motivated are you to eat a ‘healthy’ diet?)
at Po0.001, indicating good temporal stability (Table 2). and their own perceived knowledge of a healthy eating
The Item Difficulty Index was re-assessed for the revised (‘How would you rate your own knowledge of ‘healthy
Section C. The P-value’s highlighted moderate scores for the eating’ on a scale of 1–10? (1 ¼ no knowledge and 10 ¼ very
new questions indicating that they were not as easy as the knowledgeable)’). The final version of the questionnaire
initial draft and appropriate for inclusion in the final (excluding Section E—the demographic items) is shown in
questionnaire (Table 3). Appendix Table 1.
With regards to Item Discrimination Index analysis, all
items in Section C achieved correlations above 0.20 (Kline,
1986), with r ranging from 0.43 to 0.70 (Table 3), indicating
that these items were suitable for inclusion in the final Discussion
questionnaire.
The questionnaire development process involved a literature
search, reviewing the findings from existing literature and
Final adjustments highlighting any gaps in the current research. The review
After the reliability and validity assessment, four further highlighted the limited availability of studies reporting the
alterations were made to the questionnaire. One amendment development of validated tools used to measure consumers’
was made to question 1 in Section A, the tick box was food label reading habits. To our knowledge, this is the first
removed from ‘Other please specify’ (as the majority of study to report the validation and reliability testing of a
participants omitted this question, despite earlier font and short questionnaire to assess consumers’ use, understanding
formatting alterations). The age range (question 23) was and perception of food labels.
In the development of this instrument, special considera- between survey contact; therefore, no changes would be
tion was given to developing a questionnaire, which could expected unless enquiry of the subject area was prompted
be completed independently, without assistance. It was, by the initial questionnaire exposure. The Spearman’s
therefore, important that the readability level of the correlation coefficients (0.51 to 0.97) were statistically
questionnaire was pitched appropriately. Care was taken to significant at Po0.001, indicating strong reliability of these
avoid long sentences, complex terminology, acronyms or items assessed.
abbreviations, double-barrelled and leading questions. Although the optimum value for Cronbach’s alpha varies
It is recommended that independently administered throughout the literature (DeVon et al., 2007), a value 40.70
(self-completion) questionnaires are short in length and are is the most commonly accepted figure (Bland and Altman,
composed mostly of closed structure questions (McColl et al., 1997). In this study, all values were 40.70 indicating high
2001). Postal questionnaires short in length (Edwards et al., levels of internal consistency in this instrument.
2002) and containing o1000 words (Jepson et al., 2005) are The majority of the participants used in the reliability and
more likely to yield a high response rate. Care was taken to validity testing of this questionnaire were both White and
ensure that the questionnaire was short in length (only four Scottish. This is recognized as a limitation of this study, as we
sides of A4), contained o1000 words and comprised mainly cannot presume that all ethnic groups and nationalities
closed end questions (47 out of 49 items). The response would use nutrition labels in the same manner, nor interpret
categories in the close structure questions were considered this questionnaire in the same way. Future feasibility testing
mutually exclusive and exhaustive after the literature review in different geographic areas or with alternative ethnic
of the topic area and the considerable effort used during the groups should be conducted before using this tool as a guide
face and content validity process. to inform changes in policy.
Various other formatting characteristics were consciously A further limitation of this study is the use of closed
applied in the questionnaire construction. Using a booklet questions when assessing consumers’ attitudes to nutrition
format with double-sided printing, vertical formats for information in catering establishments. Closed questions are
closed structure responses, a font size of 10 points and a not considered suitable when investigating topics we have
high text/background contrast were all incorporated into the limited knowledge of (Rattray and Jones, 2007). Although
instrument layout, in line with best practice recommenda- accepting the possibility that information yielded from this
tions for questionnaire design (McColl et al., 2001). instrument on nutrition information in catering establish-
Demographic questions were purposively inserted at the ments may not be representative of the entire population, it
end of the questionnaire, as it has been suggested that these is notable that robust content and face validity were
questions can be considered threatening (Sudman and performed in the development of this questionnaire, and
Bradburn, 1982) or boring (Rattray and Jones, 2007). The future qualitative studies in the topic area will expand on the
demographic questions included in this questionnaire were findings derived from this instrument.
adapted from existing tools, for example The 2001 Census, The questionnaire developed in this study was intended to
The Food Standards Agency Consumer Attitudes Survey and be administered by post, nationwide throughout the UK.
The Family and Children’s Study. Further research should involve examining other question-
Although the questionnaire underwent vigorous reliability naire administration methods in various settings. For
testing, limitations may still remain in the reliability example, administering the questionnaire in a supermarket
methodology used. Repeat reliability testing (test–retest) or retail setting may provide valuable information when
presumes that exposure to the questionnaire at time 1 does comparing the choices and decisions consumers make
not trigger direct investigation of the topic area by the within their own home environment and in a busy retail
respondent. In this study, no intervention took place or catering setting.