REPORT. Evaluation and Design of Nutrition Knowledge Measures KATHRYN PARMENTER AND JANE WARDLE ICRF Health Behaviour Unit, Department of Epidemiology & Public Health, University College London, London, United Kingdom ABSTRACT This article sims to help health professionals, who do not have a knowledge of survey design to evaluate and design nutrition knowledge measures. Historically, these mea- sures have been poorly designed and therefore may not assess nutrition knowledge accurately, with the consequence that the results may have liede meaning. This paper highlights the fac- tots to look for in existing measores of ifa suitable measure can- not be found, shows how to modify of design one thae will ive useful and interpretable results. General design concepts are cov- ered including the generation of items, the use ofa pilor study and item analyses, and assessment of validity and reliability as, ‘well as particular issues posed in the nutrition field, such as the interpretability of food terms, Researchers are advised co use an existing measure, or modify an existing measure if possible, before developing a new one. Whether evaluating or develop- ingea measure the importance of the content, structure, and sta- tistical evaluation is discussed, Collaboration between nutei= tional and behavioral scientists is recommended to provide complementary areas of expertise in the evaluation/develop- ment of nutrition knowledge measures, (INE 32:269-277, 2000) INTRODUCTION ‘Since the unequivocal evidence that dietary behavior is linked to illness and risk of chronic diseases, there has been. ‘much interest from health researchers and practitioners in the measurement of nutrition knowledge asa possible mediator of behavior." Seudies measuring nutrition knowledge typ- ically focus on a particular population (e.g, children,* the elderly,” or medical inpatients?) and/or knowledge about a particular food or nutrient (e.g, fit or fiber’). Inevitably, these instruments have limited application outside the special field for which they were designed. Furthermore, investigators ‘who are interested in looking at nutrition knowledge typi~ cally have expertise in nutrition and less experience in psy- ‘Thisresech was supported by gran et the Biotechnol an Bile Se ces Reech Counc ‘Addie for eonespondene: Jane Well Ph.D. HORE Hea Hehaioue Unit Deparment of Epemilogy & Public Heak, Unser College London 2-16 “orig Pe London, WCE BT, UR: Te 440) 720 62:4 (120 7413 284K; Eom j wade uc ak {S20 SOCIETY FOR NUTRITION EDUCATION 269 chometric issues such as validity and reliability. These issues are often described by and for experts in psychological test- ing and may not be easily understood by those unfamiliar with these concepts. The consequence of measuring mutt tion knowledge with a scale of unknown validity or relabi ity is that itis impossible to know whether itis actually me: suring what it claims to measure (:e., nutrition knowledge) Ic is common for investigators wishing to study an aspect of nutrition knowledge to write their own questionnaires. ‘The advantage of this is that the questions asked should be ‘exactly relevant to that particular study. Given that existing nutrition knowledge questionnaires are usually written for such specific purposes this may be the best or only option for a researcher working in a new area or with a new sector of the community. The disadvantages of developing 2 ques- tionnaire from scratch are (1) the time involved in doing so; (2) the possible limitations in expertise and resources, result ing in a poorly designed questionnaire; nd (3) results that are difficult to compare with those obtained from past studies Different outcomes may reflect real differences in knowledge between samples or just differences in the measures used. ‘This paper advises that an investigator wishing to measure nutrition knowledge should first see whether a suitable, valid, and reliable measure has already been developed that could be used, although, as discussed later, this may not pre- clade the necessity of carrying out further validation checks. Alternatively, an existing measure could be modified to suit particular needs. This paper describes the steps involved in developing a measure of nutrition knowledge that can either be used as a guide for evaluating existing measures or 0 develop a new measure should this be necessary. We then dis- cuss some of the issues involved in modifying an existing STEPS INVOLVED IN DESIGNING A MEASURE OF NUTRITION KNOWLEDGE Designing a new measure should be resorted to only if an existing inserumnent cannot be found. There are plenty of textbooks on this subject," and many aspects of question naire design ate easily comprchensible and so will be refer- enced. The aim ofthis setion isto outline the diferent stages in questionnaire design, focusing on those aspects that are not270 Parmenter and Wardle NUTRITION KNOWLEDGE MEASURES so emily understood and pointing out particular issues con= cerning nutrition knowledge that are not covered in general textbooks. This will be usefal in evaluating existing measures and designing new ones. Questionnaires may be sel administered or administered by an interviewer face to fice or over the telephone." This paper will primarily be concerned with self-administered ‘questionnaires, as these pose the most challenges. or exam- ple, without an incerviewer, the instructions, layout, and for~ mat of the questionnaire become extremely important to motivate recipients to complete the questionnaire and com plete it accurately: The items abo have to be absolutely clear and unambiguous. Defining the scope of the measure. First, it is neces sary t0 decide which areas of knowledge ate to be assessed. It is important to distinguish between “knowledge” and “beliefs” These two constructs are often confused. Some- times, investigators will use the two terms interchangeably in their studies. For example, Whichelow"® referred to the assessment of beliefs, but asked respondents whether each of 10 common foods contained dietary fiber. It is important to be clear that itis knowledge that is being measured and not knowledge and/or beliefs as a confused measure will pro- duce confused results. distinction can be made as follows: knowledge can be conceptualized as factually true" and knowledge items can be judged as correct or incorrect (ovithin the constraints of present science). Beliefs, on the other hand, may not be factually true (e.., the belief that all food additives cause medical disonders)."* Commonly, stud= ies looking a nutrition knowledge ask questions concerning, the current dietary recommendations and use dichotomous ‘or multiple-choice items. In contrast belief are usually stu ied within the framework of social cognition models such as the Theory of Reasoned Action.'* Within this model, belief typically relate to the healthiness or taste of foods and are often measured using a scale ranging fom strongly agree «0 strongly disagece."" Ie is important to be precise in defining the areas of knowledge to be measured. It is helpful to lst these and mark. hhow many items you write for each one to ensure adequate coverage. You may wish to decide at this point how many items to include for each area Generation of items. Generating an item pool is con- cerned with deciding which items should be included in the instrument (-e., whether the items cover all aspects of what, is being measured and how the questions are asked or con- seructed).The issue of validity is discussed in detail below, but itis partly determined by the choice of items. Questionnaire developers should state which areas of nutrition they have chosen to include in their instrument and why. Nevertheless, as this is a subjective judgment, potential users need to check, whether they agree that the material chosen is a fair repre~ sentation of the area under study. For example, despite a lengthy and impressive explanation of how and why items ‘were chosen to be included in an instrument designed t© measure nutrition knowledge in patients with eating disor dots” items included “a half hour of cross-country skiing (6 km) burns 450 keal"“most of the aromatics are fa sola- ble;”and100 g of peas contain 10 g of carbohydrates,” none Cf which has obvious relevance to actual eating behavior Ac the first stage, many items as possible should be gen~ erated, many more than will be used in the final measure.As a rule of thumb, twice as many items are needed for the pilot study as forthe final version, so that poor items ean be elim- inated. thorough review of the literature should be used to guide this process, drawing on existing questionnaires and ther relevant publications such as health education leaflets. I is important to keep the target population in mind at this stage: for example, if you are writing items for pregnant ‘women, look at the leaflets they are given regarding nutri- tional advice and ensure chat the source used is appropriate for the respondents of the questionnaire. If your question naire is forthe general public, use the guidelines written for the public (eg,, in the UK.,a Health Education Authority Jeaffet* would be more appropriate than a report aimed at government policy makers). Other sources may be used to inform this process, such as needs assessments or qualitative research conducted with the target population, ‘There are several types of items used in questionnaires. The first choice is whether items should be open or closed." If chey are to be closed, knowledge items may be alternate choice, with two responses to choose from (crue/false, yes/no, agree/ disagree), or multiple choice, which consists of a statement or question and a list of possible ‘options with one correct answer. Iti usually clear which type to use, For example, it may be difficult to think of enough plausible distracter options for a multiple-choice-style ques tion. Ifivis not clear, it may help t0 try to construct an item using all methods, and the most appropriate one usually becomes obvious. Make sure chat alternate-choice items ean be classified without doubt (e.g.,28 either yes or no),and for ‘multiple-choice items, make sure chat there is only one eor- rect response with credible distracter options. This ean some times be a challenge in the area of nutrition knowledge where precise dietary recommendations may vary between ‘organizations (c.g, the discrepancy between recommended chily fruit and vegetable intake between the Health Educa~ tion Authority” and the government Department of Health” im the UK.) Common problems in existing measures, which therefore risk inaccurate assessment of knowledge, are ambiguous wording and poor interpretability, inadequate control of ‘guessing, and poor instructions and unclear format, Each of these will be addressed in the context of designing a nutri- tion knowledge measure Ioterpretability can be a problem for measures in general ‘To minimize misinterpretation, questions should not be ambiguous, should not be negatively phrased (because they are difficult to understand), and should not be double~ barrelled (i.e.,asking two questions in one). Examples of poorJournal of Nutrition Eduestion Volume 32 Number 5 items include “Semi-skimmed milk contains very little fat” (che answer is dependent on respondents’ interpretation of “very litte”),"*Cholesterol in food is not the most important dietary factor influencing blood cholesterol” (the negative phrasing is dificult o understand), and “There is more pro- tein and calcium in a ghss of whole milk than a glass of skimmed milk” (should be two separate questions, one on calcium and one on protein). There ate also particular problems associated with mutri- tional terms. For example, Sobal and Cassidy found that people associated fiber with muscle fibers in meat, and that 40% of respondents believed that fiber and roughage were different concepts. Many foods may be ambiguous. For example, “beans” may refer to baked beans, green beans, broad beans, tinned or dried mixed beans, and 50 on. Clari- fication of foods may be necessary. If using the term “oily fish;"an example would be wise (¢.g., mackerel) Precision is also necessary; for example, “bread” may refer to white, brown, granary; etc. Ina pilot study, Parmenter and Wardle” found that respondents did not answer an item concerning rice because it did noe stipulate whether it was white or ‘brown, even though it made no difference to the answer. The difficulty in constructing nutrition knowledge items is illus- tated very well in the study by Cremer and Kessler Respondents’ answers to their seemingly unambiguous item “Which foods (ftom a lst of eight) were high in fat?” sug- _gested that some respondents interpreted the item as"which foods were fattening?” Guessing is a hazard in questionnaire design, and there is no easy solution. Some respondents will guess and others will leave the item blank when they are not sure of the answer, “don't know” or “not sure” category may be included in the response choices to discourage guessing. There are, how- ever, potential pitfalls of including such a category. First, respondents who know the right answer but pethaps are not ‘confident that it is right may select the “don't know/not, sure” response. Second, respondents who could work out the right answer given a little thought may select “don't know/not sure” as the easier and quicker option. The advan- tages of having a “don’t know/not sure” category, however, are that it does give respondents the chance co make a response, and ie reduces guessing. Furthermore, the option “not sure” rather than “don’t know” may appear less neya~ tive and also help to reduce guessing, Similarly, where appro priate, it may be a good idea to use the response format “agree/disagree/not sure” rather than “true false," again to try and reduce the threatening nature of knowledge ques- tions Instructions can set the tone of the questionnaire, so itis important to invest time in writing clea, simple insructions, making sure that the questionnaire docs not intimidate respondents (e.g., do not call the questio Specifically, in a nutrition knowledge questionnaire, the introduction could attempt to get respondents “on side” by telling them that their answers would help the experts iden- tify the topics in mutrition that people were finding the most September # October 2000, 271 confusing.** This may also help to elicit honest answers, Fur- thermore, it may reduce the possibility of respondents in postal surveys asking other people or looking up the answers. ‘The instructions should be as brief as posible, as respondents, will lose interest in and cease to read lengthy ones, ‘The layout of the questionnaire should be attractive and casy to follow. If more than one area of knowledge is to be assessed, Keep these areas separate since a randomly ordered questionnaire will be disjointed and confusing to respon- dents, Inroductory comments before each new section will also help orient respondents’ attention, but, again, these should be bref. If different types of items are used itis prob ably best to geoup them into open, dichotomous, and mul- tiple choice to make chem easier for respondents co complete ‘The order of questions needs to be considered. Begin with casy items and make sure that an item early in the question- naire does not answer an item later on. For example,"Do you think these are fats?... polyunsaturates, etc.” could be answered from the item “Are polyunsaturated fats mainly found in. ..” Ie may be better to include only one of these jtemvs as respondents may return tothe first question and cor rect it ifnecessary. Any demographic questions may be bet- ter placed at the end of the survey instrument as some respondents clsike answering these questions, seeing them as intrusive or threatening. By placing them at the end, not only will recipients be more likely to complete the instrument, but any negative feelings they may have at being asked these questions will not interfere with their performance on the rest of che knowledge items. For further information on the format and onder of questionnaires, see Dillman’ and Sud- rman and Bradburn.” ‘There are addtional specific issues to look out for in gues~ tions concerning natrition, One reason for measuring nutri~ sion knowledge is to establish the relationship between knowledge and dietary behavior, so itis important to check thatthe items are relevant to dietary behavior. Compare the following ewo items included in questionnaires developed to ‘measure nutrition knowledge in the general public:"One of the hormones involved in controlling bunger is adotrophic, cholecystokinin, angiotensin, or adrenocorticotrophic"™ and “If you wanted to reduce your risk of heart disease, which of the following would be best «0 choose? Cheddar cheese, cottage cheese, Edam, of Silton?”” The first item is not rl evant to dietary behavior and inappropriate to include in a measure assessing general nutrition knowledge. In contrast, the second item concerns nutrition knowledge and the real issue of dietary choice. Nutrition questions should also have an appropriate level of dificulty (xe., not too easy and not t00 difficult). Resni- cow and Reinhard, for example, asked children as young as 5 whether various foods contained cholesterol and fiber. The children’s responses suggested that if they thought the food. od," it contained fiber: if they thought the food was “bad,” it contained cholesterol. The children were respond- ing using a different interpretation to the one intended because the questions were too difficult for ths age group to272 Parmenter and Wanlle/NUTRITION KNOWLEDGE MEASURES understand, By contrast, Anliker et al used role play to check that children had understood the questions before they were answered, ‘Another problem with asessing nutrition knowledge is that not everyone may agree with the current recommenda tions or be able to carry them ont for personal health rea sons. Ifa question reads"'you should eat les fit. .agree/dis- agree,” respondents may choose “disagree” because they personally should not (e.g, because they are underweight or are already on a very low-fat diet). There is no way of know= ing, whether questions worded in this way elicit valid answers {ie., based on their general nutrition knowledge rather than. based on their personal nutritional requirements). To help remove these risks, itis wise to word items inthe third per son. For example,"Do you think health experts recommend that people should be eating more, the same amount, oF less of these foods. . vegetables, sugary foods, ete” Is also pos- sible chat respondents will answer according to their own likes and dislikes, perhaps picking their favorite food rather ‘han the healthiest one in items asking about the best dietary choices. Be certain that you know how you are going to score your questionnaire. This is particularly important with open- ‘ended questions. For example, ifone of the correct responses ‘you were looking for to the item "What diseases or health problems do you think are related to fat intake?” was heare disease, other possible answers could include clogged arter- ies, hardening arteries, blood clots, and arteriosclerosis. You need to be sure which responses are going to be marked as correct. The answers collected from a pilot study will provide a fairly exhaustive list of possible responses. Finally.a measure is typically scored with a single summed score, but this may not be appropriate.A single score assumes that a single con- struct is being measured, but nutrition knowledge measures often cover a variety of topics, and knowledge of one topic does not necessarily mean knowledge of another. For exam- ple, knowledge of food groups might be low, whereas knowl- edge of diet-disease relationships may be high. One total Score in this ease would give an inaccurate assessment and may fail to predict specific dietary behaviors. A score is, needed for each topic being measured. Before piloting your questionnaire, organize a panel of independent “experts” and ask them to review the first draft, ooking at both content and item construction. It is a good idea to give them a checklist of what to look for (e-g.,ade- quate coverage of the subject area, layout, item ambiguity, item comprehension, ec.) For a nutrition knowledge mea- sure, itis a good idea to have a mixture of dietitians and sue- vey design experts. Dietitians or nutritionists will have the expertise in nutritional matters so will be able to advise on the representativeness of the items as well as points of nutri- tional ambiguity and accuracy. Survey design experts (6.8. psychologists) will be able to advise on the item construc tion and response formats. Your panel may find items con- fusing, irelevant, or controversial. As a result of this exercise, you may discard, modify, or add new items. Ideally, after you have made these changes, you should give the second version ‘of your questionnaire tothe panel again, fora second review. ‘You may find further minor changes are necessary Pilot study and item analyses. This is the process by ‘which the initially large number of potential items is culled and the best items are selected for the final questionnaire. A pilot study involves having the items completed by a sample ‘of people who are similar to those for whom the measure is intended. The pilot sample should be as large as possible but should be at least one greater than the number of items in the measure.” Analyses should then be carried out on the ‘completed items in order to select the best ones forthe final version. Such analyses may include measurements of item facility, item discrimination, and homogeneity: These will be ‘explained in turn, although many pilot studies may not, in fact, include them all, All analyses can be performed easily ‘with a statistical package such as SPSS. Ifyou are not famil- iar with a statistical package, it may be possible to enlist the help of someone from your expert panel who is. Analyzing item facility (or item difficulty) indicates the extent co which respondents answer an item in the same way. Clearly if all of the responslents answer an item either cor= rectly or incorrectly, this item is not capable of discriminat- ing among respondents. Developers will sometimes give the results of this analysis in terms of the “facility (or difficulty) index."This index should lie between two values.The values ‘most often used are 0.2, ess than which the item is too dif- ficult, and 0.8, more than which the item is too easy?! ‘These indices may also be given as a percentage of respon- dents giving the correct answer (i.e. 20% and 80%), Item discrimination isthe ability of an individual item to discriminate between those who do well on the measure and those who do not. IF high-scoring respondents tend to get a particular item wrong, whereas low-scoring respondents tend to get che same item right, the item is clearly not behav- ing as it howd. The results of an item discrimination analy- sis are given as a Pearson’s correlation, as itis assessed by correlating each item with the total score. Experts say thatthe ‘minimum item-total correlations should be 0.2" or 0.3.* ‘Those items not meeting these criteria should be discarded Homogeneity refers to the extent to which all ofthe items in a scale are measuring different aspects of the same auuribute. For example, each item in a measure testing caffeine knowledge should relate to caffeine, as in the following items taken from a 19-item scale: “eaffeine is a stimulant,” ‘caffeine is found in chocolate,” and “colas are the only car- bbonated beverages that contain caffeine” Homogeneity is sometimes referred to as internal consistency and sometimes, confusingly as internal reliability. Some questionnaires have a number of different sections, each measuring a different aspect of knowledge (e.g. dietary recommendations and dict-discase relationships). If this is the case, measures of homogeneity should be applied to the individual sections as cach one is asessing a different area of nutrition knowledge. Homogeneity is usually asessed using the Kuder-Richard~Journal of Netrtion Education Volume 32 Number § son formula 20" for bipolar items and Cronbach’ coefficient alpha for items with more than two response options. Developers may not state that they have assessed the homo- geneity or internal consistency oftheir measure but may sim~ ply give the results (e.g., KR. 20 = 0,82, 0 = 0.84).The rec- ommended minimum requirement for homogeneity/ internal consistency is 0.7" or 0.8. The discrepancy in the minimum requirement for these analyses is not something to worry about for the following, reason. There isan argument for considering these indices as, a guide rather than absolute. For example, the item facility indices of 0.2 and 0.8 advocate that if fewer than 20% of responclents answer an item correctly itis too difficult, and if more than 80% of respondents answer an item correctly it, is too easy: However, there may be times when itis consid ered important to find out what proportion of the popula~ sion knows a particular piece of knowledge regardless of ‘whether the item is too easy or too difficult. For instance, the dietary recommendation to cat less fat may be known by more than 80% of the population, but as a core aspect of dietary knowledge it could be considered inappropriate not to include it in a nutrition knowledge measure. This trade~ off between content validity and more statistical criteria comes down to the discretion ofthe developers of measures, and there are no hard and fast rules. If multiple-choice items have been used, the distracter “options should also be checked to ensure their credibility and therefore tility. Distracters should look as though they could be right. Each distracter option should ideally be endorsed by a similar proportion of respondents who do not choose the correct option. The examination of distracter options is rarely carried out, ori ts, may not be reported. It is there- fore a good idea for potential users to check the distracter items themselves and at least judge whether they are plausi- ble. Ifthey are not, respondents may be choosing the correct option because the other choices are so obviously incorrect, and not because they know the correct one to be true. Items can, and should, be analyzed qualitatively in addi tion to the quantitative (statistical) analyses described above. Qualitative analysis looks at those issues that should have been covered when the items were generated initially (see the section on generation of items). These issues cover how rep- resentative the items are ofthe area under investigation, how clear the format of the questionnaire is, how clear the instructions are, the interpretability of the items, and their response formats. The pilot study is a good opportunity to observe diffi~ culties respondents have in answering any of the items. If the questionnaire is self-administered with an interviewer present, then respondents can be asked co complete the items out loud. Any misinterpretation of the items will then be heard, and respondents will be more likely to ask the interviewer ifan item is unclear. Otherwise, comments may be made on the questionnaire. You may also find that 2 particular item is consistently not answered, which may bbe because of the layout of your items. These comments, September * October 2000 273, and observations are valuable and should be collected and acted on, Reliability. The assessment of reliability is intended to ‘establish thatthe questionnaire is measuring something in a reproducible and consistent way: 1e can be evaluated by either the correlation with parallel measures (which is rarely, if ever, practiced as two very similar measures have «0 be developed) or the test-retest method. Some developers eval- uate reliability using, he split-half method, but this method is based on a score obtained on only one occasion, whereas reliability is che extent to which results are consistent over time. Test-retest reliability involves administering the same measure to the same respondents under the same conditions ‘on two separate occasions and correlating the scores. The trick is to select an appropriate time interval: too long and things may have changed, too short and respondents may remember theie first responses. The recommended time interval varies from 2 t0 14 days" to 3 months." The inter- val should be guided by how much change might be ‘expected in the interim. As quite rapid change may occur in nutrition knowledge because of the media, for example, a shorter interval between the two administrations may be ‘more appropriate. Pearson's correlation between scores atthe two sittings should be atleast 0.7. Validity. Validity is quite often not understood or misun= derstood. Validity simply concerns the extent to which a ‘questionnaire measures what it claims to measure. So is a nutrition knowledge questionnaire actually measoring nutri- tion knowledge? For example, a measure assessing general nutrition knowledge in HIV-infected patients included items asking how many vitamin A and D supplements the respon= dents were taking. These items are clearly not measuring nutrition knowledge. Validity is described within a three category classification of content validity criterion-related validity, and construct validity.” Each of chese will be cexphined in turn, Content validity, as already mentioned, concerns the rep~ resentativeness of the items, or how well the items sample the ‘whole subject under investigation, and the way in which the items are written." Ie is buile into a questionnaire during its development, and although there is no statistical test asses, itsevidence of content validity can be accumulated in the fol- lowing waysrby a detailed description and justification of the content; by careful construction of items, response formats, and layout; and by independent reviews from a panel of ‘experts on both content and interpretability. Construct validity concerns whether respondents’ scores provide a good measure ofa specific construct (or eoncepe)." Unlike physica attributes such as height, which are directly observable, abstrace attributes, or constructs, represent ideas constructed by scientists to describe or explain behavior. Nutrition knowledge has been described as “a scientific construct chat nutrition educators have created to represent individuals’ cognitive processes related to information about274 Parmenter and Wardle/NUTRITION KNOWLEDGE MEASURES food and nutrition."*” One of the easiest ways €0 assess con= struct validity isto give the measure to two groups, one of which is known to have good nutrition knowledge (e.g.. dietitians) and the other not.This is sometimes called “vali dation by extreme groups." The former group should score significantly higher, which can be ascertained statistically using an independent samples t-test Some demogeaphic characteristics are predictive of knowing more about nucti- tion: it has been found that higher scores are achieved among women," higher socioeconomic classes,” and those with ‘more educational qualifications." so the two comparison ‘groups for establishing criterion validity should be matched as faras possible in these respects. Another way to asses con struct validity isto show chat scores have changed following an educational program. However, a problem with this type ‘of study is that if scores do not change, the problem could lie notin the sensitivity of che questionnaire but in the pro gram. Criterion validity is the correlation of a newly developed questionnaire with another questionnaire measuring the same general construct—ideally one that has been used and accepted in the field ("2 gold standard”). This type of val= idation is rarely appropriate for nutrition knowledge mea sures because the rationale for criterion validity isthe need to replace one measure with a shorter, cheaper, oF less inva sive one and there is no nutrition knowledge measure of wide use and acceptance that can be used as the criterion Nevertheless it could be relevant in developing a shortened version of a longer measure. ‘A fourth aspect of validity, ahhough not in the technical sense, i face validity, which refers to how relevant the igems appear to be to the respondents." The advantage of high face validity is that it gives respondents confidence in the per~ ceived effectiveness of the questionnaire and thereby increases their motivation and accuracy. However, the disad- ‘vantage is that, by definition, subjects can guess what a face valid questionnaire is measuring and such a measure could therefore potentially induce faking or socially desirable answers, Ifa nutrition knowledge questionnaire did not have hgh face validity it would be questionable whether the mea~ sure had content validity, and the fact that respondents know ‘he items are measuring nutrition knowledge does not afleet their ability to answer them correctly Ieis hoped that the above paragraphs will ot only be use~ ful asa guide to designing a questionnaire from scratch bue will also act as a glosary to help nutrition investigators understand what developers have or have not included in the development of their measures. It should also provide a benchmark to determine how well a questionnaire has been thought out and designed. However, a word of caution. The quality of a measure is dependent both on whether it has met certain statistical criteria and on its content. in terms of item. inclusion and item construction. Its perhaps easier to judge the qualicy ofa measure by is statistical achievements, which are quantifiable, than on its item content and construction, which are qualitative, and to some extent subjective, assess- ments, However, the importance of content vali be overstated, and a measure cannot be judged by its statis tical achievements alone, Some measures have had extensive quantitative analyses performed on them," but the items are sill not entirely relevant to both dietary behavior and the target population, Conversely. few statistical analyses were cattied out during the development of other measures but the content was described in detail and the items have been well constructed and are relevant to dietary behavior. I i therefore important that the measure is checked for content validity and the items for interpretability so thatthe measure {snot judged solely on its statistical achiovements. Existing questionnaires such as those just mentioned!" vwith well-constructed items bue few statistical analyses, are more than likely to be valid and reliable, and an assessment of these properties would be an easier task than to design a questionnaire from scratch. Or it may be the case that an existing questionnaire is found but is either going to be used oon a different population than the one for which ie was developed or it needs some modifications (e-g..adding some extra questions or making changes to the wording). The next section will look at modifying an existing questionnaite. MODIFYING AN EXISTING MEASURE Icis imporcanc to realize that a measure is not valid for every situation: the issue of validity arises with each new use of the measure. Although an instrument may have had a validity study carried out during its development, ifa potential user plans co alter its content in any significant way of use it on a different population, a further validation scudy is necessary. If the wording, order of items, or response formats are changed or new items are added, the validity may have been compromised. In nutrition knowledge questionnaires, itis very likely that questions may have to be modified or added as new dietary reports are published quite frequently with new or slightly different informacion in them. Items of information co be included in a measure of nutrition knowl- edge will differ for each population. Items appropriate for a group of dietitians, doctors, pregnant women, or shoppers will differ considerably one from another. A questionnaire developed and validated in America may not be valid in the UK. If extra items are needed, the least disruptive way of adding chem is atthe end of the existing questionnaire. If the population differs from the one for which the ques- tionnaire was originally designed, itis important to check that the language used will be understood by the new sam- ple of respondents. For example, is the level of difficulty appropriate? Are there any colloquialisms/terms that will not be understood by a sample living in a different region? ‘Anderson et al." based some questions on local educational ‘material and discussed modifications that might be required if used in other regions (e-g., awareness of a particular branded low-fat spread)Journal of Nutrition Education Volume 32 Number 5 DISCUSSION This paper has discussed the issues involved in designing measures of nutrition knowledge, although the principles are equully applicable to measures of attitudes, belief and other areas of knowledge, It has dealt with the fundamental stages—the generation of items, a pilot study and item analy- ses, reliability and validity—to enable health professionals to make an informed choice as to whether an existing ques- tionnaire is good enough to be used again, could be modi- fied, and, if not, how to design a respectable questionnaire that will produce meaningful results, These steps are sum= marized in Figure 1 Ithas been written as a user-friendly guide to question naire evaluation and design, bringing together the most important issues in psychological testing with particular ref erence to nutrition knowledge. It has not been possible to cover every aspect of questionnaire development in detail and topics that are easily understood (e.g. format and order of items) have been referenced for further information. Te muse be emphasized that it will be unusual to find a measure of nutrition knowledge that has all or even most of the attributes discussed in this paper. However, this paper gives investigators the knowledge necessary to evaluate a September * October 2000 275, ‘measure, rather than relying on the judgment of the devel- ‘pers themselves. For example, developers may have described how and why ther items were chosen for incli- sion, thus describing content validity, but not report it as such, and so it is important to be able to recognize these aetributes. No matter what the developer has reported, i is equally important chat potential users check the stems them selves Whether looking at the suitability of existing question- naires or developing. a new one, there are some essential points to bear in mind Validity and reliability have to be built Into a questionnaite from the outset. The importance of con= cent validity cannot be overemphasized, First, if 2 question naire does not include a representative sample of tems ofthe whole domain being measured, respondents’ scores will not bean accurate assessment oftheir knowledge. Furthermore, individuals may be discriminated against or favored, depending on which areas are covered. An indlvidal wish ing to lose weight may know more about the calorie con- tent of foods, whereas an individual wishing to reduce his/her cholesterol level may know more about the cheles~ terol and saturated fat content in foods. Second, the way in which items are written is fundamental fo an accurate asess= iment of knowledge. [Fan item is ambiguous, for example, ') subject area 1) target population Is there an existing measure suitable for: Yer oF [Check reliability and validity and re-pil If necessary before using Is there an existing measure that could be modified for your purposes? Ges] De] ‘+ make modifications Design measure forthe study. bbe aware of content validity . 1 + repilot to ensure statistical wording, ner, response rellability and validity and format layout ‘norpretability of items exer pane ll of ters plot with stable group heck elit and validity, Sscarding tems pyrite check forclanyof tems * epilot tina venice Figure 1. Summary of steps involved in selecting or designing a nutrition knowledge measure,276 Parmenter and Wandle/NUTRITION KNOWLEDGE MEASURES respondents may not answer it as intended, thus compromis- ing the accuracy of the result. If time and resources are limited when designing a ques- tionnaire, it is worth investing in the early stages of is devel- ‘opment, Resources should be directed towand careful con- sideration of the material the items will cover, item construction, and format of the questionnaire, including lay ‘out and instructions This is because itis more than likely that questionnaire whose content has been carefully thought out and whose items have been careflly constructed will be valid and reliable. A pilot study is important to highlight any dif ficulties that respondents encounter with the items. Diffi- culties are easily overlooked by developers or independent experts who may not have the same perspective as their respondents. [Fe is not possible to obtain a pilot sample that meets the suggested minimum (i.c.,one more than the num= ber of items) itis beter Co use as many respondents as pos- sible than to omic this stage altogether. Similarly if an ade~ quate sample proves difficult to obtain for the validity and reliability studies, itis better co use a smaller sample chan none at all. Itis alo worth remembering that construct valid ity accumulates with each new successful use of the ques- tionnaire, so if the expected results are achieved in the main study using the questionnaire this is further evidence of its construct validity. ‘There are several issues to bear in mind when examining or writing nutrition items in particular. There are many dif- ferent levels of nutrition knowledge. Research is carried out by scientists who produce scientific papers. Findings from these studies are incorporated into reports for policy makers, who produce dietary recommendations. Nutrition experts then translate these reports into dietary guidelines to be used by their professions. Finally, these guidelines are translated into dietary advice for the public. Ie s important that items are appropriate for the knowledge level of respondents; items appropriate for health professionals, for example, would be inappropriate for the general public. It is easy for nutrition items to be misinterpreted. Eating is a very personal experi- ence, and so respondents may answer questions according 60 their own experiences or likes and dislikes. Care needs to be taken co ensure that tems are answered according to national recommendations rather than personal assessments of dietary requirements. In self-administered instruments, it might be especially important co stress that respondents should answer the questions themselves, as health and nutrition can be a favorite topic of social conversation with friends and relatives. IMPLICATIONS FOR RESEARCH AND PRACTICE This paper has advocated chat health professionals wishing to use a nutrition knowledge questionnaire should try to find. 4 suitable one that has already been developed before either modifying one or developing one themselves. Developing, any measure is time consuming, even if it is not well designed, but the sime spent on developing a valid and reli- able measure will reap rewards of more accurate and there- fore meaningful results, and will enable results to be compa- rable across studies.A collaboration between nutritional and behavioral scientists makes an ideal team for the development of nutrition knowledge measures. REFERENCES 1. Anderson AA, Cox DN, MeKella 8, Resa JL Take Five, aa 1 MEJ, Mela Dj education intervention co increase fait and veg- cable nese: impact om atinedes roads dietary change: Br} Ne 1998:80:133-40. 2, Gibson FL, Worle Was), Frit and vegetable consumption, ut sional kuowledge and belief in others and children, Appetite 1998;31-205-28, 3. MeDonnel GE, Roberts DCK, Lee C. Stages of change and reduction of ciety ft eect af knowledge on atieudes ivan Austrian wi ‘versity population. J Nutr Educ 1998;30:37-44, 4, Stew A.Van Staveren WA, De Graaf C, Burema J. Not ‘edge and attitudes towards high-Gt foods and low-fit alternaives in tee generations of women, Eur} Clin Nutr 1996;80:33-41 5, Wardle |, Parmenter K, Willer Nutrition knowledge and fod inte Appetite 2000:34:1-8, 6, Resnicow K, Reinhadt J. What do chikren know abou it, be tholestera? a survey of3,116 primary and secondary school sudens. J Nowe Ede 1991:2565-71. 17. Thomas SE, Kendrick OW, Eddy JM. Modication of « nusiional 198 now Jed {questionnaire fr older adults andthe ably ofits knowledge and a= tude evaluations to predict dietary adequacy. J Nutr Elde 19909.35-68. 18 Anderson AS, Umapsthy D, Palumbo L, Pearson D. Nuttin know! cee assessed bya suuctured questionnaire in 3 group of medical in- tiens.J Haim Nutr Diet 1988,1.39-46, 9, Gromer SA, Kesler LG. The ft ad Ser content of foods what Amet~ cans know J Nott Ee 1992;28:149-52 11, Oppentein AN, Questionnaire design, nterewing and atu mea surement London: Pinter, 1992, 11, Diliman D. Mii and telephone sorveys: the ttl design method. New York: Wiley, 1978 12, Whichelow MI.Which foods contain dietary fibre? The belie ofan ‘dom sample of the British population. Eur J Clin Nutr 1988 42.945-51 13, Kem J Heath education and the problems oF knowledge Health Pro- motion Int 199136:291-6 14, Shepherd The effects of nrtonl belie and values on food accep- tance. In: Solms J Booth DA, Panghorn RM, Rauahardt ©, es. Food acceptance and nutstion, London: Academic Pret, 19873387402 Libcin M, Anon | Holi, titude, intention, and behavioue New ‘York Wiley 1975. 16, Shepierd R, Towler G. Nutrition knowledge, tbtades and fi inake: application of the theory of remioned action. J Hum Nutr Dist 15538797 17, Lage RG, Schweiger U, Date-HetoldU, Seeger M, Fichter MM, Pirke KM, Neritonal knowledge in paiens with eating disorders. oe {J Bat Dison 1988:7:69-73.Journal of Nutrition Education Volume 32 Number 5 18, Heath Education Authority, Enjoy healthy cating. London: Hak Education Authosity, 1993 19, Sadan, Bradburn NM. Asking guctionssa practical guide to ques tionnaire design, San Francitco:Josiey-Bast, 1991 20, Payne SLThe at of asking questions Princeton: Princeton University Pres, 195 21, Heath Eatin Authority The national Food guide-The Balance of 00d health, London: Health Education Authority, 1994 Department of Health. Nutritional pects of cacdinascular dsc, Report on heath and weil secs 46, London: HMSO, 1994 23, Sobal J. Canidy CM, Pale perceptions of Eber in foods, Eeal Food, Nae 1991.26:303-12, 24, Sobal|,Caidy CM. Public Beli about the amount of een ood, Appeste 1993:20:21-92. Parmenter K Wardle. Development of gener nutrition noledge questionnaire for aduls, Eur J Clin Nutr 1999:53:298-308, 26, Towler G, Shepherd R. Development ofa nursonal knowledge ques tionnaire} Hum Nutr Diet 1990;3:255-68 27. Tate J,Cade ] Public knowledge of dietary fa and coronary hear dis exe Health Ede J 1990319:32-5 28, AnlkerJA, Las MJ, Samonds KW, BealVA. Parental mesages and the 49, tution awareness of preschool chien. Nate Ed 1990; September © October 2000 277 29, Rast J, Golorbok S, Modern psychometric: dhe science of psycho: logical asessment, London: Routledge, 1992. 30, Sueinet DL, Noeman GR, Health mestrement sales: 2 practical side to their development and uss, Oxond: Oxf University Pres, 9, Kline B-The handbook oF psychological testing London: Routledge, 1993, 32, Coolican H, Research methods and statis in psychology. Srl Ea London: Hodder & Stoughton, 1999. 33. Bergman EA, Erickson ML, Boyungs JC. Caffeine knowledge, atitudes and consumption in adult women. J Note Educ 1992/24179-84. 34, Nunsally JC. Prychometrie theory 2nd Ed. New York: McGraw-Hill 7s. 35, Mantero- tenes E, Baum MK, Javier Jet Nattional knowledge, beliefs and practices in the HIV 991113340 136, Anata A. Poycholugical ening. Sch Ed New York: Mactlln, 1982 37. Axelson ME, Brier D. The measurement and coneepessization of nutrition knowledge, Nu Educ 1992,24.239-46 38, Finnegan JR. Viewanath K, Rooney B a. Predictors of knowledge shout heey easing in a rural midwestern US city Health Educ Res 199054213 a fected patient, Nutr Res