Experimental Design & NATIONAL, ; MATH + SCIENCE Student Saturday Session INITlATIVE Student Notes - Prep Session Topic: Experimental Design {A free response question dealing with sampling or experimental design has appeared on every AP Statistics exam. ‘The question is designed to assess your understanding of fundamental concepts and generally consists of multiple pasts. Important vocabulary and key concepts related to experimental design include: ‘+ The difference between an experiment and an observational study te » Characteristics of a well-tlesigned experiment; specifically: bet oe ‘© Control—keeping extraneous variables as constant as possible; ‘© Replication—having at least two experimental units in each treatment group; (© Randomization—making sure that treatments are allocated to experimental nits at random; © and (Sometimes) Blocking —see a few more details below. + Contol group +) Explanatory variables: factors, levels, and treatments +) Experimental units and subjects + Response variable + Sources of possible bias + Placebo effect Confounding + Blinding (single blind and double blind) * Blocking, blocks © Completely randomized design + Randomized block design, including matched pairs design «Types of conclusions that can be drawn from experiments and observational studies + Scope of inference Be sure you understand that --- + Experiments are studies in which the researcher imposes a treatment on experimental units. Ifno treatment is assigned or imposed, the study is called an observational study. © Experimental units are the smallest independent “objects” to which treatments are assigned and on which a response is measuzed. Consider an experiment that is designed to determine which of several types of fish food will result in the greatest weight gain for fish. If tanks contain several fish, and food is added to the water in the tank, then the tank is the experimental unit (aot the individual fish), since the fish in a tank are not independent of one another, but tanks are independent of one another. © Some experiments have a control group (a group of experimental units that receive no treatment or receive only a placebo), but this is not necessary for a well designed experiment. + Sometimes different treatments are simply compared with one another. + Replication refers to having multiple expesimental units in each treatment group (tepeating the treatment), not o repeating the entire experiment. + Inan experiment, randomization refers to randomly assigning experimental units tothe treatments. Often the experimental units are not a random sample of the population of interest. While this is nota problem with the experimental design, it may limit the scope of inference for the experimental results. (Note that random samples are necessary in surveys.) ‘* The purpose of random assignment (of experimental units to treatment groups or of treatments to experimental units) is to even out extraneous variables and make treatment groups that are approximately similar in all respects except forthe treatment + Ina double blind experiment, someone must know which treatment the experimental unit received! The subjects (assuming they are people) are blind to which treatments they are receiving, and anyone who interacts with the subjects should also be blinded to treatments. + Also, ifthe response variable isin any way a subjective evaluation, then the person performing that evaluation must be blind to what treatments were applied. But obviously, some person or people on the research team must have a record of what treatments have been applied to what subjects! Submitted by Glorta Barrett and Floyd Bullrd, Viginia Advonced Study Strategies lenvany, 2011’ NaTiONAt Experimental Design Nari SCIENCE ‘tudent Saturday Session INITIATIVE + Aconfounding variable is one that affects the response variable and also is related to group membership. A variable that affects the response variable and is nor related to group membership (that is, the variable would be expected to even out across the groups) is not a confounding variable. You may refer to this type of variable as an extraneous variable. + For example: It has been observed that people who take long vacations have, on average, significantly longer lifespans than people who don’t. Can we conclude that vacationing is a way to extend your lifespan? ‘Not necessar#lysa‘person’s income could be a confounding variable—people with higher incomerws# more likely to be able to take long vacations, and they're also more likely to afford health care that could lead to longer lifespans. Note that something like exercise would probably be an extraneous variable and not a confounding variable. Exercise may indeed be associated with longer lifespans, but is there an association between getting exercise and taking long vacations? * _Itis best to avoid using the term lurking variable. It will almost always get you in trouble! * Blocks are groups of experimental units that are homogeneous with respect to some inherent characteristic that is expected to affect the response to treatments. * Blocks are considered # form of control — blocks help control known sources of variability among the experimental units so that the experimenter is better able to detect differences in the response variable that are due to the treatments. + “Blocking is used to control the factors you can see; randomization helps balance the ones you cannot see.” Richard L. Schaffer, AP Statistics Chief Faculty Consultant, 1997-1999 Submitted by Gloria Barrett and Floyd Bullard, Virginia Advanced Study Strategies January, 2011Student Saturday Session INITIATIVE . .. NATIONAL, Experimental Design & MATH + SCIENCE 1 22, 25. Multiple Choice Questions from 2002 Exam Which of the following is a key distinction between well designed experiments and observational studies? ‘More subjects are available for experiments than for observational studies. Ethical-ganstraints prevent large-scale observatioaabetudics. end Experiments are less costly to conduct than observational studies. ‘An experiment can show a direct cause-and-effect relationship, whereas an observational study cannot. Tests of significance cannot be used on data collected from an observational study. moaw> A study of existing records of 27,000 automobile accidents involving children in Michigan found that about 10 percent of children who were Wearing a seatbelt (group SB) were injured and that about 15 percent of children who were not wearing a seatbelt (group NSB) were injured. Which of the following statements should NOT be included in a summary report about this study? Driver behavior may be a potential confounding factor. ‘The child's location in the car may be a potential confounding factor. ‘This study was not an experiment, and cause-and-effect inferences are not warranted, ‘This study demonstrates clearly that seat belts save children from injury. ‘Concluding that seatbelts save children from injury is risky, at least until the study is independently replicated. moADP A new medication has been developed to treat sleep-onset insomnia (difficulty in falling asleep). Researchers ‘want to compare this drug to a drug that has been used in the past by comparing the length of time it takes, subjects to fall asleep. Of the following, which is the best method for obtaining this information? ‘A. Have subjects choose which drug they are willing to use, then compare the results B. Assign the two drugs to the subjects on the basis oftheir past sleep history without randomization, then ‘compare the results. C. Give the new drug to all subjects on the first night. Give the old drug to all subjects on the second night. Compare the results. D. Randomly assign the subjects to two groups, giving the new drug to one group and no drug to the other group, then compare the results. Randomly assign the subjects to two groups, giving the new drug to one group and the old drug to the other group, then compare the results. m Submitted by Gloria Barrett and Floyd Bullard, Virginia Advanced Study Strategies January, 2011Experimental Design NATIONAL MATH + SCIENCE Student Saturday Session Leelee 20068 #5 5. When a tractor pulls a plow through an agricultural field, the energy needed to pull that plow is called the deaftyThe draft is affected by environmental conditions suchtssoil type, terrain, andaseisture, A study was conducted to determine whether a newly developed hitch would be able to reduce draft compared to the standard hitch. (A hitch is used to connect the plow to the tractor.) Two large plots of land were used in this, study. [twas randomly determined which plot was to be plowed using the standard hitch. As the tractor plowed that plot, a measurement device on the tractor automatically recorded the draft at 25 randomly selected points in the plot. After the plot was plowed, the hitch was changed from the standard one to the new one, a process that takes a substantial amount of time. Then the second plot was plowed using the new hitch. ‘Twenty-five measurements of draft were also recorded at randomly selected points in this plot A. What was the response variable in this study? Identify the treatments. What were the experimental units? B. Given that the goal of the study is to determine whether a newly developed hitch reduces draft compared to the standard hitch, was randomization used properly in this study? Justify your answer. Submitted by Gloria Barrett and Floyd Bullard, Vtginta Advanced Study Strategies January, 201Experimental Design NATIONAL, MATH + SCIENCE Student Saturday Session hare C. Given that the goal of the study is to determine whether a newly developed hitch reduces draft compared to the standard hitch, was replication used properly in this study? Justify your answer, mag OD hat et D. Plot of land is a confounding variable in this experiment. Explain why, Submitted by Gloria Barrett and Floyd Bullard, Virginia Advanced Study Strotegies January, 2011Experimental Design NATIONAL MATH + SCIENCE Student Saturday Session Maes 2003 #4 4, Because of concems about employee stress, a large company is conducting a study to compare two programs (tai chi or yoga) that may help employees reduce their stress levels. Tai chi is a 1,200-year-old practice, originating in China, that consists of slow, fluid movements. Yoga is a practice, originating in India, that consists of breathing exercises and movements designed to stretch and relax muscles. The company has assembled tgcbup of volunteeremaployees to participate in the study during the first half of thetwekanch hour ceach day for a 10-week period. Each volunteer will be assigned at random to one of the two programs. ‘Volunteers will have their stress levels measured just before beginning the program and 10 weeks later at the completion of it. ‘A. A group of volunteers who work together ask to be assigned to the same program so that they can participate in that program together. Give an example of a problem that might arise if this is permitted, Explain to this volunteer group why random assignment to the two programs will address this problem. B, Someone proposes that a control group be included in the design as well. The stress level would be measured for each volunteer assigned to the control group at the start ofthe study and again 10 weeks later What additional information, if any, would this provide about the effectiveness of the two programs? C. Isitreasonable to generalize the findings of this study to all employees of this company? Explain. Submitted by Gloria Barrett and éloyd Bullard, Virginia Advanced Study Strategies January, 2011