Sampling, Reliability…

Sampling, Reliability, and Validity in Education: A study document

Victoria L. Howard

February 3, 2020

Aspen University
Sampling, Reliability…
2

ABSTRACT
To improve the success of research, it is imperative that the researcher go above and beyond the
basic premise of research, as a whole. This occurs when the researcher develops a project with an
approximate knowledge of the intended outcome in mind. By using more than one type of
reliability method, validity techniques, and internal validity methodology, with a clear
understanding of the research process steps, along with a solid understanding, the participants
will understand more fully the specifics of their roles in the research process and what will be
evaluated for understanding. This reduces the confusion for the researcher who specifies the
sampling criteria, the reliability of the findings, and how valid the outcome could potentially be.
It is through the collaborative efforts of the researcher and the comprehension levels of the
participants in the sampling groups, where the outcome intended could potentially be achieved.
Sampling, Reliability…
3

Introduction

Inaccuracies in measurement will typically occur in most research projects and most

researchers anticipate variation in scoring results when the tool used is dispersed to the same or

similar groups repeatedly (Fraenkel, Wallen, & Hyun, 2016, p. 156). To best prepare your

research environment and the human subject or group designated by the researcher to execute the

testing and analyzation of the research project, the information provided to the subjects must be

clear and described in plain language, to include who is to do what and why. Asking the why

question and openly conveying the anticipated intent of the experiment to those who have been

chosen to participate will aid in minimizing disparities in the testing results, but will not

eliminate them.

The sample subjects who are representative of the population, for example in a school system

such as the one listed below should pare to verify and validate the schools population, because its

relevance is the commonality of the sampling group to its origin population (Šimundić, 2013).

In the following scenario, several disparities are apparent. Let us examine:

1. A researcher conducted a study to determine if projecting visual illustrations onto

a screen during science class would enhance students' understanding of scientific

principles. The researcher chose two classes for the study: a biology class and a

chemistry class. He told one class that they would get to see color illustrations

projected onto a screen. He told the other class that they would be the "control"

group so they would not get the illustrations. Since the chemistry classroom

already was technologically equipped, the researcher chose the chemistry class

as the one that would see the illustrations. At the end of the semester, the

researcher asked for volunteers from the two classes to take a test to help him see
Sampling, Reliability…
4

if the illustrations were effective. Most of the students in the chemistry class, and

about half of the students in the biology class, volunteered to take the test.

Classes in # of students Control Group Color Technologically # of Mean

the Study Illustrations Equipped students

that took

the Tested
Biology 20 Yes No No 10 0.5

class
Chemistry 20 No Yes Yes 16 0.8

class
Type of Size of test Imbalanced Course Vague research Social Is test/

threat to population experimental content history desirability research

internal imbalanced treatment variability factors repeatable?

Validity
Note: The testing measurement used to gain insight of whether the students with illustrations

learned more effectively than the students who did not have illustrations is distorted, because

there were more students taking the test that had illustrative learning.

 Identifiable threat to internal validity #1; with the explanation of steps taken to

minimizing potential threats to internal validity.

In section 1, the content measured only represents a fraction of what the researcher is trying to

ascertain (Cohen, 2013). Streamlining the measure ranges will yield a more effective result when

you streamline the concept hone in the focus of the test. Disparities pose threats to internal

validity, thus the importance of the researcher to manage all aspects of the project. Not

controlling the possible threats to internal validity within a research project reduces the viability
Sampling, Reliability…
5

of the data collected during the study (Slack & Draugalis, 2001): Which is the case in the above

scenario.

 Identifiable threat to internal validity #2; with the explanation of steps taken to

minimizing potential threats to internal validity.

Validity is the measuring of the information or data the researcher is trying to measure. If the

inquiry makes sense, the hypothesis is verified, if it does not make sense to the participants, the

test results and other collected information or internal validity is questionable and creates cause

for concern that the criterion; the outside measures that relate to the inside variables or criterion

(Cohen, 2013) used to support the researcher’s findings, can be questioned. The researcher needs

to restructure the question in a format that would yield a predictive result. Variances in the

number of students in the test groups create a lack of construct because the variables may not be

reliable or repeatable.

2. Developing a new research instrument requires careful consideration. Suppose

that a state department of education has developed a new instrument for

evaluating teacher performance. What characteristics constitute a good

instrument? Why?

Effectively collecting material that would best outline teacher performance requires an

instrumentation tool equipped with procedural information (the steps to follow to correctly

administer the research) and conditional preferences (conducive to compliance and the favored

conditions for evaluating teacher performance) to be a relevant tool (Fraenkel, Wallen, & Hyun,

2016, p. 113). The new research instrument that the state implemented to evaluate teacher

performance more effectively must also mirror the instructor’s job process and responsibilities if
Sampling, Reliability…
6

the intended results of the evaluation are to be representative of the teachers’ job performance

during the school year.

Create and briefly describe several ways that the developers could ensure that the instrument

has construct validity.

The value of an instrument lies within its effectiveness to distribute a numerical score to the

persons being evaluated with similar job processes, in this case; teachers. Building a viable

research instrument that can effectively measure teacher key performance will include validity

criteria that are:

Valid in nature with measurability qualities that have context, are predictive in nature,

constructive in concept and raw format (materially), and that runs concurrent with all evaluative

indicators, including content validity, the measuring of the questions or research inquiry and it’s

fullness of interpretation and to what extent (Price, Jhangiani, Chant, & Chiang, 2015).

Ways for the developer to evaluate instrument/tool has construct validity.

Construct validity measures the effectiveness of the instrument or tool and its capabilities and

the effectiveness of its design. In other words, does it do what it is designed to do (Fraenkel et

al., 2016, pg. 149)? For example, if the instrument is designed to measure teacher performance

by the week, month, or semester, is it equipped with the facts necessary to make a fair analysis of

the instructor’s performance based on this criterion, not their gender or subject matter taught.

Conducting an effective research before compiling and designing the teacher performance

evaluation form prepares the researcher for success during the collection phase of the project.

Knowing what you want to evaluate and how you want to score the results are conclusions that

inevitability influence the quality and the quantity of the information retrieved (Abowitz &

Toole, 2010).
Sampling, Reliability…
7

See sample instrument below:

(Example 6. Teacher performance evaluation report, n.d.)

Sampling, Reliability…
8

3. Work through some of the reasons why ecological generalizability is almost

impossible to attain with a single research study.

There continues to be misperceptions encompassing generalizability from the posture of how

true or valid is the data compared to its efficacy being tainted, due to the framework and

execution of the research study (Kukull & Ganguli, 2012). Oftentimes, researchers conceivably

compare varied ecological groups of subjects with similar characteristics anticipating the same

results that were ascertained from their pilot test subjects and groups. Although in theory, it

would seem logical that adjunct instructors who teach online have the same approach to teaching

their students as an adjunct instructor who teaches at a brick and mortar school. True, both

groups of instructors are qualified to teach their respective subjects in ways that each group

would see fit; however, expecting that the testing instrument used will yield the same results

from these different groups of instructors is a muse, as is testing the similarities of these

instructors within their individual subgroups. The results and findings will vary and cannot be

guaranteed to produce the same result. Ecological generalizability is the inability of the

researcher to gain insight into trends of the results of contextualizing this way. As a result,

researchers must refrain from generalizing the contents/results of research that they have

conducted in and within other formats, according to the authors of our textbook (Fraenkel et al.,

2016, pg. 106).

4. What are some of the reform trends emerging in multicultural education today?

What type of evidence do you think is the easiest to obtain among that which is

content, construct, and criterion-related? Why?

Too much emphasis is placed on the differences in multicultural education environments

verses their similarities to include the common dominator - learning. The differences are not as
Sampling, Reliability…
9

varied as some educators think, yet it is evident that multicultural education and collaborative

efforts to teach all students is the concern and focus of some reform efforts. Customizing

learning efforts to include rich historical data, historical events, people of influence, and other

cultural backdrop information of the various cultures that today’s classrooms represents,

provides information for the entire classroom. This shifts the focus from one race or ethnicity

appearing more important than their fellow students does, to learning arena representative of its

complete and diverse student body (Researchgate, 1993).

Current trends in multicultural education and education as a whole are: Standards- Based

Reform designed to foster improved educational results for all students; improved and cultural

designed curriculum to include all ethnicities in the learning process in multicultural learning

environments; STEM course content aligned with basic educational standards for improved

assessment outcomes in multicultural educational environments.

Inclusion efforts- Created to enrich learning outcomes for all students including those with

multicultural differences as well as students with and without disabilities, representative of our

country (Voltz, Sims, & Nelson, 2010).

Conclusion

Improving research and research methodologies begins and ends with planning, evaluating, and

understanding the intricacies of the subject (human), the data/subject matter, the parameters of

the research that is to be examined, and intended or anticipated outcome data (if applicable).

Preparing and planning your research aids in the reduction of internal and external validity issues

and concerns, giving the research process the scientific edge necessary to make public the

results. Without public exposure, the research will not be classified as scientific, gaining insight

into current trends in research and data collection revitalizes the research process and allows its
Sampling, Reliability…
10

process to remain relevant and applicable to future research trends. It is through the collaborative

efforts of the researcher and the comprehension levels of the participants in the sampling groups,

that the anticipated outcomes can potentially be achieved.

Sampling, Reliability…
11

References

Abowitz, D. A., & Toole, T. M. (2010). Mixed Method Research: Fundamental Issues of Design,

Validity, and Reliability in Construction Research. Journal of Construction Engineering

& Management, 136(1), 108–116. https://doi.org/10.1061/(ASCE)CO.1943-

7862.0000026

Cohen, J. N. (2013, March 6). Validity [Video file]. Retrieved from

https://www.youtube.com/watch?v=ExAICQFeFJI

Example 6. Teacher performance evaluation report. (n.d.). Retrieved from

https://images.app.goo.gl/PjCa9RCF8Yhd8TBt6

Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2016). The nature of research. In How to Design

and Evaluate Research in Education (9th ed., p. 12). New York, NY: McGraw-Hill

Education.

Kukull, W. A., & Ganguli, M. (2012). Generalizability: The trees, the forest, and the low-

hanging fruit. Neurology, 78(23), 1886-1891. doi:10.1212/wnl.0b013e318258f812

Price, P. C., Jhangiani, R. S., Chant, I., & Chiang, A. (2015, October 13). Reliability and

Validity of Measurement – Research Methods in Psychology. Retrieved from

https://opentextbc.ca/researchmethods/chapter/reliability-and-validity-of-measurement/

Researchgate, Multicultural Education in the US: Current Issues and Suggestions for Practical

Implementations. (1993, January 1). Retrieved from

https://www.researchgate.net/publication/317367781_Multicultural_Education_in_the_U

S_Current_Issues_and_Suggestions_for_Practical_Implementations
Sampling, Reliability…
12

Šimundić, A. M. (2013). Bias in research (23457761). Retrieved from Biochemia medica

website: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900086/