Research in Language

Education

Dr. Marwa Zuhri, M.A

Basic Principles of Research (2, Dornyei, pp 13-
21)
Why conducting one’s own data-based
investigation?
(1)No two language learning are exactly the same;
the literature rarely provides the exact answers
to one’s specific questions.
(2)There is progress in research; new advances are
continuously made by building on the results of
past research
(3)For teachers to become more effective teachers;
providing new insights into the teaching and
learning process.
 Basic Principles of Research (3, Dornyei, pp 13-21)
What are characteristics of good researchers?
(1)Genuine curiosity
(2)A lot of common sense
(3)Good ideas
(4)Combination of discipline, reliability, and social
responsibility
Three main types of primary data:
(1)Quantitative (numbers)
(2)Qualitative (spoken, written, documents)
(3)Language data (language sample)
 Definitions of Research
Compare the previous definition with this definition:
“Research may be defined as the
systematic and objective analysis
and recording of controlled
observations that may lead to the
development of generalization,
principles, or theories, resulting in
prediction and possibly ultimate
control of events”.
 kinds of Research
QUANTITATIVE APPROACH
(1)Well established designs
(2)Includes 3 big categories of research designs: (a)
experimental (b) ex post facto/causal
comparative, and (c) descriptive
(3)The terms quantitative and qualitative may be
used to refer to the approach OR to the kinds of
data collected.
(4)If the conclusion is based on analyzing the
quantitative data (numbers) it will be
quantitative research, and if the conclusion is
based on analyzing the qualitative data (non-
numbers) it will be qualitative.
 kinds of Research

(1)The most important difference between quantitative

and qualitative is in the final goal: the final goal of
conducting quantitative research is to verify theory,
and the final goal of conducting qualitative research is
to generate theory
(2)In other words, quantitative=theory verification
whereas qualitative=theory generation
(3)Verifying a theory is providing evidence to see
whether the evidence supports or doesn’t support the
theory. If the evidence supports the theory, the theory
will become stronger; but if the evidence does not
support the theory, the theory may be
modified/revised/changed
 kinds of Research

(1)How does a quantitative researcher verify theory? (a)

he formulates hypothesis based on the theory (b) test
the hypothesis by using certain statistics (c) the
hypothesis testing is based on the data collected and
analyzed.
(2)Generating a theory is producing a theory.
(3)How does a qualitative researcher generate a theory?
(a) collect data, (b) based on the data collected, he
formulates the hypothesis, (c) based on the
hypothesis, he collects more data to check the
hypothesis; and this will continue until all the data are
redundant or saturated (same data again and again)
 Criteria of Good Research Questions
(1)Feasible/researchable/manageable (in terms of time,
money, and energy)
(2)Researcher must have the knowledge about the topic
(3)Researcher must have the skill to conduct the study/to
answer the question
(4)Novel/original/interesting
(5)Significant/important/useful
In brief, it must be feasible and significant. It is feasible if
the researcher has the knowledge in conducting the
study/in answering the question. It is significant if it is
new/if the answer is not available yet, so it can
contribute to the body of knowledge or the
development of science
 VARIABLES AND HYPOTHESIS
Variable is anything that can vary: can be different or change within the same
or different objects or persons.
Examples of variables of a person: age, motivation, gender, skill, knowledge
Examples of variables of a school: programs, facilities, teachers, students,
parents
Kinds of variables: (1) independent variables (=the “cause variable”) (2)
dependent variables (= “effect variable”)
Independent variable is a variable which affects another variable; where as
dependent variable is a variable which is affected by another variable.
Some variables can be treated as either independent variables or dependent
variables: eg: motivation can be treated as (1) independent variable in a
study on the effect of motivation on the students’ learning outcomes, or a
(2) dependent variable in a study on the effect of giving rewards on the
students’ motivation to learn a language
But some variables cannot be manipulated: some variables can only be
independent variables (eg: age, gender). These are sometimes called
classifying or attributive variables
 HYPOTHESIS in Quantitative Research
Hypothesis is a tentative answer to the research
question
E.g. If the research question is “is Method A
better than Method B? the tentative answer
(=the hypothesis) can be: “Method A is better
than Method B.”
Later if the researcher analyses the data using
inferential statistics, he/she has to change the
hypothesis into “Null Hypothesis” (=statement
of no difference): “There is no difference
between Method A and Method B.”
 REVIEW OF RELATED LITERATURE
• What is meant by reviewing related literature?
• Literature: any sources of information, printed
or not printed (web sources), which can be
used as basis to conduct a research study. The
information is relevant to the topic that the
researcher wants to investigate.
• The potential source to get the materials:
garuda.kemdiknas.go.id (articles from journals
published in Indonesia, S-1/S-2/S-3 theses of
Indonesian students, and other materials)
 REVIEW OF RELATED LITERATURE
• Steps of reviewing related literature (how to
review related literature): (1) identify the
potential sources (2) select the materials which
are relevant to the topic (3) decide which
materials to read based on the selection of the
most relevant materials (4) read and then
evaluate whether they are good in quality or not
(5) use the materials which are good in quality to
support our research study, and then (6) cite the
source in the text of our research report (e.g.
theses) and write the bibliographical information
about the sources in the REFERENCES.
 REVIEW OF RELATED LITERATURE
• What kinds of sources: (1) primary sources
(because it is (a) first hand information (b)
original, and (c) the quality has been checked
e.g. journals, thesis, research reports), and (2)
secondary sources (e.g. books, magazines,
newspapers)
• Criteria of good sources to review and use for
our research study: (1) relevance, (2) primary,
and (3) most recently published
 DATA COLLECTION
• Data should be collected to answer research questions
• Data to be collected is based on the variables involved in our
study; e.g “the effect of teaching English on the students’
English Proficiency”, we need to collect data about the
students’ English proficiency at the end of the instructional
program by using “an English Proficiency Test”.
• How can we collect data about the students? It depends on
the variables. If it is (1) cognitive/language skills, the best way
is by using a test, (2) opinions/ perceptions/preferences/
motivation, the best way is by using interview (if the number
of respondents is limited) or using questionnaires (if a large
number of respondents is involved), (3) activities (students’
participations, interactions, how the teachers teach, how the
teachers respond to students’ questions/responses), the best
way is by using observations (observation sheets, audio or
audio-visual recorders), (4) what has been written/recorded
(like novels), the best way is documentary analysis
 DATA COLLECTION
• If you use a test or questionnaires, you have to
make sure that they are good instruments. What
are criteria of good instruments (1) validity (=you
get what you really want) (2) reliability (=it can
give the same results from time to time) (3)
practicality
• In collecting data for quantitative research, we
have to decide the most appropriate source of
data: (1) population, and (2) sample; but
• In qualitative research, population and sample
are not relevant (they don’t need to have a
population or sample)
 POPULATION AND SAMPLE
• Sample is taken because the researcher cannot
collect data from the whole population
• What we want to study is about the population, so
the sample that we take must represent the
population
• The conclusion drawn from analyzing data from the
sample is going to be applied to the population
• Drawing conclusions from the sample and apply the
conclusions to the population is called generalization
• To represent = having the same characteristics which
can be done through random sampling techniques
• Random is a concept which guarantees that
everybody/everything in the population has an equal
chance to be selected as a sample
• In other words, in order to have a sample which
represents the population, we should use one of the
following random sampling techniques:
1. Simple random sampling (used when the population is
homogeneous): use random digit numbers found at
the end of statistics/research books, Example
82726151 20937464 etc
53384020 35421881 etc
30497591 09835621 etc
etc. etc.
2. Systematic random sampling (when the pop is
homogeneous): start taking the first student randomly
(pick a number from the list of the students; then go
on with the next student using certain interval
3. Stratified random sampling (used when the
population consists of strata/levels): a
proportional number of sampling
units/students should be taken from each
level/stratum (you also use simple random
sampling technique)
4. Cluster random sampling (used when the
sampling units are organized in groups, e.g.
students in schools, especially when you
want to draw conclusions about the schools)
5. Multi-stage random sampling (used when the
population is very complex, e.g. students of
Senior High schools in East Java: The students
are organized in schools within cities in East
Java): start selecting cities in East Java, then the
schools within the cities, then the classes within
each school, then students in each class. So, the
sampling units/students are selected through
different stages using a combination of random
sampling techniques.
Which technique is the best? Depends on the
characteristics of the population (homogeneous,
heterogeneous, or complex)
 Summary of Random Sampling Techniques:
When do you use simple random sampling? (1)
When the population is homogeneous: only the first
grade of a school; all water in the well;
When do you use stratified random sampling? (3)
When the population consists of levels or strata: all the
students in a school
When do you use multi-stage random sampling? (5)
When the population is very complex: all the students of
schools in a district (cities or provinces)
When do you use cluster random sampling? (4)
When the sampling units are organized in groups: the
parallel classes in a school; you have 3A, 3B, 3C, 3D, 3E,
and you want to take only one or two classes.
 QUANTITATIVE RESEARCH DESIGNS
3 CATEGORIES OF QUANTITATIVE RESEARCH
DESIGNS:

1.Experimental
2.Ex-post facto/causal-comparative,
3.Descriptive (survey, correlational)
 Experimental research is intended to establish cause and effect
relationship between independent and dependent variables.
• How can you establish cause and effect relationship?
– By manipulating the independent variable/s and look at the effect
on the dependent variable after controlling the other variables from
also affecting the dependent variable
– How do you control the other variables? By making the other
variables the same (same teacher, same individuals, etc). But it is
hard to control when it deals with long-term effect (e.g. learning
outcomes); it will be easier if it deals with short-term effect (e.g. the
effect of outline/no outline on the quality of compositions)
• What is meant by “manipulating the independent variable/s”?
– Making the independent variable/s vary (different). E.g:
manipulating method of teaching (so that we can have Method A
and Method B) and look at the effect on the dependent variable (e.g.
English proficiency). Another example: using different languages (as
a result of manipulating the language as the independent variable)
to bargain to see the effect on the price.
 Ex-post facto (Causal-Comparative) is also intended to establish
cause and effect relationship but the researcher does not
manipulate the independent variables. The independent variable
varies naturally (e.g. language lab at schools, some schools have
language lab and some do not have language labs).
When the students’ achievement in schools with language lab is
better than that in schools without language lab, we can conclude
that language labs can affect the students’ achievement (in
English). The weakness of this conclusion is that their achievement
may also be affected by other factors (e.g the quality of the
teachers, the students input, and other facilities)

In addition, the researcher cannot control all other variables.

• Therefore, the statement of cause and effect relationship is not as
strong as when it is experimental
 Descriptive Research (survey, correlational) is
NOT intended to establish cause and effect
relationship. It is intended to describe the
phenomena (events, status, etc) as they are,
answering “what”.
Example: to describe the English proficiency of first
year, second year, and third year students of senior
schools without being interested in why they are
different/same.
In correlational study, the researcher is not interested in
cause and effect relationship, but is co-relational (one
can be associated with the other) so that the
researcher can predict one from the other. E.g
correlation between the masteries of English and
Math
What is the similarity between experimental and ex
post facto/causal comparative? Both are intended
to establish cause-effect relationship.
What is the difference between the two? In
experimental design, the independent variable can
be manipulated by the researcher, where as in
expost facto the independent variable is NOT
manipulated by the researcher, it is manipulated
naturally/has existed before
What is the difference between experimental&ex post
facto and the descriptive research? In
experimental/ex post facto the purpose is to
esetablish cause and effect relationship, but in
descriptive research, the researcher wants to
describe the phenomena, not interested in cause-
effect relationship
 DATA ANALYSIS
– Statistics is used to analyze data for quantitative
research
– Data should be analyzed in order to answer
research questions
– Basically, all kinds of data are analyzed in order to
make the data more meaningful
– Data to be collected should be about the variables
involved

26
In qualitative research, the data are in the form of
verbal description as a result of collecting using
interviews, observation etc; but in quantitative
research the data are in the form of numbers as
a result of collecting data using tests,
questionnaires, etc.
In quantitative research all information/data should
be converted into numbers because statistics is
used to analyze the data.
Quantitative researchers have to finish collecting all
the data before they analyze them; but
qualitative researchers can (or even have to)
analyze the data in the process of collecting the
data.
Analyzing qualitative data may include: data
reduction (selecting which is relevant and
dropping which is not relevant), data
categorization (by comparing and contrasting
resulting in grouping which are the
same/similar and which are different) and
conclusion drawing
In qualitative research, data which the researcher
collects can be categorized as (1) descriptive
data (as originally expressed by the subjects) ,
and (2) reflective data (as a result of the
researcher’s interpretation)
RESEARCH STATISTICS can be (1) descriptive
statistics, or (2) inferential statistics:
Descriptive Statistics is used when the researcher
does not want to make generalization (e.g. as a
teacher interested in studying about his/her
programs)
OR: when the researcher can collect data from
the whole population (all the samplings
unit/students in the population)
Inferential Statistics is used when the researcher
wants to make generalization (drawing a
conclusion from the sample and apply it to the
population)
 DESCRIPTIVE STATISTICS
It includes:
(1) Measures of Central Tendency (e.g. mean to indicate
group performance), and
(2) Measures of Variability (e.g. Standard Deviation/
Variance/range) to indicate heterogeneity/
homogeneity of the group/s
Mean of a group indicates the performance of the group
in relation to the variable
Mean=average score
Group A, mean=75
Group B, mean=65
Endang is from Group A, and Susi is from Group B. Which
one is better, Endang or Susi? We don’t know, we have
to know their individual scores
Measures of variability indicate differences among
the individuals in a group, so we know that our
group is homogeneous or heterogeneous by
knowing the measures of variability (=range,
variance, standard deviation)
Range= (100,65,40,35,25,22,20,15,10) the
maximum score minus the minimum score (plus
one or without plus one)
Variance (S2) vs Standard Deviation (S)= The square
root of S2 is S, S square is S2, once you know the
S you can find the S2 and vice versa
 RESEARCH REPORT WRITING
Based on the steps (questions, data collection, data
analysis, and conclusion) you have been through, you
have to write:
(1)Introduction: context of your topic, what previous
researchers have found about the topic, and the
purpose of your own study. In the introduction, you
have to show that your research question is good,
fulfilling the criteria of a good research question
(especially that the answer is not available yet and that
it will contribute to the body of knowledge)
(2)Method: what kind of data you collect, from whom
you collect the data, how you collect the data, how
you analyze the data
(3) Results of Analyzing the Data: display the
data (if necessary), process of analysis (if
necessary), the result of analysis (the result of
statistical procedure if the data are in the
form of numbers)
(4) Discussions: give meanings to the results of
data analysis based on your own
interpretation, compare the results with the
theory and/or the previous research findings
(5) Conclusion: formulate the conclusion using
statements which are meaningful so that the
question “so what” can be answered clearly.
(6) Reference: a list of literature sources which are
mentioned in the text. There should be a perfect match
between what is mentioned in the text and the list:
what is mentioned in the text should put in the list, and
what is in the list should be mentioned in the text.
The literature reviewed and mentioned in the text
(which parts of the text? The background and the
discussion) should be (a) relevant to the topic, (b) from
primary sources (journals and research reports, theses),
and (c) from the most recently published sources (up to
date).
All those elements in a research report are usually written
in a structured format (with certain headings, like
“Introduction”, “Methods”, “Results”, “Discussions”,
and “Conclusion”) if it is a formal research report like,
theses, or articles in a scholarly journal.