RESEARCH TRAINING & BACHELOR THESIS

CHAPTER 3:
CASE SELECTION AND
MEASUREMENT

ROBERT ROODERKERK
CHECKLIST FOR CRITICAL EVALUATION OF A REPORT

4. Which population is studied?

a. Is the population a part of the theoretical domain?
b. Is the population exactly defined and are its characteristics specified?
c. Is the whole population studied (“census”) or a sample?
d. Is the sample a probability sample?
e. Are there missing cases (“non-response”)? How many?

5. How are X and Y measured?

a. Does measurement rely on informants or respondents, or on other
sources?
b. Are informants or respondents trustworthy?
c. Are measurements valid?
d. Are measurements reliable?
CASE SELECTION
DATA MATRIX

 After having defined the research strategy, we are interested in

constructing the data matrix.
 This data matrix contains one column for each variable being
studied.
 The research strategy in combination with the data matrix define the
final effect size parameter in the study.

 Example of a data matrix:

Respondent ID Brand loyalty Purchase intention

1 7 8
2 6 4
3 6 3
4 4 6
POPULATION

 The first step is selecting cases for any study:

Choose a population from the theoretical domain (Chapter 1).

 Theoretical domain := the total set of all cases to which the

hypothesis applies.

 Population = a set of cases defined by one characteristic or by a

set of characteristics, subset of the theoretical domain

 An example:
 Theoretical domain = all alliances, all over the world, in all economic
sectors, at all times.
 Population = US alliances of airlines with a total turnover of at least [n]
dollars a year.

region industry size

CHARACTERISTICS
 SAMPLING

 When it is impossible to study the whole population (a census), one

has to resort to a sample (subset of the population)
 The process of selecting cases from the population into the sample is
called sampling

Examples  Two types of sampling techniques:

1. Probability sampling – elements in the population have a known
simple probability of ending up in the sample
random • Requires a complete and correct list of all elements of the population
sampling • Results in random/probability samples
(‘lottery’) • Allows for statistical inference from sample to population

2. Non-probability sampling – elements in the population have an

convenience unknown probability of ending up in the sample
sampling • Results in nonrandom/nonprobability samples
• Results only hold for data set
DATA SET ≠ SAMPLE

 A requirement for inferring the effect size in a population from the

effect size in the sample is that the sample is complete

 However we could have missing (measures of) cases

 We should establish whether these missing cases are at random

and not the result of any selective mechanism (selection bias or
non-response bias)

 Example:
o Study on the effect of workload on employee satisfaction
o Employees with a high workload are too busy to participate in a study
FROM DOMAIN TO DATA SET (DUL & HAK, 2008: 46)

x
x xx

= theoretical domain

= population of instances of the object of study

(“sampling frame” = list of all instances)

= sample of a population

= data set (smaller than sample due to missing instances)

X = instance of the object of study

 CHECKLIST FOR CRITICAL EVALUATION OF A REPORT

4. Which population is studied?

a. Is the population a part of the theoretical domain?
b. Is the population exactly defined and are its characteristics specified?
c. Is the whole population studied (“census”) or a sample?
d. Is the sample a probability sample?
e. Are there missing cases (“non-response”)? How many?
EXAMPLE: FONG ET AL. (2010)
4. WHICH POPULATION IS STUDIED?

motivation for population choice

page 634, Fong et al. (2010)

4A. IS THE POPULATION A PART OF THE THEORETICAL DOMAIN?

ABSTRACT, Fong et al. (2010)

4A. IS THE POPULATION A PART OF THE THEORETICAL DOMAIN?

Theoretical domain:
All firms with CEOs, at all times, across all locations & industries.

YES: the population is a subset of the domain!

4B. IS THE POPULATION DEFINED AND ARE ITS
CHARACTERISTICS SPECIFIED?
The population has been defined in terms of
firm size, location, time period
4C. IS THE WHOLE POPULATION STUDIED (“CENSUS”)
OR A SAMPLE?

Can be seen as a census of

the population
4C. IS THE WHOLE POPULATION STUDIED (“CENSUS”)
OR A SAMPLE?

A sample is studied.
4D. IS THE SAMPLE A PROBABILITY SAMPLE?

random (probability) sample

Note: the size of the initial sample is not provided

 4E. ARE THERE MISSING CASES (“NON-RESPONSE”)?
HOW MANY?

final sample = data set

YES: There are missing cases, due to for instance data availability

The authors describe the sample sizes of the different models, from which
we can derive the relative numbers of missing data, but there is no
indication of the absolute amount of missing data
 Compustat contains census
VISUAL SUMMARY of this population

All firms with CEO’s, at all times, across all locations and industries
1. Domain
X X X X
X
X X
X X
X
X
X
X X
X X
X
X
X X
X
X

2. Population: US publicly traded firms in 1995 with assets > $ 10 mio

3. Sample: random sample of population
4. Data set: per analysis after removing missing observations
MEASUREMENT
 MEASUREMENT

 Data matrix = the core of any study.

 The number of cases (i.e. rows in a data matrix) are decided upon prior
to conducting the study. This is why missing cases have to be reported.

 Each column represents a variable.

 How do we fill the cells in the data matrix? This is known as

“measurement”.

 The values that each variable can take must be clearly specified by
the researcher (e.g. satisfaction as an ordinal variable).

 This specification must be valid and the resulting scores must be

reliable.
EVIDENCE, DATA AND SCORES

It is useful to make a distinction between the following stages:

1. Collect relevant evidence
E.g. what a respondent has said in an interview is evidence that is collected
through a voice recorder.

2. Store it for further processing

E.g. The recording might be transcribed or summarized in a document, with the
voice recording being stored as a source of information for further processing
(e.g. tone of voice etc.).

3. Generate a final score

E.g. This data is then interpreted and categorized or coded in order to typify the
opinions that have been measured in for that particular respondent.
EVIDENCE, DATA AND SCORES

 A good measurement requires a lot of decisions which will influence:

1. The quality of the evidence;
2. The quality of the data;
3. The scores that will be analyzed in order to generate the effect size

 E.g. the measurement protocol is usually decided upon from the

outset of a study: How will “project success be operationalized?”
A. Financial success
B. Timely delivery
C. Satisfaction
EVALUATING THE QUALITY OF THE MEASUREMENT

1. Measurement validity = the extent to which procedures of data

collection and of coding can be considered to meaningfully capture
the ideas contained in the concept of which the value is measured.
Not assessed objectively, but rather as an outcome of argumentation
and discussion. I.e. can every detail of the procedure be justified in terms of
the researcher’s definition of the variable being measured?

2. Measurement reliability = the precision of the scores obtained by

the measurement.
This can be measured by, for example, re-measuring the same thing to
assess how much the two scores differ. The average difference between
these scores is the reliability score. This can be expressed as an inter-
observer, inter-rater, or test-retest reliability rate.
 CHECKLIST FOR CRITICAL EVALUATION OF A REPORT

5. How are X and Y measured?

a. Does measurement rely on informants or respondents, or on other
sources?
b. Are informants or respondents trustworthy?
c. Are measurements valid?
d. Are measurements reliable?
EXAMPLE: FONG ET AL. (2010)
RELEVANT HYPOTHESIS

 Hypothesis in theme:
“CEO compensation is positively related to firm performance”

CEO + Firm
overpayment profitability

 Corresponding hypothesis in article:

“H3: CEO overpayment is associated with subsequent
increases in firm profits”
Hence,
X: CEO overpayment
Y: subsequent change in firm profitability
5. HOW ARE X AND Y MEASURED?
Y = change in firm profitability

page 635, Fong et al. (2010)

 5. HOW ARE X AND Y MEASURED? X = CEO overpayment

page 635, Fong et al. (2010)

 5A. DOES THE MEASUREMENT RELY ON INFORMANTS
OR RESPONDENTS (OR OTHER SOURCES)?
Y = change in firm profitability

Measurement relies on observations

found in databases

page 634, Fong et al. (2010)

5A. DOES THE MEASUREMENT RELY ON INFORMANTS
OR RESPONDENTS (OR OTHER SOURCES)?
X = CEO overpayment

Inputs for the wage equation are

obtained from databases, proxy
statements, annual reports etc.
 5B. ARE THE INFORMANTS OR RESPONDENTS
TRUSTWORTHY?

These are well known databases that are very trustworthy

THE RELIABILITY OF (FINANCIAL) DATABASES

1. Most (if not all) databases offered by the EDSC (Erasmus Data
Service Center) are considered to be trustworthy.
2. If there is only a limited amount of data available, this may
hamper the intended statistical analysis as well as raise selection
bias issues.
3. The best way to assess whether a database is appropriate for
your research, is to read the database manual and (if necessary)
to make individual appointments.
4. The difficulty lies in understanding what the data represents,
and what processing steps you will take to address your research
question.

Source: Erasmus Data Service Center

THE RELIABILITY OF (FINANCIAL) DATABASES
5C. ARE MEASUREMENTS VALID?

Definition of CEO overpayment (X):

Measurement of CEO overpayment

 Using the wage score (residual) seems a valid approach

 However, equating CEO overpayment with a wage score in
the highest quartile can be contested
5C. ARE MEASUREMENTS VALID?
5C. ARE MEASUREMENTS VALID?
Y: subsequent change in firm profitability

On the one hand ROA is

commonly used measure
for firm profitability

But not without criticism,

market-based proxies for
profitability could have
been used as well
5D. ARE MEASUREMENTS RELIABLE?

Y: subsequent change in firm profitability

It seems safe to assume that the consulted databases result in

reliable measures of Return on Assets, which is the only moving
part in the definition of change in firm profitability.
5D. ARE MEASUREMENTS RELIABLE?

X: CEO overpayment

Hard to assess how reliable the “wage score is in highest quartile”

score is. Among others depends on:

1) Quality of the wage regression model – any omitted variables,

model fit?
2) Quality of the data set used for estimating the wage model –
would missing observations have changed the ranking of the
residuals?