Efficient designs

Lecture 3 (SEN1221 – part II)

Eric Molin – TU Delft

16-12-2019

Delft
University of
Technology

Challenge the future

Recap orthogonal fractional factorial designs

• orthogonal fractions from full factorial design

• main effects only
• no interactions (in principle)
• some interactions may be added

• characteristics
• Orthogonality (no correlations among attributes)
• assure high reliability of parameters
• attribute level balance
• all levels are observed an equal number of times

Regret in Traveler Decision Making 2

2
Recap constructing choice sets

• sequential construction
• no correlations within each alternative, but correlations
between alternatives
• apply for unlabeled alternatives
• all alternatives have the same attributes & levels

• simultaneous construction
• all correlations are 0
• apply for labeled alternatives
• attributes & levels (may) differ among alternatives
• allows estimation alternatieve specific parameters
• labels represent attributes that are not varied
• captured by alternative specific constant
Regret in Traveler Decision Making 3
3
Focus this lecture

• efficient designs
• experimental designs based on prior information

• advantages
• helps avoiding dominance
• may increase reliability parameters
• may reduce number of choice sets
• helps determining number of required respondents

• disadvantage
• if prior information is incorrect, parameters may be biased
• see lecture 6
Regret in Traveler Decision Making 4
4
why efficient designs?

stated choice
Regret experiments
in Traveler Decision Making 5
Assignment 1

• Which of the following choice sets will reveal more information

about the trade-offs between time and costs?

• choice set 1 A B
• time in min. 10 20
• costs €1 €2

• choice set 2 A B
• time in min. 10 20
• costs €2 €1

Regret in Traveler Decision Making 6

6
Answer to assignment 1
• in choice 1 A is a dominant alternative
• it is cheaper and faster
• at least better on one attribute, not worse on all others
• provides no information about trade-offs

• while choice set 2 reveals information:

• if A is chosen: willing to pay at least €1 to save 10 min
• Value of Time (VoT) is at least €6 per hour

• choice set 1 A B
• time in min. 10 20
• costs €1 €2

• choice set 2 A B
• time in min. 10 20
• costs €2 €1
Regret in Traveler Decision Making 7
7
dominant alternatives

• problem dominant alternatives

• reveals no information about trade-offs
• should be avoided  remove

• if dominant choice sets are removed:

• introduces correlations
• design becomes less efficient

• alternative
• efficient designs may reduce the problem

Regret in Traveler Decision Making 8

8
efficient designs

• aim to balance the utilities of alternatives in choice sets

• to avoid dominance (pA=100%, pB=0%)
• not exactly 50%-50%  why not?
• more like 30%-70% or 20%-80%

• efficient designs
• to maximize information about trade-offs
• to minimize standard errors of parameters

Regret in Traveler Decision Making 9

9
recap standard error (s.e.)

• s.e. indicates reliability of parameter b:

• 95% reliability interval: {b-1.96*s.e., b+1.96*s.e.}

• illustration:
• bcosts = -0.241 (in 100 euro) ; s.e. = -0.084
• 95% confidence interval: {-0.406, -0.076}

• implication VoT (btime = 0.231)

• VoT = 0.231/-(-0.241/100) = 95 €/hour
• VoT with highest and lowest bcosts-values: {57, 302}
• imagine s.e. was reduced by 2 (-0.042):
• 95% confidence interval would be: {71, 145}

• thus: smaller standard errors, higher reliability (less uncertainty)

Regret in Traveler Decision Making 10
10
standard error

• efficient designs result in smaller s.e.

with the same number of respondents
compared to orthogonal designs

orthogonal design

efficient design

source: Bliemer and Rose, 2006

Regret in Traveler Decision Making 11
11
Constructing efficient designs in Ngene

stated choice
Regret experiments
in Traveler Decision Making 12
priors

• priors
• best guesses on parameter values
• needed to balance utilities of the choice alternatives

• how to obtain priors?

• from previous research  literature
• conduct small pilot research (N=30)

• also needed:
• model (data generating process) & utility function
• we assume the model is MNL
• hence, a design is only efficient for a particular model
• and if priors are correct
Regret in Traveler Decision Making 13
13
 Ngene code for efficient designs

rows: number of choice sets (see later)

design eff: construct an efficient design
;alts = alt1, alt2, alt3 mnl: of model type MNL
;rows = 8 d: minimize the d-error (= d-efficient design)
;eff = (mnl,d)
;con con: include constant in utility calculations
;model: (this is not default!)
U(alt1) = b1[1.2] + b2[-0.6]*A[6,8,10,12] + b3[-0.4]*B[4,8] + b4[0.3] *C[0,1] /
U(alt2) = b5[0.8] + b2 *A + b3 *B + b6[0.8] *C /
U(alt3) = b2 *A + b7[-1.0]*C
$

bracket after parameter: [ ] contains the prior value

Note: these are all linear parameters for interval & ratio level variables
Regret in Traveler Decision Making 14
14
 Assignment 2
prior values placed in brackets: [ ]

U(alt1) = b1[1.2] + b2[-0.6]*A[6,8,10,12] + b3[-0.4]*B[4,8] + b4[0.3] *C[0,1] /

U(alt2) = b5[0.8] + b2 *A + b3 *B + b6[0.8] *C /
U(alt3) = b2 *A + b7[-1.0]*C

1. How many alternatives?

2. Are alternative specific constants included?
3. Which alternative is the reference?
4. Which attributes have the same levels for all alternatives?
5. Which parameters are estimated in a generic way?
6. Which parameters are estimated alternative specific?
7. How many parameters & constants are estimated?

Regret in Traveler Decision Making 15

15
 Answer to assignment 2

U(alt1) = b1[1.2] + b2[-0.6]*A[6,8,10,12] + b3[-0.4]*B[4,8] + b4[0.3] *C[0,1] /

U(alt2) = b5[0.8] + b2 *A + b3 *B + b6[0.8] *C /
U(alt3) = b2 *A + b7[-1.0]*C

1. Three alternatives
2. Yes, b1 and b5, which have different prior values
3. Alt.3  no prior included, so its constant = 0
4. All attributes have the same levels across the alternatives
• but B is not included in alt.3
5. b2 is estimated for all three alternatives
b3 is generic for alt.1 and alt.2 but does not play a role in alt.3
6. b4, b6, and b7 estimated alternative specific for attribute C
7. 7: five attribute parameters : b2, b3, b4, b6 and b7 and two
constants: b1 and b5 Regret in Traveler Decision Making 16
16
iterative process • Because in the syntax we
asked to minimize the D-
error (see later), designs are
evaluated on this criterion

• search algorithms constantly

change the design (other
value combinations)
• trail and error
• takes time
• iteration history is
shown
• it runs endlessly

• only a design with a lower D-

error is saved
• bold ones are in memory
Regret in Traveler Decision Making 17
and can be checked 17
checking utility balance

stated choice
Regret experiments
in Traveler Decision Making 18
predicting choice probabilities

• efficient design allows predicting choice probabilities:

• parameters  prior values
• alternatives  attribute values generated by the design
• model  MNL is assumed model utility function

• allows:
• checking whether dominance occurs
• calculating utilities & choice probabilities in choice sets
• determining the required number of respondents
• calculate standard errors of parameters

Regret in Traveler Decision Making 19

19
checking for dominance

• the closer an expected probability of an alternative is to 1…

• …the more limited the information about the trade-offs

• rough rule of thumb:

• probability of an alternative should be < 0.90

• steps:
• calculate expected utilities
• substitute design attribute levels in utility function
• calculate expected probabilities by MNL
• check for extreme values

Regret in Traveler Decision Making 20

20
 Assignment 3
Assume Ngene found the following efficient design

What is the expected utility for alt.1 in choice set 1 given

the assumed priors?
U(alt1) = b1[1.2] + b2[-0.6]*A[6,8,10,12] + b3[-0.4]*B[4,8] + b4[0.3] *C[0,1] /
U(alt2) = b5[0.8] + b2 *A + b3 *B + b6[0.8] *C /
U(alt3) = b2 *A + b7[-1.0]*C
Regret in Traveler Decision Making 21
21
 Answer to assignment 3

U(alt1) = b1[1.2] + b2[-0.6]*A[6,8,10,12] + b3[-0.4]*B[4,8] + b4[0.3] *C[0,1] /

U(alt2) = b5[0.8] + b2 *A + b3 *B + b6[0.8] *C /
U(alt3) = b2 *A + b7[-1.0]*C

substitute the design attribute values for A, B & C of alt1 in choice 1

in assumed utility function:
Valt1 = 1.2 + -0.6*A -0.4*B +0.3*C
Valt1 = 1.2 + -0.6*10 -0.4*4 +0.3*0 = -6.4
Regret in Traveler Decision Making 22
22
Ask Ngene to calculate expected utilities

check ‘design properties MNL’

and ‘Utilities’

Regret in Traveler Decision Making 23

23
Assignment 4

What is the expected probability for alt1 in choice set 1?

recall that the design is optimized for the MNL model

Regret in Traveler Decision Making 24

24
 Answer to assignment 4

MNL model: p(a1) = exp(Va1) / (exp(Va1) + exp(Va2) + exp(Va3))

p(a1) = exp(-6.4) / (exp(-6.4) + exp(-7.6) + exp(-4.6)) = 0.136

Regret in Traveler Decision Making 25

25
Assignment 5
Are some choice sets at risk of revealing limited information about
trade-offs?

Is any of the expected probabilities > 0.90?

Regret in Traveler Decision Making 26

26
Answer to assignment 5

No, none of the expected probabilities is > 0.90

across all choice sets, each alternative has a fair chance of being
Regret in Traveler Decision Making 27
chosen 27
number of respondents

stated choice
Regret experiments
in Traveler Decision Making 28
Number of respondents required

• question of interest
• how many respondents are needed to assure that
parameters are statistically significant?

• needed: standard errors (s.e.) of parameter (b)

• parameter is statistically significant if │t │ > 1.96
• recall: t = b / s.e.
• thus: │ b │ > 1.96 * s.e.

• s.e.
• is normally only known if you have data
• however, based on the priors we can predict utilities and
therefore choices for all constructed choice sets
• and thus calculate s.e. for every
Regret inparameter for
Traveler Decision N = 1 29
Making
29
how to obtain standard errors?

• standard error (s.e.) of parameter

• square root (√) of the variance

• asymptotic variance-covariance (AVC) matrix of parameters:

• variance of each parameter on diagonal
• covariances between the parameters off-diagonal

• AVC can be calculated

• taking the second derivative of the loglikelihood function
• depends on:
1. data generation process (=model)
2. (prior) parameter values
3. generated experimental design
Regret in Traveler Decision Making 30
30
AVC derivation

• E denotes that the expected value is calculated

• I = Fisher Information matrix
• X = attribute levels (thus the design)
• Y = choices (expected: based on priors & design)
• LL for many choice models (no panel structure):

• n = respondent; s = choice task; j = alternative

• If all respondents face the same choices, only one design needs
to be evaluated (N=1):

Regret in Traveler Decision Making 31

31
calculating the required N

• t(N=1) = b / s.e.
• b = prior
• expected s.e.

• statistical significance of b:│t │> 1.96

 higher t-value  increases significance
• realized by small s.e.

• s.e. of parameters of actual data is dependent on N

• N = number of respondents that completes the full design
• s.e. decreases with factor: 1/√N
• thus, t increases by factor √N: tN=1 * √N > 1.96
• N > (1.96 / tN=1)2 Regret in Traveler Decision Making 32
32
Assignment 6
• prior value of parameter b1 = 1.2

• Calculate the s.e. (square root of variance of parameter)

• Calculate the t-value for N=1 (t(N=1) = prior / s.e.)
• Calculate the required N to obtain statistical significance
• (N > (1.96 / t(N=1))2

Regret in Traveler Decision Making 33

33
Answer to assignment 6

1. Variance of parameter b1 (for N=1) = 5.7167

• standard error = root of variance  √ 5.7167 = 2.391

2. t(N=1) = b1 / s.e. = 1.2 / 2.391 = 0.5019

3. N > (1.96 / t(N=1))2

N > (1.96 / 0.5019)2
N > 15.25

• assuming the prior is correct then 16 respondents would result in

a statistically significant parameter b1
• if choice sets are blocked in 3 blocks: you need > 46!

Regret in Traveler Decision Making 34

34
Ngene output

SP estimate = required N to obtain statistical significance

Regret in Traveler Decision Making 35
35
Number of choice sets

stated choice
Regret experiments
in Traveler Decision Making 36
number of choice sets

• Ngene requires you to specify the number of choice sets (rows)

• typically fewer choice sets needed than for orthogonal designs

• because orthogonality is not preserved
• though attribute level balance is typically preserved
• may be relaxed, but is tricky

• number of required choice sets is determined by

• number of parameters to be estimated
• information obtained from each choice

Regret in Traveler Decision Making 37

37
information obtained from each choice

• estimation of one parameter requires 1 df.

• every choice set adds to df, depending on # alternatives

• if option A is chosen:
• choice set {A,B}  A ‘is better’ than B
• adds 1 degree of freedom
• choice set {A,B,C}  A ‘is better’ than B and C
• adds 2 degrees of freedom
• choice set {n alt.}  A ‘is better’ than n-1 alt.
• adds n-1 degrees of freedom

• requirement: degrees of freedom > number of parameters!

Regret in Traveler Decision Making 38
38
Minimum number of choice sets

• (# alternatives per choice set– 1) * # choice sets > # parameters

• # choice sets > #parameters / (#alternatives per choices set-1)

• thus, in case of 2 alternatives per choice set:

• min. number of choice sets = number of parameters + 1

• Warning (more details in lecture 4)

• testing for linearity requires two parameters
• nominal (qualitative) attributes may require more parameters
• these need to be included in the utility function
• and counted as parameters!

Regret in Traveler Decision Making 39

39
Assignment 7

What is the required absolute minimum number of choice sets?

(assuming that we do not wish to preserve attribute level balance)

1. 8 parameters, 2 alternatives per choice set

2. 8 parameters, 3 alternatives per choice set

Regret in Traveler Decision Making 40

40
Answer to assignment 7

1. 8 parameters, 2 alternatives per choice set

• every choice set adds 1 df.
• min. # choices sets = # parameters +1 = 9

2. 8 parameters, 3 alternatives per choice set

• every choice set adds 2 df.
• min. # choices sets = (# parameters / 2 ) + 1 = 4 + 1 = 5

Regret in Traveler Decision Making 41

41
preserving attribute level balance

• efficient designs do not automatically preserve attribute level

balance:
• choose number of choice sets that can be divided by the
number of levels of any attribute

• examples
• 8 can be divided by 2 and 4, but not by 3
• 9 can be divided by 3, but not by 2 and 4
• 12 can be divided by 2, 3 and 4

Regret in Traveler Decision Making 42

42
Assignment 8

• Assume:
• choice sets with a car and a train alternative
• attributes cost, time and comfort, each varies in 3 levels
• estimate linear alternative specific parameters

1. What is the required minimum number of choice sets?

2. What is the required minimum number if you wish to preserve

attribute level balance?

3. How many choice sets would an orthogonal design require?

Regret in Traveler Decision Making 43

43
Answer to assignment 8

1. number of parameters:
• 2 alt. with 3 attributes each = 6 + 1 constant = 7
• every choice set adds 1 df
• thus at least 8 choice sets required

2. 8 cannot be divided by 3
• 9 is the nearest larger number that can be divided by 3

3. Basic Plan 4 = 18 choice sets

• preserving orthogonality generally requires more choice sets

Regret in Traveler Decision Making 44

44
Assignment 9
• attributes
• ticket price: €600, €800, €1000
• Comfort: low, medium, high
• Travel time: 11, 13, 15 minutes
• Safety rating: 1 (=very unsafe), 4, 7 (=very safe)

• assume two unlabelled alternatives per choice set

• & utility function with generic linear parameters

1. What is the required minimum number of choice sets?

2. Would this result in attribute level balance?
3. Which nearest larger number would?
4. Would this allow to test for linearity?
5. To summarize: why is 9 chosenRegret
instead of 5?
in Traveler Decision Making 45
45
Answers to assignment 9

1. Two alts per choice set: add 1 df per choice set

• 4 parameter, thus at least 5 choice sets

2. No, 5 cannot be divided by 3?

3. 6

4. No, test for linearity requires 2 parameters per attribute. So this

would require estimating 8 parameters (see lecture 4)

5. a) it preserves attribute level balance

b) it allows testing for linearity (see lecture 4)
Regret in Traveler Decision Making 46
46
Example Ngene syntax (assignment 9)

Design
;alts = alt1,alt2
;rows = 9
;eff = (mnl,d)
;model:
U(alt1) = b1[-0.00441] *price[600,800,1000] +
b2[0.650] *comfort[0,1,2] +
b3[0.511] *safety[1,4,7] +
b4[-0.1] *time[11,13,15] /
U(alt2)= b1*price + b2*comfort + b3*safety + b4*time
$

Regret in Traveler Decision Making 47

47
Summary number of choice sets

• determine degrees of freedom per choice set

• N alternatives per choice set: adds N-1 df.

• determine number of parameters

• note: testing for non-linearity requires 2 parameters
• see next lecture

• select number of choice sets (rows) such that:

• number * df per choice set > number of parameters
• preferably preserve attribute level balance
• choose number that can be divided by number of levels

Regret in Traveler Decision Making 48

48
Use D-efficient or S-efficient design?

stated choice
Regret experiments
in Traveler Decision Making 49
type of efficient designs

• A-efficient designs:
• Minimizes the trace of AVC matrix
• trace = sum of values on main diagonal (variances= s.e.2)
• ignores the covariates between attributes
• not very useful if measurement units of attributes differ

• D-efficient designs:
• based on determinant of the AVC matrix
• takes both variance & covariances into account
• generally preferred

• S-efficient designs:
• minimizes the s.e. of the parameter for which it is hardest to
reach statistical significanceRegret in Traveler Decision Making 50
50
Assignment 10
1. Which design requires the least number of respondents?
2. Is S-efficient design better for all parameters?

S-efficient

D-efficient

Regret in Traveler Decision Making 51

51
 1. S-estimate (min. SP estimate)
lowest for S-efficient
Answer to assignment 10
2. But SP estimates of all other
S-efficient attributes are much higher!
• thus, you win reliability on
one parameter, but you
loose reliability on most
other parameters!

• Note that the D-

D-efficient error of the S-
efficient design is
much larger,
which indicates
an overall less
efficient design
Regret in Traveler Decision Making 52
52
Assignment 11

• We consider the same utility function with 3 alts. and attr. A, B, C

• Imagine attributes A and B are generic time and cost attributes
• Attribute C is an alternative-specific comfort attribute

Consider the two objectives below. For which objective would you
choose a D-efficient design and for which an S-efficient design?

1. objective 1: estimate the most reliable value-of-time indicators

comfort is mainly included to arrive at more realistic alternatives

2. objective 2: estimate the most reliable comfort parameter

Regret in Traveler Decision Making 53

53
Answer to assignment 11

1. choose D-efficient if objective is to estimate the most reliable

value-of-time indicators
• increases overall reliability, thus of both time and costs
• you accept that comfort may not become significant

2. choose S-efficient design if the objective is to estimate the most

reliable comfort parameter
• increases the reliability of comfort
• accept the lower reliability of the costs and time parameters

• thus: efficient design type choice depends on the objective!

Regret in Traveler Decision Making 54

54
Assignment 12

• Parameter b4 has highest SP estimate

• requires the highest number of observations to obtain statistical
significance

• Why is this the case? Could we have expected this just by looking
at the utility function?

U(alt1) = b1[1.2] + b2[-0.6]*A[6,8,10,12] + b3[-0.4]*B[4,8] + b4[0.3] *C[0,1] /

U(alt2) = b5[0.8] + b2 *A + b3 *B + b6[0.8] *C /
U(alt3) = b2 *A + b7[-1.0]*C

Regret in Traveler Decision Making 55

55
Answer to assignment 12

• yes, because:
1. parameters for attribute C are estimated alternative specific
• whereas all information about trade-offs for generic attribute
A can be used for estimating a single parameter
• the same amount of information regarding attribute C needs
to be divided across three parameters
• the estimation algorithms will then have more trouble finding
the exact values, thus more uncertainty (higher s.e.)
• thus, less reliable parameters, but parameters may be more
valid (less bias) compared to a single generic parameter for C

2. of the C attribute, parameter b4 has the smallest value

• suppose s.e. = 0,2 for b4, b6 and b7 (priors: 0,3; 0,8; -1.0)
• 95% reliability interval: b4 +/- 1.96*0,2 
• of {-0,92; 6,92} Regret
thusinnot statistically significant
Traveler Decision Making 56
56
summary

• D-error
• minimizes overall all attribute variances and covariances
• most parameters ‘benefit’
• default option

• S-error
• minimizes the s.e. of least reliable parameter
• only use if there is special interest in this parameter

Regret in Traveler Decision Making 57

57
Comparing design types

stated choice
Regret experiments
in Traveler Decision Making 58
efficiency

• efficiency in design
• a design is more efficient if it results in smaller s.e.’s of
parameters (for a given number of observations)

• lower d-error (is minimized)

• more efficient design

• if number of choice sets increases, d-error decreases

• does this require fewer observations?
= number of choice sets multiplied by the number of
required respondents

Regret in Traveler Decision Making 59

59
Assignment 13
• D-efficient designs are generated for the utility function on sl.14
• two- and four level attributes
• the number of choice sets (rows) is gradually increased
• required observations = S-estimate * # choice sets

1. Does the D-error decrease with increase in # choice sets?

2. And does the number of required observations decrease
considerably?
3. So does increasing the number of choice sets result in a more
efficient design?

Regret in Traveler Decision Making 60

60
Answer to Assignment 13

1. Yes, the D-error decreases with increase in # choice sets

2. No, the number of required observations is about the same,
around 1000 required observations
3. So increasing the number of choice sets does not result in a
more efficient design

• Note that the D-error cannot be compared among designs that

have different number of choice sets!

Regret in Traveler Decision Making 61

61
Assignment 14
1. Which design type requires the least number of observations
for all parameters to become statistically significant?
2. Is this also the most efficient design?

this is an orthogonal
designs, but the most
efficient of all possible
orthogonal fractions

Regret in Traveler Decision Making 62

62
Answer to assignment 14
1. S-efficient
- because it maximizes the
reliability of the most
difficult parameter.
- this is the basis for the S-
estimate

2. No, comparing the D-

errors among the designs
for the same number of
choice sets shows: the D-
error is always smallest for
D-efficient, followed by
orthogonal-efficient and
the least for S-efficient.

Regret in Traveler Decision Making 63

63
advantages efficient designs (vs. orthogonal)

• helps avoiding dominance

• by balancing utilities

• more reliable parameter

• with same number of respondents
• (or fewer respondents to arrive at the same reliability)

• requires a smaller number of choice sets

• Information about trade-offs is maximized
• And orthogonality is not required

• allows easy calculation of required number of observations

• assuming priors are correct
Regret in Traveler Decision Making 64
64
disadvantages efficient designs (vs. orthogonal)

• requires more effort

• finding good priors – may require pilot research
• requires many extra decisions
• warning: higher risk at making mistakes!

• risk of wrong priors

• then less efficient & biased parameters
• thus priors may affect parameters
• but not fully understood yet

• may be less flexible

• typically only optimized for one model type and one utility
specification
• e.g. if you only specify linear parameters, you may
not be able to test Regret
for non-linearities
in Traveler Decision Making 65
65
If uncertainty about priors

• certainty about sign, but no idea about magnitude

• choose very small priors with correct sign

• certainty about sign, but uncertainty about magnitude:

• consider Bayesian efficient designs (not covered)
• max. 6-8 parameters

• no idea about sign and magnitude

• choose prior = 0
• warning: meant for a single attribute  don’t do this for
all attributes because this will result in a random design!
• then better choose another design strategy
Regret in Traveler Decision Making 66
66
Efficient design always more efficient?

• only if:
• assumed priors are approximately correct
• wrong sign?  orthogonal more efficient
• assumed data generation process is correct
• MNL is a good approximation for mixed logit
• but not for panel mixed logit

• particularly use if:

1. number of respondents is limited
2. dominance is a severe problem

• otherwise, use orthogonal designs!

• traditional, but is still good!
Regret in Traveler Decision Making 67
67
Next lecture

• linearity
• testing for linearity

• coding nominal variables (dummy & effects)

• to include qualitative attributes
• to include priors for non-linear parameters in efficient
designs

• designs for complex attributes

• attributes of which the values are determined by other
attributes (e.g.: comfort, safety)

Regret in Traveler Decision Making 68

68