(Original PDF) Categorical Data

Analysis 3rd Edition by Alan Agresti

Visit to download the full and correct content document:
https://ebooksecure.com/download/original-pdf-categorical-data-analysis-3rd-edition-
by-alan-agresti/
 Contents

Preface

1 Introduction: Distributions and Inference for Categorical Data

1.1 Categorical Response Data, 1

1.2 Distributions for Categorical Data, 5
1.3 Statistical Inference for Categorical Data, 8
1.4 Statistical Inference for Binomial Parameters, 13
1.5 Statistical Inference for Multinomial Parameters, 17
1.6 Bayesian Inference for Binomial and Multinomial Parameters, 22
Notes, 27
Exercises, 28

2 Describing Contingency Tables

2.1 Probability Structure for Contingency Tables, 37

2.2 Comparing Two Proportions, 43
2.3 Conditional Association in Stratified 2 x 2 Tables, 47
2.4 Measuring Association in / x J Tables, 54
Notes, 60
Exercises, 60

3 Inference for Two-Way Contingency Tables

3.1 Confidence Intervals for Association Parameters, 69

3.2 Testing Independence in Two-way Contingency Tables, 75
3.3 Following-up Chi-Squared Tests, 80
3.4 Two-Way Tables with Ordered Classifications, 86
3.5 Small-Sample Inference for Contingency Tables, 90
3.6 Bayesian Inference for Two-way Contingency Tables, 96
3.7 Extensions for Multiway Tables and Nontabulated Responses, 100
Notes, 101
Exercises, 103
viii CONTENTS

4 Introduction to Generalized Linear Models 113

4.1 The Generalized Linear Model, 113

4.2 Generalized Linear Models for Binary Data, 117
4.3 Generalized Linear Models for Counts and Rates, 122
4.4 Moments and Likelihood for Generalized Linear Models, 130
4.5 Inference and Model Checking for Generalized Linear Models, 136
4.6 Fitting Generalized Linear Models, 143
4.7 Quasi-Likelihood and Generalized Linear Models, 149
Notes, 152
Exercises, 153

5 Logistic Regression 163

5.1 Interpreting Parameters in Logistic Regression, 163

5.2 Inference for Logistic Regression, 169
5.3 Logistic Models with Categorical Predictors, 175
5.4 Multiple Logistic Regression, 182
5.5 Fitting Logistic Regression Models, 192
Notes, 195
Exercises, 196

6 Building, Checking, and Applying Logistic Regression Models 207

6.1 Strategies in Model Selection, 207

6.2 Logistic Regression Diagnostics, 215
6.3 Summarizing the Predictive Power of a Model, 221
6.4 Mantel-Haenszel and Related Methods for Multiple 2 x 2 Tables, 225
6.5 Detecting and Dealing with Infinite Estimates, 233
6.6 Sample Size and Power Considerations, 237
Notes, 241
Exercises, 243

7 Alternative Modeling of Binary Response Data 251

7.1 Probit and Complementary Log-log Models, 251

7.2 Bayesian Inference for Binary Regression, 257
7.3 Conditional Logistic Regression, 265
7.4 Smoothing: Kernels, Penalized Likelihood, Generalized
Additive Models, 270
7.5 Issues in Analyzing High-Dimensional Categorical Data, 278
Notes, 285
Exercises, 287
CONTENTS IX

8 Models for Multinomial Responses 293

8.1 Nominal Responses: Baseline-Category Logit Models, 293

8.2 Ordinal Responses: Cumulative Logit Models, 301
8.3 Ordinal Responses: Alternative Models, 308
8.4 Testing Conditional Independence in / x / x K Tables, 314
8.5 Discrete-Choice Models, 320
8.6 Bayesian Modeling of Multinomial Responses, 323
Notes, 326
Exercises, 329

9 Loglinear Models for Contingency Tables 339

9.1 Loglinear Models for Two-way Tables, 339

9.2 Loglinear Models for Independence and Interaction in Three-way
Tables, 342
9.3 Inference for Loglinear Models, 348
9.4 Loglinear Models for Higher Dimensions, 350
9.5 Loglinear—Logistic Model Connection, 353
9.6 Loglinear Model Fitting: Likelihood Equations and Asymptotic
Distributions, 356
9.7 Loglinear Model Fitting: Iterative Methods and Their Application, 364
Notes, 368
Exercises, 369

10 Building and Extending Loglinear Models 377

10.1 Conditional Independence Graphs and Collapsibility, 377

10.2 Model Selection and Comparison, 380
10.3 Residuals for Detecting Cell-Specific Lack of Fit, 385
10.4 Modeling Ordinal Associations, 386
10.5 Generalized Loglinear and Association Models, Correlation Models,
and Correspondence Analysis, 393
10.6 Empty Cells and Sparseness in Modeling Contingency Tables, 398
10.7 Bayesian Loglinear Modeling, 401
Notes, 404
Exercises, 407

11 Models for Matched Pairs 413

11.1 Comparing Dependent Proportions, 414

11.2 Conditional Logistic Regression for Binary Matched Pairs, 418
11.3 Marginal Models for Square Contingency Tables, 424
11.4 Symmetry, Quasi-Symmetry, and Quasi-Independence, 426
X CONTENTS

11.5 Measuring Agreement Between Observers, 432

11.6 Bradley-Terry Model for Paired Preferences, 436
11.7 Marginal Models and Quasi-Symmetry Models for Matched Sets, 439
Notes, 443
Exercises, 445

12 Clustered Categorical Data: Marginal and Transitional Models 455

12.1 Marginal Modeling: Maximum Likelihood Approach, 456

12.2 Marginal Modeling: Generalized Estimating Equations (GEEs)
Approach, 462
12.3 Quasi-Likelihood and Its G E E Multivariate Extension: Details, 465
12.4 Transitional Models: Markov Chain and Time Series Models, 473
Notes, 478
Exercises, 479

13 Clustered Categorical Data: Random Effects Models 489

13.1 Random Effects Modeling of Clustered Categorical Data, 489

13.2 Binary Responses: Logistic-Normal Model, 494
13.3 Examples of Random Effects Models for Binary Data, 498
13.4 Random Effects Models for Multinomial Data, 511
13.5 Multilevel Modeling, 515
13.6 G L M M Fitting, Inference, and Prediction, 519
13.7 Bayesian Multivariate Categorical Modeling, 523
Notes, 525
Exercises, 527

14 Other Mixture Models for Discrete Data 535

14.1 Latent Class Models, 535

14.2 Nonparametric Random Effects Models, 542
14.3 Beta-Binomial Models, 548
14.4 Negative Binomial Regression, 552
14.5 Poisson Regression with Random Effects, 555
Notes, 557
Exercises, 558

15 Non-Model-Based Classification and Clustering 565

15.1 Classification: Linear Discriminant Analysis, 565

15.2 Classification: Tree-Structured Prediction, 570
15.3 Cluster Analysis for Categorical Data, 576
Notes, 581
Exercises, 582
CONTENTS xi

16 Large- and Small-Sample Theory for Multinomial Models 587

16.1 Delta Method, 587

16.2 Asymptotic Distributions of Estimators of Model Parameters and Cell
Probabilities, 592
16.3 Asymptotic Distributions of Residuals and Goodness-of-fit Statistics, 594
16.4 Asymptotic Distributions for Logit/Loglinear Models, 599
16.5 Small-Sample Significance Tests for Contingency Tables, 601
16.6 Small-Sample Confidence Intervals for Categorical Data, 603
16.7 Alternative Estimation Theory for Parametric Models, 610
Notes, 615
Exercises, 616

17 Historical Tour of Categorical Data Analysis 623

17.1 Pearson-Yule Association Controversy, 623

17.2 R. A. Fisher's Contributions, 625
17.3 Logistic Regression, 627
17.4 Multiway Contingency Tables and Loglinear Models, 629
17.5 Bayesian Methods for Categorical Data, 633
17.6 A Look Forward, and Backward, 634

Appendix A Statistical Software for Categorical Data Analysis 637

Appendix B Chi-Squared Distribution Values 641

References 643

Author Index 689

Example Index 701

Subject Index 705

Appendix C Software Details for Text Examples (text website)

(www.stat.ufl.edu/~aa/cda/cda.html)

Appendix D Solutions to Selected Exercises (text website)

(www. s t a t . u f 1 . e d u / ~ a a / c d a / c d a . h t m l )
 Preface

The explosion in the development of methods for analyzing categorical data that began in
the 1960s has continued apace in recent years. This book provides an overview of these
methods, as well as older, now standard, methods. It gives special emphasis to generalized
linear modeling techniques, which extend linear model methods for continuous variables,
and their extensions for multivariate responses.

O U T L I N E OF TOPICS

Chapters 1 - 1 0 present the core methods for categorical response variables. Chapters 1 - 3
cover distributions for categorical responses and traditional methods for two-way contin-
gency tables. Chapters 4-8 introduce logistic regression and related models such as the
probit model for binary and multicategory response variables. Chapters 9 and 10 cover
loglinear models for contingency tables.
In the past quarter century, a major area of new research has been the development of
methods for repeated measurement and other forms of clustered categorical data. Chapters
11-14 present these methods, including marginal models and generalized linear mixed
models with random effects. Chapter 15 introduces non-model-based methods for classi-
fication and clustering. Chapter 16 presents theoretical foundations as well as alternatives
to the maximum likelihood paradigm that this text adopts. Chapter 17 is devoted to a
historical overview of the development of the methods. It examines contributions of noted
statisticians, such as Pearson and Fisher, whose pioneering efforts—and sometimes vocal
debates—broke the ground for this evolution.
Appendices illustrate the use of statistical software for analyzing categorical data. The
website for the text, w w w . s t a t . u f l . e d u / ~ a a / c d a / c d a . h t m l , contains an appendix
with detailed examples of the use of software (especially R, SAS, and Stata) for performing
the analyses in this book, solutions to many of the exercises, extra exercises, and corrections.

C H A N G E S IN THIS EDITION

Given the explosion of research in the past 50 years on categorical data methods, it is an
increasing challenge to write a comprehensive book covering all the commonly used meth-
ods. The second edition of this book already exceeded 700 pages. In including much new

xiii
xiv PREFACE

material without letting the book grow much, I have necessarily had to make compromises
in depth and use relatively simple examples. I try to present a broad overview, while pre-
senting bibliographic notes with many references in which the reader can find more details.
In attempting to make the book relatively comprehensive while presenting substantive new
material, every chapter of the first two editions has been extensively rewritten. The major
changes are:

• A new Chapter 7 presents alternative methods for binary response data, including
some regularization methods that are becoming popular in this age of massive data
sets with enormous numbers of variables.
• A new Chapter 15 introduces non-model-based methods of classification, such as
linear discriminant analysis and classification trees, and cluster analysis.
• Many chapters now include a section describing the Bayesian approach for the meth-
ods of that chapter. We also have added material (e.g., Sections 6.5 and 7.4) about ways
that frequentist methods can deal with awkward situations such as infinite maximum
likelihood estimates.
• The use of various software for categorical data methods is discussed at a much ex-
panded website for the text, www. s t a t . u f 1 . e d u / ~ a a / c d a / c d a . h t m l . Examples
are shown of the use of R, SAS, and Stata for most of the examples in the text, and
there is discussion also about SPSS, StatXact, and other software. That website also
contains many of the text's data sets, some of which have only excerpts shown in the
text itself, as well as solutions for many exercises and corrections of errors found in
early printings of the book. I recommend that you refer to this appendix (or special-
ized software manuals) while reading the text, perhaps printing the pages about the
software you prefer, as an aid to implementing the methods. This material was placed
at the website partly because the text is already so long without it and also because it
is then easier to keep the presentation up-to-date.

In this text, I interpret categorical data analysis to refer to methods for categorical
response variables. For most methods, explanatory variables can be categorical or quan-
titative, as in ordinary regression. Thus, the focus is intended to be more general than
contingency table analysis, although for simplicity of data presentation, most examples use
contingency tables. These examples are simplistic, but should help you focus on under-
standing the methods themselves and make it easier for you to replicate results with your
favorite software.
Other special features of the text include:

• More than 100 analyses of data sets.

• About 600 exercises, some directed toward theory and methods and some toward
applications and data analysis.
• Notes at the end of each chapter that provide references for recent research and many
topics not covered in the text, linked to a bibliography of more than 1200 sources.

INTENDED AUDIENCE AND USE AS A TEXTBOOK

I intend this book to be accessible to the diverse mix of students who take graduate-level
courses in categorical data analysis. But I have also written it with practicing statisticians
PREFACE XV

and biostatisticians in mind. I hope it enables them to catch up with recent advances and
learn about methods that sometimes receive inadequate attention in the traditional statistics
curriculum.
The development of new methods has influenced—and been influenced by—the in-
creasing availability of data sets with categorical responses in the social, behavioral, and
biomedical sciences, as well as in public health, genetics, ecology, education, marketing and
the financial industry, and industrial quality control. And so, although this book is directed
mainly to statisticians and biostatisticians, I also aim for it to be helpful to methodologists
in these fields.
Readers should possess a background that includes regression and analysis of variance
models, as well as maximum likelihood methods of statistical theory. Those not having
much theory background should be able to follow most methodological discussions. Those
with mainly applied interests can skip most of Chapter 4 on the theory of generalized linear
models and proceed to other chapters. However, the book has a distinctly higher technical
level and is more thorough and complete than my lower-level text, An Introduction to
Categorical Data Analysis, Second Edition (Wiley, 2007).
Today, because of the ubiquity of categorical data in applications, most statistics and
biostatistics departments offer courses on categorical data analysis or on generalized linear
models with strong emphasis on methods for discrete data. This book can be used as a text
for such courses. The material in Chapters 1 - 6 forms the heart of most courses. There is
too much material in this book for a single course, but a one-term course can be based on
the following outline:

• Basic contingency table analysis, covering Chapters 1 - 3 , perhaps skipping some

tangential sections such as 1.5.7, 1.6, 2.4, 3.4-3.7.
• Logistic regression and related methods for binary data, covering Chapters 4 - 6 ,
perhaps skipping some tangential sections such as 4.4-4.7 and 6.4-6.6.
• Multinomial response models, covering at least Sections 8.1 and 8.2.
• Matched pairs and clustered data, covering at least Sections 11.1-11.2.

Courses with biostatistical orientation may want to include bits f r o m Chapters 12 and 13
on marginal and random effects models. Courses with social science emphasis may want
to include some topics on loglinear modeling f r o m Chapters 9 and 10. Some courses may
want to select specialized topics from Chapter 7, such as probit modeling, conditional
logistic regression, Bayesian binary data modeling, smoothing, and issues in the analysis
of high-dimensional data.

ACKNOWLEDGMENTS

I thank those who commented on parts of the manuscript or provided help of some type.
Special thanks to Anna Gottard, David Hoaglin, Maria Kateri, Bernhard Klingenberg, Keli
Liu, and Euijung Ryu, who gave insightful comments on some chapters and made many
helpful suggestions, and Brett Presnell for his advice and resources about R software and his
comments about some of the material. Thanks to people who made suggestions about new
material for this edition, including Jonathan Bischof, James Booth, Brian Caffo, Tianxi
Cai, Brent Coull, Nicholas Cox, Ralitza Gueorguieva, Debashis Ghosh, John Henretta,
xvi PREFACE

David Hitchcock, Galin Jones, Robert Kushler, Xihong Lin, Jun Liu, Gianfranco Lovison,
Giovanni Marchetti, David Olive, Art Owen, Alessandra Petrucci, Michael Radelet, Gerard
Scallan, Maura Stokes, Anestis Touloumis, and Ming Yang. Thanks to those who com-
mented on aspects of the second edition, including pointing out errors or typos, such as
Pat Altham, Roberto Bertolusso, Nicholas Cox, David Firth, Rene Gonin, David Hoaglin,
Harry Khamis, Bernhard Klingenberg, Robert Kushler, Gianfranco Lovison, Theo Nijsse,
Richard Reyment, Misha Salganik, William Santo, Laura Thompson, Michael Vock, and
Zhongming Yang. Thanks also to Laura Thompson for preparing her very helpful manual
on using R and S-Plus for examples in the second edition. Thanks to the many who re-
viewed material or suggested examples for the first two editions, mentioned in the Prefaces
of those editions. Thanks also to Wiley Executive Editor Steve Quigley and Associate
Editor Jacqueline Palmieri for their steadfast encouragement and facilitation of this project.
Finally, thanks to my wife Jacki Levine for continuing support of all kinds.

A L A N AGRESTI

Gainesville, Florida and Brookline, Massachusetts

February 2012
CHAPTER 1

Introduction: Distributions and

Inference for Categorical Data

From helping to assess the value of new medical treatments to evaluating the factors that
affect our opinions and behaviors, analysts today are finding myriad uses for categorical
data methods. In this book we introduce these methods and the theory behind them.
Statistical methods for categorical responses were late in gaining the level of sophistica-
tion achieved early in the twentieth century by methods for continuous responses. Despite
influential work around 1900 by the British statistician Karl Pearson, relatively little de-
velopment of models for categorical responses occurred until the 1960s. In this book we
describe the early fundamental work that still has importance today but place primary
emphasis on more recent modeling approaches.

1.1 C A T E G O R I C A L R E S P O N S E DATA

A categorical variable has a measurement scale consisting of a set of categories. For

instance, political philosophy is often measured as liberal, moderate, or conservative. Diag-
noses regarding breast cancer based on a mammogram use the categories normal, benign,
probably benign, suspicious, and malignant.
The development of methods for categorical variables was stimulated by the need to
analyze data generated in research studies in both the social and biomedical sciences.
Categorical scales are pervasive in the social sciences for measuring attitudes and opinions.
Categorical scales in biomedical sciences measure outcomes such as whether a medical
treatment is successful.
Categorical data are by no means restricted to the social and biomedical sciences. They
frequently occur in the behavioral sciences (e.g., type of mental illness, with the categories
schizophrenia, depression, neurosis), epidemiology and public health (e.g., contraceptive
method at last sexual intercourse, with the categories none, condom, pill, IUD, other),
genetics (type of allele inherited by an offspring), botany and zoology (e.g., whether or
not a particular organism is observed in a sampled quadrat), education (e.g., whether a stu-
dent response to an exam question is correct or incorrect), and marketing (e.g., consumer

Categorical Data Analysis, Third Edition. Alan Agresti.

© 2 0 1 3 John Wiley & Sons, Inc. Published 2013 by John Wiley & Sons, Inc.

1
2 INTRODUCTION: DISTRIBUTIONS AND INFERENCE FOR CATEGORICAL DATA

preference among the three leading brands of a product). They even occur in highly quan-
titative fields such as engineering sciences and industrial quality control. Examples are the
classification of items according to whether they conform to certain standards, and subjec-
tive evaluation of some characteristic: how soft to the touch a certain fabric is, how good a
particular food product tastes, or how easy a worker finds it to perform a certain task.
Categorical variables are of many types. In this section we provide ways of classifying
them.

1.1.1 Response-Explanatory Variable Distinction

Statistical analyses distinguish between response (or dependent) variables and explana-
tory (or independent) variables. This book focuses on methods for categorical response
variables. As in ordinary regression modeling, explanatory variables can be any type. For
instance, a study might analyze how opinion about whether same-sex marriages should be
legal (yes or no) changes according to values of explanatory variables, such as religious
affiliation, political ideology, number of years of education, annual income, age, gender,
and race.

1.1.2 Binary-Nominal-Ordinal Scale Distinction

Many categorical variables have only two categories. Such variables, for which the two
categories are often given the generic labels "success" and "failure," are called binary
variables. A major topic of this book is the modeling of binary response variables.
When a categorical variable has more than two categories, we distinguish between
two types of categorical scales. Variables having categories without a natural ordering are
said to be measured on a nominal scale and are called nominal variables. Examples are
mode of transportation to get to work (automobile, bicycle, bus, subway, walk), favorite
type of music (classical, country, folk, jazz, rock), and choice of residence (apartment,
condominium, house, other). For nominal variables, the order of listing the categories is
irrelevant to the statistical analysis.
Many categorical variables do have ordered categories. Such variables are said to be
measured on an ordinal scale and are called ordinal variables. Examples are social class
(upper, middle, lower), political philosophy (very liberal, slightly liberal, moderate, slightly
conservative, very conservative), patient condition (good, fair, serious, critical), and rating
of a movie for Netflix (1 to 5 stars, representing hated it, didn't like it, liked it, really liked
it, loved it). For ordinal variables, distances between categories are unknown. Although
a person categorized as very liberal is more liberal than a person categorized as slightly
liberal, no numerical value describes how much more liberal that person is.
An interval variable is one that does have numerical distances between any two values.
For example, systolic blood pressure level, length of prison term, and annual income are
interval variables. For most such variables, it is also possible to compare two values by
their ratio, in which case the variable is also called a ratio variable.
The way that a variable is measured determines its classification. For example, "educa-
tion" is only nominal when measured as (public school, private school, home schooling);
it is ordinal when measured by highest degree attained, using the categories (none, high
school, bachelor's, master's, doctorate); it is interval when measured by number of years
of education completed, using the integers 0, 1 , 2 , 3,
CATEGORICAL RESPONSE DATA 3

A variable's measurement scale determines which statistical methods are appropriate.

It is usually best to apply methods appropriate for the actual scale. In the measurement
hierarchy, interval variables are highest, ordinal variables are next, and nominal variables
are lowest. Statistical methods for variables of one type can also be used with variables at
higher levels but not at lower levels. For instance, statistical methods for nominal variables
can be used with ordinal variables by ignoring the ordering of categories. Methods for
ordinal variables cannot, however, be used with nominal variables, since their categories
have no meaningful ordering. The distinction between ordered and unordered categories
is not important for binary variables, because ordinal methods and nominal methods then
typically reduce to equivalent methods.
In this book, we present methods for the analysis of binary, nominal, and ordinal
variables. The methods also apply to interval variables having a small number of distinct
values (e.g., number of times married, number of distinct side effects experienced in taking
some drug) or for which the values are grouped into ordered categories (e.g., education
measured as < 1 2 years, > 1 2 but < 1 6 years, > 1 6 years).

1.1.3 Discrete-Continuous Variable Distinction

Variables are classified as discrete or continuous, according to whether the number of

values they can take is countable. Actual measurement of all variables occurs in a discrete
manner, due to precision limitations in measuring instruments. The discrete-continuous
classification, in practice, distinguishes between variables that take few values and variables
that take lots of values. For instance, statisticians often treat discrete interval variables having
a large number of values (such as test scores) as continuous, using them in methods for
continuous responses.
This book deals with certain types of discretely measured responses: (1) binary vari-
ables, (2) nominal variables, (3) ordinal variables, (4) discrete interval variables hav-
ing relatively few values, and (5) continuous variables grouped into a small number of
categories.

1.1.4 Quantitative-Qualitative Variable Distinction

Nominal variables are qualitative—distinct categories differ in quality, not in quantity. In-
terval variables are quantitative—distinct levels have differing amounts of the characteristic
of interest. The position of ordinal variables in the qualitative-quantitative classification
is fuzzy. Analysts often treat them as qualitative, using methods for nominal variables.
But in many respects, ordinal variables more closely resemble interval variables than they
resemble nominal variables. They possess important quantitative features: Each category
has a greater or smaller magnitude of the characteristic than another category; and although
not possible to measure, an underlying continuous variable is often present. The political
ideology classification (very liberal, slightly liberal, moderate, slightly conservative, very
conservative) crudely measures an inherently continuous characteristic.
Analysts often utilize the quantitative nature of ordinal variables by assigning numerical
scores to the categories or assuming an underlying continuous distribution. This requires
good judgment and guidance from researchers who use the scale, but it provides benefits
in the variety of methods available for data analysis.
4 INTRODUCTION: DISTRIBUTIONS AND INFERENCE FOR CATEGORICAL DATA

1.1.5 Organization of Book and Online Computing Appendix

The models for categorical response variables discussed in this book resemble regres-
sion models for continuous response variables; however, they assume binomial or multi-
nomial response distributions instead of normality. One type of model receives special
attention—logistic regression. Ordinary logistic regression models apply with binary re-
sponses and assume a binomial distribution. Generalizations of logistic regression apply
with multicategory responses and assume a multinomial distribution.
The book has four main units. In the first, Chapters 1 through 3, we summarize descriptive
and inferential methods for univariate and bivariate categorical data. These chapters cover
discrete distributions, methods of inference, and measures of association for contingency
tables. They summarize the non-model-based methods developed prior to about 1960.
In the second and primary unit, Chapters 4 through 10, we introduce models for cate-
gorical responses. In Chapter 4 we describe a class of generalized linear models having
models of this text as special cases. Chapters 5 and 6 cover the most important model for bi-
nary responses, logistic regression. Chapter 7 presents alternative methods for binary data,
including the probit, Bayesian fitting, and smoothing methods. In Chapter 8 we present
generalizations of the logistic regression model for nominal and ordinal multicategory
response variables. In Chapters 9 and 10 we introduce the modeling of multivariate cate-
gorical response data, in terms of association and interaction patterns among the variables.
The models, called loglinear models, apply to counts in the table that cross-classifies those
responses.
In the third unit, Chapters 11 through 14, we discuss models for handling repeated
measurement and other forms of clustered data. In Chapter 11 we present models for
a categorical response with matched pairs; these apply, for instance, with a categorical
response measured for the same subjects at two times. Chapter 12 covers models for more
general types of repeated categorical data, such as longitudinal data f r o m several times
with explanatory variables. In Chapter 13 we present a broad class of models, generalized
linear mixed models, that use random effects to account for dependence with such data. In
Chapter 14 further extensions of the models from Chapters 11 through 13 are described,
unified by treating the response as having a mixture distribution of some type.
The fourth and final unit has a different nature than the others. In Chapter 15 we consider
non-model-based classification and clustering methods. In Chapter 16 we summarize large-
sample and small-sample theory for categorical data models. This theory is the basis for
behavior of model parameter estimators and goodness-of-fit statistics. Chapter 17 presents
a historical overview of the development of categorical data methods.
Maximum likelihood methods receive primary attention throughout the book. Many
chapters, however, contain a section presenting corresponding Bayesian methods.
In Appendix A we review software that can perform the analyses in this book. The
website w w w . s t a t . u f l . e d u / ~ a a / c d a / c d a . h t m l for this book contains an appendix
that gives more information about using R, SAS, Stata, and other software, with sample
programs for text examples. In addition, that site has complete data sets for many text
examples and exercises, solutions to some exercises, extra exercises, corrections, and links
to other useful sites. For instance, a manual prepared by Dr. Laura Thompson provides
examples of how to use R and S-Plus for all examples in the second edition of this text,
many of which (or very similar ones) are also in this edition.
In the rest of this chapter, we provide background material. In Section 1.2 we review the
key distributions for categorical data: the binomial and multinomial, as well as another that
DISTRIBUTIONS FOR CATEGORICAL DATA 5

is important for discrete data, the Poisson. In Section 1.3 we review the primary mechanisms
for statistical inference using maximum likelihood. In Sections 1.4 and 1.5 we illustrate
these by presenting significance tests and confidence intervals for binomial and multinomial
parameters. In Section 1.6 we introduce Bayesian inference for these parameters.

1.2 D I S T R I B U T I O N S F O R C A T E G O R I C A L DATA

Inferential data analyses require assumptions about the random mechanism that generated
the data. For regression models with continuous responses, the normal distribution plays the
central role. In this section we review the three key distributions for categorical responses:
binomial, multinomial, and Poisson.

1.2.1 Binomial Distribution

Many applications refer to a fixed number n of binary observations. Let yi, y2,..., y„
denote observations f r o m n independent and identical trials such that P(Yj — 1) = n and
P(Yj: = 0) = 1 — n . We refer to outcome 1 as "success" and outcome 0 as "failure."
Identical trials means that the probability of success it is the same for each trial. Independent
trials means that the {F,} are independent random variables. These are often called Bernoulli
trials. The total number of successes, Y = i ^ , has the binomial distribution with index
n and parameter tc, denoted by bin(n, n).
The probability mass function for the possible outcomes y for Y is

y = 0, 1 , 2 , . . . , « , (1.1)

where the binomial coefficient ( = n\/[y\{n - y)!]. Since E(Yj) = E ( Y f ) = 1 X J T +

0 x (1 —7t) = n ,

E(Yi) = 7T and var(F ; ) = n(l - n).

The binomial distribution for Y = Yi h a s mean and variance

lx = E(Y) = nn and a2 = var(K) = nn{\ — it).

The skewness is described by E(Y - /h)3/CT3 = (1 - In j/^nnd - it). The distribution

is symmetric when n — 0.50 but becomes increasingly skewed as n moves toward either
boundary. The binomial distribution converges to normality as n increases, for fixed 7r, the
approximation being reasonable 1 when n[min(7r, 1 — 7r)| is as small as about 5.
There is no guarantee that successive binary observations are independent or identical.
Thus, occasionally, we will utilize other distributions. One such case is sampling binary
outcomes without replacement f r o m a finite population, such as observations on whether a
homework assignment was completed for 10 students sampled f r o m a class of size 20. The

'See w w w . s t a t . t a m u . e d u / ~ w e s t / a p p l e t s / b i n o m i a l d e m o 2 . h t m l .
6 INTRODUCTION: DISTRIBUTIONS AND INFERENCE FOR CATEGORICAL DATA

hypergeometric distribution, studied in Section 3.5.1, is then relevant. In Section 1.2.4 we

discuss another case that violates the binomial assumptions.

1.2.2 Multinomial Distribution

Some trials have more than two possible outcomes. Suppose that each of n independent,
identical trials can have outcome in any of c categories. Let y,, = 1 if trial i has outcome
in category j and y,> = 0 otherwise. Then y, = (>', i, y^, • • •, y,-r) represents a multinomial
trial, with J ^ j >'/; = 1; for instance, (0, 0, 1 , 0 ) denotes outcome in category 3 of four
possible categories. Note that y, c is redundant, being linearly dependent on the others.
Let n j — yij denote the number of trials having outcome in category j. The counts
(«i, «2. • • •, nc) have the multinomial distribution.
Let 7ij = P(Yjj = 1) denote the probability of outcome in category j for each trial. The
multinomial probability mass function is

p{n\,n2, ..., nc-i) = (—-—^ ) 7t"'7r"2 • ••it"''. (1.2)

n
Since j = n< this is (c — l)-dimensional, with nL• = n — (n\ + • • • + «<•-1). The bino-
mial distribution is the special case with c = 2.
For the multinomial distribution,

E{nj) = njZj, v&r(nj) = n7Tj(\—7Tj), cov(n¡, n^) =—nji¡71^. (1-3)

We derive the covariance in Section 16.1.4. The marginal distribution of each n , is binomial.

1.2.3 Poisson Distribution

Sometimes, count data do not result from a fixed number of trials. For instance, if Y =
number of automobile accidents today on motorways in Italy, there is no fixed upper bound n
for Y (as you are aware if you have driven in Italy!). Since Y must take a nonnegative integer
value, its distribution should place its mass on that range. The simplest such distribution
is the Poisson. Its probabilities depend on a single parameter, the mean p. The Poisson
probability mass function (Poisson 1837, p. 206) is

P(y) = —p-, y = 0,1,2,.... (1.4)

It satisfies E{Y) = var(K) = p. It is unimodal with mode equal to the integer part of p.
Its skewness is described by E(Y — pf /cr* = 1 / ^ f p . The Poisson distribution approaches
normality as p increases, the normal approximation being quite good when p is at least
about 10.
The Poisson distribution is used for counts of events that occur randomly over time or
space, when outcomes in disjoint periods or regions are independent. It also applies as an
approximation for the binomial when n is large and TT is small, with p = nn. For example,
suppose Y = number of deaths today in auto accidents in Italy (rather than the number of
accidents). Then, Y has an upper bound. If each of the 50 million people driving in Italy
is an independent trial with probability 0.0000003 of dying today in an auto accident, the
DISTRIBUTIONS FOR CATEGORICAL DATA 7

number of deaths Y is a bin(50000000, 0.0000003) variate. This is approximately Poisson

with p = nn = 50000000(0.0000003) = 1 5 .
A key feature of the Poisson distribution is that its variance equals its mean. Sample
counts vary more when their mean is higher. When the mean number of daily fatal accidents
equals 15, greater variability occurs from day to day than when the mean equals 2.

1.2.4 Overdispersion

In practice, count observations often exhibit variability exceeding that predicted by the
binomial or Poisson. This phenomenon is called overdispersion. We assumed above that
each person has the same probability each day of dying in a fatal auto accident. More
realistically, these probabilities vary from day to day according to the amount of road traffic
and weather conditions and vary from person to person according to factors such as the
amount of time spent in autos, whether the person wears a seat belt, how much of the
driving is at high speeds, gender, and age. Such variation causes fatality counts to display
more variation than predicted by the Poisson model.
Suppose that Y is a random variable with variance v a r ( K \ p ) for given p, but p itself
varies because of unmeasured factors such as those just described. Let 0 = E(p.). Then
unconditionally,

E(Y) = E[E(Y\p)], var(K) = £ [ v a r ( y |/i)] + v a r [ £ ( F

When Y is conditionally Poisson (given p), then E(Y) = E(p) = 0 and var(7) = E(p) +
var(/i) = 0 + v a r ( p ) > 0.
Assuming a Poisson distribution for a count variable is often too simplistic, because of
factors that cause overdispersion. The negative binomial is a related distribution for count
data that has a second parameter and permits the variance to exceed the mean. We introduce
it in Section 4.3.4.
Analyses assuming binomial (or multinomial) distributions are also sometimes invalid
because of overdispersion. This might happen because the true distribution is a mixture
of different binomial distributions, with the parameter varying because of unmeasured
variables. To illustrate, suppose that an experiment exposes pregnant mice to a toxin and
then after a week observes the number of fetuses in each mouse's litter that show signs of
malformation. Let nt denote the number of fetuses in the litter for mouse i. The pregnant
mice also vary according to other factors, such as their weight, overall health, and genetic
makeup. Extra variation then occurs because of the variability from litter to litter in the
probability TZ of malformation. The distribution of the number of fetuses per litter showing
malformations might cluster near 0 and near «,, showing more dispersion than expected
for binomial sampling with a single value of rt. Overdispersion could also occur when n
varies among fetuses in a litter according to some distribution (Exercise 1.17). In Chapters
4, 13, and 14 we introduce methods for data that are overdispersed relative to binomial and
Poisson assumptions.

1.2.5 Connection Between Poisson and Multinomial Distributions

For adult residents of Britain who visit France this year, let Y\ = number who fly there,
Y2 = number who travel there by train without a car (Eurostar), = number who travel
there by ferry without a car, and Y4 = number who take a car (by Eurotunnel Shuttle or
Another random document with
no related content on Scribd:
had had to do with the death of my rabbit, and I hope you'll forgive
me for thinking it. I had no right to think so badly of you as that, and
now I'm dreadfully ashamed to remember that I did. Do forgive me,
won't you?"

"I—I've nothing to forgive," gasped Tim, utterly taken aback by the

little girl's words and the remorseful expression of her face. "I—I wish
you wouldn't speak like that, and—"

"Just say you forgive her, Shuttleworth, and she'll be satisfied,"

interposed Bob. "It's been on her mind that she's thought worse of
you than you deserve," he added with a smile.

"Yes, I ought to have taken your word, Tim," said Kitty distressfully. "I
know now that you wouldn't tell a story for the world. Why, what's the
matter?"

Well might she inquire, for Tim's plain, freckled countenance was
quivering with strong emotion. The next moment he disappeared
from view on the other side of the wall, leaving the brother and sister
staring at each other in blank amazement.

"He could hardly keep from crying," said Bob, "that's why he's gone. I
should never have thought he was such a moody-hearted chap as
that."

"I didn't say anything to hurt him, did I?" asked Kitty anxiously. Then,
as Bob shook his head, she proceeded, "Fancy his keeping the
secret about Fluffy all this time! If father had allowed me to tell him
that I saw him take away the covered basket, we should have found
out the truth before. I was right about that, you see. Perhaps he'll
come back in a few minutes and talk to us again."

But Tim did not return. He had rushed indoors and upstairs to his
own room; and whilst Kitty and Bob lingered in the garden, waiting
and hoping to see his red head appear over the wall he was sobbing
by his bedside and shedding the bitterest, most repentant tears that
had ever dimmed his eyes in his short life, as he recalled the words
Kitty had spoken in such a tone of conviction—"I know now that you
wouldn't tell a story for the world."

CHAPTER XII.
TIM ENTERTAINS.

"TIM, there's a letter from your father by this evening's post," said Mr.
Shuttleworth, opening his nephew's bedroom door, "and it contains
news which will interest you. Why, dear me, what's amiss?" he
inquired, as Tim rose hastily from his knees by the bedside and
turned his tear-stained face towards him.

Receiving no response, Mr. Shuttleworth entered the room, and,

seating himself on a chair by the window, surveyed the little boy
through his spectacles, his expression one of deep concern.

"You are not home-sick?" he asked dubiously. Then, as Tim shook

his head, he continued: "That's well. Cannot you tell me what
troubles you? Perhaps I can help you."

Tim hesitated. His uncle's voice was very kind, and the near-sighted
eyes which met his were full of sympathy. A great longing possessed
Tim to confide in some one, and, obeying the impulse of the
moment, he opened his heart to his uncle, and told him how he had
killed Kitty's rabbit and subsequently denied having overturned the
hutch, how he had tried to make amends, and how Kitty's humble
request for forgiveness for her suspicion of him and her assurance
that she knew he would not tell a lie for the world had made him feel
the falsity and cowardice of his past conduct. Mr. Shuttleworth
listened without remark until his nephew had finished his confession;
then he said:

"It seems to me, Tim, there's only one course to be taken—to make
a clean breast of everything to the people next door."

"I should like them to know, but I can't tell them—I can't!" cried Tim in
great distress. "Think what a bad opinion they'll have of me, and
they've all been so kind. I never meant to kill the rabbit, Uncle John; I
didn't know what was in the box."

"It was a spiteful trick anyway, Tim, and I don't wonder you were
ashamed afterwards when your temper had cooled down; but I don't
comprehend why, instead of owning to the truth, you were led to tell
a lie."

"It was because I was afraid the Glanvilles wouldn't have anything to
do with me if they knew what I'd done, and I did so want to be
friendly with them, Uncle John."

Mr. Shuttleworth was puzzled. He found himself incapable of

entering into the workings of his nephew's mind; but he saw he was
very conscience-stricken, and in great trouble, and longed to comfort
him. "I wish I could set matters on a right footing for you, Tim," he
said thoughtfully. "Would you like me to repeat to the Glanvilles all
you have imparted to me to-night?"

"Oh, Uncle John, if you only would I should be so glad! I expect they
will all despise me, but I can't have Kitty asking me to forgive her,
when—Oh, I don't think I ever before told such a big lie as the one I
told her, and it's worried me ever since."

"I'm glad to hear it, my boy; it shows me you're not in the habit of
uttering falsehoods, It's a grand thing to be truthful, for it takes a deal
of courage to be that."

"Yes, indeed it does," Tim rejoined with a sigh.

"The path of truth is difficult to walk because it's steep at times," Mr.
Shuttleworth said gravely, "but you know the old adage about putting
a stout heart to a steep hill, don't you? And the path of falsehood
seems easy travelling at first, because it's all down hill, and we don't
see the obstacles that are sure to rise before us as we go on. Take
my advice and don't turn from the uphill path in the future, my boy."

"I will try not to," Tim responded earnestly, much touched by his
uncle's little sermon. "It is very good of you not to be harder on me,
Uncle John. Did you not say you had heard from father?"

"Dear me, I had forgotten that. Yes, I heard from your father by to-
night's post; he is going to have his holiday from the bank next
month, and proposes to come and fetch you home. What do you
think of his plan?"

"I call it a capital one," said Tim, his face brightening.

"Your father will spend his holidays—a fortnight—here. It will be a

great pleasure to me to have his society. I shall be sorry to lose you,
Tim; and I have been thinking—"

But what Mr. Shuttleworth had been thinking he did not inform his
nephew then, for he broke off in the middle of his sentence, and a
few minutes later he went downstairs.

That night, after Tim had gone to bed, Mr. Shuttleworth made his call
next door. He saw Mr. and Mrs. Glanville, and repeated Tim's
confession to them. Needless to say they were greatly surprised; but
much to Mr. Shuttleworth's relief, Mrs. Glanville seemed to
understand Tim in the matter.

"I see he has a very passionate temper," she said, "and I can quite
believe he did not know it was a rabbit's hutch he was overturning,
for I am sure he would not wilfully injure a living creature. If only he
had owned the truth and not told Kitty that lie!"
"To me that seems the worst part of the business," said Mr.
Shuttleworth gravely. "But I think the lie brought its punishment,
judging by the distress of mind I found Tim in to-night; I do not
believe he will so easily give way to temptation again. I am afraid he
has exhibited himself to you all in a very bad light in this matter, but I
assure you he is sincerely repentant, and if your little girl and boy
can find it in their hearts to forgive him and continue to be his friends,
I shall be truly glad."

"I shall explain everything to Kitty and Bob," Mrs. Glanville replied,
"and I am sure they will not be harsh in their judgment of Tim. Poor
Tim! I must have a little talk to him myself."

Mrs. Glanville kept her word, and the following morning, whilst her
children were at school, she made a point of seeing Tim and
speaking to him as gently and sympathetically as his own mother
would have done under such circumstances, pointing out to him that
the fact of his having given way to his passionate temper and having
sought revenge had brought about the death of an innocent creature,
and had thus led him into further sin. No more was to be said on the
subject, she told him, but she hoped he had learnt a lesson he would
never forget.

"And Kitty and Bob?" Tim asked falteringly. "I suppose they won't be
friends with me any longer now?"

"Oh, yes, indeed I hope they will," Mrs. Glanville replied. "You are
going to entertain them on Saturday, you know. Had you forgotten
that?"

"No," Tim answered in a low tone. "But I thought—I was afraid that
they might not care to come. What did they say, Mrs. Glanville, when
they heard it was I who overturned the hutch?" he inquired with
considerable anxiety.

"Of course they were greatly astonished, but neither of them believe
you meant to injure the rabbit; they know you are not in the least
cruel. Kitty was hurt because you had told her a falsehood and Bob
was incredulous at first—it seems he had formed a very high opinion
of your sense of honour and truth."

Tim's cheeks burnt with shame on hearing this, and he wondered if

he would ever be able to restore Bob's faith in him again. Well was it
for him that he had not heard the words in which Bob had remarked
on his conduct; he would have greeted his visitors on Saturday even
more nervously than he did, had he known that it was to their mother
he owed their having kept their appointment. For they had been
justly indignant against the boy next door, who, as Kitty had taken
care to point out, had acted the enemy's part, at least on one
occasion, and by his own confession had proved that she had not
been far from the truth after all.

But Kitty and Bob were kind-hearted children, and it was not in their
natures to harbour resentment against any one for long; so that
when Tim, rather falteringly, put forward the plan for making a tent in
the back garden, they agreed to it cordially, and set to work with a
will, and by the aid of a clothes' prop for a centre pole and several
old wrappers, they managed to erect a kind of awning under which
they decided they would have their tea.

"Won't Mr. Shuttleworth have tea with us?" asked Kitty, when
Deborah arrived with the tea-things, which were placed on a small
gipsy table inside the tent. "There's only room for two chairs, but he
can have one, can't he?"

"And you must have the other, Kitty," said Tim. "For, of course, you'll
pour out the tea."

"If you wish it, I will," the little girl replied, a look of extreme
gratification settling on her face.

"I'll tell master you're beginning tea," remarked Deborah. "Then he

can join you or not as he likes."

A few minutes later Mr. Shuttleworth appeared upon the scene. He

thought it would have been more comfortable to have had the meal
in the house; but as the children assured him it was much nicer to
have it outdoors, he good-humouredly took the second chair in the
tent, whilst Kitty poured out the tea, and the boys sat cross-legged
on the ground. They were all very friendly and merry, and by-and-by
a glad bark was heard, and Snip, who had been waiting for an
opportunity of joining his young mistress and master all the
afternoon, and had just gained admittance into the garden, jumped
into Kitty's lap.

"Oh, Snip, you weren't invited, sir!" she cried reproachfully, kissing
him on the top of his head.

"Nevertheless, he must have some tea," Mr. Shuttleworth said with a

smile, as he poured out a saucer of milk for the little dog. "Dear me,
he has been in the wars, has he not?"

Kitty immediately explained how Snip had come by his wounds, and
went on to give Mr. Shuttleworth a graphic description of old Jacob
Dottin and his shop, finally telling him that Fluffy had come from
there.

"Who is Fluffy?" Mr. Shuttleworth inquired.

"The beautiful white rabbit Tim gave me in place of the one he

killed," she replied. Whereat Tim turned crimson and looked terribly
abashed, and Bob changed the conversation.

It was not until shortly before his visitors left that Tim plucked up
sufficient courage to speak to them about the bad turn he had served
them at the commencement of his acquaintance with them, and to
beg their forgiveness, which was willingly granted.

"It was an unneighbourly trick, and we didn't deserve you should

have done it," Bob said frankly, "but, of course, you never dreamt it
was something living we had in the box. We'll never mention it
more."

"And the lie I told," faltered Tim, looking at Kitty.

"Why didn't you own up to me?" she questioned.

"Because I was so afraid if I did you and Bob would never let me be
your friend, that was my reason," he rejoined. "I bitterly repented
afterwards that I hadn't told the truth."

"Well, you can be our friend still, if you like," Bob remarked, glancing
at his sister, who nodded her head assentingly. "I'm sure we'd both of
us much rather have a friend than an enemy next door."

CHAPTER XIII.
A REGULAR BRICK.

"KITTY! Kitty!" called Bob. "Oh, there you are! Keep Snip here with
you, will you, till I'm gone? Don't let him get away and follow me!"

"Why not?" demanded Kitty.

It was again a Saturday afternoon, several weeks having passed

since the sister and brother had been entertained next door. The little
girl, who had been watching Fluffy eating a sprig of parsley, turned
away from the rabbit hutch as she spoke, and joined Bob, who,
fishing rod in hand, and accompanied by Snip, had addressed her
from the back door.

"Because I'm going to Lang's pond, with Shuttleworth," he explained,

"and if Snip went with us he'd for certain spoil our sport by hunting
for water-rats and disturbing the fish. He seems to have an idea that
I want to get rid of him, for he's keeping as near me as he can, and
—"

"Oh, do let me go with you this afternoon!" interposed Kitty, eagerly.

"Why can't I?" she asked, as her brother shook his head.

"Well," he said, "it's possible that we may meet Richards, and that he
may ask us to his house to tea—that is, if you're not there."

"Then you'd better go your own way!" snapped Kitty, adding under
her breath, "Such selfishness!"

"But will you keep Snip with you?" questioned Bob.

Kitty assented. She took Snip into the house, and shut herself into
the dining-room with him, feeling deeply aggrieved. Mr. and Mrs.
Glanville had gone away for the day, therefore it was exceedingly
unkind of Bob, she considered, to go off with Tim Shuttleworth and
leave her.

"He makes a great deal too much of Tim," she reflected. "It's all very
well to let bygones be bygones, but Bob seems to have forgotten
altogether how disgusted he was with Tim at one time, and is making
him his chief friend. I haven't such a short memory as Bob."

This was true. Bob had a happy knack of putting anything he wished
to forget away from him. But Kitty, though she had agreed never
again to mention the ill-turn Tim had done her, often allowed her
mind to dwell on it, and the consequence was that she was
considerably less genial to him than was her brother.

By-and-bye, when she judged the boys must have taken their
departure, she decided to go out with Snip. It had rained earlier in
the day, but the weather had cleared about noon, and now the sun
was shining brilliantly. It was a really perfect May afternoon.

"I'll get some wild hyacinths for mother," the little girl thought,
remembering that she had heard Mrs. Glanville remark at breakfast
that they must now be in bloom and she would like some. "And on
my way home, I'll pass by Lang's pond and pay the boys a surprise
visit. I'll see Snip doesn't interfere with them."

Accordingly she sallied forth, with a basket on her arm, and

accompanied by the faithful Snip. First she went to a confectioner's,
where she purchased some chocolate creams, then left the town by
a side street, and a quarter of an hour later found her in a pretty
wood, carpeted with masses of wild blue hyacinths. She filled her
basket with the fragrant flowers, after which she sat down on the
moss-covered trunk of a fallen tree, and ate her chocolates, sharing
them with Snip.

"There!" she cried, as she popped the last chocolate into her mouth,
and rose. "Now we'll go to Lang's pond—we've as much right there
as anyone, eh, Snip? Still, I think I'd better lead you. Bob can't
complain if I do that."

Nevertheless Bob was not altogether pleased when, ten minutes

later, standing by the side of Tim Shuttleworth, on the edge of Lang's
pond, his eyes fixed on the float of his fishing rod, he heard a shout
and recognised his sister's voice.

"It's Kitty," Tim informed him. "She's brought Snip with her, but she
has him on the leash."

"Well, boys, what sport?" Kitty questioned, as she joined them. "The
fish are biting well, I suppose, after the rain? Let me see what you've
caught!"

She opened a fishing-basket which lay on the ground, and gave a

little laugh as she viewed its contents.

"Three dace, two roach, and an eel!" she cried. "The dace are the
smallest I ever saw. You ought to have put them back into the water
again, they aren't worth cooking. The roach are not so bad, but—"
"I daresay you think you could catch bigger ones!" Bob broke in,
sarcastically. He was disappointed that they had not had better sport,
and his sister's candid comments had vexed him.

"Richards is not here, I see," observed Kitty, after a few minutes'

silence.

"No, he has not turned up," Tim answered. "He did not promise to
come—only said that he might. What lovely hyacinths you have
there!"

"Yes, haven't I?" said Kitty.

She placed her basket on the ground, and stood watching the boys
for some while, but she soon tired of inactivity, and strolled around
the edge of the pond, still leading Snip. By-and-bye she caught sight
of some fine forget-me-nots growing close to the water, and stooped
to gather them. At that exact moment Snip spied a water-rat, and,
jerking the leash out of his mistress' hand, sprang Into the water in
pursuit of it.

"There!" cried Bob. "I knew what it would be!" He threw aside his rod
as he spoke, and glared angrily at his sister. "You must have let him
go on purpose, Kitty!" he asserted.

"I did not!" the little girl declared. "Indeed I did not! Oh, Bob, I'm so
sorry! Come here, Snip! Come here, sir!"

But Snip paid not the least attention to her. He was now swimming
around the edge of the pond, and sniffing into the rat-runs which
undermined the bank. Tim was secretly as vexed with Kitty as was
Bob, but out of pity for her he did not show his feelings, and stood
listening in silence to the hot words which were now passing
between the sister and brother.

"Well," Bob said at length, his anger beginning to cool, "of course, if
you say you didn't let him go on purpose, I believe you, and we'll
drop the subject. We'd better go home to tea now, I suppose."
"I'm sorry I came," murmured Kitty, almost tearfully. "I know you
didn't want me. Oh, oh!" she cried, her tone suddenly changing to
one of great excitement. "Look at Snip, boys! Oh! Do look! What's
the matter with him? Oh, see how he's struggling! Oh, Bob! Oh, Tim!
He'll drown—he'll drown!"

Snip had turned a rat out of its lair in the bank, and had been
swimming across the pond, which was very deep in the centre, after
it, when something had become entwined about his hind legs, and
he was now struggling in vain to get them free.

"Don't be silly, Kitty!" Bob said, speaking sharply, because he was

frightened and could not think what had happened to the dog. "How
can Snip drown when he can swim like—"

"Oh, you don't understand!" broke in Kitty, wringing her hands in her
distress. "It's the leash! It's got around his legs, and—Oh, Bob, go to
him, go to him!"

It was all very well to say "Go to him!" But Bob could not swim, and
knowing that he would be out of his depth in the water before he
could reach Snip, he naturally hesitated. At that minute, evidently
realizing the peril of his situation, the little dog uttered a piteous cry,
which nearly drove his mistress frantic; she would have dashed into
the pond herself if her brother had not prevented her.

"Oh, what can we do?" she wailed. "Oh, this is awful—awful! Oh,
Bob, can't you do anything? Oh, Tim!—where's Tim?"

"Here!" answered Tim, behind her. "It'll be all right, Kitty! Don't you be
frightened! I'll get him!"

Kitty now perceived that Tim had divested himself of his coat and
waistcoat, and was hurriedly unlacing his boots.

"I can swim—a little," he told her. "I think I shall be able to reach him.
There!"—As he pulled off his last boot. "Now I'm ready!"
He waded into the pond till the water reached his chin, Kitty and Bob
watching him with deepest anxiety, then swam a few strokes which
brought him within reach of Snip. He grasped the little dog by the
back of the neck, and, turning, essayed to swim back with him, but
Snip continued to struggle, and no progress was made.

"Can you get off the leash?" shouted Bob. "Try! Try!"

Tim did try, and, after several vain attempts, succeeded in

disentangling the leash, and setting the dog free. That done Snip
swam ashore of his own accord, and his rescuer followed him, to be
greeted with tearful thanks from Kitty as he waded out of the pond,
and a warm hand-clasp from Bob, who told him that he was "a brave
chap, and no mistake!"

"Yes, indeed," agreed Kitty. "But for you, Tim, poor Snip would
certainly have been drowned. Oh! I wonder what mother and father
will say when they hear about this? Oh, dear me, how dreadfully wet
you are, to be sure!"

"It would be a miracle if he was not!" Bob exclaimed, with a laugh,

which hid deep feeling. "What silly remarks you do make, Kitty! We
must get you home as quick as we can, Tim, or you'll catch cold!"

Tim had pulled on his boots over his wet stockings with some
difficulty by this time, and was now kneeling to lace them up.

"Oh, I shan't catch cold!" he declared cheerfully.

"I hope not," said Kitty, her voice full of concern, "because if you did
you might get pneumonia again. Oh, Tim, I haven't thanked you for
what you've done."

"Please don't!" Tim interposed, adding—"I'm so glad I could swim!"

He rose from his knees as he spoke, and, meeting Kitty's eyes, still
rather tearful, smiled at her.
"Oh," the little girl cried, "you don't know how grateful I feel to you."

"You're a brick, Tim—yes, a regular brick!"

CHAPTER XIV.
CONCLUSION.

"YOU'LL find me in the garden when you've finished your lessons,"

said Kitty, one evening, a few weeks later, as she poked her fair
head around the dining-room door to address her brother, who was
seated at the table, at work on his French translation. "I want to
consult you about something," she added impressively.

"All right," Bob answered. "I'll be with you before long."

Kitty went into the garden, and took up her position on a seat under
the apple tree. There, in a very few minutes, Bob joined her.

"Well?" he said interrogatively.

"Tim has gone out with his uncle," Kitty informed him. "I saw them
start together; so I thought this would be a good opportunity to
consult you about something—Tim not being about, you know."

"The 'something' has to do with Shuttleworth, then?" questioned

Bob.

Kitty nodded. "He'll be going home soon now," she observed. "Dear
me, it's quite remarkable we should both of us be so friendly with him
after—"

"Not remarkable at all," her brother broke in, rather impatiently. "He's
a jolly nice fellow—he's proved it. Speak out! What is it you want to
consult me about?"

"Well, I've been thinking that before Tim leaves we ought to give him
a present," Kitty said eagerly. "You know, he really did save Snip's
life. Father says that the poor little thing would certainly have
drowned if Tim hadn't gone to him and freed him from the leash."

"And you think he should have a testimonial as a reward?"

suggested Bob, with a teasing smile.

"I think nothing of the kind," the little girl answered, flushing. "You are
pretending to misunderstand me. I meant this—that we may never
see Tim again, and I should like to give him a present for a
keepsake, something that will put him in mind of us when he looks at
it."

"Well, I don't think that's a bad idea," Bob said, now speaking
seriously. "Let me consider. What could we give him? How much
money have you by you?"

"One shilling and two-pence, but I shall be having my sixpence

pocket-money on Saturday, of course."

"So you'll be good for one shilling and eight-pence. If I put a couple
of shillings to that, what could we buy? I know! A pocket-knife! We
could get a beauty, with three blades, the best that's made, for three
shillings and sixpence or four shillings."

"Oh, Bob, that would be the very thing! Tim has only an old bone-
handled pocket-knife with one blade, and that blade is broken! When
shall we get it! The new one, I mean? On Saturday?"

"Yes, if you like."

"And how shall we give it to him? I mean, will you, or shall I? You are
the elder, but it was my idea that we should make him a present.
Still, if you'd like to give it to him—" Kitty broke off, and looked at her
brother inquiringly.

"We can draw lots to decide that point," he answered, "then

everything will be fair."

Thus it came to pass that the following Saturday, shortly before one
o'clock, when Tim spoke to Kitty over the garden wall, and inquired
how she and Bob were going to spend the afternoon, she replied,
with an air of reserve, which puzzled him and rather aroused his
curiosity, that they were going into the town on private business.

"But we shall not be away very long," she added. "And I daresay we
shall see you as soon as we get back."

"I don't know about that," he answered, rather piqued because the
little girl seemed afraid that he might suggest accompanying her and
her brother. "I may not be at home then."

After the mid-day meal Tim had half a mind to go for a walk in the
country by himself, for his uncle betook himself as usual to his study.
But, as Kitty had spoken of seeing him on her return from the town,
he decided to remain at home, write to his mother, and afterwards
stroll out into the garden, so that the young folks next door could hail
him if they wanted him. Accordingly, he settled himself in the dining-
room, and had barely finished his letter, when, to his great
amazement, Deborah opened the door, and announced:

"Miss and Master Glanville!"

"Kitty! Bob!" he exclaimed, rising from his seat at the table, and
looking quickly from one to the other of his visitors. "Why didn't you
call to me from the garden as you usually do? I should have heard
you."
"If you're writing to your people perhaps you don't want to be
disturbed, and we'd better not stay?" suggested Bob. "Hurry up,
Kitty!" he whispered to his sister.

"Oh, do stay!" cried Tim. "Sit down, won't you? Here, Kitty, have this
easy chair?"

Kitty sank into the chair indicated, but not before she had put her
hand in her pocket, and drawn therefrom a small package. It had
fallen to her lot to present the pocket-knife.

"I daresay you thought it was rather odd of me not to tell you where
Bob and I were going this afternoon," she commenced, addressing
Tim. "Now, didn't you?"

"Well, yes, I did," he admitted, candidly. "Of course, I saw you didn't
want me to go with you—"

"That was just it," Kitty broke in. "We didn't want you to go with us
because we were thinking of buying you a present. We've bought it!
Bob chose it! It's a pocket-knife, and, oh, I do hope you'll like it!
There was one with a tortoiseshell handle I rather fancied, but Bob
said he was sure you'd prefer this one—if not, the man in the shop
won't mind changing it. This one has three blades, all of the very
best steel, they cut like lances, and it has a buck-horn handle. Oh,
look at it, do!" She thrust the little package into Tim's hands as she
spoke.

Tim, who had become quite red in his surprise, opened the package
with fingers which trembled with eagerness, and a very handsome
pocket-knife was revealed to view.

"You have bought this for me?" he cried, glancing first at Kitty, who
was watching him anxiously, trying to read from the expression of his
countenance if he approved of her brother's choice, and then at Bob,
who nodded assent. "Oh, how kind—how very, very kind of you both!
Oh, thank you—thank you! Oh, it's a grand knife! Three blades! Oh, I
say, I shall never be able to thank you half enough for this!" His eyes
were sparkling with delight.

"Do you prefer it to one with a tortoiseshell handle?" questioned

Kitty.

"Of course he does!" Bob exclaimed, without giving Tim time to

answer for himself, "Any boy would! That tortoiseshell handled knife
wasn't half as strong as this one; it was nice enough in its way, but it
was only fit for a girl."

"You couldn't have given me one I should have liked better than
this," Tim said earnestly, "I'm quite sure of that."

Kitty was satisfied. "That's all right then!" she said.

"Mother told us to ask you to tea with us, Shuttleworth," remarked

Bob, a few minutes later, after Tim had tried all three blades of his
new pocket-knife on a lead pencil. "Will you come?"

"Oh, thank you, I should like to—you know that," Tim replied. "Just
wait whilst I put away my writing things and tell uncle where I'm
going. I say, whatever made you think of giving me a present?"

"It was Kitty who thought of it," said Bob. "And when she mentioned
it to me, I considered it a very good idea and wondered I hadn't
thought of it myself. We're awfully pleased you like the knife—it's a
keepsake, you know—something for you to remember us by."

Tim was much touched—so touched, indeed, that he could find no

words to answer, but his face was eloquent of all the feelings in his
warm Irish heart.

"We shall be very sorry when you're gone," said Kitty, looking at him
with friendly blue eyes. "We shall miss you, oh, ever so much! You'll
think of us all, and of Snip—oh, I shall never forget that you saved
Snip's life! But, come, I'm sure it's near tea-time. I'm getting so
hungry!"
The three young people were very happy and merry together during
the hour or so which followed. After tea they repaired to the garden,
and on the trunk of the big apple tree Tim carved Kitty's initials, and
then Bob's, and then his own, with the big blade of his new treasure.

"There!" he cried, as he surveyed his finished handiwork, "How's

that!"

"Beautifully done!" answered Kitty, admiringly.

"Ah," said Tim, "I've a good tool—I never before handled such a
splendid knife." He shut it with a snap, and returned it to his pocket.

"Glad it suits you, old fellow!" said Bob, whilst Kitty looked very
gratified. "Yes, the initials look all right—they're very well done
indeed."

*****

Shortly before midsummer, Tim's father arrived at B—, and, after his
fortnight's holiday, returned to Dublin, taking his son with him. Tim
had been quite low-spirited at parting with his friends next door
although he had been going home; he had been sorry to say good-
bye to his uncle, too. But, as it happened, he was not to be absent
from B— long; for his uncle had offered to take him to live with him
and to send him to the B— Grammar School. And as he was to go to
his own people for the Christmas and the summer holidays, he was
simply delighted with this arrangement. So the end of September
found him once more a resident beneath his uncle's roof; and at the
beginning of the autumn term he was entered as a pupil at B—
Grammar School, where he was introduced to the boys by Bob
Glanville as "my friend, Shuttleworth."
Kitty and Bob scarcely ever think of the old days now when they
were so prejudiced against Tim Shuttleworth, and there is no one for
whom Kitty has a greater liking and regard than the boy whom she
once looked upon as her enemy.

THE END.

HEADLEY BROTHERS, 18, DEVONSHIRE STREET, E. C. 2; AND

ASHFORD, KENT.