3-D Computer Vision: Principles,

Algorithms and Applications 1st Edition

Yu-Jin Zhang
Visit to download the full and correct content document:
https://ebookmass.com/product/3-d-computer-vision-principles-algorithms-and-applic
ations-1st-edition-yu-jin-zhang/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Computer vision: theory, algorithms, practicalities

Fifth Edition Davies

https://ebookmass.com/product/computer-vision-theory-algorithms-
practicalities-fifth-edition-davies/

Hidden Semi-Markov models : theory, algorithms and

applications 1st Edition Yu

https://ebookmass.com/product/hidden-semi-markov-models-theory-
algorithms-and-applications-1st-edition-yu/

New Optimization Algorithms and their Applications:

Atom-Based, Ecosystem-Based and Economics-Based 1st
Edition Zhang

https://ebookmass.com/product/new-optimization-algorithms-and-
their-applications-atom-based-ecosystem-based-and-economics-
based-1st-edition-zhang/

Building Computer Vision Applications Using Artificial

Neural Networks, 2nd Edition Shamshad Ansari

https://ebookmass.com/product/building-computer-vision-
applications-using-artificial-neural-networks-2nd-edition-
shamshad-ansari/
Feature extraction and image processing for computer
vision Fourth Edition Aguado

https://ebookmass.com/product/feature-extraction-and-image-
processing-for-computer-vision-fourth-edition-aguado/

Probabilistic graphical models for computer vision Ji Q

https://ebookmass.com/product/probabilistic-graphical-models-for-
computer-vision-ji-q/

Transformers for Natural Language Processing and

Computer Vision, Third Edition Denis Rothman

https://ebookmass.com/product/transformers-for-natural-language-
processing-and-computer-vision-third-edition-denis-rothman/

Computer Vision Technology for Food Quality Evaluation,

Second Edition Sun

https://ebookmass.com/product/computer-vision-technology-for-
food-quality-evaluation-second-edition-sun/

Algorithms For Dummies (For Dummies (Computer/Tech))

2nd Edition Mueller

https://ebookmass.com/product/algorithms-for-dummies-for-dummies-
computer-tech-2nd-edition-mueller/
Yu-Jin Zhang

3-D Computer
Vision
Principles, Algorithms and Applications
3-D Computer Vision
Yu-Jin Zhang

3-D Computer Vision

Principles, Algorithms and Applications
Yu-Jin Zhang
Department of Electronic Engineering
Tsinghua University
Beijing, China

ISBN 978-981-19-7579-0 ISBN 978-981-19-7580-6 (eBook)

https://doi.org/10.1007/978-981-19-7580-6

Jointly published with Publishing House of Electronics Industry

The print edition is not for sale in China (Mainland). Customers from China (Mainland) please order the
print book from: Publishing House of Electronics Industry.

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore
Pte Ltd. 2023

Jointly published with Publishing House of Electronics Industry, Beijing, China

This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or
dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publishers, the authors, and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publishers nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publishers remain neutral with
regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface

This book is a specialized textbook that introduces the basic principles, typical
methods, and practical techniques of 3-D computer vision. It can provide an
advanced/second course service of computer vision for graduate students of related
majors in higher engineering colleges and universities, after studying 2D Computer
Vision: Principles, Algorithms and Applications.
This book mainly covers the higher level of computer vision from the selection of
materials. This book is self-contained, mainly for information majors, but also takes
into account learners of different professional backgrounds, and also considers the
needs of self-study readers. After learning the fundamental content of computer
vision in this book, readers can carry out scientific research and solve more and even
difficult specific problems in practical applications.
This book pays more attention to practicality in writing. Considering that com-
puter vision technology has been involved in many professional fields in recent
years, but many working people are not specialized in computer vision technology,
this book does not emphasize the theoretical system too much, minimizes the
formula derivation, and focuses on commonly used techniques. This book has
many sample questions and uses intuitive explanation to help readers understand
abstract concepts. A subject index list is given at the end of the book, and those
corresponding terms are marked in bold in the text.
This book provides a large number of self-test questions (including hints and
answers). In terms of purpose: on the one hand, it is convenient for self-study to
judge whether they have mastered the key content; on the other hand, it is also
convenient for teachers to carry out online teaching and strengthen teacher-student
interaction during lectures. The types of questions are multiple-choice questions,
which can be easily judged by a computer. In terms of content, many questions
express the basic concepts in a different way, supplementing the text, so that learners
can deepen their understanding. Some questions list descriptions that are similar but
not the same, or even have opposite meanings. Through dialectical thinking of pros
and cons, learners can also deeply understand the essence. The hints have been
provided for all self-test questions, allowing readers to obtain more information to

v
vi Preface

further identify the meaning of the questions. At the same time, each question can be
said to be divided into two levels in this way. Readers can complete the self-test after
reading the hints to show that they basically understand it, and to complete the self-
test without looking at the hints indicates that they have an even better grasp.
From the structure of this book, there are 12 chapters in total, plus one appendix,
answers to self-test questions, and subject index. Under these 15 first-level headings,
there are a total of 103 second-level headings (sections), and there are 141 third-level
headings (subsections) underneath. The book has a total of about 500,000 words
(including pictures, drawings, tables, formulas, etc.) and a total of (numbered)
228 figures, 22 tables, and 566 formulas. In order to facilitate teaching and learning,
this book provides a total of 68 examples of various types and 157 self-test questions
(all with hints and answers). In addition, there are a list of more than 100 directly
related references and a list of more than 500 subject terms for indexing, at the end of
the book.
This book can consider three aspects from the knowledge requirements of the
prerequisite courses: (1) Mathematics: including linear algebra and matrix theory, as
well as basic knowledge about statistics, probability theory, and random modeling;
(2) Computer science: including the mastery of computer software technology, the
understanding of computer structure system, and the application of computer pro-
gramming methods; (3) Electronics: On the one hand, the characteristics and prin-
ciples of electronic equipment; on the other hand, circuit design and other content. In
addition, the book 2D Computer Vision: Principles, Algorithms and Applications
can be counted as the discipline prerequisite of this book.
Thanks to the editors of the publisher for carefully composing the manuscript,
seriously reviewing, and attentively modifying.
Finally, the author thanks his wife Yun HE and daughter Heming ZHANG for
their understanding and support in all aspects.

Beijing, China Yu-Jin Zhang

Contents

1 Computer Vision Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Human Vision and Characteristics . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Visual Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Brightness Properties of Vision . . . . . . . . . . . . . . . . . . 3
1.1.3 Spatial Properties of Vision . . . . . . . . . . . . . . . . . . . . . 6
1.1.4 Temporal Properties of Vision . . . . . . . . . . . . . . . . . . . 8
1.1.5 Visual Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2 Computer Vision Theory and Framework . . . . . . . . . . . . . . . . . 13
1.2.1 Reaserch Goals, Tasks, and Methods of Computer
Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.2.2 Visual Computational Theory . . . . . . . . . . . . . . . . . . . 14
1.2.3 Framework Problems and Improvements . . . . . . . . . . . 20
1.3 Three-Dimensional Vision System and Image Technology . . . . . 23
1.3.1 Three-Dimensional Vision System Process . . . . . . . . . . 23
1.3.2 Computer Vision and Image Technology Levels . . . . . . 24
1.3.3 Image Technology Category . . . . . . . . . . . . . . . . . . . . 25
1.4 Overview of the Structure and Content of This Book . . . . . . . . . 27
1.4.1 Structural Framework and Content of This Book . . . . . 27
1.4.2 Chapter Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 29
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.1 Linear Camera Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.1.1 Complete Imaging Model . . . . . . . . . . . . . . . . . . . . . . 38
2.1.2 Basic Calibration Procedure . . . . . . . . . . . . . . . . . . . . 40
2.1.3 Internal and External Parameters . . . . . . . . . . . . . . . . . 41
2.2 Non-Linear Camera Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2.1 Type of Distortion . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

vii
viii Contents

2.2.2 Calibration Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

2.2.3 Classification of Calibration Methods . . . . . . . . . . . . . 48
2.3 Traditional Calibration Methods . . . . . . . . . . . . . . . . . . . . . . . . 49
2.3.1 Basic Steps and Parameters . . . . . . . . . . . . . . . . . . . . . 49
2.3.2 Two-Stage Calibration Method . . . . . . . . . . . . . . . . . . 51
2.3.3 Precision Improvement . . . . . . . . . . . . . . . . . . . . . . . . 56
2.4 Self-Calibration Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 61
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3 Three-Dimensional Image Acquisition . . . . . . . . . . . . . . . . . . . . . . . 67
3.1 High-Dimensional Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.2 Depth Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2.1 Depth Image and Grayscale Image . . . . . . . . . . . . . . . 69
3.2.2 Intrinsic Image and Non-Intrinsic Image . . . . . . . . . . . 70
3.2.3 Depth Imaging Modes . . . . . . . . . . . . . . . . . . . . . . . . 72
3.3 Direct Depth Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.3.1 Time-of-Flight Method . . . . . . . . . . . . . . . . . . . . . . . . 74
3.3.2 Structured Light Method . . . . . . . . . . . . . . . . . . . . . . . 77
3.3.3 Moiré Contour Stripes Method . . . . . . . . . . . . . . . . . . 79
3.3.4 Simultaneous Acquisition of Depth and Brightness
Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.4 Stereo Vision Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.4.1 Binocular Horizontal Mode . . . . . . . . . . . . . . . . . . . . . 85
3.4.2 Binocular Convergence Horizontal Mode . . . . . . . . . . . 89
3.4.3 Binocular Axial Mode . . . . . . . . . . . . . . . . . . . . . . . . 92
3.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 95
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4 Video Image and Motion Information . . . . . . . . . . . . . . . . . . . . . . . 101
4.1 Video Basic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.1.1 Video Expression and Model . . . . . . . . . . . . . . . . . . . 102
4.1.2 Video Display and Format . . . . . . . . . . . . . . . . . . . . . 104
4.1.3 Color TV System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.2 Motion Classification and Representation . . . . . . . . . . . . . . . . . 108
4.2.1 Motion Classification . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.2.2 Motion Vector Field Representation . . . . . . . . . . . . . . 109
4.2.3 Motion Histogram Representation . . . . . . . . . . . . . . . . 111
4.2.4 Motion Track Description . . . . . . . . . . . . . . . . . . . . . . 113
4.3 Motion Information Detection . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.3.1 Motion Detection Based on Camera Model . . . . . . . . . 114
4.3.2 Frequency Domain Motion Detection . . . . . . . . . . . . . 118
4.3.3 Detection of Movement Direction . . . . . . . . . . . . . . . . 121
Contents ix

4.4 Motion-Based Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

4.4.1 Motion Detection-Based Filtering . . . . . . . . . . . . . . . . 123
4.4.2 Motion Compensation-Based Filtering . . . . . . . . . . . . . 124
4.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 130
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5 Moving Object Detection and Tracking . . . . . . . . . . . . . . . . . . . . . . 135
5.1 Differential Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.1.1 Calculation of Difference Image . . . . . . . . . . . . . . . . . 136
5.1.2 Calculation of Accumulative Difference Image . . . . . . . 138
5.2 Background Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.2.1 Basic Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.2.2 Typical Practical Methods . . . . . . . . . . . . . . . . . . . . . . 140
5.2.3 Effect Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.3 Optical Flow Field and Motion . . . . . . . . . . . . . . . . . . . . . . . . 145
5.3.1 Optical Flow Equation . . . . . . . . . . . . . . . . . . . . . . . . 145
5.3.2 Optical Flow Estimation with Least Square Method . . . 146
5.3.3 Optical Flow in Motion Analysis . . . . . . . . . . . . . . . . . 147
5.3.4 Dense Optical Flow Algorithm . . . . . . . . . . . . . . . . . . 150
5.4 Moving Object Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
5.4.1 Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
5.4.2 Particle Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.4.3 Mean Shift and Kernel Tracking . . . . . . . . . . . . . . . . . 160
5.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 164
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6 Binocular Stereo Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.1 Stereo Vision Process and Modules . . . . . . . . . . . . . . . . . . . . . 170
6.1.1 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.1.2 Image Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.1.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.1.4 Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.1.5 3-D Information Recovery . . . . . . . . . . . . . . . . . . . . . 172
6.1.6 Post-Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
6.2 Region-Based Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . 173
6.2.1 Template Matching . . . . . . . . . . . . . . . . . . . . . . . . . . 174
6.2.2 Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
6.3 Feature-Based Stereo Matching . . . . . . . . . . . . . . . . . . . . . . . . 185
6.3.1 Basic Steps and Methods . . . . . . . . . . . . . . . . . . . . . . 186
6.3.2 Scale Invariant Feature Transformation . . . . . . . . . . . . 190
6.3.3 Dynamic Programming Matching . . . . . . . . . . . . . . . . 192
6.4 Error Detection and Correction of Parallax Map . . . . . . . . . . . . 194
6.4.1 Error Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
x Contents

6.4.2 Error Correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

6.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 198
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
7 Monocular Multiple Image Recovery . . . . . . . . . . . . . . . . . . . . . . . 205
7.1 Photometric Stereo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
7.1.1 Light Source, Scenery, Lens . . . . . . . . . . . . . . . . . . . . 207
7.1.2 Scene Brightness and Image Brightness . . . . . . . . . . . . 208
7.1.3 Surface Reflection Characteristics and Brightness . . . . . 213
7.2 Shape from Illumination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
7.2.1 Representation of the Surface Orientation
of a Scene . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
7.2.2 Reflectance Map and Brightness Constraint
Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
7.2.3 Solution of Photometric Stereo . . . . . . . . . . . . . . . . . . 222
7.3 Optical Flow Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
7.3.1 Optical Flow and Motion Field . . . . . . . . . . . . . . . . . . 226
7.3.2 Solving Optical Flow Equation . . . . . . . . . . . . . . . . . . 229
7.4 Shape from Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
7.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 240
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
8 Monocular Single Image Recovery . . . . . . . . . . . . . . . . . . . . . . . . . 247
8.1 Shape from Shading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8.1.1 Shading and Orientation . . . . . . . . . . . . . . . . . . . . . . . 248
8.1.2 Gradient Space Method . . . . . . . . . . . . . . . . . . . . . . . 250
8.2 Solving Brightness Equation . . . . . . . . . . . . . . . . . . . . . . . . . . 253
8.2.1 Linearity Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
8.2.2 Rotational Symmetry Case . . . . . . . . . . . . . . . . . . . . . 256
8.2.3 The General Case of Smoothness Constraints . . . . . . . . 258
8.3 Shape from Texture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
8.3.1 Monocular Imaging and Distortion . . . . . . . . . . . . . . . 260
8.3.2 Orientation Restoration from the Change of Texture . . . 263
8.4 Detection of Texture Vanishing Points . . . . . . . . . . . . . . . . . . . 271
8.4.1 Detecting the Vanishing Point of Line Segment
Texture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
8.4.2 Determine the Vanishing Point Outside the Image . . . . 273
8.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 276
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
9 Three-Dimensional Scenery Representation . . . . . . . . . . . . . . . . . . 283
9.1 Local Features of the Surface . . . . . . . . . . . . . . . . . . . . . . . . . . 284
9.1.1 Surface Normal Section . . . . . . . . . . . . . . . . . . . . . . . 284
Contents xi

9.1.2 Surface Principal Curvature . . . . . . . . . . . . . . . . . . . . 285

9.1.3 Mean Curvature and Gaussian Curvature . . . . . . . . . . . 286
9.2 Three-Dimensional Surface Representation . . . . . . . . . . . . . . . . 288
9.2.1 Parameter Representation . . . . . . . . . . . . . . . . . . . . . . 288
9.2.2 Surface Orientation Representation . . . . . . . . . . . . . . . 291
9.3 Construction and Representation of Iso-surfaces . . . . . . . . . . . . 294
9.3.1 Marching Cube Algorithm . . . . . . . . . . . . . . . . . . . . . 295
9.3.2 Wrapper Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 298
9.4 Interpolating Three-Dimensional Surfaces from Parallel
Contours . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
9.4.1 Contour Interpolation and Tiling . . . . . . . . . . . . . . . . . 300
9.4.2 Problems That May Be Encountered . . . . . . . . . . . . . . 302
9.4.3 Delaunay Triangulation and Neighborhood Voronoï
Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
9.5 Three-Dimensional Entity Representation . . . . . . . . . . . . . . . . . 306
9.5.1 Basic Representation Scheme . . . . . . . . . . . . . . . . . . . 306
9.5.2 Generalized Cylinder Representation . . . . . . . . . . . . . . 309
9.6 Key Points and References for Each Section . . . . . . . . . . . . . . . 311
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
10 Generalized Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
10.1 Matching Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
10.1.1 Matching Strategies and Categories . . . . . . . . . . . . . . . 318
10.1.2 Matching and Registration . . . . . . . . . . . . . . . . . . . . . 321
10.1.3 Matching Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 323
10.2 Object Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
10.2.1 Measure of Matching . . . . . . . . . . . . . . . . . . . . . . . . . 324
10.2.2 Corresponding Point Matching . . . . . . . . . . . . . . . . . . 327
10.2.3 String Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
10.2.4 Matching of Inertia Equivalent Ellipses . . . . . . . . . . . . 329
10.2.5 Shape Matrix Matching . . . . . . . . . . . . . . . . . . . . . . . 331
10.3 Dynamic Pattern Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
10.3.1 Matching Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
10.3.2 Absolute Pattern and Relative Pattern . . . . . . . . . . . . . 334
10.4 Graph Theory and Graph Matching . . . . . . . . . . . . . . . . . . . . . 336
10.4.1 Introduction to Graph Theory . . . . . . . . . . . . . . . . . . . 336
10.4.2 Graph Isomorphism and Matching . . . . . . . . . . . . . . . . 340
10.5 Line Drawing Signature and Matching . . . . . . . . . . . . . . . . . . . 343
10.5.1 Contour Marking . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
10.5.2 Structural Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . 344
10.5.3 Labeling via Backtracking . . . . . . . . . . . . . . . . . . . . . 346
10.6 Key Points and References for Each Section . . . . . . . . . . . . . . . 347
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
xii Contents

11 Knowledge and Scene Interpretation . . . . . . . . . . . . . . . . . . . . . . . . 355

11.1 Scene Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
11.1.1 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
11.1.2 Attribute Hypergraph . . . . . . . . . . . . . . . . . . . . . . . . . 358
11.1.3 Knowledge-Based Modeling . . . . . . . . . . . . . . . . . . . . 360
11.2 Logic System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
11.2.1 Predicate Calculation Rules . . . . . . . . . . . . . . . . . . . . . 362
11.2.2 Inference by Theorem Proving . . . . . . . . . . . . . . . . . . 366
11.3 Fuzzy Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
11.3.1 Fuzzy Sets and Fuzzy Operations . . . . . . . . . . . . . . . . 370
11.3.2 Fuzzy Reasoning Method . . . . . . . . . . . . . . . . . . . . . . 372
11.4 Scene Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
11.4.1 Bag-of-Words/Feature Model . . . . . . . . . . . . . . . . . . . 375
11.4.2 pLSA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
11.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 383
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
12 Spatial-Temporal Behavior Understanding . . . . . . . . . . . . . . . . . . . 389
12.1 Spatial-Temporal Technology . . . . . . . . . . . . . . . . . . . . . . . . . 390
12.1.1 New Research Field . . . . . . . . . . . . . . . . . . . . . . . . . . 390
12.1.2 Multiple Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
12.2 Spatial-Temporal Interest Point Detection . . . . . . . . . . . . . . . . . 392
12.2.1 Detection of Points of Interest in Space . . . . . . . . . . . . 392
12.2.2 Detection of Points of Interest in Space and Time . . . . . 393
12.3 Spatial-Temporal Dynamic Trajectory Learning and
Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
12.3.1 Automatic Scene Modeling . . . . . . . . . . . . . . . . . . . . . 396
12.3.2 Path Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
12.3.3 Automatic Activity Analysis . . . . . . . . . . . . . . . . . . . . 403
12.4 Spatial-Temporal Action Classification and Recognition . . . . . . 405
12.4.1 Motion Classification . . . . . . . . . . . . . . . . . . . . . . . . . 405
12.4.2 Action Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 407
12.5 Key Points and References for Each Section . . . . . . . . . . . . . . . 414
Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418

Appendix A: Visual Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

A.1 Shape Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
A.2 Spatial Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
A.2.1 Nonvisual Indices of Depth . . . . . . . . . . . . . . . . . . . . . 421
A.2.2 Binocular Indices of Depth . . . . . . . . . . . . . . . . . . . . . 422
A.2.3 Monocular Indices of Depth . . . . . . . . . . . . . . . . . . . . . 424
Contents xiii

A.3 Motion Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427

A.3.1 The Condition of Motion Perception . . . . . . . . . . . . . . . 427
A.3.2 Detection of Moving Objects . . . . . . . . . . . . . . . . . . . . 429
A.3.3 Depth Motion Detection . . . . . . . . . . . . . . . . . . . . . . . 429
A.3.4 Real Motion and Apparent Motion . . . . . . . . . . . . . . . . 430
A.3.5 Correspondence Matching of Apparent Motion . . . . . . . 430
A.3.6 Aperture Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
A.3.7 Dynamic Indices of Depth . . . . . . . . . . . . . . . . . . . . . . 433
A.4 Key Points and References for Each Section . . . . . . . . . . . . . . . 434
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
Answers to Self-Test Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435
Chapter 1 Computer Vision Overview . . . . . . . . . . . . . . . . . . . . . . . . 435
Chapter 2 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
Chapter 3 Three-Dimensional Image Acquisition . . . . . . . . . . . . . . . . . 436
Chapter 4 Video Image and Motion Information . . . . . . . . . . . . . . . . . 437
Chapter 5 Moving Object Detection and Tracking . . . . . . . . . . . . . . . . 437
Chapter 6 Binocular Stereo Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
Chapter 7 Monocular Multiple Image Recovery . . . . . . . . . . . . . . . . . 439
Chapter 8 Monocular Single Image Recovery . . . . . . . . . . . . . . . . . . . 439
Chapter 9 Three-Dimensional Scene Representation . . . . . . . . . . . . . . 440
Chapter 10 Scene Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
Chapter 11 Knowledge and Scene Interpretation . . . . . . . . . . . . . . . . . 441
Chapter 12 Spatial-Temporal Behavior Understanding . . . . . . . . . . . . . 442
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
Chapter 1
Computer Vision Overview

Computer vision is an information subject/discipline that uses computers to realize

the functions of human vision system (HVS). This book mainly introduces the high-
level content of computer vision, which can be used as a textbook for in-depth study
of computer vision. Readers can read this book after studying 2D Computer Vision:
Principles, Algorithms and Applications.
The process of human vision (in brief, vision) can be regarded as a complex
process from sensation (obtaining the image by the 2-D projection of the 3-D world)
to perception (gaining the content and meaning of the 3-D world from the 2-D
image) process. The ultimate goal of vision in a narrow sense is to make a mean-
ingful explanation and description of the scene for the observer, and in a broad sense,
it is also to make a behavior plan based on these explanations and descriptions, as
well as the surrounding environment and the wishes of the observer. Computer
vision is artificial vision or human-made vision. It is to use computers to realize
human visual functions, hoping to make meaningful judgments about actual targets
and scenes based on the perceived images.
The sections of this chapter are arranged as follows:
Section 1.1 introduces the characteristics of human vision, the brightness properties
of vision, the spatial properties of vision, and the temporal properties of vision, as
well as makes some discussions on visual perception.
Section 1.2 discusses the research purpose, research tasks, and research methods of
computer vision. It also introduces the visual computational theory proposed by
Marr in more detail. Moreover, a combined presentation for some improvement
ideas is provided.
Section 1.3 gives a general introduction to the 3-D vision system that obtains 3-D
spatial information and realizes the understanding of the scene. It compares and
discusses the layers of computer vision and image technology, leading to the
main content of this book.
Section 1.4 presents the structure of the book and gives the brief summaries of each
chapter.

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 1
Y.-J. Zhang, 3-D Computer Vision, https://doi.org/10.1007/978-981-19-7580-6_1
2 1 Computer Vision Overview

1.1 Human Vision and Characteristics

Computer vision (also called artificial vision or human-made vision) is developed on

the basis of human vision. There are two meanings here. One is that computer vision
needs to realize the functions of human vision, so it could imitate the system
structure and functional modules of human vision; the other is that computer vision
needs to extend the functions of human vision, so it could use the features of human
vision to improve the efficiency and effectiveness of these functions.
The following first gives an overview of the functional characteristics of human
vision and then discusses some important visual brightness, spatial, and temporal
properties.

1.1.1 Visual Characteristics

First compare vision with some related concepts.

1.1.1.1 Vision and Other Sensations

It is generally believed that humans have five senses of vision, hearing, smell, taste,
and touch and the corresponding sensory organs, for obtaining information from the
objective world. Among them, vision provides humans with most of the data they
receive; in other words, humans often rely more on vision than other senses when
they acquaint the world. For example, the input information obtained by humans
from eye observation often reaches several million bits, and the data rate during
continuous viewing can exceed tens of million bits per second. The human brain has
more than 1010 cells/neurons, some of which have more than 10,000 connections
(or synapses) with other neurons. It is estimated that the amount of visual informa-
tion the brain receives from the eyes is at least two orders of magnitude larger than all
the information obtained from other sense organs.

1.1.1.2 Vision and Computer Vision

Although computer vision needs to realize the functions of human vision, there are
still differences between human vision and computer vision. Human vision first
receives light stimulation in a certain wavelength range from the external environ-
ment through the sensory organs (eyes) of the visual system to the retina and then
encodes and processes the perceptive organs (brain or cerebral visual cortex) of the
visual system to obtain subjective feelings. Therefore, vision involves not only
physics and chemistry but also psychophysiology. Computer vision mainly relies
on photoelectric conversion for image acquisition, then uses processing and analysis
1.1 Human Vision and Characteristics 3

functions to obtain objective data, and makes stricter reasoning and judgments based
on these results.

1.1.1.3 Vision and Machine Vision

In the early days, computer vision put more emphasis on the research of vision
science and design system software, while machine vision not only considered
design system and software but also considered hardware environment and image
acquisition technology as well as the integration of vision systems. Therefore, from
the perspective of vision system integration, machine vision systems are more
comparable to human vision systems (human visual systems). However, with the
development of electronic technology and computer technology, true and real-time
applications can be realized to a considerable extent on a stand-alone PC. Due to the
increase in knowledge in related fields, the difference between machine vision and
computer vision has been significantly reduced, and they are more interchangeably
used nowadays.

1.1.1.4 Vision and Image Generation

Vision can be regarded as a process of obtaining description and explanation of the

scene from the image of the scene with the help of the knowledge of the law of image
formation. The image generation in graphics can be seen as the process of generating
images from the abstract description of the scene with the help of the knowledge of
the law of image formation. Although they have some parallels/similarities, certain
people regard them as inverse processes, but their complexity is quite different. The
image generation process is completely deterministic and predictable, and the visual
process involves not only providing a list of all possible explanations but also
providing the most likely interpretation. This search process is one to many and
may be accompanied by combinatorial explosion. Therefore, vision is inherently
more complex than image generation in graphics.

1.1.2 Brightness Properties of Vision

The brightness of vision corresponds to the light intensity that the human eye feels
from the scene. A psychological term closely related to brightness is subjective
brightness or subjective luminance The subjective brightness refers to the bright-
ness of the observed object, which is judged by the human eyes according to the
intensity of the light stimulation of the retina. Examples of three typical brightness
properties in which the perception of brightness is related to multiple factors are as
follows.
4 1 Computer Vision Overview

1.1.2.1 Simultaneous Contrast

The subjective brightness felt from the surface of an object is not only related to the
brightness of the surface itself but also to the relative relationship (ratio) between the
brightness of the surface and the surrounding environment (background). If two
objects with different brightness have a similar relative relationship with their
respective backgrounds, they can appear to have the same brightness. At this time,
the subjective brightness perceived by people has nothing to do with the absolute
value of the object brightness. Conversely, the surface of the same object will appear
brighter if it is placed on a darker background and will appear darker if it is placed on
a brighter background. This phenomenon is called simultaneous contrast, also
called conditional contrast.
Example 1.1 Simultaneous Contrast Example
All the small squares in the center of big squares (background) in Fig. 1.1 have
exactly the same brightness. However, it looks brighter when it is on a dark
background and looks darker when it is on a bright background. So, it feels like
when these four pictures are viewed from left to right, the small square in the center
gradually darkens. This is just the result of simultaneous contrast.

1.1.2.2 Mach Band Effect

At the boundary of the touch regions with different brightness in the object, human
vision may estimate the brightness value too high or too low. In other words, the
subjective brightness felt from the surface of an object is not a simple proportional
function of the illuminance the object receives. This phenomenon was discovered by
Mach, so it is called the Mach band effect.
Example 1.2 Example of Mach Band Effect
Figure 1.2a is a Mach band pattern, which includes three parts: the left side is a
uniform low-brightness area, the right side is a uniform high-brightness area, and the
middle is a gradual transition from low brightness to high brightness. Fig. 1.2b
shows the actual brightness distribution (three straight lines) from left to right. If one
looks at Fig. 1.2a with the eyes, one will find that there is a darker band at the
junction of the left and middle bands than the left band and a darker band at the
junction of the middle and right bands than the right band. The subjective brightness

Fig. 1.1 Example of simultaneous contrast

1.1 Human Vision and Characteristics 5

Fig. 1.2 Schematic

diagram of Mach band effect
(a)

Brightness

(b)
Distance

Subjective
brightness
(c)
Distance

obtained should be as shown in Fig. 1.2c. In fact, the darker band and the brighter
band do not exist objectively; it is the result of subjective brightness perception.

1.1.2.3 Contrast Sensitivity

Contrast sensitivity (also called contrast susceptibility) reflects the ability of the
human eye to distinguish differences in brightness. It is affected by the observation
time and the size of the observed object. If a grid composed of lines of different
thickness and contrast is used for testing, the closer the contrast between the bright
and dark lines of the grid perceived by the eyes is to the contrast between the bright
and dark lines of the original test grid, the greater the contrast sensitivity. Under ideal
conditions, people with good eyesight can distinguish a brightness contrast of 0.01,
which means that the contrast sensitivity can reach up to 100.
If the abscissa represents the thickness of the bright and dark lines of the test grid
and the ordinate represents the contrast sensitivity, then the measured result will give
the modulation transfer function of the visual system, that is, the ability that human
visual system accurately converts the test grid into an optical image. Here the grid
thickness can be expressed by spatial frequency, and its unit is the number of circles
(number of lines) contained in each degree of viewing angle, that is, circle/degree
(CPD).
The contrast sensitivity can be regulated by the light modulation factor M. If Lmax,
Lmin, and Lav are, respectively, the maximum, minimum, and average brightness
values, then
6 1 Computer Vision Overview

L max - L min
M= ð1:1Þ
Lav

1.1.3 Spatial Properties of Vision

Vision is first and foremost a spatial experience, so the spatial properties of vision
have a great influence on visual effects.

1.1.3.1 Spatial Cumulative Effect

Vision has a cumulative effect in space. The range of light stimulus intensity
experienced by the human eye can reach about 13 orders of magnitude. If described
by light illuminance, the lowest absolute stimulus threshold is 10-6 lx (lux), and the
highest absolute stimulus threshold exceeds 107lx. Under the best conditions, every
light quantum in the peripheral region of the retina will be absorbed by a rod cell, and
a visual response can be caused by only a few light quanta. This is considered to have
taken place in a complete spatial accumulation, and it can be described by the law of
the inverse ratio of light intensity and area. This law can be written as

E c = kAL ð1:2Þ

where Ec is the absolute threshold of vision, which is the critical light energy
required for 50% detection probability (i.e., the light energy when the light stimu-
lation is observed once in every two tests in multiple experiments); A is the
cumulative area; L is the brightness; and k is a constant, which is related to the
units used by Ec, A, and L. Note that the area that can satisfy the above law has a
critical value Ac (corresponding to a round solid angle with a diameter of about
0.3 rad). When A < Ac, the above law holds; otherwise the above law does not hold.
It can be seen that the spatial cumulative effect can be understood as follows:
when a small and weak light spot is presented alone, it may be invisible (cannot
cause a visual response), but when multiple such light spots are connected together
as a large spot at the same time, one can see it. Its functional significance lies in: large
objects may be seen in a dark environment even if the outline is blurred.

1.1.3.2 Spatial Frequency

The spatial frequency corresponds to the changing speed of the visual image in
space. This can be tested with stripes whose brightness sinusoidally changes in
space. The brightness function Y(x, y) = B(1 + mcos2πfx). Among them, B is the
basic brightness, m is the amplitude (corresponding to black and white contrast), and
1.1 Human Vision and Characteristics 7

f is the fringe frequency (corresponding to the fringe width). The spatial resolution
capability can be tested by changing m when f is given as a fixed value. Obviously,
the larger m, the stronger the spatial resolution ability. In practice, to test the
minimum m value that can distinguish bright and dark stripes at different angles
and frequencies, one can define 1/m minute (1′) as the contrast sensitivity. Usually
the human eye, in terms of spatial frequency felling, is equivalent to a band-pass
filter (more sensitive to the intermediate thickness of the stripes), the most sensitive
is 2 ~ 5 CPD, and the spatial cut-off frequency is 30 CPD.
When a person observes a still image, the eyeball is not still in one place, usually
after staying in one place for a few hundred milliseconds to complete the image
acquisition, the eyeball will move to another place to take another image, and so
on. This kind of movement is called saccadic eye movement. Studies have shown
that jumping movement can increase contrast sensitivity, but the peak sensitivity will
decrease.

1.1.3.3 Visual Acuity

Visual acuity is usually defined as the reciprocal of the viewing angle value
corresponding to the smallest detail that can be distinguished under certain condi-
tions. The smaller the viewing angle, the greater the visual acuity. If V is used for
visual acuity, then V = 1/(viewing angle). It represents the ability of the human eye
to correctly distinguish the details and contours of objects. A visual acuity of
1 indicates the resolution capability at a standard distance when the corresponding
viewing angle is 1°. The actual viewing angle of the human eye is 30″ ~ 60″ (this is
basically consistent with the cone cell diameter of about 0.004 mm), that is, the best
visual acuity can reach 2.0.
Visual acuity is affected by many factors, including the following:
1. Distance: When the distance of the object from the observer increases, the visual
acuity of the human eye decreases. This phenomenon is most obvious at about
10 m, and beyond a certain distance limit, the details of the object can no longer
be recognized.
2. Brightness: Increasing the brightness of the object (or enlarging the pupil) will
increase the visual acuity. The relationship between visual acuity and brightness
I is

V = a log I þ b ð1:3Þ

where a and b are constants. Visual acuity increases with increasing brightness, and
the relationship between the two is logarithmic. If the brightness continues to
increase to a certain level, the visual acuity will be close to saturation and will not
increase.
Another random document with
no related content on Scribd:
 But Susie shook her head. "No, you don't," she said.
"God wants us to know and love him—you and me, Elfie."

"I know them police that stands at the door, and that's
enough for me," said the girl. "You can go if you like.
Church, and tables, and chairs, and eating off plates, and
sleeping in beds, is all one, I guess; and them that gets
used to it can't do without it. But I can, and I shan't run to
the police for that."

But although Elfie would not go with Susie, she willingly

consented to show her the way; for she had not been to a
church in this neighbourhood, and only knew the road to
take the work backwards and forwards. So, after carefully
washing her face and brushing her hair, and making herself
as tidy as possible, Susie went out, carrying her prayer-
book in her pocket-handkerchief, and trying to fancy that
her mother was with her still.

Elfie would not come near the church; but after pointing
it out, and watching Susie go in, she ran back to play with
her companions, wondering all the time what could be going
on inside the church to make Susie so anxious to go there.
This was her first question when she met her as she came
home.

"What do you look at—what do you do," she asked,

"when you go to church?"

"We pray and sing, and hear what the minister says,"
answered Susie.

"What does he say?" asked Elfie.

Susie thought for a minute, and then answered, "Well,

he reads out of the Bible, and says, 'Our Father.' You know
that, don't you?"
 But Elfie shook her head. "Who is 'Our Father'?" she
asked.

"God, who lives up in heaven, where mother's gone,"

answered Susie.

"He's your Father, then, I suppose," said Elfie.

"Yes, and yours too," said Susie quickly.

"No, he ain't; I don't know him," said Elfie, shaking her

head with a little sigh.

"But he knows you, Elfie—knows you, and loves you,

and wants you to love him."

But Elfie shook her head persistently. "I don't know

nothing about him, and nobody ever loved me," she' said.

And to end the conversation, she ran away to finish her

game of buttons, while Susie walked quietly home.

She ate a slice of dry bread for her dinner, and saved
one for Elfie; and then took her mother's Bible out of the
little box, and sat down to read a chapter just as she used
to do before her mother died. But the sight of the familiar
old book upset all her firmness, and she sat down with it in
her lap, and burst into tears. She was still crying when Elfie
came rushing in to ask if she would not come out and join
their play.

"What's the matter?" she exclaimed when she saw

Susie in tears. "Are you so hungry?" she asked—for hunger
seemed the only thing worth crying for to Elfie; and then,
seeing the slice of bread on the table, and guessing it had
been left for her, she put it on the Bible, saying, "You eat it,
Susie; I've had some cold potatoes, and I ain't very hungry
now."

But Susie put it back into her hands. "No, no, Elfie; you
must eat that," she said. "I'm not crying because I'm
hungry."

"What is it then?" said Elfie.

Susie looked down at the book lying in her lap. "I was
thinking about mother," she said.

"Are you getting tired of living with me?" asked Elfie

quickly.

"Oh no; you're very kind. I don't know what I should do

without you, Elfie; but I do want my mother," said Susie
through her tears.

Elfie looked puzzled. She was beginning to understand

that all the mothers in the world were not like hers—that
Susie's was not; and she could not understand why Mrs.
Sanders had gone away and left her.

"What made her go away?" she asked.

Susie left off crying to look at her companion in

surprise. "Don't you know God took her to heaven?" she
said.

"Yes, I know, you said that before," answered Elfie

impatiently; "but what made him take her?"

"Because he loved her," said Susie.

"But you said just now he loved you; why didn't he take
you up there as well?"
 "I asked mother about that one day, when she was
telling me she should have to go away; but she said she
thought God had some work for me to do in the world first
before he took me home." And Susie dried her tears, and
tried to be brave and choke back her sobs as she spoke.

"What work will you have to do?" asked Elfie, sitting

down on the floor close to Susie's stool. Elfie always
preferred rolling on the floor to sitting on any kind of seat;
and she greatly enjoyed questioning Susie.

"Mother said God would teach me that if I asked him,"

answered Susie. "I don't know yet what it will be."

"Then why don't you ask him?" said Elfie in her

straightforward fashion.

"I do," whispered Susie. "I ask him every night;

because I want to do it, and then go home to mother."

"Is that what you do when you kneel down before you
get into bed?" asked Elfie.

Susie nodded. "God hears what I say, too," she

answered.

"Well, then, why didn't your mother ask him to let her
stay and help you to do the work, if she didn't want to go
away?" said Elfie sharply.

Susie knew not what to answer. The question puzzled

her not a little; and to escape from Elfie's saying any more,
she proposed reading a chapter from the Bible.

Elfie had grown tired of playing, and was quite willing to

listen. She could not read herself, and was full of wonder
that Susie could; and for some time she chattered and
questioned so much about this that Susie could not begin;
but at last she grew quiet, and Susie turned to her favourite
verses in St. Matthew—the story of young children being
brought to Jesus.

"That was kind of him to say, 'Let the children come to

me,'" said Elfie when Susie paused.

"Yes; the Lord Jesus is always kind," said Susie.

"I wish he was here in London; I'd go to him," said Elfie.

"It's nice to have anybody speak kind to you."

"You can go to him, Elfie," said Susie. "The Lord Jesus

has gone up to heaven again now; but he'll hear you just as
plain as though he was in the room here."

Elfie stared. "You don't think I'm going to believe that,

do you?" she said sharply.

"Why not? It's the truth," said Susie.

"Maybe it is for fine folks that wants a lot of things to

live, but not for a poor little street girl like me," answered
Elfie.

"Why don't you think it's for you, Elfie?" asked her
companion.

"Because I know what I am, and I guess he'd soon find

out I was street rubbish, as the fine folks call me in the
market." And Elfie clenched her fist angrily as she spoke.

"O Elfie, Jesus don't think you're street rubbish!" said

Susie. "I think he cares for people all the more when he
knows they're poor, because he was a poor man himself
once."
 "A poor man!" exclaimed Elfie. "Why, you said he was
God's Son, and all the world was his."

"So it is; but when he came down here, the people

wouldn't believe he was God's Son, and so he lived like a
poor man—as poor as you and I, I think, Elfie."

But Elfie shook her head. "I'm street rubbish, but you
ain't," she said.

"I found a verse about it," said Susie, "where Jesus says
how poor he was—'The foxes have holes, and the birds of
the air have nests, but the Son of man hath not where to
lay his head.' There; that means Jesus had no home or
comfortable bed, he was so poor," said Susie.

Elfie sat looking at her in dumb surprise.

"He was just as poor as me," she said. "Why didn't he

go away, and leave the people, if he was God's Son?"

"Because he loved them, and he wanted them to know

it; and to know that God loved them too, and wanted them
to love him and be happy."

Elfie had never had any one to love her in all her life,
and she could but dimly understand what Susie meant; but
she did understand it a little, and all the vain longings she
had felt when looking at a mother kissing her child sprung
up in her heart now, as she said, in a subdued, gentle voice,
"I wish he'd love me just a little."

"He does love you," said Susie, "not a little, but a great
deal."

"Did he tell you to tell me so?" asked Elfie eagerly.

 Susie knew not what to reply to this; but the thought
stole into her heart—Was this the work her mother had
spoken of—was she to tell Elfie of the love of God, try to
make her understand it, and lead her to love him?

But her silence made Elfie think she had no message for
her, and she said, "You need not be afraid to tell me, Susie;
nobody ever did love me, and nobody ever will; and I don't
want any love either." But in spite of these words, so
sharply and angrily spoken, Elfie burst into tears.

Susie had never seen her cry before, and for very
sympathy she burst into tears herself, as she threw her
arms round her companion's neck, and drew her closely
towards her. "Don't cry, Elfie; I'll love you," she said. "I'll
love you ever so much; and you'll believe God loves you
too; won't you?" she added coaxingly.

Elfie clung to Susie, and held her in a passionate

embrace. "Say it again—" she whispered, "say you love me,
Susie; it's what I've been wanting ever so long, I think."

"Everybody wants it," said Susie. "God puts the feeling

in our heart, mother said; and then he gives us people to
love us, just that we may know how he loves us himself."

"Tell me some more about it," said Elfie, still in the

same subdued voice, and clinging fast round Susie's neck,
her dirty tangled head of hair resting on her shoulder.

"I don't know how to tell it, Elfie, but just as the Bible
tells it. Mother made me learn a good many verses about
the love of God. I'll tell you some of them. 'God is love';
'Like as a father pitieth his children, so the Lord pitieth
them that fear him'; 'God so loved the world, that he gave
his only begotten Son, that whosoever believeth in him
should not perish, but have everlasting life.' Now, don't you
see God must love you, for you're in the world, and God so
loved the world that he sent Jesus Christ to die that we
might be saved?"

"Saved?" repeated Elfie.

"Yes; saved from our sins—the wicked things we do that

make God sorry, and angry too," said Susie.

But Elfie did not care to hear about this; she wanted to
know whether it was possible for God to love her—whether
he had told Susie, he would love her.

"I'd do anything for that," she said, pushing back her

tangled hair. "Do you think he'd like me better if I was to
keep my face clean and comb my hair like you do?" she
asked.

Susie smiled. "I think God does like people to be clean,"

she said; "and I'd like it, Elfie."

"Then I'll do it," said Elfie in a determined tone. "I've

thought it was no good. Before, I was just street rubbish,
and nobody cared for me; but if you do, and God will, I'll
wash my face; and perhaps he will by-and-by, as the Lord
Jesus his Son was a poor man himself."

And Elfie went at once to fetch some water to wash her

face, and Susie promised to help her to do her hair.

CHAPTER III.
 OUR FATHER.

FROM this time Elfie began to pay some attention to her

personal appearance. She washed her face and hands, and
combed her hair every morning, before she went out, and,
of course, looked less wild; but her rags, poor child, were
past mending, and there seemed no hope of ever being able
to replace these with better clothes now. New ones—new
frocks, new shoes, that gave other little girls so much
pleasure—Elfie had never had. Sometimes she wore a pair
of old shoes or boots picked up in the street, and
sometimes she went barefoot. And it was much the same
with frocks and bonnets: sometimes she picked up a rag
that would cover her, or had one given her, and she wore it
until it dropped to pieces. She had never been quite naked;
but many times she had been almost so, until some one
had given her something to put on.

She began to wish now that some one would do so

again; and formed all sorts of plans for saving enough
money to buy herself a frock at a second-hand clothes' stall
—plans that always failed, for winter was drawing near, and
the two girls found it harder work than ever to pay the rent
and buy bread to eat.

"The rent must be paid," Elfie said over and over again,
as if to convince herself of a fact she half doubted.

Susie said nothing, but stitched away as fast as she

could, and always contrived to have the shilling for the
landlord when he called; for she knew if it were not paid,
they would be turned into the street, and for Elfie's sake, as
well as her own, she did not wish this to happen. Elfie said
she did not care, she had always been used to a street life,
but that it would never do for Susie; and so for her sake—to
keep Susie's home for her—she grew more careful and
steady, that she might be trusted by people to do odd jobs
for them, and thus bring in a few pence to add to the
weekly store.

But with all Elfie's care and steadiness, and Susie's

close stitching, they had a hard time of it to make ends
meet; and Susie grew pale and weak, and often suffered
from pain in her side. She went regularly to church on
Sunday, but she could never persuade Elfie to do so. Church
was for decent folk, not for her, she said; but she looked
forward to sitting down with her arms round Susie's neck,
to listen to her reading from the Bible, on Sunday
afternoon.

Sometimes they contrived to have a fire on Sunday, but

it was not often they could have one all the week, except to
boil the kettle occasionally; for Susie still kept up the habit
of having regular meals, and was gradually winning Elfie to
like this plan too.

People began to notice the pale, pinched little face

under the shabby black bonnet, that was seen so regularly
every Sunday in a quiet corner of the church; and at length,
a lady spoke to her as she was coming out one day.

"Where do you live, little girl?" asked the lady kindly.

It was very cold, and the lady could not help shivering
in her warm furs, and she noticed that Susie had only a thin
cape on.

"In Fisher's Lane, please, ma'am," answered Susie,

dropping a courtesy and blushing.
 But the lady did not know Fisher's Lane. "Do you go to
the Ragged School?" she asked.

Susie shook her head. "I don't know where it is," she
said.

"That is a pity," said the lady, "for there is a Sunday

school there afternoon and evening, in a nice, warm room,
and the teachers would be glad to see you, I am sure."

"Would they?" said Susie. "I used to go to Sunday

school before we came to live here. Perhaps Elfie knows
where it is, and maybe she'll come with me."

"Ask her," said the lady; "we shall be very glad to see
you both."

She did not stay to ask who Elfie was; but she looked
after Susie as she ran down the street, and was surprised to
see her join poor, ragged, neglected-looking Elfie—for Susie
still contrived to keep a decent appearance, although her
clothes were so thin and old.

The lady's invitation was repeated to Elfie; but to

Susie's surprise she did not look at all pleased.

"Do you know where the school is?" asked Susie.

Elfie nodded. "Yes, I know where it is, but I shan't go."

"O Elfie, do," said Susie coaxingly.

"No, I shan't. You may, if you want to leave me all alone

on Sunday afternoons," said Elfie sulkily.

"But I don't want to leave you, Elfie; want you to come

with me," said Susie.
 "I don't want to come," said Elfie doggedly.

"Why not?" persistently asked Susie.

"I don't like schools, nor them that go to 'em." And to

end all further discussion on the subject, Elfie ran home,
leaving Susie to follow more leisurely.

There was nothing for her to hurry home for. The room
looked cold, bare, and desolate, for they could not indulge
in a fire to-day; they had not been able to make up the rent
money, and the thought of this had troubled Susie until she
went to church. There, however, she had heard the
message bidding her to cast her care upon God; and she
came home to the cheerless room, and her dinner of dry
bread, feeling as blithe as a bird.

"Why, what's come to you, Susie?" asked Elfie. "You was

crying and fretting about the rent money before you went
out, and now you look as though you'd got it all safe in the
tin box."

The mention of the rent brought a little cloud into

Susie's face, but it was quickly dispelled as she answered,
"O Elfie, I wish you could have heard the minister to-day,
and what he said about God taking care of us."

"It don't seem as though he took much care of you and

me," said Elfie sulkily, as she looked at the empty grate,
and tried to draw her rags over her bare shoulders.

"Are you very cold, Elfie?" asked Susie tenderly.

"I shouldn't think you was very warm," said Elfie

crossly. "Your frock ain't in rags perhaps, but it's as thin as
mine."
 "Yes, it is thin," said Susie, "and I'm cold; but it seems
to me God does care even for our being cold, for he's sent
to tell us we may go where there is a fire this afternoon."

"Where's that?" asked Elfie sharply.

"At the school the lady told me about," answered Susie.

"She said there was a fire there, and that they would be
very glad to see us."

"Well, I shan't go," said Elfie. "I'd rather stop here in

the cold."

This seemed unreasonable to Susie. "Do tell me why

you won't go?" she said.

"No, I shan't. And if you go, don't you tell anybody you
know me," said Elfie.

"Why not? Have you been to the school before?" asked

Susie.

"I shan't tell you, and I won't go," said Elfie doggedly.

Susie was puzzled. She hardly knew what to do, for she
did not like to leave Elfie, and yet she wanted to go to
school. But at length she decided to stay at home and read
to her companion, and go to the school in the evening, if
Elfie would show her the way; for they had no fire and no
candle to burn to-night, and it would be very dull to sit
there in the dark listening to the noises in the other lodgers'
rooms, for there was rarely a Sunday evening passed
without a quarrel in the house. Elfie would go out to play
with some of her companions as soon as it grew dusk; but
Susie had given up going out to play on Sunday.
 After a little persuasion, Elfie agreed to take Susie to
the corner of the street where the school was; but she
would not go any further, and she promised to meet her at
the same corner when she came out after school.

"But I don't know what time the school will be over,"

said Susie.

"I do," said Elfie with a short laugh; "but mind you ain't
to tell any of 'em who showed you the way," she added in a
more serious tone.

Susie promised not to mention her name, and she

hoped the lady who had invited her would forget that she
had said she would bring Elfie with her; but she could not
help thinking it very strange that Elfie should dislike the
idea of coming so much.

The children had begun to assemble when she reached

the school; and hardly knowing where she was going, Susie
went into the large, light, warm room, and looked round for
the lady whom she had seen in the morning. She was not
there, but another teacher came forward and asked her
name, and where she lived; and on hearing she could read
put her into the Bible class at once.

Susie looked shyly at her companions, who were, of

course, looking at her, but not very shyly, for many of them
looked as though they were used to a street life, and most
of them were older than herself. What a treat it was to
these poor girls to sit down in a warm, light room, Susie
could only guess. To her it was very delightful—the mere
sensation of light and warmth; and the only drawback to
her enjoyment was the thought that poor Elfie was not
sharing it.
 She could join in singing the opening hymn; and then,
when the books were given out, she found her place more
quickly than the rest, and ventured to lift her eyes to the
teacher's face for a minute, and then saw that the lady was
looking at her.

"You have not been to the school before, have you, my

dear?" she said in a gentle voice.

"No, ma'am," answered Susie.

"I hope we shall see you very often now. Can you come
every Sunday?" said the lady.

"Yes, ma'am," replied Susie.

And then, the others having found their places, the

reading commenced. The lady explained the meaning of
each verse as they went on, but spoke more particularly of
God's care for his children.

When school was over, and Susie met Elfie, she told her
of the evening lesson, and how like it was to what she had
heard in the morning; but Elfie answered, "I'm going to
take care of myself now, and then perhaps God will do it for
me by-and-by."

"I think we need God's care now," sighed Susie,

thinking of the deficient store of halfpence in the tin box at
home.

"Well, we don't get it," said Elfie defiantly; "and going to

that school won't bring it neither. Don't go again, Susie,"
she added.

"But I like it; and I must go now, because I've

promised," said Susie. "I do wish you would go with me, it
is so nice, Elfie. We sing, and read, and pray to God; and
the room is so beautiful with the fire and the gas."

"I know all about it," said Elfie sulkily; "and I know just
what you'll do too: you'll go to that school, and then you
won't like me. Some of 'em 'll tell you I'm a bad girl, and
then you won't speak to me." And the thought of this so
overcame poor Elfie that she burst into tears.

Susie put her arm round her neck, and drew her own
thin cape over her shoulders. "Nobody shall make me say
that about you, Elfie," she said. "Don't cry. I'll love you
always; and you shall come to school with me, and learn to
read."

But Elfie still shook her head about going to school. "I
can't go there," she said.

"Yes, you shall, Elfie. I know why you don't like to go;
it's because your frock is so old. But we'll try and make
another this week. I think mother would like you to have
her frock to go to school in," she added. "And there's her
shawl; perhaps we could make two of it; and I don't think
she'd mind, as we are so cold."

Susie was determined that nothing should damp her

happiness to-night, and she would not listen to Elfie's
refusal to go to school. She felt brave, too, or she could not
have spoken about cutting up her mother's dress and
wearing her shawl as she did. Yes, the little girl was brave
and hopeful. What she had heard of God's care and tender
love to-day had brought back all the lessons of her
childhood; and she could believe that God was her Father,
and cared—really cared for and loved her.

When they reached home she said, "I wish you'd kneel
down and say 'Our Father' of a night, like I do, Elfie."
 "But I don't know it," said Elfie.

"Well, I'll teach you, shall I? You can say it after me in

bed until you know it by yourself; only, I'd like you to kneel
down and say it first, like I did to mother."

Elfie was generally willing to do anything to please her

companion, and she very readily consented to this. And so,
after shutting the door, the two girls knelt down in the pale
moonlight beside a chair, and Elfie repeated the words
slowly and reverently as Susie uttered them—the divine
words that make all men brothers and all women sisters.

There must have been some such thought as this in

Susie's mind, for as she crept into bed after Elfie she said,
"I did not think of it before, but you are my sister, Elfie, so I
shall never forget to love you;" and she kissed her as she
spoke.

Elfie threw her arms round her. "Say you'll love me

always," she whispered; "for there's nobody else in all the
world if you don't."

"I do love you," said Susie. "But oh, Elfie, I wish you'd
believe God loves you too—that he is our Father."

"I don't know nothing about fathers; I never had a

father," said Elfie. "But if you'll love me, perhaps I shall
believe that God does, by-and-by—especially as the Lord
Jesus was a poor man. I like to hear about that, because,
you see, it makes it seem somehow that he knows all about
poor people—even street rubbish like I am, if he had no bed
and no home."

Before they went to sleep that night, Elfie had learned

to repeat the Lord's Prayer almost perfectly (she could learn
quickly if she liked); and at last dropped to sleep
murmuring the words, "Our Father—our Father." And Susie
thought over all she had heard that day of the heavenly
Father's love; and at last fell asleep, to dream that her
mother had come back to lift all the care off her shoulders,
and shelter her from every rough wind that blew.

But Monday morning brought the every-day anxiety

with it; and Susie's first thought was of the landlord, and
what he would say when he came in the afternoon and
found she had only tenpence of the rent saved up in the tin
box. She tried to recall something of what she had heard
the previous day—tried to cast her care upon God; but it
was very hard; and it was not until she had knelt down and
prayed, ay, and sobbed out her trouble before him, that she
could believe any of it this morning, although she had felt
so sure of it the day before.

Elfie had woke up first and gone out. She often did this
if there was only a small piece of bread in the house,
because then she could leave the bread for Susie, and pick
up her breakfast at the market, or about the streets.

So, after eating her bread, Susie took out her work,
sitting upon the low stool, with the blanket of the bed
wrapped round her, for it was bitterly cold this morning, and
they had no fire. They had been afraid to buy coals or
wood, as they could not make up the rent. This was Susie's
great anxiety this morning. What the landlord would say,
she did not know. He was a gruff, cross man; and Susie
dreaded his visit—sat trembling with fear at the thought of
hearing him come up the stairs; and again and again lifted
her heart in asking that they might not be turned out of
their home.
 CHAPTER IV.
ELFIE'S SIXPENCE.

SUSIE'S suspense as to the result of the landlord's visit

came to an end sooner than she expected. He called earlier
than usual to-day, and the poor girl's last faint hope that
Elfie would be able to earn twopence and get back before
he came was cut off as she heard his halting footsteps
coming up the stairs. He knocked at the opposite door first,
and Susie hoped he would be detained there, and she crept
to the top of the stairs and looked over, in the hope of
seeing Elfie coming up.

But Elfie was not to be seen; and with a sinking heart,

Susie went back and took down the tin box, and then sat
down to her work again, waiting for the door to open and
Elfie to come in, for somehow she had persuaded herself
that she would come in yet. But in a minute or two, the
opposite door closed, and then there was a knock at her
own. Susie could hardly walk across the room to open it,
she trembled so violently.

"Good morning," said the landlord pleasantly, as he

stepped in and looked round the room. "You keep the place
nice and clean," he said approvingly. "But why don't you
have a fire, child? It's cold to-day, and you sitting at your
sewing."

"Yes, sir," said Susie meekly, glancing at the empty

grate, and hardly knowing how to tell him she had not been
able to make up the rent.
 "You ought to have a fire," went on the man, not
noticing her confusion, and wishing to say something kind
to the poor little orphan. "You ought to have a fire this cold
day; every other room in the house has one."

"Have they, sir?" said Susie, thinking the man was

displeased. "I'm very sorry I can't get one too; but I don't
think the place will get damp—we have one sometimes."

"The place get damp!" repeated the landlord. "What do

you mean, child?"

"Please, sir, I thought you was afraid the room would

spoil," said Susie, still dreading to make the revelation that
she had only tenpence of the rent.

"Spoil!" repeated the man. And he looked round on the

patched, discoloured walls, and laughed. "Why, child, you
keep your room nicer than any other in the house. I was
thinking you must be cold."

"I don't mind that much, sir, if I can only stay here,"
said Susie; "but—but please, sir, I've only got tenpence of
the rent to-day. I hope you won't turn us out for the other
twopence. I'll try and pay it next week, sir," she added.

The man took the halfpence and counted them, and

then looked at the little pale, pinched face before him. He
loved money, and was used to scenes of misery, but was
not quite without human feeling, and Susie's mute distress
was almost more than he could look upon unmoved. "Who
told you I should turn you out, child?" he said.

"No one, but—but I was afraid you would if I didn't keep

the rent paid," said Susie.