Design for Embedded Image

Processing on FPGAs, 2nd Edition

Donald G. Bailey
Visit to download the full and correct content document:
https://ebookmass.com/product/design-for-embedded-image-processing-on-fpgas-2n
d-edition-donald-g-bailey/
DESIGN FOR EMBEDDED
IMAGE PROCESSING
ON FPGAs
DESIGN FOR EMBEDDED
IMAGE PROCESSING
ON FPGAs

Second Edition

Donald G. Bailey
Massey University, Palmerston North, New Zealand
This second edition first published 2024
© 2024 John Wiley & Sons, Ltd

Edition History
John Wiley & Sons (Asia) Pte Ltd (1e, 2011)

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any
means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain
permission to reuse material from this title is available at http://www.wiley.com/go/permissions.

The right of Donald G. Bailey to be identified as the author of this work has been asserted in accordance with law.

Registered Office(s)
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK

For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content that appears in standard
print versions of this book may not be available in other formats.

Trademarks
Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates in the United
States and other countries and may not be used without written permission. All other trademarks are the property of their respective
owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.

Limit of Liability/Disclaimer of Warranty

While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with
respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without
limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by
sales representatives, written sales materials or promotional statements for this work. This work is sold with the understanding that
the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for
your situation. You should consult with a specialist where appropriate. The fact that an organization, website, or product is referred
to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the
information or services the organization, website, or product may provide or recommendations it may make. Further, readers should
be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read.
Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to
special, incidental, consequential, or other damages.

Library of Congress Cataloging-in-Publication Data

Names: Bailey, Donald G. (Donald Graeme), 1962- author.

Title: Design for embedded image processing on FPGAs / Donald G. Bailey.
Description: Second edition. | Hoboken, NJ : Wiley, 2024. | Includes index.
Identifiers: LCCN 2023011469 (print) | LCCN 2023011470 (ebook) | ISBN
9781119819790 (cloth) | ISBN 9781119819806 (adobe pdf) | ISBN
9781119819813 (epub)
Subjects: LCSH: Embedded computer systems. | Field programmable gate
arrays.
Classification: LCC TK7895.E42 B3264 2024 (print) | LCC TK7895.E42
(ebook) | DDC 621.39/5–dc23/eng/20230321
LC record available at https://lccn.loc.gov/2023011469
LC ebook record available at https://lccn.loc.gov/2023011470

Cover Design: Wiley

Cover Image: © bestfoto77/Shutterstock

Set in 9/11pt TimesLTStd by Straive, Chennai, India

Contents
Preface xiii

Acknowledgments xix

About the Companion Website xxi

1 Image Processing 1
1.1 Basic Definitions 1
1.2 Image Formation 3
1.2.1 Optics 3
1.2.2 Colour 5
1.3 Image Processing Operations 6
1.4 Real-time Image Processing 8
1.5 Embedded Image Processing 9
1.6 Computer Architecture 10
1.7 Parallelism 11
1.7.1 Temporal or Task Parallelism 12
1.7.2 Spatial or Data Parallelism 13
1.7.3 Logical Parallelism 14
1.7.4 Stream Processing 14
1.8 Summary 15
References 16

2 Field-programmable Gate Arrays 19

2.1 Hardware Architecture of FPGAs 19
2.1.1 Logic 21
2.1.2 DSP Blocks 22
2.1.3 Memory 23
2.1.4 Embedded CPU 23
2.1.5 Interconnect 24
2.1.6 Input and Output 24
2.1.7 Clocking 26
2.1.8 Configuration 26
2.1.9 FPGAs vs. ASICs 27
2.2 Programming FPGAs 28
2.2.1 Register Transfer Level 30
2.2.2 Hardware Description Languages 32
2.2.3 High-level Synthesis 33
2.3 FPGAs and Image Processing 38
2.3.1 Choosing an FPGA or Development Board 39
2.4 Summary 40
References 41
vi Contents

3 Design Process 45
3.1 Problem Specification 45
3.2 Algorithm Development 47
3.2.1 Algorithm Development Process 47
3.2.2 Algorithm Structure 48
3.2.3 FPGA Development Issues 51
3.3 Architecture Selection 51
3.3.1 System Architecture 52
3.3.2 Partitioning Between Hardware and Software 53
3.3.3 Computational Architecture 55
3.4 System Implementation 60
3.4.1 Mapping to FPGA Resources 60
3.4.2 Algorithm Mapping Issues 62
3.5 Testing and Debugging 63
3.5.1 Design 63
3.5.2 Implementation 64
3.5.3 Common Implementation Bugs 64
3.5.4 Timing 66
3.5.5 System Debugging 68
3.5.6 Algorithm Tuning 70
3.5.7 In-field Diagnosis 71
3.6 Summary 72
References 73

4 Design Constraints 77
4.1 Timing Constraints 77
4.1.1 Low-level Pipelining 78
4.1.2 Process Synchronisation 80
4.1.3 Synchronising Between Clock Domains 82
4.1.4 I/O Constraints 83
4.2 Memory Bandwidth Constraints 84
4.2.1 Memory Architectures 84
4.2.2 Caching 86
4.2.3 Row Buffering 87
4.3 Resource Constraints 88
4.3.1 Bit-serial Computation 89
4.3.2 Resource Multiplexing 89
4.3.3 Arbitration 92
4.3.4 Resource Controllers 94
4.3.5 Reconfigurability 95
4.4 Power Constraints 97
4.5 Performance Metrics 98
4.5.1 Speed 99
4.5.2 Resources 99
4.5.3 Power 99
4.5.4 Cost 100
4.5.5 Application Metrics 100
4.6 Summary 101
References 102
Contents vii

5 Computational Techniques 105

5.1 Number Systems 105
5.1.1 Binary Integers 105
5.1.2 Residue Systems 106
5.1.3 Redundant Representations 107
5.1.4 Fixed-point Numbers 107
5.1.5 Floating-point Numbers 108
5.1.6 Logarithmic Number System 110
5.1.7 Posit Numbers 110
5.2 Elementary Functions 111
5.2.1 Square Root 111
5.2.2 Trigonometric Functions 112
5.2.3 Linear CORDIC 116
5.2.4 Hyperbolic Functions 117
5.2.5 Logarithms and Exponentials 118
5.2.6 Lookup Tables 118
5.2.7 Polynomial Approximations 122
5.2.8 Iterative Techniques 123
5.3 Other Computation Techniques 124
5.3.1 Incremental Update 124
5.3.2 Separability 124
5.4 Memory Structures 124
5.4.1 FIFO Buffer 124
5.4.2 Zigzag Buffers 126
5.4.3 Stacks 126
5.4.4 Linked Lists 127
5.4.5 Trees 128
5.4.6 Graphs 129
5.4.7 Hash Tables 129
5.5 Summary 130
References 131

6 Interfacing 135
6.1 Camera Input 135
6.1.1 Analogue Video 136
6.1.2 Direct Digital Interface 137
6.1.3 MIPI Camera Serial Interface 138
6.1.4 Camera Link 139
6.1.5 USB Cameras 139
6.1.6 GigE Vision 139
6.1.7 Camera Processing Pipeline 140
6.2 Display Output 143
6.2.1 Display Driver 143
6.2.2 Display Content 146
6.3 Serial Communication 147
6.3.1 RS-232 147
6.3.2 I2 C 148
6.3.3 Serial Peripheral Interface (SPI) 149
6.3.4 Universal Serial Bus (USB) 150
viii Contents

6.3.5 Ethernet 150

6.3.6 PCI Express 151
6.4 Off-chip Memory 151
6.4.1 Static RAM 152
6.4.2 Dynamic RAM 152
6.4.3 Flash Memory 155
6.5 Processors 155
6.5.1 AXI Interface 155
6.5.2 Avalon Bus 156
6.5.3 Operating Systems 157
6.5.4 Implications for System Design 157
6.6 Summary 157
References 158

7 Point Operations 161

7.1 Point Operations on a Single Image 161
7.1.1 Contrast and Brightness Adjustment 161
7.1.2 Global Thresholding and Contouring 164
7.1.3 Lookup Table Implementation 166
7.2 Point Operations on Multiple Images 167
7.2.1 Image Averaging 168
7.2.2 Image Subtraction 170
7.2.3 Background Modelling 172
7.2.4 Intensity Scaling 175
7.2.5 Masking 175
7.2.6 High Dynamic Range (HDR) Imaging 177
7.3 Colour 179
7.3.1 False Colour 179
7.3.2 Colour Space Conversion 180
7.3.3 Colour Thresholding 192
7.3.4 Colour Enhancement 193
7.3.5 White Balance 194
7.4 Multi-spectral and Hyperspectral Imaging 197
7.4.1 Hyperspectral Image Acquisition 197
7.4.2 Processing Steps 198
7.5 Summary 199
References 199

8 Histogram Operations 203

8.1 Greyscale Histogram 203
8.1.1 Building the Histogram 203
8.1.2 Data Gathering 205
8.1.3 Histogram Equalisation 209
8.1.4 Automatic Exposure 214
8.1.5 Threshold Selection 215
8.1.6 Histogram Similarity 220
8.2 Multidimensional Histograms 220
8.2.1 Triangular Arrays 221
8.2.2 Multidimensional Statistics 222
8.2.3 Colour Segmentation 225
Contents ix

8.2.4 Colour Indexing 228

8.2.5 Texture Analysis 229
8.3 Summary 231
References 231

9 Local Filters 235

9.1 Window Caching 235
9.1.1 Border Handling 237
9.1.2 Filter Latency 239
9.2 Linear Filters 239
9.2.1 Filter Techniques 240
9.2.2 Noise Smoothing 243
9.2.3 Edge Detection 246
9.2.4 Edge Enhancement 248
9.3 Nonlinear Filters 249
9.3.1 Gradient Magnitude 249
9.3.2 Edge Orientation 250
9.3.3 Peak Detection and Non-maximal Suppression 251
9.3.4 Zero-crossing Detection 252
9.3.5 Bilateral Filter 252
9.3.6 Adaptive Thresholding 253
9.3.7 High Dynamic Range Tone Mapping 255
9.4 Rank Filters 256
9.4.1 Sorting Networks 258
9.5 Adaptive Histogram Equalisation 262
9.6 Morphological Filters 262
9.6.1 Binary Morphology 262
9.6.2 Greyscale Morphology 266
9.7 Colour Filtering 268
9.7.1 Colour Morphology and Vector Median 269
9.7.2 Edge Enhancement 269
9.7.3 Bayer Pattern Demosaicing 271
9.7.4 White Balancing 272
9.8 Summary 273
References 274

10 Geometric Transformations 281

10.1 Reverse Mapping 282
10.1.1 Anti-alias Filtering 283
10.1.2 Interpolation 284
10.2 Forward Mapping 291
10.2.1 Separable Mapping 292
10.2.2 Hybrid Approach 296
10.3 Common Mappings 297
10.3.1 Affine Transformation 297
10.3.2 Perspective Mapping 297
10.3.3 Polynomial Mapping 298
10.3.4 Lens Distortion 299
10.3.5 Non-parametric Mappings 302
x Contents

10.4 Image Registration 302

10.4.1 Feature-based Methods 303
10.4.2 Area-based Methods 307
10.4.3 Applications 314
10.5 Summary 315
References 315

11 Linear Transforms 321

11.1 Discrete Fourier Transform 322
11.1.1 Fast Fourier Transform (FFT) 323
11.1.2 Goertzel’s Algorithm 331
11.1.3 Applications 332
11.2 Discrete Cosine Transform (DCT) 336
11.3 Wavelet Transform 338
11.3.1 Filter Implementations 340
11.3.2 Applications 344
11.4 Summary 345
References 345

12 Image and Video Coding 349

12.1 Compression Techniques 350
12.1.1 Colour Conversion 350
12.1.2 Prediction and Transformation 350
12.1.3 Motion Estimation and Compensation 351
12.1.4 Quantisation 352
12.1.5 Run-length Coding 353
12.1.6 Entropy Coding 354
12.2 DCT-based Codecs 357
12.2.1 DCT Block Processing 357
12.2.2 JPEG 357
12.2.3 Video Codecs 358
12.3 Wavelet-based Codecs 359
12.4 Lossless Compression 360
12.5 Perceptual Coding 361
12.6 Coding Hyperspectral Images 362
12.7 Summary 362
References 363

13 Blob Detection and Labelling 367

13.1 Bounding Box 367
13.2 Run-length Coding 369
13.3 Chain Coding 369
13.3.1 Sequential Implementation 370
13.3.2 Single-pass Stream Processing Algorithms 370
13.3.3 Feature Extraction 372
13.4 Connected Component Labelling (CCL) 374
13.4.1 Random Access Algorithms 374
13.4.2 Multiple Pass Algorithms 374
13.4.3 Two-pass Algorithms 375
13.4.4 Parallel Algorithms 377
13.4.5 Hysteresis Thresholding 377
13.5 Connected Component Analysis (CCA) 377
Contents xi

13.5.1 Basic Single-pass Algorithm 378

13.5.2 Reducing Memory Requirements 379
13.5.3 Eliminating End-of-row Overheads 379
13.5.4 Parallel Algorithms 380
13.5.5 Further Considerations and Optimisations 381
13.6 Distance Transform 381
13.6.1 Morphological Approaches 381
13.6.2 Chamfer Distance 382
13.6.3 Euclidean Distance 384
13.6.4 Applications 386
13.6.5 Geodesic Distance Transform 386
13.7 Watershed Transform 387
13.7.1 Flow Algorithms 388
13.7.2 Immersion Algorithms 389
13.8 Hough Transform 391
13.8.1 Line Hough Transform 391
13.8.2 Circle Hough Transform 394
13.8.3 Generalised Hough Transform 395
13.9 Summary 396
References 396

14 Machine Learning 403

14.1 Training 403
14.1.1 Loss and Cost Functions 404
14.1.2 Model Optimisation 405
14.1.3 Considerations 406
14.2 Regression 409
14.2.1 Linear Regression 409
14.2.2 Nonlinear Regression 409
14.2.3 Neural Networks 409
14.3 Classification 411
14.3.1 Decision Trees 411
14.3.2 Random Forests 412
14.3.3 Bayesian Classification 412
14.3.4 Quadratic Discriminant Analysis 414
14.3.5 Linear Discriminant Analysis 414
14.3.6 Support Vector Machines 415
14.3.7 Neural Networks 416
14.3.8 Clustering 417
14.4 Deep Learning 418
14.4.1 Building Blocks 419
14.4.2 Architectures and Applications 421
14.4.3 Training 427
14.4.4 Implementation Issues 428
14.5 Summary 433
References 433

15 Example Applications 441

15.1 Coloured Region Tracking 441
15.2 Foveal Sensor 443
15.2.1 Foveal Mapping 444
15.2.2 Using the Sensor 447
xii Contents

15.3 Real-time Produce Grading 448

15.3.1 Software Algorithm 448
15.3.2 Hardware Implementation 450
15.4 Stereo Imaging 453
15.4.1 Rectification 454
15.4.2 Calculating the Depth 456
15.4.3 Stereo Matching Design 457
15.5 Face Detection 459
15.5.1 Design 460
15.6 Summary 461
References 461

Index 465
Preface
Image processing, and in particular embedded image processing, faces many challenges, from increasing reso-
lution, increasing frame rates, and the need to operate at low power. These offer significant challenges for imple-
mentation on conventional software-based platforms. This leads naturally to considering field-programmable
gate arrays (FPGAs) as an implementation platform for embedded imaging applications. Many image process-
ing operations are inherently parallel, and FPGAs provide programmable hardware, also inherently parallel.
Therefore, it should be as simple as mapping one onto the other, right? Well, yes … and no.
Image processing is traditionally thought of as a software domain task, whereas FPGA-based design is firmly
in the hardware domain. There are a lot of tricks and techniques required to create an efficient design. Perhaps
the biggest hurdle to an efficient implementation is the need for a hardware mindset. To bridge the gap between
software and hardware, it is necessary to think of algorithms not on their own but more in terms of their under-
lying computational architecture. Implementing an image processing algorithm (or indeed any algorithm) on an
FPGA therefore consists of determining the underlying architecture of an algorithm, mapping that architecture
onto the resources available within an FPGA, and finally mapping the algorithm onto the hardware architecture.
While the mechanics of this process is mostly automated by high-level synthesis tools, the underlying design
is not. A low-quality design can only go so far; it is still important to keep in mind the hardware that is being
implied by the code and design the algorithm for the underlying hardware.
Unfortunately, there is limited material available to help those new to the area to get started. While there
are many research papers published in conference proceedings and journals, there are only a few that focus
specifically on how to map image processing algorithms onto FPGAs. The research papers found in the literature
can be classified into several broad groups.
The first focuses on the FPGA architecture itself. Most of these provide an analysis of a range of techniques
relating to the structure and granularity of logic blocks, the routing networks, and embedded memories. As
well as the FPGA structure, a wide range of topics are covered, including underlying technology, power issues,
the effects of process variability, and dynamic reconfigurability. Many of these papers are purely proposals, or
relate to prototype FPGAs rather than commercially available chips. While they provide insights as to some of
the features which might be available in the next generation of devices, most of the topics within this group are
at too low a level.
A second group of papers investigates the topic of reconfigurable computing. Here, the focus is on how an
FPGA can be used to accelerate some computationally intensive task or range of tasks. While image processing
is one such task considered, most of the research relates more to high-performance computing rather than
low-power embedded systems. Topics within this group include hardware and software partitioning, hardware
and software co-design, dynamic reconfigurability, communications between an FPGA and central processing
unit (CPU), comparisons between the performance of FPGAs, graphics processing units (GPUs) and CPUs,
and the design of operating systems and specific platforms for both reconfigurable computing applications and
research. Important principles and techniques can be gleaned from many of these papers even though this may
not be their primary focus.
The next group of papers considers tools for programming FPGAs and applications, with a focus on improv-
ing the productivity of the development process. A wide range of hardware description languages have been
proposed, with many modelled after software languages such as C, Java, and even Prolog. Many of these are
developed as research tools, with very few making it out of the laboratory to commercial availability. There has
xiv Preface

also been considerable research on compilation techniques for mapping standard software languages to hard-
ware (high-level synthesis). Techniques such as loop unrolling, strip mining, and pipelining to produce parallel
hardware are important principles that can result in more efficient hardware designs.
The final group of papers focuses on a range of applications, including image processing and the implemen-
tation of both image processing operations and systems. Unfortunately, as a result of page limits and space
constraints, many of these papers give the results of the implementation of various systems but present rela-
tively few design details. This is especially so in the case of many papers that describe deep learning systems.
Often the final product is described, without describing many of the reasons or decisions that led to that design.
Many of these designs cannot be recreated without acquiring the specific platform and tools that were used or
inferring a lot of the missing details. While some of these details may appear obvious in hindsight, without this
knowledge, many are far from obvious just from reading the papers. The better papers in this group tended to
have a tighter focus, considering the implementation of a single image processing operation.
So, while there may be a reasonable amount of material available, it is quite diffuse. In many cases, it is
necessary to know exactly what you are looking for, or just be lucky to find it. The intention of this book,
therefore, is to bring together much of this diverse research (on both FPGA design and image processing) and
present it in a systematic way as a reference or guide.

Intended Audience
This book is written primarily for those who are familiar with the basics of image processing and want to
consider implementing image processing using FPGAs. Perhaps the biggest hurdle is switching from a software
mindset to a hardware way of thinking. When we program in software, a good compiler can map the algorithm in
the programming language onto the underlying computer architecture relatively efficiently. When programming
hardware though, it is not simply a matter of porting the software onto hardware. The underlying hardware
architecture needs to be designed as well. In particular, programming hardware usually requires transforming
the algorithm into an appropriate parallel architecture, often with significant changes to the algorithm itself. This
requires significant design, rather than just decomposition and mapping of the dataflow (as is accomplished by
a good high-level synthesis tool). This book addresses this issue by not only providing algorithms for image
processing operations but also discusses both the design process and the underlying architectures that can be
used to implement the algorithms efficiently.
This book would also be useful to those with a hardware background, who are familiar with programming
and applying FPGAs to other problems, and are considering image processing applications. While many of the
techniques are relevant and applicable to a wide range of application areas, most of the focus and examples are
taken from image processing. Sufficient detail is given to make many of the algorithms and their implementation
clear. However, learning image processing is more than just collecting a set of algorithms, and there are any
number of excellent image processing textbooks that provide these.
It is the domain of embedded image processing where FPGAs come into their own. An efficient, low-power
design requires that the techniques of both the hardware engineer and the software engineer be integrated tightly
within the final solution.

Changes Since the First Edition

Although many of the underlying design principles have not changed, the environment has changed quite sig-
nificantly since the first edition. In general, there has been an increase in the requirements within applications.
Resolutions are increasing, with high-definition television (HDTV) becoming ubiquitous, and growing demand
within 4K and 8K resolutions (the 8K format has 33.2 Mpixels). Sensor resolutions are also increasing steadily,
with sensors having more than 100 Mpixels becoming available. Frame rates have also been increasing, with up
to 120 frames per second being part of the standard for ultra-high-definition television (UHDTV). The dynamic
range is also increasing from commonly used 8 bits per pixel to 12–16 bits. All of these factors lead to more
data to be processed, at a faster rate.
Preface xv

Against this, there has been an increasing awareness of power and sustainability issues. As a low-power
computing platform, FPGAs are well placed to address the power concerns in many applications.
The capabilities of FPGAs have improved significantly as technology improvements enable more to be
packed onto them. Not only has there been an increase in the amount of programmable logic and on-chip mem-
ory blocks, but FPGAs are becoming more heterogeneous. Many FPGAs now incorporate significant hardened
logic blocks, including moderately powerful reduced instruction set computing (RISC) processors, external
memory interfacing, and a wide range of communication interfaces. Digital signal processing (DSP) blocks
are also improving, with the move towards supporting floating-point in high-end devices. Technology improve-
ments have seen significant reductions in the power required.
Even the FPGA market has changed, with the takeover of both Altera and Xilinx by Intel and AMD, respec-
tively. This is an indication that FPGAs are seen as a serious contender for high-performance computing and
acceleration. The competition has not stood still, with both CPUs and GPUs increasing in capability. In par-
ticular, a new generation of low-power GPUs has become available that are more viable for embedded image
processing.
High-level synthesis tools are becoming more mature and address many of the development time issues
associated with conventional register transfer level design. The ability to compile to both software and hardware
enables more complex algorithms to be explored, with faster debugging. They also allow faster exploration of the
design space, enabling efficient designs to be developed more readily. However, the use of high-level synthesis
does not eliminate the need for careful algorithm design.
While the use of FPGAs for image processing has not become mainstream, there has been a lot of activity in
this space as the capabilities of FPGAs have improved. The research literature on programming and applying
FPGAs in the context of image processing has grown significantly. However, it is still quite diffuse, with most
papers focusing on one specific aspect. As researchers have looked at more complex image processing opera-
tions, the descriptions of the implementation have become higher level, requiring a lot of reading between the
lines, and additional design work to be able to replicate a design.
One significant area that has become mainstream in image processing is the use of deep learning models.
Deep learning was not around when the previous edition was written and only started becoming successful in
image processing tasks in the early 2010s. Their success has made them a driving application, not only for
FPGAs and FPGA architecture but also within computing in general. However, deep learning models pose a
huge computational demand on processing, especially for training, but also for deployment. In an embedded
vision context, this has made FPGAs a target platform for their deployment. Deep learning is a big topic on its
own, so this book is unable to do much more than scratch the surface and concentrate on some of the issues
associated with FPGA-based implementation.

Outline of the Contents

This book aims to provide a comprehensive overview of algorithms and techniques for implementing image
processing algorithms on FPGAs, particularly for low- and intermediate-level vision. However, as with design
in any field, there is more than one way of achieving a particular task. Much of the emphasis has been placed on
stream-based approaches to implementing image processing, as these can efficiently exploit parallelism when
they can be used. This emphasis reflects my background and experience in the area and is not intended to be
the last word on the topic.
A broad overview of image processing is presented in Chapter 1, with a brief historical context. Many of
the basic image processing terms are defined, and the different stages of an image processing algorithm are
identified. The problem of real-time embedded image processing is introduced, and the limitations of conven-
tional serial processors for tackling this problem are identified. High-speed image processing must exploit the
parallelism inherent in the processing of images; the different types of parallelism are identified and explained.
FPGAs combine the advantages of both hardware and software systems, by providing reprogrammable
(hence flexible) hardware. Chapter 2 provides an introduction to FPGA technology. While some of this will
be more detailed than is necessary to implement algorithms, a basic knowledge of the building blocks and
underlying architecture is important to developing resource efficient solutions. The synthesis process for build-
ing hardware on FPGAs is defined, with particular emphasis on the design flow for implementing algorithms.
xvi Preface

Traditional hardware description languages are compared with high-level synthesis, with the benefits and limi-
tations of each outlined in the context of image processing.
The process of designing and implementing an image processing application on an FPGA is described in
detail in Chapter 3. Particular emphasis is given to the differences between designing for an FPGA-based
implementation and a standard software implementation. The critical initial step is to clearly define the image
processing problem that is being tackled. This must be in sufficient detail to provide a specification that may
be used to evaluate the solution. The procedure for developing the image processing algorithm is described in
detail, outlining the common stages within many image processing algorithms. The resulting algorithm must
then be used to define the system and computational architectures. The mapping from an algorithm is more
than simply porting the algorithm to a hardware description language. It is necessary to transform the algorithm
to make efficient use of the resources available on the FPGA. The final stage is to implement the algorithm
by mapping it onto the computational architecture. Several checklists provide a guide and hints for testing and
debugging an algorithm on an FPGA.
Four types of constraints on the mapping process are limited processing time, limited access to data, lim-
ited system resources, and limited system power. Chapter 4 describes several techniques for overcoming or
alleviating these constraints. Timing explores low-level pipelining, process synchronisation, and working with
multiple clock domains. A range of memory and caching architectures are presented for alleviating memory
bandwidth. Resource sharing and associated arbitration issues are discussed, along with reconfigurability. The
chapter finishes with a section introducing commonly used performance metrics in terms of both system and
application performance.
Chapter 5 focuses on the computational aspects of image processing designs. These help to bridge the gap
between a software and hardware implementation. Different number representation and number systems are
described. Techniques for the computation of elementary functions are discussed, with a particular focus on
those that are hardware friendly. Many of these could be considered the hardware equivalent of software libraries
for efficiently implementing common functions. Possible FPGA implementations of a range of data structures
commonly found in computer vision algorithms are presented.
Any embedded application must interface with the real world. A range of common peripherals is described
in Chapter 6, with suggestions on how they may be interfaced to an FPGA. Particular attention is given to
interfacing cameras and video output devices. Interfacing with other devices is discussed, including serial com-
munications, off-chip memory, and serial processors.
The next section of this book describes the implementation of many common image processing operations.
Some of the design decisions and alternative ways of mapping the operations onto FPGAs are considered. While
reasonably comprehensive, particularly for low-level image-to-image transformations, it is impossible to cover
every possible design. The examples discussed are intended to provide the foundation for many other related
operations.
Chapter 7 considers point operations, where the output depends only on the corresponding input pixel in the
input image(s). Both direct computation and lookup table approaches are described. With multiple input images,
techniques such as image averaging and background modelling are discussed in detail. The final sections in this
chapter consider the processing of colour and hyperspectral images. Colour processing includes colour space
conversion, colour balancing, and colour segmentation.
The implementation of histograms and histogram-based processing are discussed in Chapter 8. The tech-
niques of accumulating a histogram, and then extracting data from the histogram, are described in some detail.
Particular tasks are histogram equalisation, threshold selection, and using histograms for image matching. The
concepts of standard 1-D histograms are extended to multi-dimensional histograms. The use of clustering for
colour segmentation and classification is discussed in some detail. The chapter concludes with the use of features
extracted from multi-dimensional histograms for texture analysis.
Chapter 9 considers a wide range of local filters, both linear and nonlinear. Particular emphasis is given
to caching techniques for a stream-based implementation and methods for efficiently handling the processing
around the image borders. Rank filters are described, and a selection of associated sorting network architec-
tures reviewed. Morphological filters are another important class of filters. State machine implementations of
morphological filtering provide an alternative to the classic filter implementation. Separability and both serial
and parallel decomposition techniques are described that enable more efficient implementations.
Preface xvii

Image warping and related techniques are covered in Chapter 10. The forward and reverse mapping
approaches to geometric transformation are compared in some detail, with particular emphasis on techniques
for stream processing implementations. Interpolation is frequently associated with geometric transformation.
Hardware-based algorithms for bilinear, bicubic, and spline-based interpolation are described. Related
techniques of image registration are also described at the end of this chapter, including a discussion of feature
point detection, description, and matching.
Chapter 11 introduces linear transforms, with a particular focus on the fast Fourier transform (FFT), the
discrete cosine transform (DCT), and the wavelet transform. Both parallel and pipelined implementations of
the FFT and DCT are described. Filtering and inverse filtering in the frequency domain are discussed in some
detail. Lifting-based filtering is developed for the wavelet transform. This can reduce the logic requirements by
up to a factor of 4 over a direct finite impulse response implementation.
Image coding is important for image storage or transmission. Chapter 12 discusses the stages within image
and video coding and outlines some of the techniques that can be used at each stage. Several of the standards
for both still image and video coding are outlined, with an overview of the compression techniques used.
A selection of intermediate-level operations relating to region detection and labelling is presented in
Chapter 13. Standard software algorithms for chain coding and connected component labelling are adapted
to give efficient streamed implementation. These can significantly reduce both the latency and memory
requirements of an application. Hardware implementations of the distance transform, the watershed transform,
and the Hough transform are also presented, discussing some of the key design decisions for an efficient
implementation.
Machine learning techniques are commonly used within computer vision. Chapter 14 introduces the key
techniques for regression and classification, with a particular focus on FPGA implementation. Deep learn-
ing techniques are increasingly being used in many computer vision applications. A range of deep network
architectures is introduced, and some of the issues for realising these on FPGAs are discussed.
Finally, Chapter 15 presents a selection of case studies, showing how the material and techniques described
in the previous chapters can be integrated within a complete application. These applications briefly show the
design steps and illustrate the mapping process at the whole algorithm level rather than purely at the operation
level. Many gains can be made by combining operations together within a compatible overall architecture. The
applications described are coloured region tracking for a gesture-based user interface, calibrating and correcting
barrel distortion in lenses, development of a foveal image sensor inspired by some of the attributes of the human
visual system, a machine vision system for real-time produce grading, stereo imaging for depth estimation, and
face detection.

Conventions Used
The contents of this book are independent of any particular FPGA or FPGA vendor, or any particular hard-
ware description language. The topic is already sufficiently specialised without narrowing the audience further!
As a result, many of the functions and operations are represented in block schematic form. This enables a
language-independent representation and places emphasis on a particular hardware implementation of the algo-
rithm in a way that is portable. The basic elements of these schematics are illustrated in Figure P.1. I is generally
used as the input of an image processing operation, with the output image represented by Q.
With some mathematical operations, such as subtraction and comparison, the order of the operands is impor-
tant. In such cases, the first operand is indicated with a blob rather than an arrow, as shown on the bottom in
Figure P.1.
Consider a recursive filter operating on streamed data:
{
In , |In − Qn−1 | < T,
Qn = (P.1)
Qn−1 + k(In − Qn−1 ), otherwise,

where the subscript in this instance refers to the nth pixel in the streamed image. At a high level, this can be
considered as an image processing operation, and represented by a single block, as shown in the top-left of
Figure P.1. The low-level implementation is given in the middle-left panel. The input and output, I and Q, are
xviii Preface

Image processing operation

I Filter Q
I Register
x Counter
k Constant
k
I 1 | | Function block
Q
0 Frame buffer
Single bit signal
|| T Multi-bit signal (a number)
Signal concatenation
A B A B Signal splitting

Multiplexer
A–B A>B

Figure P.1 Conventions used in this book. Top-left: representation of an image processing operation;
middle-left: a block schematic representation of the function given by Eq. (P.1); bottom-left: representation
of operators where the order of operands is important; right: symbols used for various blocks within block
schematics.

represented by registers (dark blocks, with optional register names in white); the subscripts have been dropped
because they are implicit with streamed operation. In some instances additional control inputs may be shown, for
example CE for clock enable, RST for reset. Constants are represented as mid-grey blocks, and other function
blocks with light-grey background.
When representing logic functions in equations, ∨ is used for logical OR, and ∧ for logical AND. This is to
avoid confusion with addition and multiplication.

Donald G. Bailey
Massey University
Palmerston North, New Zealand
Acknowledgments
I would like to acknowledge all those who have helped me to get me where I currently am in my understanding of
field-programmable gate array (FPGA)-based design. In particular, I would like to thank my research students
(David Johnson, Kim Gribbon, Chris Johnston, Aaron Bishell, Andreas Buhler, Ni Ma, Anoop Ambikumar,
Tariq Khan, and Michael Klaiber) who helped to shape my thinking and approach to FPGA development as we
struggled together to work out efficient ways of implementing image processing algorithms. This book is as
much a reflection of their work as it is of mine.
Most of our early work used Handel-C and was tested on boards provided by Celoxica. I would like to
acknowledge the support provided by Roger Gook and his team, first with Celoxica and later with Agility
Design Solutions. Later work was on boards supplied by Terasic. I would like to acknowledge Sean Peng and
his team for their on-going support and encouragement.
Massey University has provided a supportive environment and the freedom for me to explore this field. In
particular, Serge Demidenko gave me the encouragement and the push to begin playing with FPGAs; he has
been a source of both inspiration and challenging questions. Other colleagues who have been of particular
encouragement are Gourab Sen Gupta, Richard Harris, Amal Punchihewa, and Steven Le Moan. I would also
like to acknowledge Paul Lyons, who co-supervised several of my students.
Early versions of some of the material in this book were presented as half day tutorials at the IEEE Region
10 Conference (TENCON) in 2005 in Melbourne, the IEEE International Conference on Image Processing
(ICIP) in 2007 in San Antonio Texas, and the 2010 Asian Conference on Computer Vision (ACCV) in Queen-
stown, New Zealand. It was then adapted to three-day workshops, which were held in Australia, New Zealand,
Japan, and as a Masters Course in Germany. I would like to thank attendees at these workshops and courses for
providing valuable feedback and stimulating discussion.
During 2008, I spent a sabbatical with the Circuits and Systems Group at Imperial College London, where
I began writing the first edition. I would like to thank Peter Cheung, Christos Bouganis, Peter Sedcole, and
George Constantinides for discussions and opportunities to bounce ideas off.
My wife, Robyn, has had to put up with my absence many evenings and weekends while working on
the manuscripts for both the first and second editions. I am grateful for both her patience and her support.
This book is dedicated to her.

Donald G. Bailey
About the Companion Website
This book is accompanied by a companion website.
www.wiley.com/go/bailey/designforembeddedimageprocessc2e
This website includes PowerPoint slides.
1
Image Processing

Vision is arguably the most important human sense. The processing and recording of visual data therefore has
significant importance. The earliest images are from prehistoric drawings on cave walls or carved on stone
monuments commonly associated with burial tombs. (It is not so much the medium that is important here; any-
thing else would not have survived to today.) Such images consist of a mixture of both pictorial and abstract
representations. Improvements in technology enabled images to be recorded with more realism, such as paint-
ings by the masters. Images recorded in this manner are indirect in the sense that the light intensity pattern is
not used directly to produce the image. The development of chemical photography in the early 1800s enabled
direct image recording. This trend has continued with electronic recording, first with analogue sensors, and
subsequently with digital sensors, which include analogue to digital (A/D) conversion on the sensor chip, to
directly produce digital images.
Imaging sensors have not been restricted to the portion of the electromagnetic spectrum visible to the human
eye. Sensors have been developed to cover much of the electromagnetic spectrum from radio waves through
to gamma rays. A wide variety of other imaging modalities have also been developed, based on ultrasound,
electrons, atomic force (Binnig et al., 1986), magnetic resonance, and so on. In principle, any quantity that can
be sensed can be used for imaging, even dust rays (Auer, 1982).
Since vision is such an important sense, the processing of images has become important too, to augment or
enhance human vision. Images can be processed to enhance their subjective content or to extract useful infor-
mation. While it is possible to process the optical signals associated with visual images using lenses and optical
filters, this book focuses on digital image processing, the numerical processing of images by digital hardware.
One of the earliest applications of digital image processing was for transmitting digitised newspaper pictures
across the Atlantic Ocean in the early 1920s (McFarlane, 1972). However, it was only with the advent of comput-
ers with sufficient memory and processing power that digital image processing became widespread. The earliest
recorded computer-based image processing was from 1957 when a scanner was added to a computer at the USA
National Bureau of Standards (Kirsch, 1998). It was used for the early research on edge enhancement and pattern
recognition. In the 1960s, the need to process large numbers of large images obtained from satellites and space
exploration stimulated image processing research at NASA’s Jet Propulsion Laboratory (Castleman, 1979). At
the same time, research in high-energy particle physics required detecting interesting events from large num-
bers of cloud chamber photographs (Duff, 2000). As computers grew in power and reduced in cost, the range
of applications for digital image processing exploded, from industrial inspection to medical imaging. Image
sensors are now ubiquitous, in mobile phones, laptops, and in video-based security and surveillance systems.

1.1 Basic Definitions

More formally, an image is a spatial representation of an object, scene, or other phenomenon (Haralick and
Shapiro, 1991). Examples of images include a photograph, which is a pictorial record formed from the light

Design for Embedded Image Processing on FPGAs, Second Edition. Donald G. Bailey.
© 2024 John Wiley & Sons Ltd. Published 2024 by John Wiley & Sons Ltd.
Companion Website: www.wiley.com/go/bailey/designforembeddedimageprocessc2e
2 Design for Embedded Image Processing on FPGAs

intensity pattern on an optical sensor; a radiograph, which is a representation of density formed through exposure
to X-rays transmitted through an object; a map, which is a spatial representation of physical or cultural features;
and a video, which is a sequence of two-dimensional images through time. More rigorously, an image is any
continuous function of two or more variables defined on some bounded region of space.
A digital image is an image in digital format, so that it is suitable for processing by computer. There are
two important characteristics of digital images. The first is spatial quantisation. Computers are unable to easily
represent arbitrary continuous functions, so the continuous function is sampled. The result is a series of dis-
crete picture elements, or pixels (for 2-D images) or volume elements, voxels (for 3-D images). Sampling does
not necessarily have to be spatially uniform, for example point clouds from a LiDAR scanner. Sampling can
represent a continuous image exactly (in the sense that the underlying continuous function may be recovered
exactly), given a band-limited image and a sufficiently high sample rate (Shannon, 1949). The second character-
istic of digital images is sample quantisation. This results in discrete values for each pixel, enabling an integer
representation. Common bit widths per pixel are 1 (binary images), 8 (greyscale images), and 24 (3 × 8 bits for
colour images). Modern high dynamic range sensors can provide 12–16 bits per pixel. Unlike sampling, value
quantisation will always result in an error between the representation and true value. In many circumstances,
however, this quantisation error or quantisation noise may be made smaller than the uncertainty in the true
value resulting from inevitable measurement noise.
In its basic form, a digital image is simply a two (or higher)-dimensional array of numbers (usually inte-
gers), which represents an object or scene. Once in this form, an image may be readily manipulated by a
digital computer. It does not matter what the numbers represent, whether light intensity, reflectance, attenu-
ation, distance to a point (range), temperature, population density, elevation, rainfall, or any other numerical
quantity.
Digital image processing can therefore be defined as subjecting such an image to a series of mathematical
operations in order to obtain a desired result. This may be an enhanced image; the detection of some critical
feature or event; a measurement of an object or key feature within the image; a classification or grading of
objects within the image into one of two or more categories; or a description of the scene.
Image processing techniques are used in a number of related fields. While the principal focus of the fields
often differs, many of the techniques remain the same at the fundamental level. Some of the distinctive charac-
teristics are briefly outlined here.

Image enhancement involves improving the subjective quality of an image or the detectability of objects within
the image (Haralick and Shapiro, 1991). The information that is enhanced is usually apparent in the original
image but may not be clear. Examples of image enhancement include noise reduction, contrast enhancement,
edge sharpening, and colour correction.
Image restoration goes one step further than image enhancement. It uses knowledge of the causes of how an
image is degraded to create a model of the degradation process. This model is then used to derive an inverse
process that is used to restore the image. In many cases, the information in the image has been degraded to
the extent of being unrecognisable, for example severe blurring.
Image reconstruction involves restructuring the data that is available into a more useful form. Examples are
image super-resolution (reconstructing a high-resolution image from a series of low-resolution images),
image fusion (combining images from multiple sources), and tomography (reconstructing a cross section
of an object from a series of projections).
Image analysis refers specifically to using computers to extract data from images. The result is usually
some form of measurement. In the past, this was almost exclusively 2-D imaging, although with the
advent of confocal microscopy and other advanced imaging techniques, this has extended to three
dimensions.
Pattern recognition is concerned with identifying objects based on patterns in the measurements (Haralick and
Shapiro, 1991). There is a strong focus on statistical approaches, although syntactic and structural methods
are also used.
Computer vision tends to use a model-based approach to image processing. Mathematical models of both the
scene and the imaging process are used to derive a 3-D representation based on one or more 2-D images of
a scene. The use of models implicitly provides an interpretation of the contents of the images obtained.
Image Processing 3

The fields are sometimes distinguished based on application:

Machine vision is using image processing as part of the control system for a machine (Schaffer, 1984). Images
are captured and analysed, and the results are used directly for controlling the machine while performing a
specific task. Real-time processing is often critical.
Remote sensing usually refers to the use of image analysis for obtaining geographical information, either using
satellite images or aerial photography (including from drones).
Medical imaging encompasses a wide range of imaging modalities (X-ray, ultrasound, magnetic resonance,
positron emission, and others) concerned primarily with medical diagnosis and other medical applications. It
involves both image reconstruction to create meaningful images from the raw data gathered from the sensors
and image analysis to extract useful information from the images.
Image and video coding focuses on the compression of an image or video, so that it occupies less storage space
or takes less time to transmit from one location to another. Compression is possible because many images
contain significant redundant information. In the reverse step, image decoding, the full image or video is
reconstructed from the compressed data.

1.2 Image Formation

While there are many possible sensors that can be used for imaging, the focus in this section will be on optical
images captured from visible part of the electromagnetic spectrum. While the sensing technology may differ
significantly for other types of imaging, many of the imaging principles will be similar.
The first requirement to obtaining an image is some form of sensor to detect and quantify the incoming light.
This could be a single sensing element, with light from different directions reflected towards the sensor using
a scanning mirror. This approach is commonly used with time-of-flight laser range scanners or LiDAR. The
output of a LiDAR is typically a three-dimensional point cloud.
More commonly, a 2-D sensor array is used, which measures the light intensity at each sensing element
and produces a 2-D image. In capturing an image, it is important to sample with at least two pixels across
the highest spatial frequency (the Nyquist sampling criterion) to avoid aliasing. The two most common
solid-state sensor technologies are charge coupled devices (CCDs) and complementary metal oxide semi-
conductor (CMOS) active pixel sensors (Fossum, 1993). The basic light sensing principle is the same: an
incoming photon liberates an electron within the silicon semiconductor through the photoelectric effect.
These photoelectrons are then accumulated during the exposure time before being converted into a voltage for
reading out.
Although the active pixel sensor technology was developed before CCDs, the need for local transistors made
early CMOS sensors impractical because the transistors consumed most of the area of the device (Fossum,
1993). Therefore, CCD sensors gained early dominance in the market. However, the continual reduction of fea-
ture sizes has meant that CMOS sensors became practical from the early 1990s. Early CMOS sensors had lower
sensitivity and higher noise than a similar format CCD sensor, although with recent technological improve-
ments there is now very little difference between the two families (Litwiller, 2005). Since CMOS sensors use
the same process technology as standard CMOS devices, they also enable other functions such as A/D conver-
sion to be directly integrated on the same chip. Most modern sensors are CMOS devices, with integrated digital
conversion, providing a digital output stream.
A relative newcomer is an imaging sensor based on single-photon avalanche diodes (SPAD) (Morimoto et al.,
2020). Each sensor element can detect single photons, which are amplified to give a pulse. These pulses can
be counted, giving a direct digital measure of the intensity. The individual pulses can also be accurately timed,
giving direct time-of-flight measurement for 3-D applications.

1.2.1 Optics
A camera consists of a sensor combined with a lens. The lens focuses a parallel beam of light (effectively from
infinite distance) from a particular direction to a point on the sensor. For an object at a finite distance, as seen
4 Design for Embedded Image Processing on FPGAs

Sensor Object
(x,y,z)
f
Image Principal axis
(xi,yi) f

di do

Figure 1.1 Optical image formation using a lens using geometric optics.

in Figure 1.1, geometric optics gives the relationship between the lens focal length, f , the object distance, do ,
and image distance, di as
1 1 1
+ = . (1.1)
di do f
Since the ray through the centre of the lens is a straight line, this ray may be used to determine where an
object will be imaged to on the sensor. This pinhole model of the camera will map a 3-D point (x, y, z) in camera
coordinates (the origin is the pinhole location; the z-axis corresponds to the principal axis of the camera; and
the x- and y-axes are aligned with the sensor axes) to the point
xdi ydi
xi = , yi = (1.2)
z z
on the sensor plane. di is also called the effective focal distance of the lens. Note that the origin on the sensor
plane is in the centre of the image (the intersection of the principal axis). This projective mapping effectively
scales the object in inverse proportion to its distance from the camera (z = do ). From similar triangles, the optical
magnification is
d d
m= i = i. (1.3)
z do
The image is generally focused by adjusting the image distance, di , by moving the lens away from the sensor
(although in microscopy, di is fixed, and the image is focused by adjusting do ). This can either be manual or,
for an electronically controlled lens, an autofocus algorithm automatically adjusts di to maximise the image
sharpness (see Section 6.1.7.2).
Objects not at the optimal object distance, do , will become blurred, depending on both the offset in distance
from best focus, Δdo , and the diameter of the lens aperture, D. The size of the blurred spot is called the circle
of confusion and is given by
|Δdo | Df |Δdo | f2
c= = , (1.4)
do + Δdo do − f do + Δdo N(do − f )

with N being the numerical aperture of the lens (N = Df ).

The depth of field of a camera is the range of distances over which the degree of blurring is acceptable. This
is often defined as the range where the size of the circle of confusion is less than one pixel. If the images are
for human use, it is also necessary to take into account how the images are displayed (especially the degree
of magnification), the viewing distance, and visual acuity. This is usually taken as 0.2 mm at 250 mm distance
(Conrad, 2006).
A second factor which affects image sharpness (of an ideal lens) is diffraction. Rather than focusing a parallel
beam to an ideal point, the focus pattern in the image plane with a circular aperture is an Airy disk. The Rayleigh
criterion (Rayleigh, 1879) indicates that two points can be resolved if the peak of the diffraction pattern from
one point is outside the first null of the diffraction pattern of the other point. The size of the blur spot from
diffraction of an object at finite distance is therefore
di d
d = 1.22𝜆 = 1.22𝜆N i = 1.22𝜆N(1 + m), (1.5)
D f
where 𝜆 is the wavelength of the light.
Image Processing 5

Equations (1.4) and (1.5) may be combined to give the effective size of the blur as (Hansma, 1996)
√
de = c2 + d2 . (1.6)

Of course, any other lens aberrations or imperfections will further increase the blur.

1.2.2 Colour
Humans are able to see in colour. There are three types of colour receptors (cones) in the human eye that respond
differently to different wavelengths of light. If the wavelength dependence of a receptor is Sk (𝜆), and the light
falling on the receptor contains a mix of light of different wavelengths, C(𝜆), then the response of that receptor
will be given by the combination of the responses of all different wavelengths:

Rk = C(𝜆)Sk (𝜆)d𝜆. (1.7)

∫
What is perceived as colour is the particular combination of responses from the three different types of recep-
tors. Many different wavelength distributions produce the same responses from the receptors and therefore are
perceived as the same colour (this is known as metamerism). In practice, it is a little more complicated than
this because of chromatic adaptation.
The incoming light depends on the wavelength distribution of the source of the illumination, I(𝜆), and the
wavelength-dependent reflectivity of the object we are looking at, 𝜌(𝜆). So, Eq. (1.7) becomes

Rk = I(𝜆)𝜌(𝜆)Sk (𝜆)d𝜆. (1.8)

∫
To reproduce a colour, it is only necessary to reproduce the associated combination of receptor responses.
This is accomplished, for example in a television monitor, by producing the corresponding mixture of red,
green, and blue light.
To capture a colour image for human viewing, it is necessary to have three different colour receptors in the
sensor. Ideally, these should correspond to the spectral responses of the cones. However, since most of what is
seen has broad spectral characteristics, a precise match is not critical except when the illuminant has a narrow
spectral content (for example sodium vapour lamps and LED-based illumination sources). Silicon has a broad
spectral response, with a peak in the near infrared. Therefore, to obtain a colour image, this response must be
modified through appropriate colour filters.
Two approaches are commonly used to obtain a full colour image. The first is to separate the red, green,
and blue components of the image using a prism and with dichroic filters. These components are then captured
using three separate sensors, one for each component. The need for three chips with precise alignment makes
such cameras relatively expensive. An alternative approach is to use a single chip sensor, with a mosaic of small
filters, with one colour filter integrated with each pixel. Since each pixel senses only one colour channel, the
effective spatial resolution of the sensor is decreased. A full colour image is created by interpolating the missing
pixels of each component. The most commonly used colour filter array is the Bayer pattern (Bayer, 1976), with
filters to select the red, green, and blue primary colours. Other patterns are also possible, for example filtering the
cyan, magenta, and yellow secondary colours (Parulski, 1985). Cameras using the secondary colours have better
low light sensitivity because each filter absorbs less of the incoming light. However, the processing required
to produce the output gives a better signal-to-noise ratio from the primary filters than from secondary filters
(Parulski, 1985; Baer et al., 1999).
One further possibility that has been considered is to stack the red, green, and blue sensors one on top of the
other (Lyon and Hubel, 2002; Gilblom et al., 2003). The Foveon sensor relies on the fact that longer wavelengths
of light penetrate further into the silicon before being absorbed. Thus, using photodiodes at different depths, a
full colour pixel may be obtained without explicit filtering.
The camera output is usually streamed following a raster scan. For an analogue camera, this is a voltage
waveform, often with combined synchronisation signals to indicate the end of each row and frame. A colour
camera can output either component video (each colour channel has a separate signal) or composite video
Another random document with
no related content on Scribd:
 ellas?
Mas no ay amor do faltan
sinrazones.
A quanto mal fortuna, trae
aquellas
que haze amar, pues no ay
quien no le enfade
ni mar, ni tierra, luna, sol, ni
estrellas.
Sino a quien ama, no ay
cosa que agrade,
todo es assi, y assi fuy yo
mezquina,
a quien el tiempo estorua y
persuade.
Cessad mis uersos ya, que
amor se indigna
en uer quán presto dél me
estoy quexando,
y pido ya en mis males
mediçina.
Quexad, mas ha de ser de
quando en quando,
aora callad uos, pues ueys
que callo,
y quando veys que amor se ua
enfadando,
cessad, que no es remedio el
enfadallo.

A las Nimphas y pastores

parescieron muy bien los versos
de la pastora Belisa, la qual con
muchas lagrimas dezia,
prosiguiendo la historia de sus
males: Mas no estaua muy lexos
de alli Arsileo quando yo estos
uersos cantaua, que auiendo
aquel dia salido a caça, y estando
en lo más espeso del bosque
passando la siesta, paresçe que
nos oyó, y como hombre
affiçionado a la musica, se fue su
passo a passo entre una
espesura de arboles, que junto a
la fuente estauan: porque de alli
mejor nos pudiesse oyr. Pues
auiendo çessado nuestra musica,
él se uino a la fuente, cosa de que
no poco sobresalto reçebi. Y esto
no es de marauillar, porque de la
misma manera se sobresalta vn
coraçon enamorado, con un
subito contentamiento, que con
una tristeza no pensada. El se
llegó donde estauamos sentadas,
y nos saludó con todo el
comedimiento possible, y con
toda la buena criança que se
puede imaginar: que
uerdaderamente (hermosas
Nimphas) quando me paro a
pensar la discreçion, graçia y
gentileza del sin uentura Arsileo,
no me paresçe que fueron sus
hados y mi fortuna causa de que
la muerte me lo quitasse tan
presto delante los ojos, mas antes
fue no meresçer el mundo gozar
más tiempo de un moço a quien
la naturaleza auia dotado de
tantas y tan buenas partes.
Despues que como digo, nos uuo
saludado, y tuuo liçençia de
nosotras, la qual muy
comedidamente nos pidio, para
passar la siesta en nuestra
compañia, puso los ojos en mí
(que no deuiera) y quedó tan
preso de mis amores como
despues se paresçio en las
señales con que manifestaua su
mal. Desdichada de mí que no
uue menester yo de miralle para
querelle, que tan presa de sus
amores estaua antes que le
uiesse como él estuuo despues
de auerme uisto. Mas con todo
esso, alçé los ojos para miralle, al
tiempo que alçaua los suyos para
uerme, cosa que cada uno
quisiera dexar de auer hecho: yo
porque la uerguença me castigó,
y él porque el temor no le dexó
sin castigo. Y para dissimular su
nuevo mal, començó a hablarme
en cosas bien diferentes de las
que él me quisiera dezir, yo le
respondi a algunas dellas, pero
más cuidado tenía yo entonçes de
mirar, si en los mouimientos del
rostro, o en la blandura de las
palabras mostraua señales de
amor, que en respondelle á lo que
me preguntaua. Ansi desseaua yo
entonçes uelle sospirar, por me
confirmar en mi sospecha: como
si no le quisiera más que a mi. Y
al fin no desseaua uer en él
alguna señal que no la uiesse.
Pues lo que con la lengua alli no
me pudo dezir, con los ojos me lo
dió bien a entender. Estando en
esto las dos pastoras que
conmigo estauan, se leuantaron a
ordeñar sus vacas: yo les rogué
que me escusassen el trabajo con
las mias: porque no me sentia
buena. Y no fue menester
rogarselo más, ni a Arsileo mayor
occasion para dezirme su mal: y
no sé si se engañó, imaginando la
occasion, porque yo queria estar
sin compañia, pero sé que
determinó de aprouecharse de
ella. Las pastoras andauan
occupadas con sus vacas,
atandoles sus mansos bezerrillos
a los pies, y dexandose ellas
engañar de la industria humana,
como Arsileo tanbien nueuamente
preso de amor se dexaua ligar de
manera que otro que la
pressurosa muerte, no pudiera
dalle libertad. Pues uiendo yo
claramente, que quatro o cinco
uezes auia cometido el hablar, y
le auia salido en uano su
comedimiento: porque el miedo
de enojarme se le auia puesto
delante, quise hablarle en otro
proposito, aunque no tan lexos
del suyo, que no pudiesse sin
salir dél, dezirme lo que
desseaua. Y assi le dixe: Arsileo,
¿hallaste bien en esta tierra? que
segun en la que hasta agora has
estado, aurá sido el
entretenimiento y conuersaçion
differente del nuestro: estraño te
deues hallar en ella. El entonçes
me respondió: no tengo tanto
poder en mí, ni tiene tanta libertad
mi entendimiento, que pueda
responder a essa pregunta. Y
mudandole el proposito, por
mostralle el camino con las
occasiones le bolui a dezir: an me
dicho, que ay por allá muy
hermosas pastoras, y si esto es
ansi, quán mal te deuemos
parescer las de por acá. De mal
conoscimiento sería (respondió
Arsileo) si tal confessasse: que
puesto caso, que allá las ay tan
hermosas como te han dicho, acá
las ay tan auentajadas, como yo
las he uisto. Lisonja es essa en
todo el mundo (dixe yo medio
riendo) mas con todo esto, no me
pesa que las naturales estén tan
adelante en tu opinion, por ser yo
una dellas. Arsileo respondió: y
aun essa sería harto bastante
causa, quando otra no uuiesse,
para dezir lo que digo. Assi que
de palabra en palabra, me uino a
dezir lo que desseaua oylle,
aunque por entonçes no quise
darselo a entender, mas antes le
rogué, que atajasse el paso a su
pensamiento. Pero reçelosa que
estas palabras no fuesen causa
de resfriarse en el amor (como
muchas uezes acaesce que el
desfauorescer en los principios de
los amores, es atajar los passos a
los que comiençan a querer bien)
bolui a templar el desabrimiento
de mi respuesta, diziendole: Y si
fuere tanto el amor (o Arsileo) que
no te dé lugar a dexar de
quererme, en lo secreto: porque
de los hombres de semejante
discreçion que la tuya, es tenello
aun en las cosas que poco
importan. Y no te digo esto,
porque de una, ni de otra manera
te ha de aprouechar de más que
de quedarte yo en obligaçion, si
mi consejo en este caso tomares.
Esto dezia la lengua, mas otra
cosa dezian los ojos con que yo le
miraua, y echando algun sospiro
que sin mi liçençia daua
testimonio de lo que yo sentia, lo
qual entendiera muy bien Arsileo,
si el amor le diera lugar. Desta
manera nos despedimos, y
despues me habló muchas uezes,
y me escriuio muchas cartas, y vi
muchos sonetos de su mano, y
aun las más de las noches me
dezia cantando, al son de su
harpa, lo que yo llorando le
escuchaua. Finalmente que
venimos cada vno a estar bien
çertificados del amor que el vno al
otro tenía. A este tiempo, su
padre Arsenio me importunaua de
manera con sus recaudos y
presentes, que yo no sabia el
medio que tuuiesse para
defenderme dél. Y era la más
estraña cosa que se vió jamás:
pues ansi como se yua más
acrescentando el amor con el hijo,
assi con el padre se yua más
estendiendo el affiçion, aunque no
era todo de vn metal. Y esto no
me daua lugar a desfauorescelle,
ni a dexar de reçebir sus
recaudos. Pues viuiendo yo con
todo el contentamiento del
mundo, y viendome tan de veras
amada de Arsileo, a quien yo
tanto queria, paresçe que la
fortuna determinó de dar fin a mis
amores, con el más desdichado
sucçesso, que jamás en ellos se
ha visto, y fue desta manera: que
auiendo yo conçertado de hablar
con mi Arsileo vna noche, que
bien noche fue ella para mí: pues
nunca supe despues acá, qué
cosa era dia, concertamos que él
entrase en una huerta de mi
padre, y yo desde vna ventana de
mi aposento, que caya enfrente
de vn moral, donde él se podia
subir por estar más çerca, nos
hablariamos: ¡ay desdichada de
mí, que no acabo de entender a
qué proposito le puse en este
peligro, pues todos los dias, aora
en el campo, aora en el rio. aora
en el soto, llevando a él mis
vacas, aora al tiempo que las
traya a la majada, me pudiera él
muy bien hablar, y me hablaua los
más de los dias. Mi desuentura
fue causa que la fortuna se
pagasse del contento, que hasta
entonçes me auia dado, con
hazerme que toda la uida biuiesse
sin él. Pues uenida la hora del
conçierto y del fin de sus dias, y
principio de mi desconsuelo, vino
Arsileo al tiempo, y al lugar
conçertado, y estando los dos
hablando, en lo que puede
considerar quien algun tiempo ha
querido bien, el desuenturado de
Arsenio su padre, las más de las
noches me rondaua la calle (que
aun si esto se me acordara, mas
quitomelo mi desdicha de la
memoria, no le consintiera yo
ponerse en tal peligro); pero asi
se me oluidó, como si yo no lo
supiera. Al fin que él acertó a
venir aquella hora por alli, y sin
que nosotros pudiessemos velle,
ni oylle, nos vió él, y conosçio ser
yo la que a la ventana estaua,
mas no entendió que era su hijo
el que estaua en el moral, ni aun
pudo sospechar quien fuesse,
que esta fue la causa prinçipal de
su mal successo. Y fue tan
grande su enojo, que sin sentido
alguno se fue a su posada, y
armando una ballesta, y
poniendola vna saeta muy llena
de venenosa yerua, se uino al
lugar do estauamos, y supo tan
bien açertar a su hijo, como sino
lo fuera. Porque la saeta le dio en
el coraçon, y luego cayó muerto
del árbol abaxo, diziendo: ¡Ay
Belisa, quán poco lugar me da la
fortuna para seruirte, como yo
desseaua! Y aun esto no pudo
acabar de dezir. El desdichado
padre que con estas palabras
conosció ser homiçida de Arsileo
su hijo, dixo con una boz como de
hombre desesperado:
¡Desdichado de mí, si eres mi hijo
Arsileo que en la boz no paresçes
otro! Y como llegasse a él, y con
la luna que en el rostro le daua le
deuisasse bien y le hallase que
auia espirado, dixo: O cruel
Belisa, pues que el sin ventura mi
hijo, por tu causa, de mis manos
ha sido muerto, no es justo que el
desuenturado padre quede con la
vida. Y sacando su misma
espada, se dio por el coraçon de
manera que en un punto fue
muerto. O desdichado caso, o
cosa jamás oida ni vista. ¡O
escandalo grande para los oydos,
que mi desdichada historia
oyeren, o desuenturada Belisa,
que tal pudieron uer tus ojos, y no
tomar el camino que padre y hijo
por tu causa tomaron! No
paresciera mal tu sangre
mixturada con la de aquellos que
tanto desseauan seruirte. Pues
como yo mezquina ui el
desauenturado caso, sin más
pensar, como muger sin sentido,
me sali de casa de mis padres, y
me uine importunando con
quexas al alto çielo, y inflamando
el ayre con sospiros, a este triste
lugar (quexandome de mi fortuna,
maldiziendo la muerte que tan en
breue me auia enseñado a sufrir
sus tiros) adonde ha seys meses
que estoy sin auer uisto, ni
hablado con persona alguna, ni
procurado uerla. Acabando la
hermosa Belisa de contar su
infelice historia, començo a llorar
tan amargamente, que ninguno
de los que alli estauan, pudieron
dexar de ayudarle con sus
lagrimas. Y ella prosiguiendo
dezia: Esta es (hermosas
Nimphas) la triste historia de mis
amores, y del desdichado
sucçesso dellos, ved si este mal
es de los que el tiempo puede
curar? ¡Ay Arsileo, quantas vezes
temi, sin pensar lo que temia! mas
quien a su temor no quiere creer,
no se espante, quando vea lo que
ha temido, que bien sabia yo que
no podiades dexar de
encontraros, y que mi alegria no
auia de durar más que hasta que
su padre Arsenio sintiesse
nuestros amores. Pluguiera a
Dios que assi fuera que el mayor
mal que por esso me pudiera
hazer fuera desterrarte: y mal que
con el tiempo se cura, con poca
difficultad puede suffrirse. ¡Ay
Arsenio, que no me estorua la
muerte de tu hijo dolerme de la
tuya, que el amor que continuo
me monstraste, la bondad y
limpieza con que me quisiste, las
malas noches que a causa mia
passaste, no suffre menos si no
dolerme de tu desastrado fin: que
esta es la hora que yo fuera
casada contigo, si tu hijo a esta
tierra no uiniera! Dezir yo que
entonçes no te queria bien seria
engañar el mundo, que en fin no
hay muger que entienda que es
uerdaderamente amada, que no
quiera poco o mucho, aunque de
otra manera lo dé a entender: ay
lengua mia, callad que más aueys
dicho de lo que os an preguntado.
¡O hermosas Nimphas! perdonad
si os he sido importuna, que tan
grande desuentura como la mia
no se puede contar con pocas
palabras. En quanto la pastora
contaua lo que aueys oydo,
Sireno, Syluano, Seluagia, y la
hermosa Felismena, y aun las
tres Nimphas fueron poca parte
para oylla sin lagrimas: aunque
las Nimphas, como las que de
amor no auian sido tocadas,
sintieron como mugeres su mal,
mas no las circunstançias dél.
Pues la hermosa Dorida uiendo
que la desconsolada pastora no
cesaua el amargo llanto, la
començo a hablar diziendo:
Cessen, hermosa Belisa, tus
lagrimas, pues uees el poco
remedio dellas: mira que dos ojos
no bastan a llorar tan graue mal.
Mas qué dolor puede auer, que no
se acabe, o acabe al mismo que
lo padesçe? Y no me tengas por
tan loca que piense consolarte,
mas a lo menos podria mostrarte
el camino por donde pudiesse
algun poco aliuiar tu pena. Y para
esto te ruego, que uengas en
nuestra compañia, ansi porque no
es cosa justa que tan mal gastes
la uida, porque adonde te
lleuaremos podras escoger la que
quisieres, y no aurá persona, que
estorualla pueda. La pastora
respondió: lugar me pareçia este
harto conveniente para llorar mi
mal y acabar en él la uida: la qual
si el tiempo no me haze más
agrauios de los hechos, no deue
ser muy larga. Mas ya que tu
uoluntad es essa, no determino
de salir della en solo un punto: y
de oy mas podeis (hermosas
Nimphas) usar de la mia, segun a
las uuestras les paresçiere.
Mucho le agradesçieron todos
auelles conçedido de irse en su
compañia. Y porque ya eran más
de tres horas de la noche aunque
la luna era tan clara, que no
echauan memos el dia çenaron
de lo que en sus çurrones los
pastores trayan, y despues de
haber çenado, cada vno escogió
el lugar de que más se contentó,
passar lo que de la noche les
quedaua. La qual los enamorados
passaron con más lagrimas que
sueño, y los que no eran
reposaron del cansançio del dia.

Fin del terçero libro.

 NOTAS:
[1248] M., buscalle.
[1249] M., grande.
[1250] M., perficion.
[1251] V., daños.
[1252] V., atreuimiento.
[1253] M., grave.
 LIBRO CUARTO
DE LA DIANA DE
GEORGE DE
MONTEMAYOR

Ya la estrella del alua començaua

a dar su acostumbrado
resplandor, y con su luz los dulçes
ruyseñores embiauan a las nuues
el suaue canto, quando las tres
Nimphas con su enamorada
compañia, se partieron de la
isleta, donde Belisa su triste uida
passaua. La qual aunque fuese
más consolada en conuersaçion
de las pastoras y pastores
enamorados, todauia le
apremiaba el mal de manera que
no hallaua remedio, para dexar de
sentillo. Cada pastor le contaua
su mal, las pastoras le dauan
cuenta de sus amores, por uer si
seria parte para ablandar su
pena. Mas todo consuelo es
escusado, quando los males son
sin remedio. La dama disimulada
yua tan contenta de la hermosura
y buena graçia de Belisa, que no
se hartaua de preguntalle cosas,
aunque Belisa se hartaua de
responderle a ellas. Y era tanta la
conuersaçion de las dos, que casi
ponia embidia a los pastores y
pastoras. Mas no uuieron andado
mucho, quando llegaron a un
espesso bosque tan lleno de
syluestres y espessos arboles,
que a no ser de las tres Nimphas
guiadas, no pudieran dexar de
perderse en él. Ellas yuan delante
por una muy angosta[1254] senda,
por donde no podian yr dos
personas juntas. Y auiendo ydo
quanto media legua por la
espessura del bosque, salieron a
un muy grande, y espaçioso llano
en medio de dos caudalosos rios,
ambos çercados de muy alta y
uerde arboleda. En medio dél
paresçia una gran casa de tan
altos y soberuios edifiçios, que
ponian gran contentamiento a los
que los mirauan, porque los
chapiteles que por ençima de los
arboles sobrepujauan, dauan de
si tan gran resplandor, que
pareçian hechos de un finissimo
cristal. Antes que al gran palaçio
llegassen, uieron salir dél muchas
Nimphas de tan gran hermosura,
que sería impossible podello
dezir. Todas uenian[1255] uestidas
de telillas blancas delicadas,
texidas con plata y oro
sotilissimamente, sus guirnaldas
de flores sobre los dorados
cabellos que sueltos trayan.
Detras dellas uenia una dueña,
que segun la grauedad y arte de
su persona, parescia muger de
grandissimo respecto, uestida de
raso negro, arrimada a una
Nimpha muy más hermosa que
todas. Quando nuestras Nhimpas
llegaron, fueron de las otras
reçebidas, con muchos abraços, y
con gran contentamiento. Como
la dueña llegasse, las tres
Nimphas le besaron con
grandissima humildad las manos,
y ella las reçibio, mostrando muy
gran contento de su uenida. Y
antes que las Nimphas le
dixessen cosa de las que auian
passado, la sábia Felicia (que asi
se llamaua la dueña) dixo contra
Felismena: hermosa pastora, lo
que por estas tres Nimphas aueys
hecho no se puede pagar con
menos que con tenerme obligada
siempre a ser en vuestro fauor:
que no será poco, segun
menester lo aueys, y pues yo sin
estar informada de nadie, sé
quien soys, y adonde os lleuan
uuestros pensamientos, contodo
lo que hasta agora os ha
sucçedido, ya entendereys si os
puedo aprouechar en algo. Pues
tened animo firme, que si yo biuo
vos uereys lo que desseays, y
aunque ayays passado algunos
trabajos, no ay cosa que sin ellos
alcançar se pueda. La hermosa
Felismena se marauilló de las
palabras de Feliçia, y queriendo
dalle las graçias que a tan gran
promesa se deuian, respondio:
Dyscreta señora mía (pues en fin
lo aueys de ser de mi remedio)
quando de mi parte no haya
mereçimiento donde pueda caber
la merçed que pensays hazerme,
poned los ojos en lo que a vos
misma deueys, y yo quedaré sin
deuda, y uos muy bien pagada.
Para tan grande mereçimiento
como el vuestro (dixo Feliçia), y
tan extremada hermosura, como
naturaleza os ha conçedido, todo
lo que por uos se puede hazer es
poco. La dama se abaxó
entonçes por besalle las manos, y
Feliçia la abraçó con grandissimo
amor, y boluiendose a los
pastores y pastoras, les dixo:
animosos pastores y discretas
pastoras, no tengays miedo a la
perseuerençia de nuestros males,
pues yo tengo cuenta con el
remedio dellos. Las pastoras y
pastores le besaron las manos, y
todos juntos se fueron al
sumptuoso palaçio, delante del
qual estaua una gran plaça
çercada de altos açipreses todos
puestos muy por orden, y toda la
plaça era enlosada con losas de
alabastro y marmol negro, a
manera de axedrez. En medio
della auia una fuente de marmol
jaspeado, sobre quatro muy
grandes leones de bronçe. En
medio de la fuente estaua una
columna de jaspe, sobre la qual
quatro Nimphas de marmol
blanco tenian sus assientos. Los
braços tenian alçados en alto, y
en las manos sendos uasos
hechos a la romana. De los
quales por vnas bocas de leones
que en ellos auia, echauan agua.
La portada del Palacio era de
marmol serrado con todas las
basas, y chapiteles de las
columnas dorados. Y ansi mismo
las vestiduras de las imagenes
que en ellos auia. Toda la casa
paresçia hecha de reluziente
jaspe con muchas almenas, y en
ellas esculpidas algunas figuras
de Emperadores, y matronas
Romanas, y otras untiguallas
semejantes. Eran todas las
ventanas cada vna de dos arcos,
las çerraduras y clavazon de
plata, todas las puertas de cedro.
La casa era quadrada, y a cada
canto auia una muy alta, y
artifiçiosa torre. En llegando la
aportada, se pararon a mirar su
estraña hechura, y las imagenes
que en ella auia, que más
parescia obra de naturaleza que
de arte, ni aun industria humana,
entre las quales auia dos
Nimphas de plata, que ençima de
los chapiteles de las columnas
estauan, y cada una de su parte
tenian una tabla de alambre, con
unas letras de oro, que dezian
desta manera:

Qvien entra, mire bien como

ha biuido
y el don de castidad, si le ha
guardado,
y la que quiere bien, o le ha
querido,
mire si a causa de otro se ha
mudado,
y si la fe primera no ha
perdido,
y aquel primer amor ha
conseruado,
entrar puede en el templo de
Diana,
cuya virtud y graçia es
sobrehumana.

Qvando esto vuo oydo la hermosa

Felismena dixo contra las
pastoras Beliza y Selvagia. Bien
seguras me paresçe que
podemos entrar en este
sumptuoso palaçio de yr contra
las leyes que aquel letrero nos
pone. Sireno se atrauessó,
diziendo: esso no pudiera hazer la