Big Data Anil Maheshwari

Visit to download the full and correct content document:

https://ebookmass.com/product/big-data-anil-maheshwari/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Data analytics Anil Maheshwari

https://ebookmass.com/product/data-analytics-anil-maheshwari/

Process Safety and Big Data Sagit Valeev

https://ebookmass.com/product/process-safety-and-big-data-sagit-
valeev/

Big Data Balamurugan Balusamy

https://ebookmass.com/product/big-data-balamurugan-balusamy/

Big Data in Astronomy: Scientific Data Processing for

Advanced Radio Telescopes 1st Edition Linghe Kong
(Editor)

https://ebookmass.com/product/big-data-in-astronomy-scientific-
data-processing-for-advanced-radio-telescopes-1st-edition-linghe-
kong-editor/
Consciousness-Based Leadership and Management, Volume
1: Vedic and Other Philosophical Approaches to Oneness
and Flourishing Anil K. Maheshwari

https://ebookmass.com/product/consciousness-based-leadership-and-
management-volume-1-vedic-and-other-philosophical-approaches-to-
oneness-and-flourishing-anil-k-maheshwari/

Big Data and Mobility as a Service 1st Edition Haoran

Zhang

https://ebookmass.com/product/big-data-and-mobility-as-a-
service-1st-edition-haoran-zhang/

Big Data Analysis of Nanoscience Bibliometrics, Patent,

and Funding Data (2000-2019) 1st Edition Yuliang Zhao

https://ebookmass.com/product/big-data-analysis-of-nanoscience-
bibliometrics-patent-and-funding-data-2000-2019-1st-edition-
yuliang-zhao/

The Big R-Book: From Data Science to Learning Machines

and Big Data Philippe J. S. De Brouwer

https://ebookmass.com/product/the-big-r-book-from-data-science-
to-learning-machines-and-big-data-philippe-j-s-de-brouwer/

Distrust: Big Data, Data-Torturing, and the Assault on

Science Gary Smith

https://ebookmass.com/product/distrust-big-data-data-torturing-
and-the-assault-on-science-gary-smith/
BIG DATA
 About the Author

Dr. Anil Maheshwari is a Professor of Management Informa-

tion Systems and Director of Center for Data Analytics at
Maharishi University of Management, Fairfield, Iowa, USA.

He received a bachelor’s degree in Electrical Engineering from

Indian Institute of Technology (IIT) Delhi, and further received
his MBA degree from Indian Institute of Management (IIM)
Ahmedabad. He earned his PhD from Case Western Reserve
University, Cleveland, Ohio. He has been a Professor at the
University of Cincinnati, City University of New York, among
others. His research has been published in prestigious journals
and conferences. He teaches data analytics, big data, leadership, and marketing. He
has authored many books in data science and leadership. He has also worked in the
global IT industry for over 20 years, including leadership roles at IBM in Austin,
Texas. He has completed various leadership and marketing training programs at
IBM and also won several awards.

He is a practitioner of Transcendental Meditation (TM) and TM-Sidhi techniques. He

blogs on IT and Enlightenment at anilmah.com, and is a popular speaker on those
topics. He is also a marathoner. He can be reached at akm2030@gmail.com.
 BIG DATA

Dr. Anil Maheshwari

Professor of Management Information Systems and
Director of Center for Data Analytics
Maharishi University of Management
Fairfield, Iowa, USA.

McGraw Hill Education (India) Private Limited

CHENNAI

McGraw Hill Education Offices

Chennai New York St Louis San Francisco Auckland Bogotá Caracas
Kuala Lumpur Lisbon London Madrid Mexico City Milan Montreal
San Juan Santiago Singapore Sydney Tokyo Toronto
 McGraw Hill Education (India) Private Limited

Published by McGraw Hill Education (India) Private Limited

444/1, Sri Ekambara Naicker Industrial Estate, Alapakkam, Porur, Chennai - 600 116

Big Data
Copyright © 2017, by Dr. Anil Maheshwari
No part of this publication may be reproduced or distributed in any form or by any means, electronic, mechanical, photocopying,
recording, or otherwise or stored in a database or retrieval system without the prior written permission of the publishers.
The program listings (if any) may be entered, stored and executed in a computer system, but they may not be reproduced for
publication.
This edition can be exported from India only by the publishers,
McGraw Hill Education (India) Private Limited
Print Edition:
ISBN-13: 978-93-5260-502-6
ISBN-10: 93-5260-502-0
E-Book Edition:
ISBN-13: 978-93-5260-454-8
ISBN-10: 93-5260-454-7
Managing Director: Kaushik Bellani

Director—Science & Engineering Portfolio: Vibha Mahajan

Lead—Science & Engineering Portfolio: Hemant Jha

Content Development Lead: Shalini Jha

Content Developer: Shehla Mirza
Production Head: Satinder S Baveja
Sr. Manager—Production: Piyaray Pandita
General Manager—Production: Rajender P Ghansela
Manager—Production: Reji Kumar

Information contained in this work has been obtained by McGraw Hill Education (India), from sources believed to be reliable.
However, neither McGraw Hill Education (India) nor its authors guarantee the accuracy or completeness of any information
published herein, and neither McGraw Hill Education (India) nor its authors shall be responsible for any errors, omissions,
or damages arising out of use of this information. This work is published with the understanding that McGraw Hill Education
(India) and its authors are supplying information but are not attempting to render engineering or other professional services.
If such services are required, the assistance of an appropriate professional should be sought.

Typeset at The Composers, 260, C.A. Apt., Paschim Vihar, New Delhi 110 063 and printed at

Cover Printer:

Visit us at: www.mheducation.co.in

 Dedicated to

My family of three lovely ladies

Wife Neerja, and daughters Ankita and Nupur
who taught me love, care and gentleness
 Preface

Big Data is a new, and inclusive, natural phenomenon, as big and messy as nature
itself. It requires a new kind of consciousness to fathom its scale and scope, and its
many opportunities and challenges. Understanding the essentials of Big Data requires
suspending many conventional expectations and assumptions about data ... such as
completeness, clarity, consistency, and conciseness. Fathoming and taming the multi-
layered Big Data is a dream that is slowly becoming a reality. It is a rapidly evolving
field that is growing exponentially in value and capabilities.

There are a growing number of books being written on Big Data. They fall mostly
in two categories. There are those that focus on business aspects, and discuss the
strategic internal shifts required for reaping the business benefits from the many
opportunities offered by Big Data. Then there are those that focus on particular
technology platforms, such as Hadoop or Spark. This book aims to bring together the
business context and the technologies in a seamless way.

Thanks to Maharishi Mahesh Yogi for creating a wonderful university whose

consciousness-based environment made writing this evolutionary book possible.
Thanks to many current and former students for contributing to this book. Dheer-
aj Pandey assisted with the Weblog analyzer application and its details. Suraj
Thapalia assisted with the Hadoop installation guide. Enkh Tseeleesuren helped
write the Spark tutorial. Thanks to my family for supporting me in this process.
My daughters Ankita and Nupur reviewed the book and made helpful comments. My
father Mr. R L Maheshwari and brother Dr. Sunil Maheshwari also read the book
and enthusiastically approved it. My colleague Dr. Edi Shivaji too reviewed the book.

May the Big Data Force be with you!

Dr. Anil Maheshwari

May 2017, Fairfield, IA
 Contents

Preface vii

1. Wholeness of Big Data 1

Introduction 1
1.1 Understanding Big Data 2
1.2 Capturing Big Data 4
1.2.1 Volume of Data 4
1.2.2 Velocity of Data 4
1.2.3 Variety of Data 5
1.2.4 Veracity of Data 5
1.3 Benefitting from Big Data 7
1.4 Management of Big Data 8
1.5 Organizing Big Data 9
1.6 Analyzing Big Data 10
1.7 Technology Challenges for Big Data 12
1.7.1 Storing Huge Volumes 12
1.7.2 Ingesting Streams at an Extremely Fast Pace 12
1.7.3 Handling a Variety of Forms and Functions of Data 13
1.7.4 Processing Data at Huge Speeds 13
1.8 Conclusion 14
1.9 Organization of the Rest of the Book 15
Review Questions 16
True/False Questions 16

SECTION 1

2. Big Data Sources and Applications 21

Introduction 21
2.1 Big Data Sources 23
x Contents

2.1.1 People-to-People Communications 23

2.1.2 People-to-Machine Communications 24
2.2 Machine-to-Machine (M2M) Communications 25
2.2.1 RFID Tags 26
2.2.2 Sensors 26
2.3 Big Data Applications 27
2.3.1 Monitoring and Tracking Applications 27
2.3.2 Analysis and Insight Applications 29
2.3.3 New Product Development 31
2.4 Conclusion 32
Review Questions 32
True/False Questions 32

3. Big Data Architecture 34

Introduction 34
3.1 Standard Big Data Architecture 35
3.2 Big Data Architecture Examples 37
3.2.1 IBM Watson 37
3.2.2 Netflix 38
3.2.3 eBay 38
3.2.4 VMWare 39
3.2.5 The Weather Company 40
3.2.6 TicketMaster 40
3.2.7 LinkedIn 42
3.2.8 Paypal 42
3.2.9 CERN 43
3.3 Conclusion 44
Review Questions 44
True/False Questions 44

SECTION 2

4. Distributed Computing Using Hadoop 47

Introduction 47
4.1 Hadoop Framework 48
4.2 HDFS Design Goals 48
4.3 Master-Slave Architecture 49
4.4 Block System 51
4.5 Ensuring Data Integrity 52
 Contents xi

4.6 Installing HDFS 52

4.6.1 Reading and Writing Local Files into HDFS 53
4.6.2 Reading and Writing Data Streams into HDFS 53
4.7 Sequence Files 54
4.8 YARN 54
4.9 Conclusion 55
Review Questions 55
True/False Questions 56

5. Parallel Processing with Map Reduce 57

Introduction 57
5.1 MapReduce Overview 58
5.2 Sample MapReduce Application: Wordcount 61
5.3 MapReduce Programming 63
5.3.1 MapReduce Data Types and Formats 64
5.3.2 Writing MapReduce Programming 64
5.3.3 Testing MapReduce Programs 66
5.4 MapReduce Jobs Execution 66
5.4.1 How MapReduce Works 67
5.4.2 Managing Failures 68
5.4.3 Shuffle and Sort 68
5.4.4 Progress and Status Updates 69
5.5 Hadoop Streaming 69
5.6 Hive Language 70
5.6.1 HIVE Language Capabilities 70
5.7 Pig Language 72
5.7.1 PIG Language Capabilities 73
5.7.2 Pig Script Example 74
5.8 Conclusion 75
Review Questions 75
True/False Questions 75

6. NoSQL Databases 76
Introduction 76
6.1 RDBMS Vs NoSQL 77
6.2 Types of NoSQL Databases 78
6.3 Architecture of NoSQL 81
6.4 CAP Theorem 82
6.5 HBase 83
6.5.1 Architecture Overview 83
6.5.2 Reading and Writing Data 85
xii Contents

6.6 Cassandra 85
6.6.1 Architecture Overview 85
6.6.2 Protocols 86
6.6.3 Data Model 87
6.6.4 Cassandra Writes 88
6.6.5 Cassandra Reads 88
6.6.6 Replication 90
6.7 Conclusion 91
Review Questions 91
True/False Questions 91

7. Stream Processing with Spark 93

Introduction 93
7.1 Spark Architecture 94
7.1.1 Resilient Distributed Datasets (RDDs) 95
7.1.2 Directed Acyclic Graph (DAG) 96
7.2 Spark Ecosystem 96
7.3 Spark for Big Data Processing 96
7.3.1 MLlib 96
7.3.2 Spark GraphX 97
7.3.3 SparkR 98
7.3.4 SparkSQL 98
7.3.5 Spark Streaming 99
7.4 Spark Applications 99
7.4.1 Spark vs Hadoop 99
7.5 Conclusion 101
Review Questions 101
True/False Questions 101

8. New Ingesting Data 102

Introduction 102
8.1 Messaging Systems 103
8.1.1 Point to Point Messaging System 103
8.1.2 Publish-Subscribe Messaging System 104
8.2 Data Ingest Systems 104
8.3 Apache Kafka 104
8.4 Use Cases 105
8.5 Kafka Architecture 106
8.5.1 Producers 107
8.5.2 Consumers 107
8.5.3 Broker 107
 Contents xiii

8.5.4 Topic 107

8.5.5 Summary of Key Attributes 108
8.5.6 Data Replication 109
8.5.7 Guarantees 109
8.5.8 Client Libraries 109
8.6 Apache ZooKeeper 110
8.6.1 Kafka Producer Example in Java 110
8.7 Conclusion 110
Review Questions 110
True/False Questions 111

9. Cloud Computing 112

Introduction 112
9.1 Cloud Computing Characteristics 114
9.1.1 In-house Storage 114
9.2 Cloud Storage 114
9.3 Cloud Computing: Evolution of Virtualized Architecture 116
9.4 Cloud Computing Myths 118
9.5 Cloud Computing: Getting Started 118
9.6 Conclusion 119
Review Questions 119
True/False Questions 120

SECTION 3

10. Web Log Analyzer Application Case Study 123

Introduction 123
10.1 Client-Server Architecture 123
10.2 Web Log Analyzer 124
10.2.1 Requirements 124
10.2.2 Solution Architecture 124
10.2.3 Benefits of Such Solution 125
10.3 Technology Stack 126
10.3.1 Apache Spark 126
10.3.2 Spark Deployment 126
10.3.3 Components of Spark 126
10.4 HDFS 127
10.5 MongoDB 127
10.6 Apache Flume 128
xiv Contents

10.7 Overall Application Logic 128

10.8 Technical Plan for the Application 128
10.9 Scala Spark Code for Log Analysis 129
10.10 Sample Log Data 130
10.10.1 Sample Input Data 130
10.11 Sample Output of Web Log Analysis 131
10.12 Conclusion and Findings 132
Review Questions 132
True/False Questions 132

11. Data Mining Primer 133

Introduction 133
11.1 Gathering and Selecting Data 134
11.2 Data Cleansing and Preparation 135
11.3 Outputs of Data Mining 135
11.4 Evaluating Data Mining Results 136
11.4.1 Predictive Accuracy = (Correct Predictions)/
Total Predictions 136
11.5 Data Mining Techniques 137
11.6 Mining Big Data 140
11.6.1 From Causation to Correlation 140
11.6.2 From Sampling to the Whole 140
11.6.3 From Dataset to Data Stream 140
11.7 Data Mining Best Practices 141
11.8 Conclusion 143
Review Questions 143
True/False Questions 143

12. Big Data Programming Primer 145

Introduction 145
12.1 Comparing Hive and Pig 145
12.2 Apache Hive 146
12.2.1 Architecture of Hive 147
12.2.2 Working of Hive 147
12.2.3 Hive Data Definition 149
12.2.4 Hive Partitioning 151
12.2.5 Hive Data Manipulation 152
12.2.6 Hive View and Indexes 156
12.3 Apache Pig 157
12.3.1 Running Pig 158
12.3.2 Pig Latin Data Model 159
 Contents xv

12.3.3 Pig Latin Operators 160

12.3.4 Pig Data Definition 161
12.3.5 Pig Diagnostic Operators 162
12.3.6 Pig Data Manipulation 162
12.3.7 Pig Built-in Functions 165
12.3.8 Pig User Defined Functions 168
12.3.9 Running Pig Scripts 168
12.4 Conclusion 169
Review Questions 169
True/False Questions 170

Appendix 1 Installing Hadoop Using Cloudera on Virtual Box 171

Appendix 2 Installing Hadoop on Amazon Web Services (AWS)
Elastic Compute Cluster (EC2) 197
Appendix 3 Spark Installation and Tutorial 222
Additional Resources 232

Index 233
 Chapter 1
Wholeness of Big Data

Learning Objectives
■ Understand Big Data and its powerful business benefits
■ Learn the differences between Big Data and Conventional Data
■ Learn the 4 Vs of Big Data – Volume, Velocity, Variety, and Veracity
■ Discover three types of business applications of Big Data
■ Learn valuable tips on how to manage Big Data
■ Conceptualize the Big Data ecosystem and appreciate its key components
■ Appreciate major technological challenges in managing Big Data
■ Relate to the key technology solutions for addressing those challenges

INTRODUCTION
Big Data is an all-inclusive term that refers to extremely large, very fast, highly diverse,
and complex data that cannot be managed with traditional data management tools.

Business Intelligence

Volume Veracity

Analytics
BIG
DATA

Variety
(from, function, Velocity
source)

FIGURE 1.1 Big data context

2 Big Data

Ideally, Big Data includes all kinds of data, which helps deliver the right informa-
tion, to the right person, in the right quantity, at the right time, to help make the
right decisions. Big Data can be harnessed by developing infinitely scalable, flexible,
and evolutionary data architectures, coupled with the use of cost-effective computing
machines. The infinite potential knowledge embedded within this Big Data cosmic
computer would help connect with and enjoy the support of all the laws of nature.

This book will provide a complete overview of Big Data for executives and data sci-
entists. This chapter will cover the key challenges and benefits of Big Data, and the
essential tools and technologies available for organizing and manipulating Big Data.

1.1 UNDERSTANDING BIG DATA

Big Data can be examined at two levels (Figure 1.1). At a fundamental level, it is
just another collection of data that can be analyzed and utilized for the benefit of the
business. On another level, it is a special kind of data that poses unique challenges
and offers unique benefits. This book will focus on the latter level.

At business level, data generated by business operations, can be analyzed to generate

insights that can help the business make better decisions. This makes the business
grow bigger, and generate even more data, so that the cycle continues. This is repre-
sented by the cycle on the top-right of Figure 1.1. This aspect is discussed in Chapter
10, ‘A Primer on Data Analytics’.

On the other level, Big Data is different from traditional data in every way, i.e. space,
time, and function. Quantitatively, Big Data is 1000 times more than the traditional
data and of the data generation and transmission speed is also of the same order.
The forms and functions of Big Data are 10 times more diverse: from numbers to
text, pictures, audio, videos, web logs, machine data, and more. There are many more
sources of data, from individuals to organizations to governments, using a range of
devices from mobile phones to computers to industrial machines. Not all Big Data
will be of equal quality and value, as represented by the cycle on the bottom left of
Figure 1.1. This aspect of Big Data, and its new technologies, is the focus of this book.

Big Data is mostly, over 90 per cent, unstructured data. Every type of Big Data is
structured differently, and should be dealt in with an appropriate way. There are
huge opportunities for technology providers to innovate and manage the entire life
cycle of Big Data; to generate, gather, store, organize, analyze, and visualize this data.
 Wholeness of Big Data 3

Caselet
IBM Watson: A Big Data System

IBM created the Watson system as a way of pushing the boundaries of Artificial Intelligence and natural
language understanding technologies. Watson beat the world champion human players of Jeopardy
(quiz style TV show) in Feb 2011. Watson reads up data about everything on the web including the
entire Wikipedia. It digests and absorbs the data based on simple generic rules such as: books have
authors; stories have heroes; and drugs treat ailments. A Jeopardy clue, received in the form of a cryptic
phrase, is broken down into many possible potential sub-clues of the correct answer. Each sub-clue is
examined to see the likeliness of its answer being the correct answer for the main problem. Watson
calculates the confidence level of each possible answer. If the confidence level reaches more than a
desirable threshold level, Watson decides to offer the answer to the clue. It manages to do all this in
a mere 3 seconds.
Watson is now being applied to many important applications such as diagnosing diseases, especially
cancer. Watson can read all the new research published in the medical journals to update its knowl-
edge base. It can diagnose the probability of various diseases, by applying factors such as patient’s
current symptoms, health history, genetic history, medication records, and other factors to recommend
a particular diagnosis. (Source: Smartest machines on Earth: youtube.com/watch?v=TCOhyaw5bwg)

FIGURE 1.2 IBM Watson playing jeopardy

1. What kinds of Big Data knowledge, technologies and skills are required to build a system like
Watson? What other resources are needed?
2. Will doctors be able to compete with Watson in diagnosing diseases and prescribing medications?
Who else could benefit from a system like Watson?
4 Big Data

1.2 CAPTURING BIG DATA

If data were only growing too large, or only moving too fast, or only becoming too di-
verse, it would have been relatively easy. However, when three Vs (Volume, Velocity,
Variety) arrive together in an interactive manner, it creates a perfect storm. While
the Volume and Velocity of data drive the major technological concerns and the costs
of managing Big Data, these two Vs are themselves being driven by the third V, that
is, the Variety of forms and functions and sources of data. The varying veracity and
value of data complicate the situation further.

1.2.1 Volume of Data

The quantity of data generated in the world has been relentlessly doubling every 12–18
months. Traditional data is measured in Gigabytes (GB) and Terabytes (TB), but Big
Data is measured in Petabytes (PB) and Exabytes (1 Exabyte = 1 Million TB). This
data is so huge that it is almost a miracle that one can find any specific thing in it,
in a reasonable period of time. Searching the world-wide web was the first true Big
Data application. Google perfected the art of this application and developed many of
the path-breaking Big Data technologies we see in use today.

The primary reason for the growth of data is the dramatic reduction in the cost of
storing data. The costs of storing data have decreased by 30–40 per cent every year.
Therefore, there is an incentive to record everything that can be observed. It is called
‘datafication’ of the world. The costs of computation and communication of data have
also been coming down. Another reason for the growth of data is the increase in the
number of forms and functions of data. We will discuss more about this in the ‘Va-
riety’ section.

1.2.2 Velocity of Data

If traditional data is like a lake, Big Data is like a fast-flowing river. Big Data is being
generated by billions of devices, and communicated at the speed of the light, through
the internet. Ingesting all this data is like drinking from a fire hose. One does not
have any control over how fast the data will come. A huge unpredictable data-stream
is the new metaphor for thinking about Big Data.

The primary reason for the increased velocity of data is the increase in internet speed.
Internet speeds available to homes and offices are now increasing from 10 MB/s to 1
GB/s (100 times faster). More people are getting access to high-speed internet around
the world. Another important reason is the increased variety of sources, such as mo-
bile devices, that can generate and communicate data from anywhere, at any time.
Another random document with
no related content on Scribd:
The Project Gutenberg eBook of The
American scene
This ebook is for the use of anyone anywhere in the United
States and most other parts of the world at no cost and with
almost no restrictions whatsoever. You may copy it, give it away
or re-use it under the terms of the Project Gutenberg License
included with this ebook or online at www.gutenberg.org. If you
are not located in the United States, you will have to check the
laws of the country where you are located before using this
eBook.

Title: The American scene

Author: Henry James

Release date: August 8, 2022 [eBook #68717]

Language: English

Original publication: United Kingdom: Chapman and Hall, 1907

Credits: Richard Tonsing and the Online Distributed

Proofreading Team at https://www.pgdp.net (This file
was produced from images generously made available
by The Internet Archive)

*** START OF THE PROJECT GUTENBERG EBOOK THE

AMERICAN SCENE ***
Transcriber’s Note:
The cover image was created by the transcriber
and is placed in the public domain.
THE AMERICAN SCENE

BY

HENRY JAMES

LONDON

CHAPMAN AND HALL, Ltd

1907
 Richard Clay & Sons, Limited,
BREAD STREET HILL, E.C., AND
BUNGAY, SUFFOLK.
 PREFACE

The following pages duly explain themselves, I judge, as to the

Author’s point of view and his relation to his subject; but I prefix this
word on the chance of any suspected or perceived failure of such
references. My visit to America had been the first possible to me for
nearly a quarter of a century, and I had before my last previous one,
brief and distant to memory, spent other years in continuous
absence; so that I was to return with much of the freshness of eye,
outward and inward, which, with the further contribution of a state
of desire, is commonly held a precious agent of perception. I felt no
doubt, I confess, of my great advantage on that score; since if I had
had time to become almost as “fresh” as an inquiring stranger, I had
not on the other hand had enough to cease to be, or at least to feel, as
acute as an initiated native. I made no scruple of my conviction that I
should understand and should care better and more than the most
earnest of visitors, and yet that I should vibrate with more curiosity—
on the extent of ground, that is, on which I might aspire to intimate
intelligence at all—than the pilgrim with the longest list of questions,
the sharpest appetite for explanations and the largest exposure to
mistakes.
I felt myself then, all serenely, not exposed to grave mistakes—
though there were also doubtless explanations which would find me,
and quite as contentedly, impenetrable. I would take my stand on my
gathered impressions, since it was all for them, for them only, that I
returned; I would in fact go to the stake for them—which is a sign of
the value that I both in particular and in general attach to them and
that I have endeavoured to preserve for them in this transcription.
My cultivated sense of aspects and prospects affected me absolutely
as an enrichment of my subject, and I was prepared to abide by the
law of that sense—the appearance that it would react promptly in
some presences only to remain imperturbably inert in others. There
would be a thousand matters—matters already the theme of
prodigious reports and statistics—as to which I should have no sense
whatever, and as to information about which my record would
accordingly stand naked and unashamed. It should unfailingly be
proved against me that my opportunity had found me incapable of
information, incapable alike of receiving and of imparting it; for
then, and then only, would it be clearly enough attested that I had
cared and understood.
There are features of the human scene, there are properties of the
social air, that the newspapers, reports, surveys and blue-books
would seem to confess themselves powerless to “handle,” and that
yet represented to me a greater array of items, a heavier expression
of character, than my own pair of scales would ever weigh, keep them
as clear for it as I might. I became aware soon enough, on the spot,
that these elements of the human subject, the results of these
attempted appreciations of life itself, would prove much too
numerous even for a capacity all given to them for some ten months;
but at least therefore, artistically concerned as I had been all my days
with the human subject, with the appreciation of life itself, and with
the consequent question of literary representation, I should not find
such matters scant or simple. I was not in fact to do so, and they but
led me on and on. How far this might have been my several chapters
show; and yet even here I fall short. I shall have to take a few others
for the rest of my story.

H. J.
CONTENTS
 PAGE
I. NEW ENGLAND: AN AUTUMN IMPRESSION 1

II. NEW YORK REVISITED 72

III. NEW YORK AND THE HUDSON: A SPRING IMPRESSION 116

IV. NEW YORK: SOCIAL NOTES 158

V. THE BOWERY AND THEREABOUTS 194

VI. THE SENSE OF NEWPORT 209

VII. BOSTON 226

VIII. CONCORD AND SALEM 256

IX. PHILADELPHIA 273

X. BALTIMORE 303

XI. WASHINGTON 332

XII. RICHMOND 365

XIII. CHARLESTON 395

XIV. FLORIDA 422

 THE AMERICAN SCENE

I
NEW ENGLAND
AN AUTUMN IMPRESSION

I
Conscious that the impressions of the very first hours have always
the value of their intensity, I shrink from wasting those that attended
my arrival, my return after long years, even though they be out of
order with the others that were promptly to follow and that I here
gather in, as best I may, under a single head. They referred partly,
these instant vibrations, to a past recalled from very far back; fell
into a train of association that receded, for its beginning, to the
dimness of extreme youth. One’s extremest youth had been full of
New York, and one was absurdly finding it again, meeting it at every
turn, in sights, sounds, smells, even in the chaos of confusion and
change; a process under which, verily, recognition became more
interesting and more amusing in proportion as it became more
difficult, like the spelling-out of foreign sentences of which one
knows but half the words. It was not, indeed, at Hoboken, on
emerging from the comparatively assured order of the great berth of
the ship, that recognition was difficult: there, only too confoundingly
familiar and too serenely exempt from change, the waterside squalor
of the great city put forth again its most inimitable notes, showed so
true to the barbarisms it had not outlived that one could only fall to
wondering what obscure inward virtue had preserved it. There was
virtue evident enough in the crossing of the water, that brave sense
of the big, bright, breezy bay; of light and space and multitudinous
movement; of the serried, bristling city, held in the easy embrace of
its great good-natured rivers very much as a battered and
accommodating beauty may sometimes be “distinguished” by a
gallant less fastidious, with his open arms, than his type would seem
to imply. But what was it that was still holding together, for
observation, on the hither shore, the same old sordid facts, all the
ugly items that had seemed destined so long ago to fall apart from
their very cynicism?—the rude cavities, the loose cobbles, the
dislodged supports, the unreclaimed pools, of the roadway; the
unregulated traffic, as of innumerable desperate drays charging upon
each other with tragic long-necked, sharp-ribbed horses (a length
and a sharpness all emphasized by the anguish of effort); the
corpulent constables, with helmets askew, swinging their legs, in
high detachment, from coigns of contemplation; the huddled houses
of the other time, red-faced, off their balance, almost prone, as from
too conscious an affinity with “saloon” civilization.
It was, doubtless, open to the repentant absentee to feel these
things sweetened by some shy principle of picturesqueness; and I
admit that I asked myself, while I considered and bumped, why what
was “sauce for the goose” should not be in this case sauce for the
gander; and why antique shabbiness shouldn’t plead on this
particular waterside the cause it more or less successfully pleads on
so many others. The light of the September day was lovely, and the
sun of New York rests mostly, with a laziness all its own, on that dull
glaze of crimson paint, as thick as on the cheek of the cruder
coquetry, which is, in general, beneath its range, the sign of the old-
fashioned. Yes; I could remind myself, as I went, that Naples, that
Tangiers or Constantinople has probably nothing braver to flaunt,
and mingle with excited recognition the still finer throb of seeing in
advance, seeing even to alarm, many of the responsibilities lying in
wait for the habit of headlong critical or fanciful reaction, many of
the inconsistencies in which it would probably have, at the best,
more or less defiantly to drape itself. Such meditations, at all events,
bridged over alike the weak places of criticism and some of the
rougher ones of my material passage. Nothing was left, for the rest of
the episode, but a kind of fluidity of appreciation—a mild, warm
wave that broke over the succession of aspects and objects according
to some odd inward rhythm, and often, no doubt, with a violence
that there was little in the phenomena themselves flagrantly to
justify. It floated me, my wave, all that day and the next; so that I still
think tenderly—for the short backward view is already a distance
with “tone”—of the service it rendered me and of the various
perceptive penetrations, charming coves of still blue water, that
carried me up into the subject, so to speak, and enabled me to step
ashore. The subject was everywhere—that was the beauty, that the
advantage: it was thrilling, really, to find one’s self in presence of a
theme to which everything directly contributed, leaving no touch of
experience irrelevant. That, at any rate, so far as feeling it went;
treating it, evidently, was going to be a matter of prodigious difficulty
and selection—in consequence of which, indeed, there might even be
a certain recklessness in the largest surrender to impressions.
Clearly, however, these were not for the present—and such as they
were—to be kept at bay; the hour of reckoning, obviously, would
come, with more of them heaped up than would prove usable, a
greater quantity of vision, possibly, than might fit into decent form:
whereby, assuredly, the part of wisdom was to put in as much as
possible of one’s recklessness while it was fresh.
It was fairly droll, for instance, the quantity of vision that began to
press during a wayside rest in a house of genial but discriminating
hospitality that opened its doors just where the fiddle-string of
association could most intensely vibrate, just where the sense of “old
New York,” of the earlier stages of the picture now so violently
overpainted, found most of its occasions—found them, to
extravagance, within and without. The good easy Square, known in
childhood, and as if the light were yellower there from that small
accident, bristled with reminders as vague as they were sweet;
within, especially, the place was a cool backwater, for time as well as
for space; out of the slightly dim depths of which, at the turn of
staircases and from the walls of communicating rooms, portraits and
relics and records, faintly, quaintly æsthetic, in intention at least, and
discreetly—yet bravely, too, and all so archaically and pathetically—
Bohemian, laid traps, of a pleasantly primitive order, for memory, for
sentiment, for relenting irony; gross little devices, on the part of the
circumscribed past, which appealed with scarce more emphasis than
so many tail-pieces of closed chapters. The whole impression had
fairly a rococo tone; and it was in this perceptibly golden air, the air
of old empty New York afternoons of the waning summer-time, when
the long, the perpendicular rattle, as of buckets, forever thirsty, in
the bottomless well of fortune, almost dies out in the merciful cross-
streets, that the ample rearward loggia of the Club seemed serenely
to hang; the glazed, disglazed, gallery dedicated to the array of small
spread tables for which blank “backs,” right and left and opposite,
made a privacy; backs blank with the bold crimson of the New York
house-painter, and playing upon the chord of remembrance, all so
absurdly, with the scarcely less simplified green of their great
cascades of Virginia creeper, as yet unturned: an admonition, this,
for piety, as well as a reminder—since one had somehow failed to
treasure it up—that the rather pettifogging plan of the city, the fruit,
on the spot, of an artless age, happened to leave even so much
margin as that for consoling chances. There were plenty of these—
which I perhaps seem unduly to patronize in speaking of them as
only “consoling”—for many hours to come and while the easy wave
that I have mentioned continued to float me: so abysmal are the
resources of the foredoomed student of manners, or so helpless, at
least, his case when once adrift in that tide.
If in Gramercy Park already, three hours after his arrival, he had
felt himself, this victim, up to his neck in what I have called his
“subject,” the matter was quite beyond calculation by the time he had
tumbled, in such a glorified “four-wheeler,” and with such an odd
consciousness of roughness superimposed upon smoothness, far
down-town again, and, on the deck of a shining steamer bound for
the Jersey shore, was taking all the breeze of the Bay. The note of
manners, the note that begins to sound, everywhere, for the spirit
newly disembarked, with the first word exchanged, seemed, on the
great clean deck, fairly to vociferate in the breeze—and not at all, so
far, as was pleasant to remark, to the harshening of that element.
Nothing could have been more to the spectators purpose, moreover,
than the fact he was ready to hail as the most characteristic in the
world, the fact that what surrounded him was a rare collection of
young men of business returning, as the phrase is, and in the pride of
their youth and their might, to their “homes,” and that, if treasures of
“type” were not here to be disengaged, the fault would be all his own.
It was perhaps this simple sense of treasure to be gathered in, it was
doubtless this very confidence in the objective reality of impressions,
so that they could deliciously be left to ripen, like golden apples, on
the tree—it was all this that gave a charm to one’s sitting in the
orchard, gave a strange and inordinate charm both to the prospect of
the Jersey shore and to every inch of the entertainment, so divinely
inexpensive, by the way. The immense liberality of the Bay, the noble
amplitude of the boat, the great unlocked and tumbled-out city on
one hand, and the low, accessible mystery of the opposite State on
the other, watching any approach, to all appearance, with so gentle
and patient an eye; the gaiety of the light, the gladness of the air,
and, above all (for it most came back to that), the unconscious
affluence, the variety in identity, of the young men of business: these
things somehow left speculation, left curiosity exciting, yet kept it
beguilingly safe. And what shall I say more of all that presently
followed than that it sharpened to the last pleasantness—quite
draining it of fears of fatuity—that consciousness of strolling in the
orchard that was all one’s own to pluck, and counting, overhead, the
apples of gold? I figure, I repeat, under this name those thick-
growing items of the characteristic that were surely going to drop
into one’s hand, for vivid illustration, as soon as one could begin to
hold it out.
Heavy with fruit, in particular, was the whole spreading bough that
rustled above me during an afternoon, a very wonderful afternoon,
that I spent in being ever so wisely driven, driven further and
further, into the large lucidity of—well, of what else shall I call it but
a New Jersey condition? That, no doubt, is a loose label for the
picture; but impressions had to range themselves, for the hour, as
they could. I had come forth for a view of such parts of the condition
as might peep out at the hour and on the spot, and it was clearly not
going to be the restless analyst’s own fault if conditions in general,
everywhere, should strike him as peculiarly, as almost affectingly, at
the mercy of observation. They came out to meet us, in their
actuality, in the soft afternoon; they stood, artless, unconscious,
unshamed, at the very gates of Appearance; they might, verily, have
been there, in their plenitude, at the call of some procession of drums
and banners—the principal facts of the case being collected along our
passage, to my fancy, quite as if they had been principal citizens. And
then there was the further fact of the case, one’s own ridiculous
property and sign—the romantic, if not the pathetic, circumstance of
one’s having had to wait till now to read even such meagre meanings
as this into a page at which one’s geography might so easily have
opened. It might have threatened, for twenty minutes, to be almost
complicating, but the truth was recorded: it was an adventure,
unmistakably, to have a revelation made so convenient—to be
learning at last, in the maturity of one’s powers, what New Jersey
might “connote.” This was nearer than I had ever come to any such
experience; and it was now as if, all my life, my curiosity had been
greater than I knew. Such, for an excited sensibility, are the
refinements of personal contact. These influences then were present,
as a source of glamour, at every turn of our drive, and especially
present, I imagined, during that longest perspective when the road
took no turn, but showed us, with a large, calm consistency, the
straight blue band of summer sea, between the sandy shore and the
reclaimed margin of which the chain of big villas was stretched tight,
or at least kept straight, almost as for the close stringing of more or
less monstrous pearls. The association of the monstrous thrusts itself
somehow into my retrospect, for all the decent humility of the low,
quiet coast, where the shadows of the waning afternoon could
lengthen at their will and the chariots of Israel, on the wide and
admirable road, could advance, in the glittering eye of each array of
extraordinarily exposed windows, as through an harmonious golden
haze.
There was gold-dust in the air, no doubt—which would have been
again an element of glamour if it had not rather lighted the scene
with too crude a confidence. It was one of the phases, full of its own
marks and signs, of New York, the immense, in villeggiatura—and,
presently, with little room left for doubt of what particular phase it
might be. The huge new houses, up and down, looked over their
smart, short lawns as with a certain familiar prominence in their
profiles, which was borne out by the accent, loud, assertive, yet
benevolent withal, with which they confessed to their extreme
expensiveness. “Oh, yes; we were awfully dear, for what we are and
for what we do”—it was proud, but it was rather rueful; with the odd
appearance everywhere as of florid creations waiting, a little
bewilderingly, for their justification, waiting for the next clause in the
sequence, waiting in short for life, for time, for interest, for
character, for identity itself to come to them, quite as large spread
tables or superfluous shops may wait for guests and customers. The
scene overflowed with curious suggestion; it comes back to me with
the afternoon air and the amiable flatness, the note of the sea in a
drowsy mood; and I thus somehow think of the great white boxes as
standing there with the silvered ghostliness (for all the silver
involved) of a series of candid new moons. It could only be the
occupants, moreover, who were driving on the vast, featureless
highway, to and fro in front of their ingenuous palaces and as if
pretending not to recognize them when they passed; German Jewry
—wasn’t it conceivable?—tending to the stout, the simple, the kind,
quite visibly to the patriarchal, and with the old superseded
shabbiness of Long Branch partly for the goal of their course; the big
brown wooden barracks of the hotels, the bold rotunda of the
gaming-room—monuments already these, in truth, of a more artless
age, and yet with too little history about them for dignity of ruin.
Dignity, if not of ruin at least of reverence, was what, at other points,
doubtless, we failed considerably less to read into the cottage where
Grant lived and the cottage where Garfield died; though they had, for
all the world, those modest structures, exactly the effect of objects
diminished by recession into space—as if to symbolize the rapidity of
their recession into time. They have been left so far behind by the
expensive, as the expensive is now practised; in spite of having
apparently been originally a sufficient expression of it.
This could pass, it seemed, for the greatest vividness of the picture
—that the expensive, for New York in villeggiatura, even on such
subordinate showing, is like a train covering ground at maximum
speed and pushing on, at present, into regions unmeasurable. It
included, however, other lights, some of which glimmered, to my
eyes, as with the promise of great future intensity—hanging
themselves as directly over the question of manners as if they had
been a row of lustres reflected in the polished floor of a ball-room.
Here was the expensive as a power by itself, a power unguided,
undirected, practically unapplied, really exerting itself in a void that
could make it no response, that had nothing—poor gentle, patient,
rueful, but altogether helpless, void!—to offer in return. The game
was that of its doing, each party to the whole combination, what it
could, but with the result of the common effort’s falling so short.
Nothing could be of a livelier interest—with the question of manners
always in view—than to note that the most as yet accomplished at
such a cost was the air of unmitigated publicity, publicity as a
condition, as a doom, from which there could be no appeal; just as in
all the topsy-turvy order, the defeated scheme, the misplaced
confidence, or whatever one may call it, there was no achieved
protection, no constituted mystery of retreat, no saving complexity,
not so much as might be represented by a foot of garden wall or a
preliminary sketch of interposing shade. The homely principle under
which the picture held at all together was that of the famous freedom
of the cat to look at the king; that seemed, so clearly, throughout, the
only motto that would work. The ample villas, in their full dress,
planted each on its little square of brightly-green carpet, and as with
their stiff skirts pulled well down, eyed each other, at short range,
from head to foot; while the open road, the chariots, the buggies, the
motors, the pedestrians—which last number, indeed, was remarkably
small—regarded at their ease both this reciprocity and the parties to
it. It was in fact all one participation, with an effect deterrent to those
ingenuities, or perhaps indeed rather to those commonplaces, of
conjecture produced in general by the outward show of the fortunate
life. That, precisely, appeared the answer to the question of manners:
the fact that in such conditions there couldn’t be any manners to
speak of; that the basis of privacy was somehow wanting for them;
and that nothing, accordingly, no image, no presumption of
constituted relations, possibilities, amenities, in the social, the
domestic order, was inwardly projected. It was as if the projection
had been so completely outward that one could but find one’s self
almost uneasy about the mere perspective required for the common
acts of the personal life, that minimum of vagueness as to what takes
place in it for which the complete “home” aspires to provide.
What had it been their idea to do, the good people—do, exactly, for
their manners, their habits, their intercourse, their relations, their
pleasures, their general advantage and justification? Do, that is, in
affirming their wealth with such innocent emphasis and yet not at
the same time affirming anything else. It would have rested on the
cold-blooded critic, doubtless, to explain why the crudity of wealth
did strike him with so direct a force; accompanied after all with no
paraphernalia, no visible redundancies of possession, not so much as
a lodge at any gate, nothing but the scale of many of the houses and
their candid look of having cost as much as they knew how.
Unmistakably they all proclaimed it—they would have cost still more
had the way but been shown them; and, meanwhile, they added as
with one voice, they would take a fresh start as soon as ever it should
be. “We are only instalments, symbols, stop-gaps,” they practically
admitted, and with no shade of embarrassment; “expensive as we
are, we have nothing to do with continuity, responsibility,
transmission, and don’t in the least care what becomes of us after we
have served our present purpose.” On the detail of this impression,
however, I needn’t insist; the essence of it, which was all that was
worth catching, was one’s recognition of the odd treachery that may
practically lie in wait for isolated opulence. The highest luxury of all,
the supremely expensive thing, is constituted privacy—and yet it was
the supremely expensive thing that the good people had supposed
themselves to be getting: all of which, I repeat, enriched the case, for
the restless analyst, with an illustrative importance. For what did it
offer but the sharp interest of the match everywhere and
everlastingly played between the short-cut and the long road?—an
interest never so sharp as since the short-cut has been able to find
itself so endlessly backed by money. Money in fact is the short-cut—
or the short-cut money; and the long road having, in the instance
before me, so little operated, operated for the effect, as we may say,
of the cumulative, the game remained all in the hands of its
adversary.
The example went straight to the point, and thus was the drama
presented: what turn, on the larger, the general stage, was the game
going to take? The whole spectacle, with the question, opened out,
diffusing positively a multitudinous murmur that was in my ears, for
some of the more subtly-romantic parts of the drive, as who should
say (the sweet American vaguenesses, hailed again, the dear old
nameless, promiscuous lengths of woodside and waterside), like the
collective afternoon hum of invisible insects. Yes; it was all actually
going to be drama, and that drama; than which nothing could be
more to the occult purpose of the confirmed, the systematic story-
seeker, or to that even of the mere ancient contemplative person
curious of character. The very donnée of the piece could be given, the
subject formulated: the great adventure of a society reaching out into
the apparent void for the amenities, the consummations, after having
earnestly gathered in so many of the preparations and necessities.
“Into the apparent void”—I had to insist on that, since without it