Demystifying the Azure

Well-Architected Framework: Guiding

Principles and Design Best Practices
for Azure Workloads 1st Edition
Shijimol Ambi Karthikeyan
Visit to download the full and correct content document:
https://ebookmeta.com/product/demystifying-the-azure-well-architected-framework-gu
iding-principles-and-design-best-practices-for-azure-workloads-1st-edition-shijimol-a
mbi-karthikeyan/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Design and Deploy a Secure Azure Environment: Mapping

the NIST Cybersecurity Framework to Azure Services 1st
Edition Puthiyavan Udayakumar

https://ebookmeta.com/product/design-and-deploy-a-secure-azure-
environment-mapping-the-nist-cybersecurity-framework-to-azure-
services-1st-edition-puthiyavan-udayakumar/

Microsoft Azure for Java Developers: Deploying Java

Applications through Azure WebApp, Azure Kubernetes
Service, Azure Functions, and Azure Spring Cloud 1st
Edition Abhishek Mishra
https://ebookmeta.com/product/microsoft-azure-for-java-
developers-deploying-java-applications-through-azure-webapp-
azure-kubernetes-service-azure-functions-and-azure-spring-
cloud-1st-edition-abhishek-mishra/

Azure Security For Critical Workloads: Implementing

Modern Security Controls for Authentication,
Authorization and Auditing 1st Edition Sagar Lad

https://ebookmeta.com/product/azure-security-for-critical-
workloads-implementing-modern-security-controls-for-
authentication-authorization-and-auditing-1st-edition-sagar-lad/

Azure SQL Hyperscale Revealed: High-performance

Scalable Solutions for Critical Data Workloads 1st
Edition Zoran Bara■

https://ebookmeta.com/product/azure-sql-hyperscale-revealed-high-
performance-scalable-solutions-for-critical-data-workloads-1st-
edition-zoran-barac/
Azure SQL Hyperscale Revealed: High-performance
Scalable Solutions for Critical Data Workloads 1st
Edition Zoran Bara■

https://ebookmeta.com/product/azure-sql-hyperscale-revealed-high-
performance-scalable-solutions-for-critical-data-workloads-1st-
edition-zoran-barac-2/

Modern Data Architecture on Azure: Design Data-centric

Solutions on Microsoft Azure 1st Edition Sagar Lad

https://ebookmeta.com/product/modern-data-architecture-on-azure-
design-data-centric-solutions-on-microsoft-azure-1st-edition-
sagar-lad/

Design and Deploy Azure VMware Solutions 1st Edition

Puthiyavan Udayakumar

https://ebookmeta.com/product/design-and-deploy-azure-vmware-
solutions-1st-edition-puthiyavan-udayakumar/

The Business of Human Rights: Best Practice and the UN

Guiding Principles 1st Edition Alex Newton

https://ebookmeta.com/product/the-business-of-human-rights-best-
practice-and-the-un-guiding-principles-1st-edition-alex-newton/

Implementing Azure Cloud Design Patterns Implement

efficient design patterns for data management high
availability monitoring and other popular patterns on
your Azure Cloud 1st Edition Oliver Michalski
https://ebookmeta.com/product/implementing-azure-cloud-design-
patterns-implement-efficient-design-patterns-for-data-management-
high-availability-monitoring-and-other-popular-patterns-on-your-
Shijimol Ambi Karthikeyan

Demystifying the Azure Well-

Architected Framework
Guiding Principles and Design Best Practices for
Azure Workloads
1st ed.
Shijimol Ambi Karthikeyan
Bangalore, Karnataka, India

Any source code or other supplementary material referenced by the

author in this book is available to readers on GitHub via the book’s
product page, located at www.apress.com/978-1-4842-7118-6. For
more detailed information, please visit www.apress.com/source-code.

ISBN 978-1-4842-7118-6 e-ISBN 978-1-4842-7119-3

https://doi.org/10.1007/978-1-4842-7119-3

© Shijimol Ambi Karthikeyan 2021

This work is subject to copyright. All rights are solely and exclusively
licensed by the Publisher, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in
any other physical way, and transmission or information storage and
retrieval, electronic adaptation, computer software, or by similar or
dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks,

service marks, etc. in this publication does not imply, even in the
absence of a specific statement, that such names are exempt from the
relevant protective laws and regulations and therefore free for general
use.

The publisher, the authors and the editors are safe to assume that the
advice and information in this book are believed to be true and accurate
at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the
material contained herein or for any errors or omissions that may have
been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Apress imprint is published by the registered company APress

Media, LLC part of Springer Nature.
The registered company address is: 1 New York Plaza, New York, NY
10004, U.S.A.
Dedicated to my dearest Amma and Achan, my guardian angels watching
over me from heaven.
Introduction
Cloud-first applications have become the norm rather than the
exception in today’s cloud-first, mobile-first world. Azure is leading the
pack among the cloud platforms in terms of portfolio of services, global
coverage, and availability of enterprise adoption frameworks. Azure
Well-Architected Framework provides best-practice guidance to
customers on how to design, build, operate, and manage the lifecycle of
applications in the cloud so as to derive the optimal outcome.
Azure Well-Architected Framework (WAF) is based on five pillars:
cost optimization, performance efficiency, operational excellence,
reliability, and security. This book will deep dive into these five pillars
of WAF and provide practical guidance on adopting them for your
Azure workloads. The book starts with an introduction chapter on WAF,
its relevance, and an overview of the five encompassing elements. The
subsequent chapters will deep dive into each of the WAF pillars in
detail. The book also covers some of the common cloud workload use
cases and the considerations while adopting WAF for the respective
architectures. The concepts are explained in a simple and concise
manner that can be comprehended by Azure cloud teams irrespective
of their level of expertise.
Acknowledgments
First, I would like to thank my parents for everything I have ever
accomplished in my life, including this book. My mother, Ambi R.,
always inspired me to work toward my goals no matter how unrealistic
others perceived them to be. My father, Karthikeyan M., taught me that
it is equally important to slow down at times and take in life as it is.
They are no longer around, but their love and blessings keep me going.
The last year has been unprecedented and taught us new ways of
living our lives during a pandemic. I cannot thank enough my support
system, my family and friends, who helped me brave this storm and
inspired me to take up the endeavor of writing this book amid multiple
challenges. Special shout-out to my daughter, Sanjana Sujai, for
inspiring me to be the best version of myself every day. I am thankful to
the mentors in my professional life (there are too many to name) for
their constant support and encouragement. Last but not least, I would
like to thank the entire team at Apress for their support during the
publishing process.
Table of Contents
Chapter 1:​Azure Well-Architected Framework:​What and Why?​
Digital Transformation Through Azure
Why Do You Need a Framework for Architecture Excellence?​
What Is the Azure Well-Architected Framework?​
Practical Annotations
Scenario 1
Scenario 2
Scenario 3
Scenario 4
Key Takeaways
Chapter 2:​Cost Optimization:​Return on Your Cloud Investment
How to Design for Optimal ROI
Design Principles
Design Process
Tools and Services for Cost Optimization
Understand the Trade-Offs
Configure Reports for Visibility
Key Takeaways
Chapter 3:​Operational Excellence:​Keep the Lights On
How to Design for Operational Excellence
Design Principles
Start with the Application Design
Adopt “Everything as Code”
Enable Peak Performance for the Deployment Ecosystem
Always “Shift Left” for Testing
 Integrated Monitoring
Use Case Analysis
Scenario 3
Chapter 4:​Performance Efficiency:​Meet the Demand Spikes
Design Principles for Performance Efficiency
Design Process
Make Best Use of Cloud Scalability
Identify Bottlenecks Through Performance Testing
Monitoring Metrics for Performance
Use Case Analysis
Scenario 2
Key Takeaways
Chapter 5:​Reliability:​Build Resilient Applications in the Cloud
Design Principles for Resiliency
Popular Adoption Strategies to Meet the Defined SLAs
Considerations While Defining Application SLAs
Conduct Failure Mode Analysis
Adoption Strategies
Testing and Monitoring
Test Strategies
Monitoring and Optimization
Use Case Analysis
Scenario 4
Key Takeaways
Chapter 6:​Security:​Protect Your Workloads in the Cloud
Understanding Threat Vectors in the Cloud
 Adaptable Security for New-World Threats
Design Principles
Secure Infrastructure, Data, Network, and Application
Identity Is the New Security Perimeter
Azure Native Security Tools for Your SOC Team
Use Case Analysis
Scenario 1
Key Takeaways
Index
About the Author
Shijimol Ambi Karthikeyan
has more than 15 years of experience in
data center management, server
administration, virtualization, and cloud
technologies. She is currently working as
a technical delivery manager focusing on
Azure infrastructure, automation,
DevOps, serverless, and related
technologies. She is a tech enthusiast and
loves writing about the latest
developments in IT infrastructure and
cloud computing on her blog at
https://thetechnologychronicl
e.blogspot.com/ .
About the Technical Reviewer
Vikas Sukhija
has more than a decade of IT
infrastructure experience with expertise
in messaging, collaboration, and IT
automations. He is a blogger, architect,
and Microsoft MVP, and he is known by
the alias TechWizard. As an experienced
professional, he assists small to large
enterprises in architecting and
implementing Office 365 and Azure.
Community contributions from him
can be found at the following locations:
Blog: TechWizard.cloud
Facebook:
https://www.facebook.com/TechWizard.cloud
Twitter: https://twitter.com/techwizardcloud
Coderepo: https://github.com/VikasSukhija
© The Author(s), under exclusive license to APress Media, LLC, part of Springer
Nature 2021
S. Ambi Karthikeyan, Demystifying the Azure Well-Architected Framework
https://doi.org/10.1007/978-1-4842-7119-3_1

1. Azure Well-Architected Framework:

What and Why?
Shijimol Ambi Karthikeyan1
(1) Bangalore, Karnataka, India

With the digital transformation of the modern world, the cloud real
estate of organizations has expanded beyond expectation. Be it big data
services, analytics, machine learning, or AI, everything has found a
home in the cloud. With this transformation, it has become a challenge
for architects to choose the best-suited architectures for each of these
workloads. There are many aspects to be considered, starting from
designing and deploying the applications to keeping the lights on at an
optimal cost. Azure, being one of the leading cloud service providers,
offers the Azure Well-Architected Framework to support organizations.
The Azure Well-Architected Framework provides definitive
guidelines for deploying your workloads in Azure that are aligned with
Microsoft-recommended best practices. The framework leverages five
architecture elements: cost optimization, performance efficiency,
operational excellence, reliability, and security. This book will deep dive
into these five elements and provide practical guidance on
incorporating them into your architecture. In this first chapter, we will
try to understand the relevance of the Well-Architected Framework and
why it should form the baseline of all your design decisions while
deploying applications in Azure.

Digital Transformation Through Azure

Digital transformation in simple terms means leveraging the latest
technologies to revamp how you run your business. The plethora of
services available in the cloud has revolutionized this process. Azure is
a clear favorite among organizations because it is a trusted partner for
digital transformation, irrespective of whether they are Fortune 500
companies or small and medium-sized businesses. Some of the key
drivers for the same are listed here:
– With a global span of 60+ regions and 6,000+ services (at the time of
writing this book), Azure offers organizations the flexibility to adopt
the cloud the way they want to.
– Azure has clearly defined hybrid cloud adoption methodologies,
which allows organizations to start small and test the waters before
taking the plunge.
– Organizations with existing investments in Windows licenses and
technologies prefer Azure as it is an easy extension of their native
processes.
– Azure provides comprehensive coverage for open source
technologies: Linux, Kubernetes, Docker, Python, Node.js, HDInsight,
TensorFlow, etc., to name a few. This enables innovation and
flexibility for cloud adoption through open source frameworks.
– Microsoft has a strong partner ecosystem to onboard your workloads
to Azure and support them. Organizations can use the managed
service model from Microsoft Cloud Solution Provider (CSP) partners
to embark on their digital transformation journey.
– It has an unwavering focus on security, with a dedicated Microsoft
Trust Center to guide you on all your security, privacy, and
compliance needs.
– It has a robust portfolio of services , with the latest and greatest on
AI, data analytics, databases, IOT, serverless, or quantum computing,
to name a few. Irrespective of industry vector and business
outcomes, there are multiple options to choose from for your digital
transformation.

Why Do You Need a Framework for Architecture

Excellence?
The reason for a digital transformation is often anchored on a business
need; it could be anything from improving the customer experience to
expanding the existing service portfolio to a demand for speed and
agility to be competitive in the market. A successful digital
transformation goes through multiple phases. It starts with mapping
out the business requirements, envisioning the end state, and
identifying the right platform and technologies, followed by planning
and executing the activities.
The Microsoft Cloud Adoption Framework can be leveraged here to
arrive at the right strategies and technologies to be used. Though not
the focus of this book, the Cloud Adoption Framework complements the
Azure Well-Architected Framework. A high-level synopsis of the Cloud
Adoption Framework is listed here for context:
1. Strategy: In this initial phase, the focus will be on identifying the
business outcomes and understanding the motivation for cloud
adoption. What are the pain points that you are trying to address
through cloud adoption? Which project do you want to prioritize
and start with?

2. Plan: Once the strategy is finalized, the next step is to create an

action plan for adoption. This requires the business stakeholder
buy-in, aligning the right people for implementing the adoption
plan and enabling them through a readiness plan.

3. Ready: This is where you start your cloud journey by creating a

landing zone. Note that there will be considerations from the Well-
Architected Framework that contributes to this process. The
landing zone should be aligned with best practices and ready by the
end of this phase to host your initial workloads.

4. Adopt: During the Adopt phase, you either migrate or modernize

your workloads for the cloud or innovate and develop new
solutions to deliver the desired business outcomes. Both
approaches require adherences to best-practice guidance and
integrating the right pillars from the Well-Architected Framework.
5. Govern: This phase involves establishing an initial governance
framework for your Azure cloud workloads and improving it
iteratively.

6. Manage: Define the parameters required for ongoing operations,

focusing on business-critical metrics, analyzing workload telemetry,
ensuring ongoing compliance, and ensuring business continuity.

While the Cloud Adoption Framework helps to accelerate your

digital transformation, the Well-Architected Framework provides
prescriptive guidance at a workload level to develop a robust
application architecture in the cloud.
Often while focused on the outcome, organizations tend to overlook
the nitty-gritty details of having a robust architecture framework to
start with. The need for having the right architecture for your
applications in the cloud cannot be emphasized enough. It forms the
foundation of your digital transformation efforts. The nuances of the
architecture vary depending on whether you want to leverage cloud-
native services focusing mostly on PaaS/serverless or rather do a lift-
and-shift of an existing architecture to the cloud by leveraging IaaS.
Irrespective of the approach you choose, a framework for architecture
excellence will act as a guiding force to deploy your workloads right the
first time.
There are some questions that every architect should ponder while
you design and build an application for Azure, and the Well-Architected
Framework helps you get the answers right. Let’s look at some of the
considerations of application architecture that outline the need for such
a framework.
– Your application might have an ever-fluctuating performance
demand, but the cloud can accommodate all of that if designed right.
For example, it is important to decide whether to scale up or scale
out. What approach fits best for your workload?
– Continuous monitoring and optimization are also required to
maintain the performance levels as the application evolves. You
might have started out with one compute instance, but would that be
enough to sustain performance as your customer base increases?
– Ensuring operational excellence starts at the application design
phase and is not an afterthought. It is closely related to how well
DevOps principles are integrated into your application lifecycle.
However, are DevOps and CI/CD practices applicable only for
application code deployment? How about infrastructure?
– Applications should be designed to be resilient and highly available
so as to keep the lights on and the experience consistent for your
customers. How robust is your disaster recovery plan? Have you
considered the RPO and RTO requirements of your business?
– The security of cloud applications is a work stream in itself, which
requires a detailed assessment of possible attack vectors and
implementing preventive measures. Have you considered the
security of control plane along with the data plane?
– Last but not the least is focus on cost optimization to ensure optimal
return on your cloud investments. Though the cloud follows a pay-
as-you-go model, it is important to plan for the right sizing and
scalability of cloud resources so that you eventually pay only for
what you use.

What Is the Azure Well-Architected Framework?

When your workloads are in the cloud, the constructs of deployment,
configuration, and operations are strikingly different from what you
would have used on-premises. Adopting the right architecture, without
doubt, is the key to successfully hosting an application in the cloud.
Azure helps you with this every step of the process through the Azure
Well-Architected Framework. Consider this as a blueprint for excellence
in the Azure cloud. It consists of five main pillars: cost optimization,
operational excellence, performance efficiency, reliability, and security.
Here are the concepts covered by the five pillars of the Azure Well-
Architected Framework in brief:
Cost optimization: The basic principle is to start small and scale as
you go. Instead of making a huge investment up front, it is
recommended to follow the approach of Build-Measure-Learn, which
is aligned with the Azure Cloud Adoption Framework (CAF). It
focuses on building a minimum viable product (MVP), measuring the
feedback, and then using a “fail fast” approach to optimize your cost.
The Azure Cost Calculator can help you to get the initial cost, and
then you can use services like Azure Cost Management to review the
ongoing operational cost.
Operational excellence : The success of a cloud deployment
depends on how well-oiled your operations engine is. Starting with
the automation of deployment to monitoring, logging, and
diagnostics, the more granular your visibility is, the better placed you
are to keep the lights on for your production environment. The
monitoring and logging approach has to be consistent across cloud
resources to achieve this goal. The raw monitoring data stored in a
central storage can be leveraged by tools like log analytics to get to
the root of operational issues, enabling faster resolution.
Visualization tools can be leveraged to spot trends such as resource
utilization and unusual traffic and alert the operations team.
Performance efficiency : End users expect a solution to perform
well irrespective of the traffic being handled by it. One of the main
perks of cloud adoption is scaling on demand, which can be vertical
scaling or horizontal scaling. Vertical scaling helps you to increase
the compute power and capacity of your resources on demand, such
as increasing the number of CPU cores when your workloads are in
peak demand. Horizontal scaling, on the other hand, is adding more
instances of the resources, automatically if possible. Horizontal
scaling embodies the true power of cloud-scale deployments and is
often cheaper than increasing capacity of a single instance.
Additional considerations for performance efficiency include
network, storage, and database performance.
Reliability : No matter how airtight the architecture is, the
possibility of downtime or a failure is not 100 percent eliminated.
Hence, it is important to design your systems to be reliable; in other
words, they should be able to recover from failures with minimal
damage. Reliability is a combined function of resiliency and
availability. Because of the distributed nature of deployments in the
cloud, the failure of one component would impact multiple other
components. The rule of thumb is to leverage the built-in resiliency
features for the native cloud services, be it your VMs, databases, or
storage services. Not just your infrastructure layer, but your
 application logic should also be built on this principle for the solution
to be completely reliable.
Security: Security in the cloud is multilayered. You need to
consider the infrastructure and network security, application layer
security, and data security at rest and in transit. As identity is
considered the new security perimeter, selecting and implementing
the right identity management solution is the first step in securing
your applications. Security of the management plane and data pane
should be taken into account here. RBAC leveraging Azure AD takes
care of the management plane, by helping you implement fine-
grained access control to Azure resources. For a data plane, there are
multiple options depending on the Azure service being used, such as
data encryption, TDE, etc. In addition to application development
security best practices, you can leverage services like application
gateways that can provide layer 7 security from common attack
vectors.
Figure 1-1 summarizes the focus area of each of the five pillars.

Figure 1-1 Each pillar will be discussed in detail in subsequent chapters along with
practical scenarios and use cases

Practical Annotations
The success of any framework depends on how easy or how difficult it
is to adapt it to real-world scenarios. Given in this section are a few
such use cases. We will explore the fit of the Well-Architected
Framework for these use cases in upcoming chapters.

Scenario 1
Industry: Health and Life Sciences
With healthcare going digital, there are different options available for
consumers such as fitness bands, online consultations, medication
tracking, etc. A health and life sciences company can leverage the Well-
Architected Framework while developing a cloud-based digital
experience for its customers. Here are some key considerations for the
scenario:
– There is lot of personally identifiable information (PII) involved, and
hence data privacy and security are two of the primary concerns.
– The services should be accessible to customers when they need it
and where they need it.
– The response time of applications should ensure a smooth customer
experience.
– Adhere to HIPAA controls.

Scenario 2
Industry: Retail
Ecommerce has become a key enabler for the retail industry. It is a
highly competitive arena, so customers are spoiled by all the choices.
Ecommerce websites should be designed for optimal customer
experience to ensure repeatable business. The Azure Well-Architected
Framework can help by identifying the right components for hosting,
scaling, and securing ecommerce solutions and also to ensure long-
term sustainability. Here are some key considerations for the scenario:
– Regular updates of catalogs to cover new product releases
– Secure payment transactions and storage personal information
– Quick retrieval of specific product information and possible
alternatives
– Search experience personalized to customer preferences and
previous purchases
– Capability to handle peak hour traffic such as Black Friday sales

Scenario 3
Industry: Any
Azure offers multiple managed container solutions, the most popular
one being Azure Kubernetes Service (AKS). The service has accelerated
the adoption of microservices-based architectures in the cloud. Be it a
lift-and-shift of existing containers from on-premises or a
modernization of existing applications for the cloud, AKS is leading the
way, and this approach is popular across almost all industry vectors.
Here are some key considerations while integrating the Well-
Architected Framework for microservices-based architectures:
– Persistent data storage and availability
– Scalability consideration (pods in the deployment versus nodes in
the cluster)
– Ongoing operational hygiene of the deployment
– Resilience and recovery from failures
– Health telemetry, analytics, and continuous optimization
– Secure access (ingress, communication between microservices,
access to application keys, etc.)

Scenario 4
Industry: Financial
Most financial-sector organizations such as banks and insurance
companies are faced with challenges of adapting their legacy systems to
cloud. There could be some hard dependencies, but most of them can
be circumvented by leveraging the right pillars of the Well-Architected
Framework. Here are some key considerations of using the Well-
Architected Framework while migrating financial-sector line-of-
business (LOB) applications to Azure:
– Identity integration for seamless experience, for both customers and
employees
– Resiliency of workloads migrated to Azure; downtimes can be
unacceptable for banking solutions
– Leveraging PaaS services for modernizing some of the legacy
components
– Security of data at rest after migration and in transit during
migration
– Flexibility to adopt hybrid architectures; some components might
still remain on-premises due to security/compliance reasons
– Visibility into security posture of the environment post migration,
identifying malicious activities, and doing proactive remediation
Key Takeaways
The success of a digital transformation lies in setting the context early
on and identifying the best-fit tools, services, and framework for
implementation. In this chapter, we explored the need for the Well-
Architected Framework to enable digital transformation. The five
pillars of the Well-Architected Framework were also introduced and
how they fit into your cloud adoption strategy. With the Azure Well-
Architected Framework, most of the work is etched out for you, and
diligence applied in adapting it will make your cloud adoption more
efficient. In subsequent chapters, we will be exploring the five pillars of
the Well-Architected Framework in detail. We will also correlate them
to some real-world adoption scenarios that will help you better
understand the concepts.
© The Author(s), under exclusive license to APress Media, LLC, part of Springer
Nature 2021
S. Ambi Karthikeyan, Demystifying the Azure Well-Architected Framework
https://doi.org/10.1007/978-1-4842-7119-3_2

2. Cost Optimization: Return on Your

Cloud Investment
Shijimol Ambi Karthikeyan1
(1) Bangalore, Karnataka, India

With an increase in cloud real estate, resource sprawl can become a

reality if environments are not managed effectively. Though converting
capex to opex is one of the major advantages of cloud computing, you
should have a laser focus on optimizing your cloud costs before you
even deploy the first resource to ensure the optimal return on
investment (ROI). It is a misnomer that monitoring your cloud usage
will help with optimizing the cost; it is only one aspect of the solution.
In this chapter, we will explore the considerations for cost optimization
while deploying your workloads to Azure.

How to Design for Optimal ROI

Designing for optimal ROI is a continuous process, as you will have to
revisit it throughout the application lifecycle. Cost optimization can be
pivoted on the design principles covered next.

Design Principles
Here are the design principles.
1. Identify the right costing model.
Azure provides a pay-as-you-go model, but there are different
flavors that you can choose from. Along with the compute charges,
 the software cost, the OS cost, etc., are included in the pricing. You
also have options to reuse your existing on-premises licenses using
BYOL options. This is applicable for Microsoft software as well as
for third-party appliances that you want to add to the architecture.
Consider the following options for reducing the cost as you design
the environment:

– Azure hybrid benefit: Customers with existing Windows Server, SQL

licenses, Red Hat, or SUSE Linux subscriptions can use the same
licenses in Azure to reduce cost.
– Azure spot virtual machines: If your workloads can be interrupted
and resumed, then Azure spot VMs can be used to get up to 90
percent discount. They leverage unused compute capacity in Azure.
You agree on a maximum price you are willing to pay and the VMs
are evicted if Azure does not have compute capacity or if the cost of
the VM is higher than the agreed price.
– Reservations: With Azure reserved instances, you can reserve
Windows or Linux VMs up front for a period of one year or three
years to get an additional discount on pay-as-you-go prices. This can
be combined with hybrid to get an up to 80 percent discount on the
overall cost. There is a flexibility to pay the cost up front or on a
monthly basis. However, if you reserve the instances and choose to
cancel before the agreed time frame, there is an early termination fee
involved.
– Azure dev/test pricing: For nonproduction dev/test environments,
customers can benefit from the special dev/test pricing provided by
Azure. There are different options for the same thing: Azure monthly
credit for Visual Studio subscribers, enterprise dev/test agreements
or pay-as-you-go dev/test subscriptions with reduced rates for a
specific set of Azure services such as VMs, cloud services, SQL
databases, Logic App, App Services, and HD Insight.
2. Choose the right size of resources for the business case.
Finding the right resource for your architecture and right sizing the
resource plays an important role in keeping the costs at a
minimum. You need to look at the required compute capability,
storage capacity, traffic, or transaction requirements while selecting
the resource. Another decision point is the level of control you want
 over the resources. For example, if you want control over the end-
to-end application stack, you might want to use IaaS resources
instead of PaaS services. Even with IaaS, you will have to drill down
on the identified components. For example, are you choosing the
right VM and disk SKUs?
The redundancy options should also be considered while
selecting the resources. For example, storage services by default
would provide three copies of your data in the same region by
default, i.e., locally redundant storage. You can go with
georedundant storage (GRS) to get three copies of the data in a
paired region. However, this would incur more than double the cost
of LRS. For your dev-test environments, you might want to stick
with LRS, while production workloads can use GRS, but only if they
need georedundancy.
With each resource type, be aware of the different billing meters
associated with it. For example, the cost of a storage account is not
just for the capacity; it also includes the cost of read/write
operations, listing and creating container operations, data write
and retrieval operations, etc. Often customers get blindsided by the
primary cost meter (capacity in the case of storage) and do not take
the other billing meters into consideration.
Overprovisioning is one of the most common reason for
increased cloud cost. Customers prefer to be cautious and start
with the larger SKU than actually required by the application. For
example, a low transaction database may not actually need
premium disks, which could eat away at your monthly cloud
budget. The recommended approach is to start small, monitor
usage, and then adjust the SKU size if you hit any bottlenecks.

3. Keep the budget in sight.

The design decisions are pivoted on the requirements of
applications; however, the cost aspects of the application
components should also be considered in the decision-making
process. Always keep the budget in sight and explore the cost
implications of design decisions. The application should adhere to
the desired levels of high availability, resiliency, scalability, security,
 etc. However, while creating the architecture pattern, make sure
you are not over-engineering the solution.
For example, consider multiregion deployment patterns that
help protect the application from regional disasters. However,
running an active-active deployment for DR would be more costly
than an active-passive deployment. If the RPO/RTO permits, you
can go for an active-passive approach where the resources can be
redeployed in the target region in the event of a disaster through
predefined and tested DevOps processes. For applications that
cannot afford downtime, this may not be a possibility, and an
active-active DR pattern would be required. In such cases, cost
optimization can be done as well by running a scaled-down version
of a DR environment, which can then be scaled up in the event of a
failover.
4. Plan for scaling.
Scalability on demand is one of the key selling points of the cloud,
and you can use it for your advantage to reduce your monthly cloud
bill. Scaling can be either vertical or horizontal, and this is one of
the architecture pattern decisions.
In a vertical scaling or scale-up approach, you add capacity to
existing resources, such as updating the VM to a higher SKU to
accommodate a spike in application usage. With the Azure SQL
database, you can do dynamic scaling by changing the DTU service
tiers or changing the vCore type.
Horizontal scaling, on the other hand, is adding instances to
handle the workload. This works best with autoscaling. Identify the
right set of metrics that will help you trigger the scaling such as a
spike in CPU/memory/IOPS usage in VMs that can trigger the
autoscaling of VMs in a VMSS. With Azure SQL databases, you can
achieve this using elastic pools. Similarly, autoscaling capabilities
are available for services such as Service Fabric, Azure App service,
Azure Functions, etc.
Tying scalability back to cost optimization, it can be
summarized as follows: allocate only the resources required for
your applications at any given point in time so that you pay only for
what you are using.
5. Implement continuous monitoring.
As mentioned earlier, cost optimization is a continuous process. You
cannot just deploy the environment aligned with best practices for
cost optimization and forget about it. Applications change,
architectures evolve over time, and new components get added.
Hence, it is important to do a periodic check of the cost incurred by
deployments and do course correction. Monitor the metrics of
resources regularly to look for indicators of potential spikes in cost.
For example, if your user base has increased over the year, the
storage cost would also increase in due time. Hence, you might
want to consider tiering unused data to a cheaper tier such as cold
storage or an archive to keep the cost in check.
For a detailed analysis of the cost, tagging of resources to the
owner of systems would help. For example, tag all Azure resources
used by the finance department using the same key-value pair so
that you can get a cumulative report of the cost incurred by
department in your monthly bill. You can also create cost
management alerts based on Azure Resource Consumption . You
can generate alerts based on predefined budgets, based on Azure
prepayment (for enterprise agreements), or even based on
spending quotas defined per department in the enterprise
agreement portal.

Design Process
Let’s revisit the case study that was introduced in Chapter 1 for retail
customers. While the listed key requirements should be met, the cost of
the solution should also be managed throughout the entire lifecycle.

Scenario 2
Industry: Retail
Ecommerce has become a key enabler for the retail industry. It is a
highly competitive arena, so customers are spoiled by all the choices.
Ecommerce websites should be designed for optimal customer
experience to ensure repeatable business. The Azure Well-Architected
Framework can help by identifying the right components for hosting,
scaling, and securing ecommerce solutions and also to ensure long-
term sustainability. Here are some key considerations for the scenario:
– Regular updates of catalogs to cover new product releases
– Secure payment transactions and storage personal information
– Quick retrieval of specific product information and possible
alternatives
– Search experience personalized to customer preferences and
previous purchases
– Capability to handle peak hour traffic such as Black Friday sales
Table 2-1 lists some of the key considerations for cost optimization
while finalizing the architecture pattern for this use case.
Table 2-1 Architecture Considerations

Consideration Rationale
Where does the The cost of resources varies across Azure regions.
customer plan to deploy
the solution?
Will there be any cross- If there will be cross-zonal traffic network,
zonal traffic from ingress/egress traffic cost should be factored in.
components in other
regions?
Is multiregion This depends on whether the ecommerce website caters
deployment required? to a customer base in only one geography. Or would it be
a site catering to customers across the globe?
If multiregion is being Understand the user base per geography and plan
considered, what is the accordingly. A full-scale deployment may not be required
capacity required per in regions with minimal customer base, or you could
region? even redirect the traffic to the closest geography.
Does the solution need An additional license cost for Azure AD should be
multifactor factored in based on the requirement.
authentication or
conditional access?
Is a DDoS standard As ecommerce websites can easily be targeted for DDoS
enabled? attacks, you might want to enable a DDoS standard,
which would incur additional cost depending on the
number of resources.
Consideration Rationale
What is the Internet This depends on whether you are planning to use Azure
edge security solution? Firewall, Azure Application Gateway, or third-party edge
solutions. If there is an existing investment in a third-
party solution with a BYOL option in the cloud, that
could bring down the overall cost.
What are the scaling The ecommerce application is expected to have spike in
requirements? usage during holiday and sales seasons. Autoscaling of
components can be used here to meet the varying
demands and also keep the cost in check.
What are the different If there are multiple dev/test environments, use Azure
environments required dev/test subscription benefits to reduce cost.
for the application?
What are the services If an Azure cognitive search solution is being used,
being planned for the estimate the capacity requirements before selecting the
intelligent search pricing tier as it cannot be changed later.
functionality of the
ecommerce website?
Is a CDN being Identify the zone for CDN and estimate the data transfer
considered to enhance rates, and choose between standard and premium flavors
user experience? of the service.
Deployment model for Weigh the pros and cons of deploying application and
services such as databases in VMs versus leveraging services such as
IaaS/PaaS Azure App Service or Azure SQL DB. For example, IaaS
gives more control over the stack but would incur
additional overhead for maintenance, patching, etc., and
hence there is an indirect manpower cost involved. Add
in a hybrid benefit to the pricing model if you have BYOL
options available for the components to calculate the
final cost.
What is the expected Understand how the application aligns with the business
lifespan of the strategy of the company. Is it part of its long-term digital
application? transformation plan or a short-term plan or stop-gap
solution? If it’s the former, consider Azure Reservation
with an up-front commitment of one year or three years.
Have you considered the Avoid falling into the trap of overprovisioning by
cost differentiating maintaining all lower environments at the same level as
options between production. Use lower-resource SKUs in nonproduction
production and environments. You can also do away with the high-
Consideration Rationale
nonproduction availability configurations that would need additional
environments for the resources.
application?
Have you finalized the Create alerts based on the budget for the ongoing
budget and identified the monitoring of cost. Add tags to resources mapping to
stakeholders if internal internal departments for cross charging.
cross charging is
required?
Have you defined the Cadence should be set for reviewing information from
plan for ongoing tools like Azure Cost Management or Azure Advisor to
governance and further optimize the cost. During the lifecycle of the
monitoring of cost? application, some resources might get replaced, and they
should be deleted in time to avoid additional charges.
Examples are cost of unused storage, orphaned disks, etc.
Additionally, Azure policies can be used to prevent the
deployment of resources that do not adhere with the
decided SKU, location, features, etc.
Is the application Ensure the integration of DevOps practices for the agility
lifecycle management of the application. This also helps in the quick
automated? deployment and deletion of test and development
environments on an on-demand basis.
What is the DR strategy Depending on the choice between active-active or
for the application? active-passive, factor in the cost of backup services or
cost of additional components in the DR region.
Figure 2-1 summarizes the different phases in the cost optimization
process.
Figure 2-1 Cost optimization process

Tools and Services for Cost Optimization

Getting into the practical nuances of cost optimization, let’s consider
some of the tools that you can use for initial estimation and also for the
ongoing monitoring.

Estimation Tools
TCO Calculator: Migrating to the cloud is part of an organization’s
digital transformation journey and is often an irreversible decision.
Along with mapping the right services to host the applications, you
would also want to understand the TCO in the long term while
migrating workloads from on-premises to Azure. The Azure TCO
Calculator is a tool that can be used for this calculation. You can add
details of your on-premises workloads, i.e., details of servers,
databases, storage, and networking. You can then adjust the default
assumptions made by the calculator. For example, adjust your storage
redundancy requirement or update your software assurance coverage.
The tool is available at https://azure.microsoft.com/en-
us/pricing/tco/calculator/.
Here I’m showing the three-step calculation process for a sample
two-tier application. For the purposes of demonstration, we are
considering two servers for the app server front end and two servers
for the database back end hosted on the VMware platform.
The first step is to define the workloads. Enter the details of the
servers, as shown in Figure 2-2.

Figure 2-2 Server details

On the same page, provide the details for the storage and
networking, as shown in Figure 2-3.
Figure 2-3 Storage and networking details
On the next page, adjust the assumptions. In addition to the
redundancy and license requirements in Azure, this page also covers
other on-premises costs such as hardware, software, electricity,
virtualization, etc., which can be adjusted if required, as shown in
Figure 2-4.

Figure 2-4 Redundancy and license requirements

 In the next step, a report will be generated that projects your TCO
and savings. Figure 2-5 and Figure 2-6 show some snippets from the
sample report.

Figure 2-5 TCO sample report, cost savings over five years
Figure 2-6 TCO report, cost breakdown
Azure Pricing Calculator: If you are deploying greenfield
environments in the cloud or migrating brownfield deployments, the
Azure Pricing Calculator will come in handy to understand the
estimated monthly cost. Note that this will be an estimate, and the
actual bill will depend on the usage of the environment. The Azure
Pricing Calculator is available at
https://azure.microsoft.com/en-
us/pricing/calculator/.
The Azure Pricing Calculator helps you understand the cost
differences when you use different SKUs, regions, or flavors of the same
service. Figure 2-7 shows an example of an estimate for the cost of an
app service using Windows OS at the Premium tier.
Figure 2-7 Azure Pricing Calculator
If you want to understand the cost difference while hosting the
same SKU in a different region and using a lower SKU for, say,
nonproduction deployments, you can simply update the details in the
calculator, as shown in Figure 2-8.

Figure 2-8 Updating the calculator details

 The Azure Pricing Calculator considers all the possible charges
while you start using the resource. For example, with storage, it is not
just the capacity but the additional charges incurred for the storage
read-write operations, container operations, and the data retrieval
operations (for cool and archival tiers) that you will be charged for, as
shown in Figure 2-9.
Figure 2-9 Azure Pricing Calculator, all possible charges
 The estimates created in the Azure Pricing Calculator can be saved
or exported for future reference.
Azure Migrate: Azure Migrate is the Microsoft-recommended tool
for lift-and-shift migrations to Azure. It can be used to migrate on-
premises physical machines, virtual machines hosted in
VMware/Hyper-V, or machines hosted in other cloud platforms like
AWS or GCP. In addition to providing a unified platform for migration,
the tools also provide a cost estimation for the migrated workloads
during the assess phase. Here you choose the VM SKUs and apply any
discount percent or EA subscription discount or hybrid benefit
discounts. Once the assessment is completed, you will get a monthly
cost estimate for compute and storage after migration.
Monitoring
Azure Cloud Cost Management: This service provides visibility into
your Azure usage patterns and trends. It also uses advanced analytics to
predict the Azure monthly cost based on the usage for the month. Any
reservations, hybrid benefit discounts, or negotiated prices are also
considered by the service. The details provided by Azure Cloud Cost
Management provide visibility into the cloud spend, and the tool helps
you drill down to the usage and take the necessary cost control steps.
Figure 2-10 shows a sample cost analysis chart with the actual cost
and the forecast cost for a billing period for a pay-as-you-go
subscription.

Figure 2-10 Sample cost analysis chart

 Cost analysis reports on which service/location/resource group
name is incurring these charges are also provided, as shown in Figure
2-11. The view can be further customized to get additional views based
on resource, tags, reservation name, service tier, and so on.

Figure 2-11 Cost analysis reports

Budgets in the cost management service help to define budgets for

usage and generate alerts when the cost exceeds the defined limits.
Note that budgets and alerts do not stop any of your Azure resources or
stop the consumption, but rather help in creating awareness and
accountability in your organization about cloud spend.
Azure Advisor: Azure Advisor evaluates your Azure environments
and provides cost optimization recommendations. It provides
recommendations for the following:
– Right sizing of VMs based on utilization trends
– Shutting down unused VMs based on usage information
– Updating SKU of databases like MariaDB, MySQL, or PostgreSQL if the
resources have been under-utilized
– Monitoring and removing unprovisioned ExpressRoute circuits
– Identifying unused virtual network gateways and reducing the cost
by deleting or reconfiguring them
– Enumerating resource usage and providing recommendations for
reducing cost through reserved instances
– Deleting unused resources that could incur cost, such as IP address,
managed disk snapshot, failed data factory pipelines, etc.
Figure 2-12 shows a cost optimization recommendation from Azure
Advisor.
Figure 2-12 Cost optimization recommendation

Governance Using Azure Policies

Azure policies can be used to enforce governance on the type of
resources, location, SKU types, etc. You can use either built-in policies
or custom-built policies to suit your business requirements. For
example, you can use a built-in policy to restrict the type of resources
that cannot be deployed in your environment. This would help in
preventing resource sprawl as well as eliminate associated cost.

Understand the Trade-Offs

While making design decisions, there could be trade-offs between the
final value proposition and the cost. Often the cost optimization would
be pitted against the other four pillars of the Well-Architected
Framework.
Reliability: Consider the decision to use multiregion or multizone
deployment for the reliability of application. This could mean
additional components and additional cost. However, for production
deployments, this could be non-negotiable as you need to meet your
defined SLAs, RPO, and RTO. You need to carefully evaluate the
options and choose lower-cost reliability options such as restoring
from backup if the downtime is acceptable.
Performance efficiency: Adding additional capacity or using high SKUs
for high performance would lead to an increase in the cost. Avoid the
trap of overprovisioning—always start with the optimal size and
then scale as required. Choose the Azure location where the costs of
resources are less, but do not greatly impact the user experience due
to access latencies. When using options like spot VMs, though the
 price is less, be mindful that the services could be interrupted any
time when they get evicted. Use such options only for services that
are OK to be interrupted.
Security: Adding additional security components would increase the
overall cost of the deployment. Security is non-negotiable, and it is
often a trade-off that all organizations would end up accepting in
their fold. However, you should be aware of overdoing security. For
example, avoid piling up security services in the architecture that
essential provide the same functionality.
Operational excellence: There are operational processes that are
required to keep the lights on in your Azure environment; however,
the investment into them could drive up the cloud cost initially.
Examples are the time and effort required to set up automated
operations, i.e., manpower, cost of tools, etc. However, over time, the
investment would pay off and would eventually help to reduce the
cost by avoiding human errors, configuration mistakes, quick
rollouts/rollbacks, etc.

Configure Reports for Visibility

Tools like Azure Cost Management and Azure Advisor provides
visibility into your Azure cloud cost. You can use Azure consumption
APIs to create custom reports for both native and marketplace
resources. In addition to out-of-the-box monitoring capabilities
provided by Azure Advisor, customers can also create custom scripts
using Azure APIs to identify unused or orphaned resources. Azure Cost
Management Connector for Power BI Desktop can be used to create
customized reports for granular analysis of Azure usage. This service is
available for customers having a Microsoft customer agreement or
enterprise agreement for Azure.

Key Takeaways
Cost optimization for Azure should be based on the principles of build-
measure-learn, where the focus is not on building the perfect solution
on day 1, but rather on optimizing as you go. It is greatly dependent on
the components and the architecture patterns. However, careful
planning of budgets, evaluation of trade-offs, and ongoing monitoring
can help in achieving the right balance between cost optimization and
the other pillars of the Well-Architected Framework. In the next chapter
of this book, we will explore another pillar of WAF, i.e., operational
excellence, that will help keep the lights on for your Azure deployments.
© The Author(s), under exclusive license to APress Media, LLC, part of Springer
Nature 2021
S. Ambi Karthikeyan, Demystifying the Azure Well-Architected Framework
https://doi.org/10.1007/978-1-4842-7119-3_3

3. Operational Excellence: Keep the

Lights On
Shijimol Ambi Karthikeyan1
(1) Bangalore, Karnataka, India

Operational excellence for any application is based on how to code and

ship the application and then manage its lifecycle. Deploying or
migrating applications to the cloud is not the end of the journey; what
is more important is how you keep the lights on and how efficiently you
can do that. This chapter will focus on framework, practices, and
processes to be followed to ensure that applications run without
disruption. It will cover end-to-end processes starting from the design,
deployment, and ongoing monitoring.

How to Design for Operational Excellence

Traditional methodologies of application development and deployment
have to give way to DevOps-focused delivery if we want to achieve
optimal operational excellence. Let’s explore some of the key design
principles that will help you achieve this.

Design Principles
These are the principles:
1. Adopt DevOps.
The siloed approach where developers and the operations team
work independently does not fit well in a cloud-first world, which
 demands agility and more frequent deployment cycles.
Organizations should focus on developing a DevOps model, which is
more often a cultural shift. The demarcation between developers
and the operations team is no longer there, and everyone is equally
accountable for a successful deployment. The DevOps DORA
metrics can be used as a quantitative method for measuring your
DevOps maturity.
– Deployment frequency (DF) : This metric indicates how often you
deploy code to product environments so that your users can benefit
from new features/updates/bug fixes, etc. A high level of maturity on
this metric would mean that your team can comfortably deploy code
as and when required, even if that means multiple deployments in a
day.
– Mean lead time for changes (MLT): This metric measures the time it
takes for code committed by a developer to be successfully deployed
in a production system. At high levels of efficiency, this should be
ideally less than one day.
– Mean time to recover (MTTR): This metric indicates how your DevOps
process can handle downtime, which could be due to defects,
unplanned outages, etc. It measures the time taken for your
application to be back online after such an incident happens. At high
levels of efficiency, this should ideally be less than an hour.
– Change failure rate (CFR): This metric indicates the percentage of
deployments to production that could lead to downtime or degraded
functionality of an application. In other words, it would also indicate
the number of deployments calling for an immediate
hotfix/rollback/patches, etc. The metrics can be in the range of 0 to
15 percent as a general standard, but the actual value would depend
on your application criticality.
2. Streamline the build and release of workloads.
Though it is included as part of DevOps, the process of build and
release requires a special callout. Along with application
deployment, deploying the underlying infrastructure through IAC
ensures that the lifecycle of all application components is
streamlined. No different teams or processes are required to handle
Another random document with
no related content on Scribd:
live in peace, with this and many other crimes staring him in the
face.”
The heart sickens at such a recital of cold-blooded murder; and the
evidence of savage, not to say inhuman, barbarity that characterized
the horrible crime is sufficient to humiliate the whole race of men
and send our much vaunted Christian civilization reeling back into
the dark ages. The shadow on the dial of Ahaz went back ten degrees
—it was a wonderful miracle—but here, in the noon of the nineteenth
century, the shadow on the dial of human progress and Christian
civilization has gone down forty degrees without a miracle, and
reaches the grosser, the darker and the baser passions of our fallen
nature, which instigate and then execute deeds of horror at which all
Christendom revolts.
 CHAPTER XXI.
REV. B. H. SPENCER.

His Character and Position as a Minister—Order of Banishment—

Interview with General Merrill—Note to Colonel Kettle—Cause of
Banishment—Letter to A. C. Stewart—Provost-Marshal at
Danville—Frank, Manly Reply—Second Letter to Mr. Stewart,
and Petition to General McKean—The Latter Treated with Silent
Contempt—Strong Loyal Petition Endorsed by H. S. Lane, U. S.
Senator, and O. P. Morton; Governor of Indiana—“Red Tape”—
Petition Returned—Hon. S. C. Wilson Counsel for the Exiles—
General Schofield Finally and Unconditionally Revokes the Order
of Banishment—Indictment for Preaching Without Taking the
“Test Oath.”—Why he Declined to Take the Oath—Prayer for his
Persecutors.
 Rev. B. H. Spencer.
Neither goodness, kindness, humility nor usefulness in a minister
of the gospel could disarm malice or shield the servant of God from
the persecutions of wicked men. It is truly astonishing how many and
how diverse the pretexts framed for the arrest, robbery, banishment,
imprisonment or murder of those whose only crime was that they
were ministers of the gospel in connection with the M. E. Church,
South. Infidelity was never at a loss for expedients and Antichrist
was never without efficient agents.
The Rev. B. H. Spencer is almost a native of Missouri, being only
six months of age when his parents came to Missouri from North
Carolina, and has received regular appointments from the Missouri
Annual Conference, M. E. Church, South, consecutively since 1843,
when he was first admitted on trial. No man has a cleaner and purer
record in the Church, both in his personal and ministerial character;
and few men have occupied so many places of high trust and
responsibility. He is one of the old Presiding Elders, and has often
been called to represent his Conference on the floor of the General
Conference, and has always proved himself to be prudent in council,
wise in legislation, correct in administration and eminently useful in
the pulpit; distinguished, perhaps, for his scriptural, practical and
forcible expositions of the distinctive doctrines and duties of Bible
Christianity. He is zealous, humble, earnest, energetic and
Methodistic in all his ministerial work; extensively known and highly
esteemed in love for his works’ sake all over the State.
Long associated with the honored names that will live in the
annals of Missouri Methodism, and taking a high rank with them,
the sentiments that introduced the Rev. W. M. Rush to these pages,
and the reader, may, with but little alteration, introduce Mr. Spencer.
Mr. Spencer is a representative man in his character and position
in Missouri, and while his persecutions were severe and protracted,
his was not an isolated case. He represents in his cruel and wanton
exile a large class of Missourians, and especially of Missouri
ministers, some of whom will, perhaps, never return to this State. B.
T. Kavanaugh, L. M. Lewis, E. K. Miller, B. R. Baxter and many
others are possibly lost to the State forever. They may have gone out
for different causes, but the peculiar proscription and persecutions to
which ministers in Missouri have been subjected kept them out.
Few if any cases of persecution in Missouri present more
deliberate meditation, cooler cruelty and more heartless inhumanity
than the one disclosed in the following narrative, made in Mr.
Spencer’s own quiet, clear and forcible style. His letters to the
various military officials, written in exile, and while all the finer
sentiments and feelings of his manly, Christian heart were writhing
under the cruel injustice he had to bear without the means of
vindication or the hope of redress, are worthy the pen of Cranmer,
and would have given a higher tone and temper to the moral courage
of Latimore.
The reader must, however, measure the man and his persecutors
by the following paper:
 “Order of Banishment.
“Dear Doctor: The first item that I send you is in regard to my
banishment, as an act of ecclesiastical persecution.
“In the town of High Hill, Mo., on the 16th January, 1863, I
received from the hands of a Federal soldier the following order, viz.:

“‘Headquarters N. E. District Missouri, }

“‘Warrenton, Mo., Jan. 13, 1863. }

“‘Provost-Marshal, or Commanding Officer, Danville, Mo.:

“‘Sir: You will cause the following persons to leave the State of
Missouri, within a reasonable time after the receipt of this order, and
reside, during the war or until permitted to return, at some place
north of Indianapolis, Indiana, and east of Illinois. They will be
required to report to you, by letter, once a month, and are not
permitted to leave the State by way of St. Louis, but directed to go by
Macon City and Hannibal, Missouri. Rev. B. H. Spencer, * * * * * * *.
“‘By command of Brigadier-General Merrill.

“‘Geo. M. Houston, A. A. G.’

“The above order was accompanied by the following:

“‘Headquarters 67th Regiment E. M. M., }

“‘Danville, Mo., Jan. 16, 1863. }

“‘Rev. B. H. Spencer:

“‘Sir: The above is a true copy of Gen. Merrill’s order to me. You
will obey said order within six days from this date. You will report to
these headquarters on the day of departure.
“‘By order of J. G. Kettle, Col. Commanding.

“‘J. F. Anderson, Adjutant.’

 “On the day of receiving this order I went to Warrenton, being
Gen. Merrill’s headquarters, to see if I could not induce him to
revoke it. I found him at the supper table, and unwilling to give me a
hearing anywhere else, when the following conversation took place
between us:
“‘Gen. Merrill, I have received from you an order of banishment
from the State, and wish to see you in regard to it.’
“‘Then what is your name and place of residence?’
“‘My name is B. H. Spencer, High Hill, Mo.’
“General (in a passion)—‘I can do nothing for you!’
“I replied—‘It seems that the tongue of slander has reached you
concerning me; will you hear evidence in my favor?’
“His reply was peremptorily, ‘No, sir!’
“I inquired, ‘Will you then read documents?’
“Answer in same manner—‘No, sir!’
“He then inquired—‘Does the order allow you to go by St. Louis?’
“I answered, ‘No, sir.’
“‘Then,’ said he, ‘see that you don’t go that way!’
“I replied, ‘I don’t expect to.’
“He said, ‘see that you don’t!’ And then added, ‘You may think
yourself very fortunate that you are not hung, and should feel that
you are very mercifully dealt by!’
So the conversation ended, and I returned home and wrote the
following note to Colonel Kettle:

“‘High Hill, Mo., Jan. 19, 1863.

“‘Col. J. G. Kettle, Danville, Mo.:

“‘Honored Sir: Some time ago I promised to marry a couple in

this vicinity on to-morrow night, and as it will not be in violation of
Gen. Merrill’s order, and will furnish me some means with which to
carry out that order, will you permit me to do so?

“‘I am, very respectfully, B. H. Spencer.’

 “The following is his reply:

“‘Headquarters 67th Regiment E. M. M., }

“‘Danville, Mo., Jan. 19, 1863. }

“‘Rev. B. H. Spencer:

“‘Sir: Your request to marry the couple and to preach is granted. I

would say that you had better not speak of your banishment in your
sermon.

“‘Yours, &c., J. G. Kettle, Colonel.’

“On the 25th of January, 1863, I preached the sermon alluded to;
and then, in company with four others, made my report to military
headquarters at Danville, Mo. But, in consequence of an accident on
the railroad, I was permitted to remain with my family until the 28th
of that month, when, with a sad heart, I was compelled to leave my
distressed wife and six little children and go into a land of strangers,
and remain in exile for ten long months.
“Dr. H. W. Pitman, Rev. D. W. Nowlin, Rev. J. D. Gregory and Rev.
Wm. A. Taylor were banished in company with me. We had no trial,
either civil or military, nor would they condescend to tell us what
were the charges against us, or whether, indeed, there were any. Nor
to this day—September 7th, 1869—have we found out why it was
done, except through private and unofficial sources. The information
thus received as to the cause of my banishment was as I expected—I
was banished because I was a Southern Methodist preacher! One of
the officers was asked by one of my friends: ‘What are the charges
against Spencer?’ He answered, ‘I never heard that there are any;
but he is a man of influence, and, if disposed, can do a great deal of
harm!’ Another officer was asked by another friend, and he replied,
‘The fact that he is a Southern Methodist preacher is all I want to
know!’ There never was a more clear case of ecclesiastical
persecution than was my banishment. Certain men sought to
produce secession, treason and rebellion in the M. E. Church, South,
by way of showing how they professed to hate these things in the
nation; I opposed them, and they became my enemies and had me
banished. If any one doubts this let him attend to the following
documents:

“‘Ashby’s Mills, Ind., April 22, 1863.

“‘Mr. A. C. Stewart, Provost-Marshal, Danville, Mo.:

“‘Sir—There are reasons which induce me to believe that my case

is wholly at the disposal of the officers and Union men of Danville
and vicinity. If this be so, I wish to solicit your attention to a few
considerations in regard to my case. And, first, I was banished from
my home and family without a trial or a knowledge of the charges
against me, or who preferred them. Now, sir, is this right? Is there
any law, civil or military, that will punish an innocent man? How
could the officer who banished me know that I was guilty of any
crime without giving me a trial and hearing evidence in the case?
Have I ever had such a trial? When? Where? Who were the judge,
jury, witnesses pro and con? Where was the prisoner during the
trial? And where was my legal counsel to see that justice was done
me? With what was I charged, and who were my accusers? Three
months have passed since my banishment, and I am still left in
ignorance of why it was done. Was it done merely to gratify official
ambition? or rather, was it not done to gratify the malice of secret
enemies? Can the interests of the Government be secured or
protected or its dignity increased by such treatment of one of its
citizens? Do you say that I am a great rebel, and therefore such
treatment is good enough for me? How do you know that I am a
rebel at all, much less a great one? Did you learn it from mere rumor,
or from a trustworthy witness, sworn to tell the truth before a proper
tribunal and in the presence of the accused? In the absence of such
evidence how can an intelligent gentleman make such a charge, if,
indeed, any one does make it? If it be stated, or insinuated, that I
have been, or am, disloyal or disobedient to the Constitution of the
United States, or to any of the laws made in pursuance thereof, or to
the constitution and laws of any State where I have ever lived, or to
any military order or edict—this most unjust and oppressive one
banishing me from my home and family not excepted—I deny the
allegation and defy proof by competent testimony! Have I not
silently borne injustice and oppression long enough? Can you blame
me for entering my earnest protest against such treatment? Has it
not been said by officers who ought to know, ‘that there are no
charges against me, but that I am a man of influence, and, if
disposed, could do a great deal of harm?’ Now, if there are no
charges against me, in the name of everything that an American
citizen holds dear, why suffer me to be thus persecuted and
oppressed without an effort to prevent it? Are you not a sworn officer
—sworn to support and defend the Constitution of the United States?
and does that Constitution allow such treatment of an American
citizen against whom there are no charges? and can you allow it to be
done without an effort to prevent it and be innocent? And suppose I
have influence, is that a crime? and what reason has any one to fear
that I would use it for evil? Is it proposed to banish men of character
and influence from the State for fear they will exert their influence
for evil? If not, why send off, and keep off, so humble a person as
myself? Is this the way an officer should fulfill his oath of office? Was
he clothed with authority for this purpose? Is this the only protection
I am to expect from the officers of my native State? Is not my
banishment, under the circumstances, an unmitigated outrage upon
civil and military order, as well as upon my liberties as a citizen? I
love and almost venerate the Government of the United States as
established by our patriotic ancestors! Among earthly institutions I
expect and want nothing better. With it I find no fault. My complaint
is against certain of its officers for the injustice and oppression with
which they treat me. If you were in my place and I in your’s, what
course would you wish me to pursue? If a peaceable and quiet
citizen, such as I have always been, is not free from imprisonment or
banishment, who is safe? Has justice forsaken the land? And is there
no place where the oppressed may find redress? If there be any place
where justice may be had, will you tell me where it is, and how to
approach it? I must candidly believe that my banishment was caused
by ecclesiastical persecution—that I am banished for an
ecclesiastical and not for a political reason! Certain persons sought
to produce secession, treason and rebellion in the M. E. Church,
South, by way of showing how they professed to hate these things in
the nation, and I opposed them, because I not only loved union in the
nation, but also in the Church—hence they became my enemies, and
for this cause alone, as I believe, they secured my banishment! I
believe the officer who did it was deceived, and induced to believe me
a bad and dangerous man, or surely he would not have acted so
hastily and rashly! But you know, and so do all my enemies, that
such is not my character. Who would be injured by my return to my
family? Can anybody tell? Does anybody fear it? Shall my secret
enemies be allowed to continue the gratification of their malignity at
my expense under pretense of friendship to the Government? Will
my continued religious persecution do the Government any good?
Why, then, suffer its continuance? Why keep a man in exile without
just cause, who is in feeble health, with limited means, and a wife
and six dependent children needing his attention? Will you not then
allow me to come home at once? Do not even the instincts of
humanity, to say nothing of the higher obligations of justice and
official duty, urge compliance with this request? I honestly believe
that you and the Union men of your vicinity can get me home if you
will—just as easily as to say the word. I may be mistaken, but I
honestly believe that my whole case is in your hands, and that I
remain in exile or return to my family, just as you will the one or the
other. I have reasons for this opinion, and if I am mistaken would
like to know it. I wish to say that in all that I have written I have not
intentionally used a single word that was disrespectful toward those
in authority. In all that I have said, I have aimed to speak plainly,
candidly and earnestly, but also respectfully. I respect you on
account of the authority with which you are invested and the
Government which you represent. But I protest against the way I am
treated, and who can blame me for it? And if this protest shall be
disregarded now, perhaps it may live and speak in vindication of my
character when I am dead, and when the voice of injured justice shall
be heard and respected. If you can not release me, will you tell me
who can? And will you answer this at your earliest possible
convenience, and let me know what you intend to do in my case.

I am, most respectfully,

“‘B. H. Spencer.’

“The answer of the Provost-Marshal was prompt, frank and

manly, and does honor to the head and heart of its author. Unlike
every other officer, civil or military, to whom I had applied for
information or redress, he did not treat me with silent contempt. He
answered. And the answer is important, because it shows clearly that
he not only had no hand in the banishment of myself and my
companions in exile, but that he also had been kept in ignorance of
the intention to do it, as also for the reasons why it was done. Surely
there could have been no public charges against us, or proper trial in
our case, or the Provost-Marshal in our immediate vicinity could not
have thus been kept in ignorance of such an intention till after it was
done.
“It proves, furthermore, that by order of Gen. McKean, it was left
to the so-called loyal men of Montgomery county, Mo., to say
whether we should return or not. And we have the names of those
who gave their sworn opinions as to whether it was proper for us to
return or not, and could give them, but in mercy we withhold them.
And, finally, it proves that our efforts to obtain a revocation of our
order of banishment, to be successful, had to be kept to ourselves.
Why? Simply because if our secret enemies found it out they would
thwart our efforts at the headquarters of the Commanding General of
the district. But the letter speaks for itself. It is as follows:

“‘Office Provost-Marshal, Danville, }

Montgomery Co., Mo., April 26, 1863. }

“‘Rev. B. H. Spencer, Ashby’s Mills, Ind.:

“‘Dear Sir—I have just received yours of 22d inst., and must
acknowledge I am utterly at a loss to comprehend it.
“‘I want to say, once for all, to yourself, as also to Doct. Pitman and
Judge Nowlin, that I had no hand in your banishment whatever,
either as a private citizen or as an officer; that I never had, either
directly or indirectly, an intimation that such a thing was
contemplated. An order was issued by General McKean, who is
Commanding General of this district, headquarters at Palmyra, to J.
G. Lane, Provost-Marshal of Wellsville district, to take the testimony
of the loyal men of Montgomery county in relation to the propriety
of your return home. Lane was removed from office and his district
thrown into mine, and the order was sent to me by General McKean,
which I executed by taking the evidence of loyal men, both at High
Hill and Montgomery City, as well as Danville. The evidence was
sworn to and sent by order of the commanding General to his
headquarters.
 “‘Now, sir, I have given you the facts in regard to everything I have
had to do with this case. And, although you protest against any
intention to insult or offend in your communication, I must frankly
admit that the whole tenor of your letter seems to savor of both.
‘How can you consent, without just cause, to keep one in exile who is
in feeble health,’ &c., is one extract from your letter. ‘Will you not
then allow me to come home at once?’ is another. Now, sir, you must
know that I have no direct control of this matter! Why ask me such
questions? Why not ask me, as a private citizen, to use my influence
to obtain a revocation of the order? The authorities that issued the
order of your banishment have never asked, neither have I given, my
opinion as to the propriety of the order. Notwithstanding I consider
your letter as invidious, and, as I understand it, full of insinuations
against me, yet, under the circumstances, I will allow humanity to
step in, discard all feeling that your letter may have excited, and give
you the best advice I am capable of.
“‘Judge Nowlin, Doct. Pitman and yourself get up a letter, directed
to Brig.-Gen. McKean, Palmyra, Mo., through me as Provost-Marshal
of Montgomery county. Take humanity for your text; appeal to him
through the tears of your wife and helpless children; let Government
officers alone; agree to report to me once a week in person, if it
should be considered necessary; give every assurance that your lips
will be sealed in future as to the utterance of treason, directly or
indirectly; send the letter to me and I will forward it, with such
recommendation as I may deem proper and right, and, if that fails, I
am at the end of my row. The success of this thing will very much
depend on keeping my advice to yourselves. I may be mistaken, but I
believe your liberation may be effected in that way. Give my respects
to Judge Nowlin and Doctor Pitman.

“‘Yours, &c.,
“‘A. C. Stewart, Prov.-Marshal.’

“To the above noble letter I made the following reply:

“‘Ashby’s Mills, Montgomery Co., Indiana, }

‘May 4, 1863. }

“‘Mr. A. C. Stewart, Provost-Marshal, Danville, Mo.:

 “‘Dear Sir: Yours of the 26th April is to hand, has been read and
contents noted. And in reply let me say, I regret that you considered
my letter in its whole tenor ‘invidious, offensive and insulting,’
notwithstanding my protest against such a construction. I knew the
task I had undertaken was difficult, for there seems to be something
about official position which is always more or less impatient of
contradiction. And hence it was reasonable to conclude that this is
true of military officers, who feel that it is theirs to command and for
others to obey or submit, and not to reason or question. The
difficulty was to so employ language as to convey some idea of my
righteous indignation at the injustice of my treatment, and which
would at the same time be respectful and courteous toward those in
authority. And I question very much whether you yourself, in my
circumstances, would, if you could, have done better. I was, with only
a few days’ notice, forced away from the fellowship and pastoral
oversight of hundreds of beloved brethren; from a most dependent
and afflicted family; from my only means of their support; from the
graves of my kindred, and every thing of earth that was dear; was
denied the privilege of going by St. Louis, where I might have
reached the ears of power and have gained a revocation of my order
of banishment; with limited means, was compelled to travel a
circuitous and expensive route to my place of exile; was denied the
privilege of living in the loyal State of Illinois, where I had kindred,
and it would have cost me nothing; was denied the sympathy of
friends who would have helped me financially, but were afraid; was
sent into a land of strangers, under Government censure, where,
without sympathy, if without money, a man had better be dead; was
not allowed to know the charges against me, who were my accusers,
or even the semblance of a trial, though I had sought one of Gen.
Merrill, of Gen. Curtis, of Gen. Halleck, of Gov. Gamble, of Attorney-
General Bates, of Secretary Stanton and of President Lincoln, and
had done this, directly and indirectly, through men of commanding
influence, whose loyalty was above suspicion, and all this without
success; felt, yea knew, that I was innocent; that there could be no
truthful evidence of my being guilty of any crime; knew that I was
suffering all this to gratify the malignity of secret enemies who had
deceived the military commander and secured my banishment;
enemies who, like the midnight assassin, did their work and then
slunk away to gloat over the misery they had caused; felt satisfied
that I was thus persecuted for an ecclesiastical and not for a political
reason; was sure the Government could not be benefited by my
persecution nor injured by my return to my family; and, finally,
became thoroughly convinced that the influence that controlled the
action of those who had the power to release me from the binding
force of this order, or to keep me in exile, was in or near Danville;
and, in a word, was satisfied that I had found out the locality of the
authors of my trouble and why they persecuted me, but the identical
names of my persecutors I did not know; and hence, in view of the
foregoing considerations, I wrote you in the way I did. Now, interpret
my letter in the light of my circumstances, and imagine yourself in
my condition, and you will be able to ‘comprehend it,’ and to excuse
anything that may seem ‘discourteous or insulting,’ especially when I
assure you nothing of the kind was intended. You have my thanks for
your prompt and manly reply to my letter. There are times when I
would rather a man would abuse me a little than not answer me at
all, and this is one of those times. You are the only officer who had
the condescension, kindness, humanity, or whatever else you may
please to call it, to answer a single one of my numerous appeals for
deliverance from oppression, or for instruction as to where or how I
might obtain it. To your praise be this spoken. It affords me much
pleasure, also, to learn from yourself that you had no hand in
securing my banishment, or knowledge of it until after it occurred. I
wish I could think the same of every other citizen of Danville.
“And now that, in accordance with your wish, I am addressing you
as a private citizen, may I ask, and confidently expect, that you will
give me the names of my accusers, and the nature of their
accusations against me, if there are any, together with the names of
those loyal men whose sworn testimony was sent to Gen. McKean in
regard to the ‘propriety’ of allowing me to come home, and the
substance of what each one said? As that is the nearest a trial of
anything else I have had, should not the accused be allowed to know
his accusers, the names of the witnesses and the nature of their
testimony against him? You reprehend me very severely for
insinuating that you have any ‘direct control of my case.’ Well, I did
not suppose you had authority to revoke the order of banishment;
but I did suppose, and do still suppose, that you and your friends of
that vicinity can influence Gen. McKean to revoke the order or not,
just as you wish; and that you have control of my case in that way.
And hence it is that I am so thankful to you, and so much encouraged
by your kind offer to use your influence with the commanding officer
to set aside this order and permit me to return home. And I am sure
if you do promptly and vigorously exert your influence in that
direction you are certain of success.
“Among your items of advice you say, ‘Give every assurance that
your lips will be sealed in future as to the utterance of treason,
directly or indirectly.’ Now, as this is, to my mind, an intimation that
some one, or all three of us, are charged with having been guilty of
treasonable utterances, and hence are required to give assurance that
we will do so no more, I wish to say for myself that, if such be the
intimation, I deny the allegation in toto; for I have neither uttered
nor acted treason, nor do I expect to do either in future. And if I am
permitted to return, and you can protect me from the tongue of
slander, and the secret enemies that with consummate mendacity
hound my steps and torture and misrepresent my language and
conduct, you will hear nothing of treason, either in utterance or
action. But, if that can not be done—if the tongue of slander and
falsehood against me can not be silenced in any other way—then give
a fair trial, and make these secret liars, who whisper falsehoods into
official ears against those they hate, ‘face the music,’ and I will
vindicate my innocence. Upon that subject I can make no further
promises. A mere charge of treason, you know, is no evidence of
guilt. The immaculate Son of Man was accused of rebellion, sedition
and treason, with blasphemy, and with being the agent of the prince
of devils! Of Innocence itself they said, ‘He is not fit to live; away
with him! crucify him! crucify him!’ And ‘If they have done these
things in the green tree, what will they not do in the dry?’ And the
same divine authority has said, ‘If any man will live godly in Christ
Jesus, he must suffer persecution,’ and I have made my calculations
accordingly. As to your other suggestions, I wish to say that I will
herewith transmit to Gen. McKean, through you, a request, or
petition, for the revocation of this order in my case, accompanied
with a few of the reasons why I make it, which I will thank you to
send to him, if you please, together with such remarks and
recommendations as you may think proper to make. Please let me
hear from you at an early day, and much oblige,

“‘Most respectfully,
 “‘B. H. Spencer.’

“The petition was sent to General McKean, through the Provost-

Marshal of Montgomery county, Mo., together with the best appeal
that he could make in our favor. But the only notice he seems to have
given it was to treat it with silent contempt.
“The following is a copy of that petition:

“‘Ashby’s Mills, Ind., May 7, 1863.

“‘Brigadier-Gen. McKean, Com., Palmyra, Mo.:

“‘Dear Sir—Will you please to revoke the order of Gen. Merrill, of

the 13th January, 1863, banishing me from the State of Missouri? A
few of the reasons why I ask you to do this are—
“’1st. The order was unjust. The General who issued this order did
not know me, was dependent upon others for his information
concerning me, and was evidently deceived by my personal enemies,
or he never would have issued it.
“’2d. I have never engaged in this rebellion in any way, nor violated
any law, civil or military; and, therefore, am not deserving of this
punishment.
“’3d. I have a wife and six small, helpless children, whose ages
range from two to twelve years, from whom I have been forcibly
separated for more than three months, and who very much need my
attention, and, therefore, humanity, to say nothing of the higher
claims of truth and justice, demands compliance with this request.
“’4th. If permitted to return, I expect to be, as I have ever been, a
law-abiding and good citizen, and, therefore, the Government can
not be benefited by my remaining in exile nor injured by my return
to my family.
“’5th. As it is the duty and glory of a Government to protect its
citizens in the possession of all their legitimate rights, I ask, and
hope it will be your pleasure to grant, that I may return to my family
in the enjoyment of the untrammeled liberty that I had before my
banishment.
“‘This petition will be sent to your headquarters by Mr. A. C.
Stewart, Provost-Marshal, Danville, Mo., accompanied by such
remarks and recommendations as he may think proper to make.
“‘In the confident expectation that you will grant this just and
reasonable request at an early day,

“‘I am, most respectfully,

“‘B. H. Spencer.’

“After being compelled to remain long enough in exile to form

character and make friends amongst strangers, at the end of nine
months some of the most prominent Union men of Indiana, on the
31st August, 1863, sent the following petition to the Provost-Marshal
General of the department of the Missouri:
“‘To Lieut.-Col. J. O. Broadhead, P. M. G. of Missouri, St. Louis,
Mo., or to whomsoever this petition should be addressed:

“‘The undersigned petitioners beg leave respectfully to represent to

the proper authorities in the State of Missouri, that we are citizens of
the United States, residents of the counties of Montgomery and
Putnam, in the State of Indiana; that we are now and ever have been
loyal and devoted to the Government of the United States; that we
are supporters of the present Administration thereof, and that we are
in favor of using all lawful ways and means for suppressing the
present rebellion and preserving the Union established by our
fathers; we, therefore, cordially endorse all and every one of the
measures of the Government having these much desired objects in
view.
“‘We beg leave further to represent that there have been residing in
our midst, in our immediate vicinity, for the past six or seven
months, three individuals, said to be citizens of Montgomery county,
in the State of Missouri, and to have been banished from that State
by the military authorities there, viz.: H. W. Pitman, B. H. Spencer
and David W. Nowlin. While we can not know the causes that led to
the banishment of these men, we would state that they came among
us under the ban of the Government, and we looked upon them as
objects of suspicion. They and their conduct have been closely
observed and narrowly scrutinized, not to say strictly watched by our
party, and we deem it but sheer justice to declare, candidly and
emphatically, that after an observation of the length of time
indicated above we have seen nothing in these men that in our
judgment would require that they longer be kept in exile.’
“‘They are represented to us as men having families dependent
greatly on them for support, and every feeling of humanity is enlisted
in their behalf, if the interests of the Government do not imperatively
require their continuance in exile. With the lights before us, and in
view of the facts that these men have resided for the past six or seven
months in a population greatly excited on political issues, and among
whom sundry disloyal practices have been rife, in which they have
had ample opportunities to have partaken if they had been so
inclined, and yet our observation has not been sufficient to detect
them as aiders or abettors in these disloyal practices; we feel free,
therefore, to declare emphatically our convictions that the interests
of the Government will not be advanced by a longer continuance of
their exile; but, on the contrary, we are satisfied that those interests
would be promoted by a revocation of the order banishing them from
Missouri. We, therefore, in behalf of these exiles, pray the authorities
in Missouri who are empowered to do so to revoke the order
banishing the said H. W. Pitman, B. H. Spencer and David W.
Nowlin from the said State of Missouri, and to release them from
further pains and penalties in the premises; and as loyal citizens in
duty bound, we will ever pray, &c.

(Signed) “‘John W. Harrison,

“‘Dr. H. Labarre,
“‘Franklin M. McMurray,
“‘Dr. George W. Miller,
“‘James Knox,
“‘J. J. Billingsley,
“‘A. D. Billingsley.’

“The undoubted loyalty of these petitioners, and their prominence

in social and political circles during Mr. Lincoln’s Administration,
received the following endorsement, which accompanied their
petition and formed a part of it:
“‘I have known the signers of this paper long and well; they are
true and loyal citizens of Indiana, and are all supporters of the
Administration. They are gentlemen of the highest character, and
their statements are entitled to full credit.

“‘H. S. Lane, U. S. Senator.’

“‘The gentlemen who signed the foregoing statement are of

undoubted loyalty, and their representations are worthy of credit.

“‘O. P. Morton, Gov. of Indiana.’

“And now, by way of showing how difficult it was for those in

prison or exile to obtain a hearing at headquarters, in consequence of
official routine, etiquette, or what is technically called ‘Red Tape,’ I
give the following inscription, which was written on the outside of
the above petition before it was returned to the petitioners. It seems
first to have come into the hands of some sub-official, who read it
and then wrote on it a digest of its contents, as follows:
“‘Petition. Citizens of Indiana. P. 102 (P. M. G.) 63. That H. W.
Pitman, B. H. Spencer and D. W. Nowlin, exiles from Montgomery
county, Mo., be permitted to return to their families and homes, as
they have been closely watched while here and have always
conducted themselves as Union men. These petitioners are indorsed
by the Governor of Indiana.’
“This sub-official then seems to have sent it to the P. M. General of
the Department, who, without granting or promising to grant the
petition, sent it back to Gov. Morton, with the following explanation
written on it:

“‘Headquarters Department of the Missouri, }

“‘Office of the P. M. G., }
“‘St. Louis, Mo., Sept. 3, 1863. }

“Respectfully returned to his Excellency, O. P. Morton, Governor

of Indiana, with the information that there are no papers on the
cases of the persons named in the within petition in this office.
Neither does their names appear upon the records. They were
probably banished by order of some district commander.
“‘By order of Lieut.-Col. J. O. Broadhead.
 “‘H. H. Haine,
“‘Lieut. and A. P. M. G. Dept. of the Missouri.’

“Upon receiving it Governor Morton sent it to Senator Lane, who

sent it to the petitioners with the following explanation:
“This paper was to-day returned to me by Governor Morton, with
the indorsements on it. Sept. 7, 1863.

“‘H. S. Lane.’

“Just think of it! No trial, no charges, nothing for us or against us,

not on the records, no papers in our cases, and yet we in exile and
compelled to stay there! But we employed one of Indiana’s noblest
lawyers, the Hon. Samuel C. Wilson, of Crawfordsville, to take that
petition and go with it in person to Gen. Schofield’s headquarters.
The result was an unconditional revocation of the order of
banishment, on the 16th Sept., 1863, which is as follows:

“‘Headq’rs Department of the Missouri, }

St. Louis, Mo., Sept. 15th, 1863. }

“‘Special Orders No. 252.]

“‘I. Dr. H. W. Pitman, David Nowlin and B. H. Spencer, citizens of

Montgomery county, Missouri, heretofore banished to Indiana, to
remain there during the war, are permitted to remain in any part of
the United States, outside of the limits of this Department. They will
report their places of residence the first of each month during the
war to the Provost-Marshal General of this Department.
“‘By command of Major-General Schofield.
“‘Wm. W. Eno, Ass’t Adj’t-Gen’l.
“‘B. H. Spencer, per Maj. Dunn.’
“The foregoing facts and documents are a mere tithing of what
might be given to the same effect, and go to show most clearly that I
was persecuted in various ways, and banished from my helpless
family for ten long months, for no higher and no other crime than
that I was a Southern Methodist preacher!