Kubernetes security Guide for

beginners from zero to hero 1st Edition

Ivan Piskunov
Visit to download the full and correct content document:
https://ebookmeta.com/product/kubernetes-security-guide-for-beginners-from-zero-to-
hero-1st-edition-ivan-piskunov/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Python For Financial Analysis From Zero to Hero 1st

Edition Van Der Post

https://ebookmeta.com/product/python-for-financial-analysis-from-
zero-to-hero-1st-edition-van-der-post/

Artificial Intelligence Programming with Python From

Zero to Hero 1st Edition Perry Xiao

https://ebookmeta.com/product/artificial-intelligence-
programming-with-python-from-zero-to-hero-1st-edition-perry-xiao/

Innopolis University From Zero To Hero Ten Years Of

Challenges And Victories 1st Edition Manuel Mazzara

https://ebookmeta.com/product/innopolis-university-from-zero-to-
hero-ten-years-of-challenges-and-victories-1st-edition-manuel-
mazzara/

Zero Trust Security An Enterprise Guide 1st Edition

Jason Garbis

https://ebookmeta.com/product/zero-trust-security-an-enterprise-
guide-1st-edition-jason-garbis/
Cisco ACI: Zero to Hero: A Comprehensive Guide to Cisco
ACI Design, Implementation, Operation, and
Troubleshooting 1st Edition Jan Janovic

https://ebookmeta.com/product/cisco-aci-zero-to-hero-a-
comprehensive-guide-to-cisco-aci-design-implementation-operation-
and-troubleshooting-1st-edition-jan-janovic/

Zero To Mastery In Environmental Studies No 1

Environmental Studies Book To Become Zero To Hero In
Environmental Studies Arnold Neel

https://ebookmeta.com/product/zero-to-mastery-in-environmental-
studies-no-1-environmental-studies-book-to-become-zero-to-hero-
in-environmental-studies-arnold-neel/

Online Security For Beginners 11th Edition 2022

https://ebookmeta.com/product/online-security-for-beginners-11th-
edition-2022/

2D Game Development From Zero To Hero A compendium of

the community knowledge on game design and development
Daniele Penazzo

https://ebookmeta.com/product/2d-game-development-from-zero-to-
hero-a-compendium-of-the-community-knowledge-on-game-design-and-
development-daniele-penazzo/

Kubernetes Security and Observability 1st Edition

Brendan Creane

https://ebookmeta.com/product/kubernetes-security-and-
observability-1st-edition-brendan-creane/
 E-Book
DevSecOps and Cloud security series
____________________________

Kubernetes
security
Guide for beginners
from zero to hero

IVAN PISKUNOV
DevSecOps Evangelist

A self-published edition

2021
_______________
Kubernetes security
Guide for beginners from zero to hero
DevSecOps and Cloud security series

e-book, self-published edition by Ivan Piskunov

___________________________________

The first edition, 2021

 Brief contents
2021

Introduction ………………………………………………………………………………. 9

Chapter 1. Basics of Kubernetes and security issues ……………………..……. 11

Chapter 2. Concept of security ………………………………………….………..….. 27

Chapter 3. Deep defense of Kubernetes …………………………….…………..…. 44

Chapter 4. Scenario attack and hand on lab ……………..………………………… 69

Chapter 5. Non-Kubernetes security ……………………………………………….... 94

Chapter 6. Tips and tricks for pass certification exam ………………………….... 113

Conclusion …………………………………………………………………….……….…. 130

References ………………………………………………………………………………… 131

Appendix A. .…………………………………………………………………….……….... 136

Appendix B. ……………………………………………………………………….…...…. 155

Kubernetes security © Ivan Piskunov

The first edition,2021

Kubernetes Security. Guide for beginners from zero to hero
by Ivan Piskunov
E-book, Self-published

Copyright © 2021 Self-published. All rights reserved.

Printed in the United States of America.

Concept and materials: Ivan Piskunov

Jun 2021. The first edition.

Revision History for the First Edition

2021-06-01: First Release

The all logo and other stuff is a registered trademark of owner of these
trademarks.

No part of this book may be reproduced, stored in a retrieval system, or

transmitted in any form or by any means, without the prior written permission of
the author (publisher), except in the case of brief quotations embedded in critical
articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author(s) or its
dealers and distributors, will be held liable for any damages caused or alleged to
have been caused directly or indirectly by this book.

The author has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of
capitals. However, not author or dealers and distributors cannot guarantee the
accuracy of this information.

DevSecOps series
 Table of Contents
Introduction ………………………………………………………... 9
Chapter 1. Basics of Kubernetes and security issues … 11
1.1 What is Kubernetes? ……………………………………………………………………………... 11

1.2 Architecture and Components ……………………………………………………………….. 12

1.3 Basic Terms and Definitions ………………………………………………………………….. 18

1.4 Kubernetes security main risks and challenges ………………………………………… 19

1.5 The biggest Kubernetes security incidents ………………………………………………. 24

Chapter 2. Concept of security …………………………….… 27

2.1 THE 4C'S of cloud native security ………………………………………………………….. 27

2.2 Thread model for Kubernetes ……………………………………………………..………… 30

2.3 K8s Attack Tree …………………………………………………………………………………... 39

Chapter 3. Deep defense of Kubernetes …………..….…. 44

3.1 Brief review of native Kubernetes security solutions ……………………….……….. 44

3.2 General recommendations and rules …………………………………………………….. 44

3.3 Set up security tweaks and force options ………………………………………………… 49

3.4 Most popular Kubernetes Security audit tools and scanners …………………….. 61

Chapter 4. Scenario attack and hand on lab …………... 69

4.1 Most popular vector attacks ………………………………………………………………….. 69

4.2 What does all these blood's lessons teach? ……………………………………………... 71

4.3 What are doing attacker that hack to Kubernetes? ………………………………….. 74

4.4 The truest tools for penetration test Kubernetes …………………………………….. 77

4.5 Hands-on Lab for improve your skills …………………………………………………… 80

4.6 Review a same main scenario attack …………………………………………………….. 84

Chapter 5. Non-Kubernetes security ……………………..… 94
5.1 The Main Risk and Challenges of Docker ……………………………………………………. 96

5.2 Docker threat model ………………………………………………………………………………… 97

5.3 What should I do for Docker security? ………………………………………………………. 102

5.4 Recommended open sources Docker security auditing tools ………………………… 111

Chapter 6. Tips and tricks for pass certification exam ..

113
6.1 How is be a security certification in DevSecOps? …………………………………….. 113

6.2 Author’s opinion about certifications …………………………………………………….. 113

6.3 What is CKS? ………………………………………………………………………………………. 115

6.4 What are get the benefits of certification? ………………………………………………. 116

6.5 Exam Details ………………………………………………………………………………………. 116

6.6 Technical details ………………………………………………………………………………….. 116

6.7 Official resources and other stuff for prepare to exam ……………………………… 117

6.8 Review of typical questions …………………………………………………………………… 121

6.9 The examples of practice task ………………………………………………………………… 124

Conclusion ……………………………………………….….……… 130

References ………………………………………………………….. 131
Appendix A. …………………………………………………………. 136
Appendix B. …………………………..…………………..………. 155
 About this book

In this comprehensive e-book, we’re going to deep dive into the Kubernetes security. This book
addressed for everyone system engineers and security expert who involved in developer’s
software process based on microservices architecture and cloud infrastructure.

From this book you will find an overview of attack surface, an analysis of main cyber risks and
potential system flaw in cluster, as well as several scenarios with different attack vectors and
demonstration cases of successful attacker penetration inside. A separate chapter in the book
will be devoted to preparing for the Certified Kubernetes Security Specialist (CKS), which
provides tips, resources, and practical tasks that will be on the exam.

AUDIENCE
This book and all materials with it are intended for a technical audience, including cyber
security experts interested in gaining a quick understanding of recent security technological
trends in DevSecOps environment such as Kubernetes orchestrion system.
All readers should be familiar with the basic concepts of virtualization, networks, and have a
good understanding of security design.

PREREQUISITES
We assume anyone who wants to understand Kubernetes should have an understating of how
the Docker works, how the Docker images are created, and how they work as a standalone unit.
To reach to an advanced configuration in Kubernetes one should understand basic networking
and how the TCP\IP protocol communication works.
If you found any errors, mismatches in this book, please notify us at e-mail:
x1.ivan.piskunov@gmail.com

RECOURCES
The official web page of self-published edition
https://gum.co/k8security

All news of my current and future publications

https://gumroad.com/ivan14piskunov

GitHub repository with all of extra materials, YAML code, policy templates and examples used
throughout the book
https://github.com/IvanPiskunov/K8s_security_book

Twitter
https://twitter.com/Ivanpiskunov14

LinkedIn
https://linkedin.com/in/ivan-piskunov
 About the author

Ivan Piskunov, cybersecurity expert who have got more than 10 years of
industry experience. Ivan has a Master's degree in Computer Science in
cybersecurity. Multiple speaker of various public security events, author
and lecturer of training courses on Linux security and technical audit of IT
infrastructure. Ivan are currently holds the position of DevSecOps in a
software development company.

You can read some of the author's materials here:

HackMag e-zine

CyberSecBastion

MY GRATITUDE
I want to thank my close family members and colleagues that they your support and believe
in me. Thanks to my all brothers of mind and spirit - you always inspire me. Thanks to
everyone on my subscribers, LinkedIn friends and all not indifferent participants who helped
me with words or deal.

Special thanks to Michael Z., Michael M. and Alexander S.

 Introduction

Why Kubernetes?
Kubernetes or known as K8s is one of the highest trending technologies in Cloud Computing
and DevSecOps as of today. Kubernetes had the fastest growth in job searches, over a 173%
from a year before as reported recently by a survey conducted by Indeed.

Learning Kubernetes is essential for any DevOps professional. DevOps engineers are always in
demand. Currently the average Silicon Valley salary for a DevOps engineer is 20% higher than
what a software engineer makes. DevOps engineers make an average of $140,000 to $200,000
annually. And One of the most in-demand skills is Kubernetes Administration.

The same survey shows that 94 percent of organizations have experienced a serious security
issue in the last 12 months in their container environment, with 69 percent having detected
misconfigurations, 27 percent experiencing runtime security incidents, and 24 percent
discovering significant vulnerabilities to remediate.

When containers first broke onto the scene, it was immediately apparent that it was a disruptive
technology. What was not clear was how containers were going to be scaled and orchestrated
effectively. In the early days, the container orchestration competition was intense, with
applications such as Docker Swarm, Apache Mesos, and Kubernetes aiming to address this
issue. However, in the last six years, Kubernetes has differentiated itself and pulled away as the
clear container orchestration choice.

The Cloud Native Computing Foundation (CNCF) backs this assertion in their yearly survey,
with 92% of respondents using containers and 83% of respondents using Kubernetes in
production. The widespread adoption can in part be credited to Cloud providers recognizing
Kubernetes as the best choice. There are over 90 verified cloud platforms that offer Kubernetes
as a service. And with all of this buy-in, it’s not surprising that 451 research predicted
a container technology market worth 4.3 Billion by 2022.

The age of containers, Kubernetes has become a popular open source project and key building block for
modern tech infrastructure. Containers need to be managed, and Kubernetes helps with tasks such as
scheduling, load balancing, and distribution, while making sure container workloads run as intended.
With Kubernetes, you can deploy and manage cloud-native applications anywhere independent of
cloud-specific requirements, avoiding infrastructure lock-in.

Kubernetes, which is Greek for helmsman or pilot, is exploding in popularity. According to a survey of
the Cloud Native Computing Foundation (CNCF) end user community, Kubernetes use has
increased from 58% in 2018 to 91% in 2020. Over the past four years, Kubernetes job searches have
increased 2,125% according to Tech Republic, and the CNCF has recently introduced a Kubernetes
security certification.

9
Why Is Kubernetes security very important?
Container security necessitates an always-on solution to protect the binary, libraries, and all it
stores. However, traditional security policies are simply not applicable to container security;
they're simply not up-to-date enough to keep up with the sophisticated DevOps environment.
Thus, companies are often left in the dark on how to build container security into their
development pipeline.

Kubernetes can be somewhat helpful in the complexity of container security, but they are also
a hotbed for attackers themselves, as it's easy for attackers to identify Kubernetes clusters. As
a result, applications across the container life cycle risk compromisation. With Kubernetes and
container security being new territory for many organisations, and fortunately, there are some
great cyber security engineers who around to take care of it for you.

10
 Chapter 1.
Basics of Kubernetes and security issues
1.1 What is Kubernetes?
Kubernetes is an open-source container-orchestration system for automating application
deployment, scaling, and management, and developed by Google to help you manage the
containerized/dockerized applications supporting multiple deployment environments like On-
premise, cloud, or standalone virtual machines.

According to Kubernetes, “Containers are a technology for packaging the (compiled) code for
an application along with the dependencies it needs at run time. Each container that you run
is repeatable; the standardisation from having dependencies included means that you get the
same behaviour wherever you run it.”

It was originally designed by Google and is now maintained by the Cloud Native Computing
Foundation. It aims to provide a "platform for automating deployment, scaling, and
operations of application containers across clusters of hosts". It works with a range of
container tools and runs containers in a cluster, often with images built using Docker.
Kubernetes was founded by Joe Beda, Brendan Burns, and Craig McLuckie, who were quickly
joined by other Google engineers including Brian Grant and Tim Hockin, and was first
announced by Google in mid-2014.

Kubernetes v1.0 was released on July 21, 2015. Along with the Kubernetes v1.0 release, Google
partnered with the Linux Foundation to form the Cloud Native Computing Foundation (CNCF)
and offered Kubernetes as a seed technology.

11
1.2 Architecture and Components
Kubernetes has a client/server architecture. Kubernetes server runs on your cluster (a group of
hosts) on which you will deploy your application. And you typically interact with the cluster
using a client, such as the kubectl CLI.

Cluster

• It is a collection of hosts(servers) that helps you to aggregate their available resources.

That includes ram, CPU, ram, disk, and their devices into a usable pool.

Node

• A node is a physical or virtual machine, running Kubernetes, onto which pods can be
scheduled.

Pods

• A pod is the basic unit that Kubernetes deals with, a group of containers. If there are two
or more containers that always need to work together, and should be on the same
machine, make them a pod.

Container

• A container is the smallest unit in the Kubernetes world. The main purpose of
Kubernetes is to manage, deploy, and, to a certain extent, monitor containers.
Kubernetes management is not limited to Docker containers.
12
Label

• A label is a key/value pair that is used to identify a resource. You could label all your
pods serving production traffic with “role=production”, for example.

Selector

• Selections let you search/filter resources by labels. Following on from the previous
example, to get all production pods your selector would be “role=production”.

Service

• A service defines a set of pods (typically selected by a “selector”) and a means by which
to access them, such as single stable IP address and corresponding DNS name.

Master

• The master is a collection of components which make up the control panel of

Kubernetes. These components are used for all cluster decisions. It includes both
scheduling and responding to cluster events.

Namespace

• It is a logical cluster or environment. It is a widely used method which is used for scoping
access or dividing a cluster.

Scheduler

• The scheduler schedules the tasks to the slave node. It stores the resource usage
information for every slave node. It is responsible for distributing the workload. It also
helps you to track how the working load is used on cluster nodes. It helps you to place
the workload on resources which are available and accept the workload.

Etcd

• etcd components store configuration detail and wright values. It communicates with the
most component to receive commands and work. It also manages network rules and
port forwarding activity.

Worker/Slave nodes

• Worker nodes are another essential component which contains all the required services
to manage the networking between the containers, communicate with the master node,
which allows you to assign resources to the scheduled containers.

13
Master Node in K8s cluster

The master node is also known as a control plane that is responsible to manage
worker/slave nodes efficiently. They interact with the worker node to

• Schedule the pods

• Monitor the worker nodes/Pods
• Start/restart the pods
• Manage the new worker nodes joining the cluster

Master Node Processes:

Every master nodes in the K8s cluster runs the following key processes

• kube-apiserver
• kubectl: kube-controller-manager
• kube-scheduler
• etcd

Let’s look into each one of the processes in detail!

14
kube-apiserver:

It is the main gateway to access the k8s cluster and act as the main gatekeeper for client level
authentication or we can say that the kube-apiserver is the front end for the Kubernetes control
plane.

15
kube-scheduler in K8s Master Node:

Every time as a K8s admin/developer if you want to schedule a new pod on the worker node,
you need to send the request to the master API server which in turn will make a call to the
Kube-scheduler process. The scheduler here will intelligently decide on which worker node this
pod should be placed.

kube-controller-manager (Kubectl):

It is one of the critical processes in a master node that monitors the status of any worker node
level failures. It keeps a close watch over the event like crashing of any pods in the worker node
and, requests the scheduler to restart or reschedule any dead /failed pods, after detecting such
event.

These control manager component of master control planer has following types of controllers:

• Node controller: Responsible to respond when any worker node goes down
• Replication controller: It ensures that the request to maintain the correct replica count
of any pod deployment is always taken care
• Endpoints controller: Populates the Endpoints object viz. Joins, Services & Pods
• Service Account & Token controllers: Create default accounts and API access tokens for
new namespaces created in the worker node.

etcd in K8s Master Node:

etcd in the master control plane is responsible to store every kind of cluster-level change in the
form of key-value pairs.

It can be easily seen as a brain of the k8s cluster which keeps the log of every minute details of
changes occurring in the cluster.

For example, if any pod crashes in the worker node and it has to be rescheduled, the same gets
stored in etcd as key-value pair, also the event of pod rescheduling on the node is also logged
here.

Worker Node In K8s Cluster

As this node does the actual work required by the cluster administrator or developer, it is
known as worker nodes. Worker node can have one or more pods, these pods are your
abstraction of a containerized application. Every worker as shown in the fig:2.0 runs these 3
key processes

• Container Runtime
• kubelet
• kube-proxy

16
Container Runtime:

Every Microservice module(micro-app) you deploy is packaged into a single pod that has its
own container runtime. One needs to install a container runtime into each worker node in the
cluster so that Pods can run there.

Some of the container runtime examples are,

• containerd
• CRI-O
• Docker

kubelet

kubelet is a primary node-agent of the worker node, which interacts with both node and the
container in the given worker node.

The kubelet is responsible for

• Maintaining a set of pods, which are composed of one or more containers, on a local
system.
• For registering a node with a Kubernetes cluster, sending events and pod status, and
reporting resource utilization.

Within a Kubernetes cluster, the kubelet watches for PodSpecs via the Kubernetes API
server.
17
A PodSpec is a YAML or JSON object that describes a pod. The kubelet takes a set of
PodSpecs that are provided through various mechanisms (primarily through the API server)
and ensures that the containers described in those PodSpecs are running and healthy.

The Kubelet is the primary and most important controller in Kubernetes. It’s responsible for
driving the container execution layer, typically Docker.

Kube-proxy:

K8s cluster can have multiple worker nodes and each node has multiple pods running, so if one
has to access this pod, they can do so via Kube-proxy.

kube-proxy is a network proxy that runs on each node in your cluster, implementing part of
the Kubernetes Service concept.

In order to access the pod via k8s services, there are certain network policies, that allow network
communication to your Pods from network sessions inside or outside of your cluster. These
rules are handled via kube-proxy

A Kubernetes cluster consists of a single master node and potentially multiple corresponding
worker nodes. The master node exposes the API, schedules deployments, and generally
manages the cluster. The worker nodes can be responsible for container runtime, like Docker
or rkt, along with an agent that communicates with the master.

1.3 Basic Terms and Definitions

API Server (general) - Exposes the features of Kubernetes over an HTTPS REST interface.
All communication with Kubernetes goes through the API Server – even cluster components
communicate via the API Server.

Manifest file - YAML file that holds the configuration of one or more Kubernetes objects. For
example, a Service manifest file is typically a YAML file that holds the configuration of the
Service. When you post a manifest file to the API Server, its configuration is deployed to the
cluster.

Namespace (general) - A way to partition a single Kubernetes cluster into multiple virtual
clusters. Good for applying different quotas and access control policies on a single cluster. Not
suitable for strong workload isolation.

Replication Controllers - A replication controller is an object which defines a pod template.

It also controls parameters to scale identical replicas of Pod horizontally by increasing or
decreasing the number of running copies.

Replication Sets - Replication sets are an interaction on the replication controller design
with flexibility in how the controller recognizes the pods it is meant to manage. It replaces
replication controllers because of their higher replicate selection capability.
18
Deployments - Deployment is a common workload which can be directly created and
manage. Deployment use replication set as a building block which adds the feature of life cycle
management.

Stateful Sets - It is a specialized pod control which offers ordering and uniqueness. It is
mainly used to have fine-grained control, which you have a particular need regarding
deployment order, stable networking, and persistent data. For stateful apps and databases

Daemon Sets - Daemon sets are another specialized form of pod controller that runs a copy
of a pod on every node in the cluster. This type of pod controller is an effective method for
deploying pods that allows you to perform maintenance and offers services for the nodes
themselves.

Ingress - To manage external communication between nodes and internal pod level
communication

ConfigMaps - To manage the end-point URLs required by the pods/ DB’s

Secrets - To keep app-level passwords and secret keys securely using based64 encoding

Volume - For Persistent data storage

PersistentVolume (PV) - Kubernetes object used to map storage volumes on a cluster.

Storage resources must bemapped to PVs before they can be used by applications.

PersistentVolumeClaim (PVC) - Like a ticket/voucher that allows an app to use a PV.

Without a valid PVC, an app cannot use a PV. Combined with StorageClasses for dynamic
volume creation.

1.4 Kubernetes security main risks and challenges

While Kubernetes orchestrated containers are replacing some more traditional cloud-based
services because they are effective in running applications from one environment to another
(public cloud, private data center, etc.), they introduce several security challenges and risks that
make the task of managing them securely quite demanding.

In published the fourth edition of State of Container and Kubernetes Security Fall
2020 survey report where we examine how companies are adopting containers, Kubernetes,
and cloud-native technologies and addressing their security challenges. Two of the findings
include:

• 90% of respondents have experienced a security incident in their container and

Kubernetes environment in the last 12 months
• 44% of respondents said they’ve delayed application deployment due to security
concerns

19
In addition, we identified the most common types of security incidents reported by respondents
and the security risks that they are concerned about the most. Taken together, organizations
need to take the necessary steps to mitigate these risks so they don’t resort to delaying
application deployment.

Images and image registries, when misused, can pose security issues

Organizations need strong governance policies regarding how images are built and stored in
trusted image registries. You must ensure that container images are built using secure and
approved base images that are regularly scanned and ensure that only images from image
registries on allow lists are used to launch containers in your Kubernetes environment.

However, images are not always easy to trust from a security perspective. The images must be
signed and originate from a trusted registry to ensure high-quality protection. They also must
get properly vetted and the code validated. Otherwise, the images are vulnerable to
cyberthreats.

Organizations need a strong governance policy around using trusted image registries. Ensuring
that only images from whitelisted image registries are being pulled in the business environment
can be challenging and must be part of any container and Kubernetes security strategy along
with security best practices, such as vulnerability scanning.

Containers create both familiar and new runtime security challenges

One of the security advantages of containers and Kubernetes is they can be treated as
immutable infrastructure – what’s running should never be patched or changed but rather
destroyed and recreated from a common template when new updates are needed.

Other properties of containers pose unique challenges, including their ephemerality and the
speed at which they can be launched or removed.

And when a potential threat is detected in a running container, such as an active breach or a
new vulnerability, you must be able to not only kill that container and relaunch a non-
compromised version but also ensure that information is used to rebuild a new container image
or to reconfigure a component within the environment that remediates the root cause of the
issue.

Other runtime security risks include a compromised container running malicious processes.
Although crypto mining has become a popular objective for malicious actors who compromise
container environments, other malicious processes can also be executed from a compromised
container, such as network port scanning to look for open paths to attractive resources.

The rapid churn of containers makes it near impossible for humans to monitor which container
processes are running at any given time, let alone identify unnecessary or malicious processes.
And when a potential threat is detected in a running container, such as an active breach or

20
vulnerability, IT teams must be able to not only kill that container and replace it with a non-
compromised version but also integrate that information into your CI/CD pipeline to inform
future build and deploy cycles.

Lastly, you must secure your Kubernetes infrastructure and its components, including the
Kubernetes API server, etcd, etc. which increase the overall attack surface with unique threat
vectors of their own.

Kubernetes offers rich configuration options, but defaults are usually the least
secure

In keeping with DevOps principles, Kubernetes is designed to speed application deployment

and simplify management and operations. Kubernetes offers a rich set of controls that can be
used to effectively secure clusters and their applications.

Kubernetes network policies, for example, behave like firewall rules that control how pods
communicate with each other and other endpoints. When a network policy is associated with a
pod, that pod is allowed to communicate only with the assets defined in that network policy.
By default, Kubernetes does not apply a network policy to a pod, meaning every pod can talk to
every other pod in a Kubernetes environment.

Another configuration risk relates to secrets management: how sensitive data such as
credentials and keys are stored and accessed. You must ensure that secrets are not being passed
as environment variables but are instead mounted into read-only volumes in your containers,
for example.

Kubernetes offers rich configuration options, but default settings are usually the least secure.
Being in accordance with the DevOps principles, Kubernetes is designed to speed application
development, not to isolate its components. For example, the default network policies allow
every asset to talk to every other asset in a Kubernetes environment. Another configuration risk
relates to how secrets such as cryptographic keys are stored and accessed, a discipline called
secrets management.

Communication and Network Policies

Containers and pods need to communicate with each other within the deployment as well as to
other endpoints to accomplish their goals. If a container is breached, the attack surface is
directly related to how broadly it can communicate with other containers and pods. In a
sprawling container environment, implementing network segmentation can be prohibitively
difficult given the complexity of configuring such policies. Therefore, developing and
implementing network policies adhering to the least privilege principle might be challenging
but it is highly recommended. Network policies specify how groups of pods are allowed to
communicate with each other and other network endpoints. We can think of them as the
Kubernetes equivalent of a firewall. The goal of such policies should be to allow containers to

21
communicate to only those containers that are absolutely required in order to minimize the
attack surface C o n t a i n e r a n d K u b e r n e t e s S e c u r i t y R i s k s

More detailed review

1) Misconfigurations/exposures

The respondents identified exposures due to misconfigurations as the most worrisome security
risk in their container and Kubernetes environments, and for good reason: 67% of respondents
have detected a serious misconfiguration in the last 12 months.

Configuration management poses a uniquely difficult challenge for security practitioners,

particularly when using Kubernetes to orchestrate containerized apps. While a host of tools are
available for vulnerability scanning of container images, configuration management requires
more consideration. People know not to expose the Kubernetes dashboard to the Internet, but
configuring a pod’s security context or implementing Kubernetes RBAC are just two examples
of more challenging settings DevOps teams need to get right.

Pay close attention to how you’re configuring

• images - don’t use non-essential software (e.g, package managers, network tools and
clients like curl, or Unix shells) that increases your security risk nor pull images from
risky sources
• secrets - don’t bake in secrets into images or expose them unnecessarily; use a secrets
management tool like Kubernetes secrets and make sure deployments mount only the
secrets they actually need
• namespaces - use them, because they provide a key boundary for network policies and
Kubernetes access control restrictions, and separating workloads into namespaces can
help contain attacks and limit the impact of mistakes or destructive actions by
authorized users.
• runtime privileges - follow best practices that adhere to the principle of least privilege
• network policies - by default Kubernetes allows pods to talk to each other unimpeded;
network policies can be used as a key security control that prevents an attacker to move
laterally through a container environment
• persistent storage - make sure you have visibility into the use and configuration of
persistent storage as this is a rare persistent vector in a mostly ephemeral container
environment
• control-plane - if you’re self-managing your Kubernetes clusters, then configuring the
control plane components is critical because they make global decisions regarding a
cluster’s operations, and compromise of any control plane component could easily result
in complete compromise of a cluster

22
The best way to address these challenges is to automate configuration management as much as
possible, so that security tools – rather than humans – provide the guardrails that help
developers and DevOps teams configure containers and Kubernetes securely.

2) Vulnerabilities

We’ve seen several instances of serious vulnerabilities impacting containers and Kubernetes in
the recent past. Common exploits of known vulnerabilities include crypto mining or other
malware installation, and privilege escalation and host access. The problem is pervasive, with
Docker Hub at one point having to remove 17 backdoored images after they had been
downloaded 5 million times.

While image scanning at the build stage is a must, vulnerabilities pose a security risk to running
deployments as well. Effective vulnerability management spans the entire container lifecycle,
and should:

3) Runtime threats

The runtime phase is critical for container security because it presents a new set of security
challenges. If you’ve shifted security to the left and minimized your security risk from
vulnerabilities and misconfigurations, then the primary threat at runtime will likely come from
external adversaries. There are a few things you can do here to mitigate your biggest security
risks.

• monitor runtime activity - start with monitoring the most security-relevant

container activities such as process activity and network communications among
containerized services and between containerized services and external clients and
servers
• leverage the declarative data - use the build and deploy time information to
evaluate observed versus expected activity during runtime in order to detect suspicious
activity
• limit unnecessary network communication - runtime is when you can see what
kind of network traffic is allowed vs what’s actually needed for the application to
function, giving you the opportunity to remove unnecessary connections
• use process allow-lists - observe the application for a period of time to identify all
processes that are executed in the normal course of the application behavior, then use
this list as your allow list against future application behavior.

4) Failed compliance audit

Compliance is one of the primary drivers behind container and Kubernetes security initiatives,
and a failed compliance audit is usually due to security becoming an afterthought in the
container adoption journey. There are several compliance standards specific to containers and
Kubernetes that apply to all organizations, including:

23
 • CIS Benchmark for Docker
• CIS Benchmark for Kubernetes
• NIST SP 800-190

Industry specific compliance standards include PCI-DSS, HIPAA, and SOC 2. A common
mistake organization make is to wait until they’re in production before considering their
compliance requirements, or only focus at the runtime phase.

The best way to mitigate your risk from failing a compliance audit is to implement your security
controls as early as possible both in your container adoption journey as well as the container
life cycle. Automate your compliance checks and evidence reporting as much as possible to
reduce overhead.

In order to reduce the security risks from containers and Kubernetes, companies first need
visibility into their cloud-native environments. They need to understand how images are built
and whether they contain any vulnerabilities, how the workloads and infrastructure is
configured to operate, and where compliance gaps exist. With this information, Security and
DevOps can begin to enforce policies to reduce the security risk to an acceptable level. You can
get started with a free container security assessment to get a complete snapshot of your
container security risk.

1.5 The biggest Kubernetes security incidents

They were hardly alone, as this short list of major Kubernetes security incidents shows below.

1. Jenkins vulnerability exploitation and make $3 million by mining

Monero (2018)
To put a dollar value on what this kind of a hijack of resources is worth, around the same time
as the Tesla incident, hackers managed to exploit a vulnerability in Jenkins to cryptomine to
the tune of about $3.5 million, or 10,800 Monero in 18 months. Monero is the same
cryptocurrency involved with the malicious Docker images we mentioned earlier. In Docker’s
case, it was discovered that six malicious images had been collectively pulled over 2 million
times, that’s 2 million users potentially mining Monero for the enemy, quite a feat.

The Jenkins Kubernetes security incident is by far one of the most audacious breaches
discovered yet, in addition to the fact that it uses vulnerable Windows machines and personal
computers running Jenkins, it also targets Jenkins CI servers. This is a recent update as the
malware goes through a number of lifecycles where it keeps updating itself and changing
mining pools to avoid detection. The fact that it can target servers now is a step-up by the
attackers who are of Chinese origin. If they could pull over $3 million from beat-up desktops,
powerful servers are going to add a few zeros to that number, at least.

24
2. Tesla Cryptojacking Epidemic (2018)
With the value of cryptocurrencies skyrocketing, and limitless compute resources located in
the cloud, hijacking resources has become a lot more lucrative than stealing info. Automaker
Tesla was one of the earlier victims of cryptojacking when a Kubernetes cluster was
compromised due to an administrative console not being password protected. The discovery
was made by RedLock Cloud Security Intelligence and made public in a report stating the
misconfiguration had helped attackers get hold of Tesla’s AWS S3 bucket credentials. Those
credentials were then used to run a cryptomining script on a pod.

What was interesting about this attack was the number of “ingenuine” precautionary measures
taken to avoid detection. Not only did the attackers refrain from using a known mining pool,
and used an unlisted one instead, they also used popular CDN service Cloudflare to hide their
IP. The attackers also made sure that the mining script didn’t use enough CPU resources to
cause an alarm or get detected, and listened on a nonstandard port, making detection based on
port traffic virtually impossible. The only way to detect such a breach is to actively monitor
configurations to ensure all policies are being followed.

3. Leak of Capital One (2019)

This Kubernetes security incident was a big one, no pun intended, and the cause for a lot of
people to wake up and take notice. Occurring exactly one year ago, this breach saw 30GB of
credit application data affecting about 106 million people being exfiltrated. What actually
caused the breach is something we see quite a lot of in the world of Kubernetes, a
misconfiguration. In particular, a misconfigured firewall that allowed an attacker to query
internal metadata and gain credentials of an Amazon Web Services’ IAM role that had no
business being that “broad” in the first place.

One of the important lessons we learned from this incident is to be more careful while assigning
IAM roles. Most people are just in a hurry to get Kubernetes to work and often sidestep
important tasks like managing secrets and services, and assigning IAM roles on a per-pod basis
as opposed to per-application. Another important task is to “roll” credentials regularly,
preferably by an automated service that puts a cap on how long before credentials need to be
renewed. This also puts an upper limit on how long a breach can continue.

4. Docker Hub Attack (2020)

With Kubernetes, containers, and distributed environments, the attack surface is only getting
bigger exponentially, and you never know where an attack is going to come from. One example
is how attackers managed to plant malicious images in the Docker hub last year, causing
anyone who uses those images to be “cryptojacked.” What this means is users unknowingly
deployed cryptocurrency miners in the form of Docker containers that then diverted compute

25
resources toward mining cryptocurrency for the attacker. This is just one in a number of similar
attacks we see of late.

Similar to the Capital One breach, changing passwords and rolling credentials is a must to avoid
this sort of situation. Additionally, for Kubernetes environments, rotating your secrets and
auditing images to ensure only verified images are being used are key steps to ensuring security.
Malicious images can be pretty hard to detect, especially since a lot of the time, the containers
work as expected. This is why additional checks that highlight any deviations in application
behavior are necessary to ensure that no stowaway processes are running in the background.
This kind of attack is quite lucrative.

5. Microsoft Azure by Kubeflow Crypto-Mining Attack (2020)

Microsoft is another organization that’s been seeing a lot of cryptojacking woes of its own. After
disclosing that there was a large-scale cryptomining attack against Kubernetes cluster in Azure
in April this year, a similar campaign was uncovered in June that targets misconfigured
Kubeflow containers to turn them into cryptominers. Similar to the compromised image
situation with Docker hub, Kubeflow uses a number of services, which in turn allow users to
use custom images like Katib and Jupyter notebook server. In the case of Jupyter, the chosen
image doesn’t have to be a legitimate notebook image, and that is where the attackers found an
entry point.

If we look into what caused this misconfiguration, it’s pretty much what causes every
misconfiguration — impatience, laziness, and lack of knowledge. Kubflow’s UI dashboard is,
by default, only accessible internally through an Istio ingress gateway. Some users,
however, took a short cut to access the dashboard directly without going through the
Kubernetes API server and didn’t realize that while what they were doing was saving time, it
was also opening a number of backdoors in the process. In this case, they were exposing the
Istio ingress gateway to the internet, allowing anyone to access the dashboard. The moral of
the story here is that there are security implications to every setting or configuration change
that takes place.

26
 Chapter 2.
Concept of security

2.1 THE 4C'S of cloud native security

You can think about security in layers. The 4C's of Cloud Native security are Cloud, Clusters,
Containers, and Code. Each layer of the Cloud Native security model builds upon the next
outermost layer. The Code layer benefits from strong base (Cloud, Cluster, Container) security
layers. You cannot safeguard against poor security standards in the base layers by addressing
security at the Code level. Cloud

What is Kubernetes Security?

• Kubernetes security is based on the 4C’s of cloud native security: Cloud, Cluster,
Container, and Code:
• Cloud (or Corporate Datacenter/Colocation facility): The underlying physical
infrastructure is the basis of Kubernetes security. Whether the cluster is built on one’s
own datacenter or a cloud provider, basic cloud provider (or physical security) best
practices must be observed.
• Cluster: Securing a Kubernetes cluster involves both the configurable components such
as the Kubernetes API and security of all the applications that are part of the cluster.
Since most cloud-native applications are designed around microservices and APIs,
applications are only as secure as the weakest link in the chain of services that comprise
the entire application.
• Container: Container design best practices consist of: starting with the smallest code
base possible (excluding unnecessary libraries or functions), avoiding granting

27
 unnecessary privileges to users in the container, and ensuring that containers are
scanned for vulnerabilities at build time.
• Code: Code presents a major attack surface for any Kubernetes environment. Simple
policies such as encrypting TCP using TLS handshakes, not exposing unused ports,
scanning, and testing regularly can help prevent security issues from arising in a
production environment.

The main questions about Kubernetes security which you should ask
your engineers team
What are the top Kubernetes security vulnerabilities during BUILD?

• Code from untrusted registries

Untrusted code can include malware or backdoors that could unintentionally grant
access to bad actors.

• Bloated base images

Less is more for containerized applications, so developers should eliminate

unnecessary packages, libraries, and shells that could be compromised.

What are the top Kubernetes security vulnerabilities during DEPLOYMENT?

• Granting unnecessary privileges.

Wherever possible, keep privileges to a minimum and mount only the secrets that a
task requires to shrink the attack surface.

• Failure to isolate applications in the cluster

Namespaces should be used to keep resources and teams separate from each other.

• Lateral motion within the cluster

Use policies that segment the network to prevent lateral movement of an attack
within the cluster.

• Unauthorized access

Ensure role-based access controls (RBAC) are properly configured to limit access.

What are the top Kubernetes security vulnerabilities during RUNTIME?

• Infrastructure attacks.

28
 During runtime, Kubernetes infrastructure elements including the API server, etcd,
and controllers all present their own attack surfaces.

• Complexity.

The ongoing health of a Kubernetes cluster has many moving parts. Compromised
containers must be quickly isolated, stopped, and replaced with healthy ones while
the source of the attack is located and remediated.

29
Another random document with
no related content on Scribd:
"I only began it in May," he said, "during a fit of insomnia. My mind
was full of scorpions, like Macbeth's, and I think I should have gone
mad if I had not summoned those shadows from the unseen world,
and set myself to anatomize them. It is a bitter book, a story of Fate's
worst irony; and in a better period of English literature—in the day of
Scott, or Dickens and Thackeray—it would have stood no chance of
being widely read. But we have changed all that. This is the day of
cruel books. Most of us have turned our pens into scalpels. And I
think this story of mine is cruel enough to hit the public taste."
"There is nothing that touches your life or mine in it?" Grace asked,
with a touch of alarm.
"No, no, no; not one thought. I wrote it while I was trying to forget
you—and trying still harder to forget myself. The shadows that move
in it bear not the faintest resemblance to you or me. It is a sordid
book, a study of human meanness, and the misery that dull minds
make for themselves: pale-grey miseries that gradually draw to a
focus and deepen to blood-red tragedy. But it has one redeeming
feature—one really good man—a city missionary, humbly born, plain,
self-educated, but a Christ-like character. I should have burnt the
book unfinished but for him. He came to my relief when my story and
I were sinking into a slough of despond."
"You talk as if the web were not of your weaving, as if you had no
power over the figures that move in it."
"I have no such power, Grace. They come to me as mysteriously as
the shadows in a dream, and their spell is strong. I cannot create
them; and I cannot change them."
She wanted him to read his story to her before it was printed; but this
was just the one thing he could not do. He could not imagine himself
reading his own words.
"It would make me hate my work," he said. "Every clumsy phrase,
every banal word, would leap out of the page and gibber at me as I
read. I will bring you the first copy fresh from the press, and when
you have read it you shall tell me afterwards whether I am ever to
write another story."
"You shall write another, and another, and go on writing," she
answered gaily. "You will give me a second world, a world peopled
with strange or lovely creatures—villains as colossal as Milton's
Satan, heroines as innocent as his Eve. My life in the world of your
imagining will be almost as intense as your own. You will give me a
second existence, better than the everyday world. You will tell me
about your dream-people, won't you, Arthur, as they spring into life?"
"The fear is that I shan't be able to refrain from talking of them, to the
other half of my soul."
"You cannot weary the other half by much talking."
"Do you think not? I can imagine a husband's art becoming an
unspeakable bore to his wife."
"Not if she loves him and loves his art."
"Ah, there's the rub."
Lady Perivale was recalled from the shadow-world of the novelist by
the substantial apparition of John Faunce, who arrived unannounced
on a sultry afternoon, and found her sitting in the garden with Mr.
Haldane and Miss Rodney, at a table strewn with all the new
magazines and some of the old poets, in those miniature editions
that so lend themselves to being carried about and not read.
"I thought I might venture to call without notice," said Faunce, "as I
have some rather important news for your ladyship."
"Indeed!"
"A libel—a most audacious libel," said Faunce, taking a paper from
his pocket.
"Where? where? What paper?" Grace and Sue exclaimed excitedly.
"Strange to say, in a society paper of most respectable character,
though of a somewhat limited circulation," replied Faunce; "a paper
which, to my knowledge, has never offended in this manner until now
—the Bon Ton and Cricket Review, a journal printed at Kennington,
and mostly circulated in the South of London."
He handed the paper to Lady Perivale, who turned the leaves
hurriedly, too agitated to read a line for the first few minutes.
It was an eminently proper paper—a paper that told of dances at
Tooting, private theatricals at Norwood, and At Homes at Tulse Hill, a
paper that described dresses and millinery, and gave receipts for
cornflower creams and jellies made without wine, for cleaning kid
gloves and making golden hair-dye. Pages were devoted to the
Oval, and other pages to school cricket. There was the usual short
story of the ultra-smart world. There was a Denmark Hill celebrity at
home. There was everything nice and proper that a Society paper
should have; and there, amidst all this respectability—like a hideous
wen upon a handsome face—appeared three atrocious paragraphs
about Lady Perivale's tête-à-tête tour with Colonel Rannock; the first
setting forth the surprise of the lady's friends on meeting her
travelling alone with a man of dubious character; the second
debating whether the freedom of fin-de-siècle manners would not
permit of any lady travelling with any gentleman without causing
scandal; the third, of a somewhat grosser tone, winding up with a
couplet from Pope:
"Nor Cæsar's empress would I deign to prove,
No, make me mistress to the man I love."
"It's abominable!" cried Grace, flushing crimson, and throwing down
the paper in a rage.
"And you tell me I'm not to horsewhip the scoundrel who wrote that!"
said Haldane, who had read the paragraphs over her shoulder.
"I do—most decidedly," answered Faunce, edging away from him
with an involuntary movement. "We wanted a libel—a gross libel—
and we've got it. We are going to bring an action against the
proprietor of the Bon Ton, but we are not going to put ourselves in
the wrong by assaulting him first. No, sir, we shall proceed against
the proprietor, editor, and printer of the Bon Ton, and we shall ask for
exemplary damages."
"Damages!" exclaimed Grace. "Do you suppose I want the
loathsome creature's money?"
"Why not make it a criminal suit, and send him to prison?" asked
Haldane.
"I think not, sir. Her ladyship's solicitors, Messrs. Harding, have gone
into the matter with me, and we are agreed that a criminal action is
not advisable."
"How does this thing happen to appear so long after the circulation
of the scandal?"
"Ah! that's the question," said Faunce, blandly. "You see, fashionable
gossip takes a considerable time to cross the Thames and filter
down to Tooting. The proprietor—and editor—lives at Tooting, and I
dare say, to his mind, the slander appeared a novelty. I'm glad he
didn't get hold of it sooner, for we should not have been prepared to
deal with the case as we are now."
Miss Rodney had picked up the Bon Ton, and was reading the
paragraphs with a frowning brow.
"How can you look at that atrocious stuff?" cried Grace, snatching
the paper from her and rolling it into a ball for her poodle, who
rushed across the lawn with it and then laid himself down and
proceeded to tear it into shreds with his paws and teeth.
"It's lucky that isn't the only copy in existence, Lady Perivale," said
Faunce.

CHAPTER XIII.
"They draw a nourishment
Out of defamings, grow upon disgraces;
And, when they see a virtue fortified
Strongly above the battery of their tongues,
Oh, how they cast to sink it!"
One of the most interesting cases in the Law Courts that winter was
Perivale v. Brown Smith, a claim of £10,000 damages on account of
a gross libel published in a paper of which the defendant was editor
and proprietor.
Brown Smith pleaded justification, and it was said that he was going
to make a good fight, and that he would produce witnesses who had
met the lady and gentleman on their travels as Mr. and Mrs. Randall.
The case came on late in November, when there were a good many
people in town, staying for the weeks before Christmas, or passing
through; and the court was packed with smart clothes and well-
known faces. Conspicuous among these curious impertinents were
two well-known figures in the little world of Belgravia and Mayfair:
Lady Morningside, whose ample person, clothed in black satin and
chinchilla, filled a considerable space on the privileged seats; and
the spare and wiry form of "the most honourable," her husband, a
man whose weather-beaten countenance, trim whiskers and keen
eye, cut-away coat and Bedford cords, indicated the indomitable
sportsman.
Eye-glasses and opera-glasses glittered across the fog, and the
point to which they were chiefly directed was the figure of Lady
Perivale, in a neat black gown, with cape and toque of Russian
sable, seated in the well of the court, with Arthur Haldane sitting
beside her.
There was much whispering among the eye-glasses about the lady
and her companion.
"She is as handsome as ever," said one; "I was told she had gone off
dreadfully. Rather audacious to bring this action, ain't it?"
"Rather a dangerous move, I should think."
"Oh! she's got Sir Joseph Jalland. He always wins when there's a
pretty woman to orate about. You'll see, he'll make the jury shed
tears."
"What odds will you give me against that fat man in the corner being
the first to weep?"
"Hush! It's going to begin."
Mr. Waltham, Sir Joseph's junior, opened the pleadings in an
undertone, which sent all the picture-hats distracted. They thought
they were losing the fun. And then a thrill ran round the Court as Sir
Joseph Jalland rose in his might, adjusted his pince-nez, trifled with
the leaves of his brief, and then slowly began to unfold his case. The
deep, grave voice made all the aigrets shiver, and every lorgnette
and binocular was turned to him.
"This greatly injured lady—this lady, whose life of blameless purity,
life spent in an exalted sphere—in the sheltered haven of a
congenial marriage, this lady whose spotless character should have
shielded her from the lightest breath of slander, has been made a
target for the salaried traducer of a venomous rag that calls itself a
newspaper, and has been allowed to drivel its poisonous paragraphs
week after week, secure in its insignificance, and a disgrace to the
Press to which it pretends to belong," flinging down the South
London Bon Ton on the desk before him, with a movement of
unutterable loathing, as if his hand recoiled instinctively from the foul
contact. "She has been made the subject of a slander so futile, so
preposterous, that one marvels less at the malice of the writer than
at his imbecility. A woman of gentle birth and exalted position,
hemmed round and protected by all those ceremonial ramparts that
are at once the restraint and privilege of wealth and social status, is
supposed to have roamed the Continent with her paramour, braving
public opinion with the brazen hardihood of the trained courtesan."
This and much more, in its proper place and sequence, did Sir
Joseph's deep voice give to the listening ears of the Court, before he
summoned his first witness, in the person of the plaintiff, Grace
Perivale.
Her evidence was given in a steady voice and with perfect self-
control.
"Did you ever travel on the Continent with Colonel Rannock?"
"Never."
"Were you in Corsica in the January of this year?"
"No."
"Or in Algiers in February?"
"No."
"Will you be so good as to say where you were living during January
and February last?"
"I was at my villa near Porto Maurizio from November last year until
the beginning of April in this year."
Sir Joseph had no more questions to ask. The defendant's counsel
exercised his right to cross-examine the witness, who stood facing
the Court, calm and proud, but deadly pale.
"Were these paragraphs in the—er"—looking at his brief—"Bon Ton,
the first you had heard of a scandal associating your name with
Colonel Rannock's, Lady Perivale?" he asked blandly.
"It was the first time such a scandal had appeared in print."
"But the scandal was not unfamiliar to you?"
"No."
"You had heard of it before?"
"Yes."
"On several occasions?"
"I was told that such a thing had been said."
"And that your friends believed it?"
"Not one!" the witness answered indignantly. "No friend of mine
believed one word of the story!"
She flushed and paled again as she spoke. She shot one involuntary
glance towards the man who was so much more than a friend, and
who had almost believed that slander.
"You will admit, I think, Lady Perivale, that the story had been
common talk for a long time before this society journal got hold of
it?"
"I know nothing about common talk."
"That will do, Lady Perivale," said the counsel.
Lady Perivale's butler and maid were the next witnesses.
They had been with their mistress at Porto Maurizio from November
to April, during which period she had never been absent from the
villa for twenty-four hours.
The defendant's counsel cross-examined both witnesses, and made
a praiseworthy—but unsuccessful—attempt to cast ridicule and
doubt upon the two old servants, whom he tried hard to place before
the jury as overpaid and venal hirelings, willing to perjure themselves
to any extent for their employer. He gratified his professional vanity
by letting off two or three forensic bon-mots, and succeeded in
raising a laugh or two at the expense of the country-bred Abigail and
the dignified London butler; but the endeavour to weaken their
testimony was an ignominious failure.
"That, my lord, would complete my case," said Sir Joseph Jalland,
"were it not essential that the falsehood and the folly of the slander in
this scurrilous rag," striking the Bon Ton with his open hand, "should
be stamped out at once and for ever; and in order that this may be
effectually done—to prove indubitably that Lady Perivale was not
with Colonel Rannock during his Continental wanderings last winter,
I shall produce the person who was with him."
Miss Kate Delmaine stepped into the box, admirably dressed, like
Lady Perivale, in a black cloth gown, and wearing a sable toque
almost of the same fashion. A murmur of surprise ran round the
Court, an excited whispering and twittering, which the usher
hastened to suppress.
Seen in that November gloom, the witness looked like Grace
Perivale's double.
Kate Delmaine! There were some among the wigs and gowns, and
some among the smart audience who remembered her in her brief
career, a girl of startling beauty, whose dazzling smile had beamed
across the footlights at the Spectacular Theatre for a season or two.
They had seen, admired, and forgotten her. She rose before them
like the ghost of their youth.
"Will you tell me where you were living last February, Miss
Delmaine?" Sir Joseph began quietly, when her carmine lips had
hovered over the Book: "from the 7th to the 25th?"
"I was at the Mecca Hotel, in Algiers."
"Alone?"
"No. Colonel Rannock was with me."
"You were in Corsica and in Sardinia before that, I believe?"
"Yes."
"Also with Colonel Rannock?"
"Yes."
"In what capacity were you travelling with him?"
The phrase produced a faint titter, and the younger of the smart
young ladies became suddenly occupied with their muffs and lace
handkerchiefs.
"We were travelling as Mr. and Mrs. Randall, if that's what you want
to know!" Miss Delmaine replied, with a look that challenged the
Court to think the worst of her.
"That is precisely what I want to know. You were going about with
Colonel Rannock as his wife—under the nom de guerre of Randall?"
"Yes."
"Good! Pray, Miss Delmaine, can you tell me where Colonel
Rannock is at this present time?"
The witness had given her evidence in an agitated and angry
manner from the beginning. The bloom on her cheeks was hectic,
and not rouge, as the smart young women thought. Her eyes were
unnaturally bright, splendid eyes, that flashed angry fire. She had
stood up boldly in her place, defying the world's contempt; but it
seemed as if the effort had been too much for her. She looked
distractedly round the court, turned white as ashes, and fell in a dead
faint, before she had answered the counsel's question, which was
irrelevant, and might not have been allowed.
There was the usual rush with glasses of water and smelling-salts,
and the witness was carried out of court.
The Court then adjourned for luncheon. The picture-hats all waited,
sniffed salts and eau de Cologne, nibbled chocolates, hungry, and
yawning for want of air, but determined to see it out.
There was bitter disappointment for the curious impertinents when,
on the judge returning to his seat, Sir Joseph Jalland informed his
lordship that Mr. Brown Smith had offered an ample apology for the
offensive article in his paper, and that his client had no desire to
continue the action in a vindictive manner.
The judge highly approved of this course.
"If Lady Perivale brought this action in order to clear her character of
a most unmerited aspersion, she has been completely successful,
and can afford to be lenient," said his lordship, with feeling.
The defendant was to publish his apology, both in his own paper and
such other papers as Sir Joseph should name. He was to destroy
every number of his paper still unsold, and to call in any numbers
remaining in the hands of the retail trade, and was further to give one
hundred guineas to any charitable institution selected by the plaintiff.
Only to Lady Perivale's solicitors and to Mr. Faunce was it known
that the defendant would not be out of pocket either by this hundred
guineas, or for the costs of the action, against which a considerable
sum had been paid into his banking account by Mr. Faunce, before
the libel—written by that very Faunce, in collaboration with one of the
ladies who did the Bon Ton gossip—appeared in Mr. Brown Smith's
popular journal.
Faunce had said there would be a libel when it was wanted, and
Faunce, who was an old friend of Brown Smith's, had produced the
libel. Nobody was any the worse, and Society was deeply humiliated
at discovering how cruelly it had misjudged a charming member of
its own privileged body. Lady Morningside and her husband made
their way to Lady Perivale directly the judge left his seat, and the old
Marquis, with an old-fashioned gallantry that recalled "Cupid"
Palmerston, bent over Grace's ungloved hand and kissed it: a
demonstration that thrilled the smart hats and eye-glasses.
Cards and letters of friendly congratulation poured in upon Lady
Perivale at Grosvenor Square that evening—letters from the people
who had cut her, making believe that the aloofness had been all on
her side.
"And now, dear, after this plucky assertion of yourself, I hope you are
not going to shut yourself from your old friends any more. It has been
so sad to see No. 101 empty all the season, and not even to know
where you were to be found," concluded one of those false friends.
Grace flung the letters into her waste-paper basket with angry scorn.
"To think people can dare to pretend they did not know I was in town,
when I drove in the park nearly every day!" she exclaimed.
"I hope you are satisfied, madam," said Faunce, when he called
upon Lady Perivale the day after the trial.
No one had seen Faunce in court, though Faunce had seen and
heard all that happened there. His work had been finished before the
case came on, and the family solicitors in Bedford Row took all the
credit of the successful result, and congratulated Lady Perivale upon
their acumen in retaining Sir Joseph Jalland.
"I hope you are satisfied, madam," Faunce said modestly, when he
called in Grosvenor Square, in response to Lady Perivale's request.
"I am more than satisfied with your cleverness in bringing the
wretched business to an issue," she said; "and now all I hope is that
I may be able to forget it, and that I shall never hear Colonel
Rannock's name again."
"I hope you will not, madam—not in any unpleasant connection,"
Faunce answered gravely.
"I must refer you for your professional charges to my lawyers, Mr.
Faunce," pursued Grace. "But I must beg you to accept the enclosed
as a token of my sincere gratitude for the trouble you have taken,
and as a souvenir of your success." She handed him an envelope.
"I assure you, Lady Perivale, I do not require anything beyond the
ordinary payment for my time and trouble."
"Oh, but you must take this, to please me," she answered. "I want
you to remember that I value your services at more than their
professional price."
She gave him her hand at parting, as she had given it at the end of
their first interview, and he thought more of that cordial handshake
than of her present, which he found to be a cheque for £500.
In the third week in December there was a very quiet wedding at St.
George's, Hanover Square, a marriage which was celebrated at half-
past eight o'clock in the morning, and at which the only witnesses
were Susan Rodney and Mr. George Howard, newly returned from
Pekin—a wedding so early and so quiet as to escape the most
invincible of the society paragraphists, the insatiable pens that had
been writing about this very marriage as an imminent event.
The bride's dark-grey cloth gown, sable-bordered travelling-cloak,
and black chip hat offered no suggestion of wedding raiment. The
breakfast was a parti carré in the dining-room at Grosvenor Square;
and the married lovers were able to leave Charing Cross at eleven
by the Continental Express without provoking any more notice from
the crowd than the appearance of a beautiful woman, perfectly
dressed for the business in hand, and leading the most perfect thing
in brown poodles, must inevitably attract. The honeymooners were
established at their hotel in Cairo before the paragraphists had wind
of the marriage.

CHAPTER XIV.
"But now with lights reverse the old hours retire,
And the last hour is shod with fire from hell.
This is the end of every man's desire."
During the four months which had elapsed since Faunce's first visit
to Kate Delmaine, alias Mrs. Randall, the detective had contrived to
keep an observant eye upon the lady; but he had not succeeded in
arriving on a more friendly footing with her, although he had obliged
her on several occasions with a small advance on account of the
promised reward.
He had called three or four times at the lodging-house in the dingy
street near the Thames, and she had received him civilly. He had
detected a lurking anxiety under the assumed lightness of her
manner—a carking care, that seemed to him of some deeper nature
than the need of money, or the sense of having fallen upon evil days.
He would not have been surprised to see her depressed and out of
spirits; but he was at a loss to understand that ever-present anxiety,
and that nervous irritability which seemed allied with fear.
He remarked to her, in a friendly way, on the state of her nerves, and
advised her to see a doctor. He urged her to live well, and to take the
utmost care of herself, to which end he was liberal with those ten-
pound notes on account.
"I want you to look your best when you appear in court," he said, "to
show that you are every bit as handsome a woman as Lady
Perivale."
"He always said I was," she answered, with a sigh.
"Colonel Rannock? He knew and admired you before he ever saw
Lady Perivale, didn't he, now?" asked Faunce, who, for reasons of
his own, was very anxious to make her talk of Rannock; but she
answered curtly—
"Whether he did or whether he didn't, it's no business of yours."
The gloomy look had come back to her face; and Faunce was more
and more convinced that, whatever her anxiety was, it was in some
way connected with Colonel Rannock.
He had brought Rannock's name into the conversation whenever he
could, and with an artful persistence, and the name had always a
depressing influence. She spoke of him reluctantly, and she seldom
spoke of him dry-eyed. Once she spoke of him in a past tense. It
could be no common fate that had left such aching memories.
Without actually "shadowing" the lady during this interval, he had
contrived to keep acquainted with her movements and associations,
and he had discovered that almost her only visitor was the man
whom he had seen on that first day—the man who had opened the
door, glanced into the room, and hurried away at sight of a stranger.
Even this person was not a frequent visitor, but he called at irregular
hours, which indicated a friendly footing.
It had not taken Faunce very long to identify this person as an
individual well-known to the patrons of the prize-ring—a pugilist
called Bolisco, who had been one of Sir Hubert Withernsea's
protégés, and had often sat at meat and drink in the very much
mixed society in the Abbey Road. Bolisco had been at the zenith of
his renown ten years ago, when Withernsea was burning that brief
candle of his days which had guttered into the grave before he was
thirty; but the pugilist's reputation had considerably declined since
then. He had been beaten ignominiously in three or four public
encounters, had seen his star go down before younger and steadier
men, and was no longer good for anything better than a glove-fight
at a second-rate tavern. One of those glove-fights had ended fatally
for Bolisco's opponent; and there had been some among the
lookers-on who accused him of brutal roughness towards a weaker
man, which had resulted in death. No blame had attached to Bolisco
in the opinion of the coroner's jury; but the patrons of the Fancy had
given him the cold shoulder since that unlucky accident, which had
happened more than a year ago.
In the course of that semi-shadowing Faunce had found out some
details of Kate Delmaine's life during the last half-year. He found that
she had occupied the shabby first-floor in Selburne Street since the
beginning of March, that she had come there straight from "abroad,"
and that her trunks were covered with foreign labels—Ajaccio,
Algiers, Marseilles, Paris, Calais. She had arrived with a great load
of personal luggage, fine clothes, and other portable property, the
greater part of which had been gradually made away with. She
would go out in a cab with a large cardboard box, and come home
half an hour afterwards on foot, having left box and contents at a
pawnbroker's in the King's Road.
Betsy, the sixteen-year-old maid-of-all-work, from whom Faunce
derived most of his information, had been a close observer of the
first-floor lodger, and was pleased to impart her knowledge and her
impressions to the amiable Faunce.
Mrs. Randall was very down-hearted, Betsy told him, and would sit
and cry for the hour together. Did she drink? Well, only a brandy-
and-soda now and then, but she used to stick a needle into her arm
that made her sleepy, and she would lie on the sofa all the afternoon
and evening sometimes, like a dead thing. The girl had heard her
moan and groan in her sleep when she took her a cup of tea in the
morning, and she would wake with a frightened look, and stare about
her "wild-like," as if she didn't know where she was.
Had she many visitors?
None, except the dark gentleman with the broken nose; and he did
not come very often, or stay long. They had words sometimes—very
high words—and once, in one of their quarrels, she went into
hysterics, and was "regular bad," and screamed at him like a lunatic.
The missus had been obliged to go upstairs to her, and tell her she
wouldn't stand such goings-on any longer. She'd have to clear out if
she couldn't behave like a lady.
All this to hear did Mr. Faunce seriously incline; and he now began to
do a little shadowing on Mr. Bolisco's account.
He knew that in all probability he was wasting his time; but the old
hunter's instinct of the Scotland Yard days was upon him, and he
wanted to know what ailed Kate Delmaine over and above the
natural depression of a woman of her class out of luck.
He had provided for her comfort, had been to her as a guardian
angel, as the time for her appearance in Court drew near. He had
advised her how to dress the part, and had ascertained what Lady
Perivale was going to wear, in order that Mrs. Randall's costume
should in some degree resemble hers. He had gone to Regent
Street on the day before the case came on, and bought a fur toque,
after the fashion of Lady Perivale's sable.
"It is only a paltry bit of skunk," Mrs. Randall declared
contemptuously, after she had blown the fur about and examined it
with a depreciatory scrutiny; but when she put it on before the cloudy
looking-glass in her parlour she owned to being pleased with herself.
"I wonder if you believe I was once a handsome woman," she said to
Faunce.
"I know you are a handsome woman now, and that you've only to
take a little more care of yourself to be as handsome as ever you
were," he answered gravely, being a kind-hearted man and really
sorry for her.
"That's skittles!" she answered. "I've come to the end of my tether.
I've nothing to live for, and I'm sick of wishing I was dead, for it don't
come off. And I don't want to kill myself; that's too cheap. I hate the
idea of an inquest, and 'The deceased was once known as this,' and
'The deceased was once t'other.' I'm a lady, Mr. Faunce, and I loathe
being magged about in the newspapers."

Now that Lady Perivale's action had ceased to be a nine days'

wonder, and the lady herself was a happy wife, travelling by easy
stages towards the land of ancient monuments and modern
amusements, pyramids and golf links, Sphinxes and croquet,
colossal sepulchres of unknown Pharaohs, and monster hotels with
unknown tariffs; now that he had accomplished his task and had
been handsomely rewarded, it might seem that John Faunce's
interest in Grace Perivale's double would cease and determine.
Strange to say that interest grew rather than diminished, and he
contrived to see his little friend, the lodging-house slavey, once or
twice a week, and so to be informed of all Mrs. Randall's
proceedings; indeed, his love of detail led him to ask Betsy for an old
blotting-book of the first-floor lodger's, which had been flung upon
the dustheap, and which the girl had retrieved from that foul
receptacle for the sake of its picture cover.
"Most people collect something," he told Betsy; "my fancy is old
blotting-paper."
"Well, I never did!" exclaimed the damsel. "I know many as collecks
postage-stamps, but I never heerd as blotting-paper was valuable!"
"It is, Betsy—sometimes," in token of which Mr. Faunce gave her a
crown piece for the ragged book, with its inky impress of Mrs.
Randall's sprawling penmanship.
Faunce had paid his witness the balance of the promised reward,
£120, in bank-notes, the evening after the trial, and he was prepared
to hear she had taken wing.
Surely with a sum of money in hand she would leave that dismal
street, and hurry away to some more attractive locality. To Paris,
perhaps, to buy fine clothes, and flaunt her recovered beauty in the
Bois; or to Monte Carlo, to try her luck at the tables. It was in the
character of such a woman to squander her last hundred pounds as
freely as if she had an unlimited capital behind it.
She had talked of leaving her lodgings, the little handmaiden Betsy
told him, but it hadn't come off. She had given a week's notice, and
then had cut up rough when the missus took a lady and gentleman
to look at the rooms. She wasn't going to be chucked out like a stray
dog, she'd go when she wanted, and not before.
"I don't believe she'll never go," Betsy said, with a wise air. "She ain't
got it in her to make up her mind about nothink. She sits in the easy-
chair all day, smokin' cigarettes and readin' a novel, or lays on the
sofa, and seems only half awake. And of a evening she gets dreadful
low. She says she hates the house, and won't sleep another night in
it, and yet when morning comes she don't offer to go. And then,
she's that under his thumb that if he say she's not to leave, go she
won't."
"You mean the dark gentleman?" said Faunce.
"Of course I do. There ain't any other as I knows on."
"Do you think she is—attached—to the dark gentleman?"
"I know she's afraid of him. I've seen her turn white at his step on the
stairs, and she's always upset after he's been to our place, and sits
and cries as if her heart was breaking. There, I do feel sorry for her!
She's a real good sort. She give me this here hat," added the slavey,
tossing her beplumed and bejewelled head. "It was bought in the
Harcade, and it ain't been worn above half a dozen times, only the
sea-water damaged it a bit when she was travelling."
So sincere and deep-rooted was Faunce's interest in Mrs. Randall,
that he took considerable pains to follow the movements of her friend
Mr. Bolisco, whom he tracked to his lair in a sporting public-house at
Battersea—an old, tumble-down building in a shabby street close to
the river, a house that had once been a respectable roadside inn,
and had once been in the country.
Faunce took some note of the famous prize-fighter's habits, which
were idle and dissolute, and of his associates, who belonged to the
lowest order, the ragged fringe of rascality that hangs upon the edge
of the sporting world. It was sad to think that so disreputable an
acquaintance could dominate the life of a fine, high-spirited creature
like Kate Delmaine. But, much as he was interested in a beautiful
woman, who was travelling on that dismal journey which is called
"going to the dogs," Mr. Faunce felt that his evening walks with
Betsy, and his occasional look in at the Gamecock at Battersea—
that sporting rendezvous where Mr. Bolisco had his "diggings,"—
were so much dilettante trifling, and mere waste of time. His work in
relation to Kate Delmaine was finished; and whatever mystery there
might be in her life, mystery involving even a crime, it was no
business of his to investigate it.
Somewhat reluctantly, therefore, like a baffled hunter who turns from
the dubious trail of the beast he has been pursuing, Mr. Faunce
discontinued his visits to Chelsea, and went no more, in his
character of a well-to-do idler interested in the prize-ring, to the
public-house across Battersea Bridge.
"I must be getting a regular amateur," he told himself, "if I can't have
done with a case when my work is finished."
Christmas came as a pleasant diversion, and during that jovial
season Faunce deserted his rooms in Essex Street, forgot that he
was a detective, and remembered that he was a citizen and a
husband. The turkey and beef, the pudding and mince-pies did credit
to Mrs. Faunce's judgment, and the skill of an unpretentious cook,
who did not scorn to bare her robust arms to the elbow, and
hearthstone the doorstep before she fried the morning rasher.
The catering had been Mr. Faunce's own work. It was his falcon
glance that had detected the finest Norfolk turkey in a row of
eighteen-pounders, the ripest York ham out of a score of good ones.
The champagne which he bought for his guests, the ten-year-old
Scotch whisky which he drank himself, were all of the best, and the
villa at Putney had the air of plenteous comfort in a small space
which pervades a well-found ship.
With his wife sitting opposite to him, and an old friend on either side,
Faunce enjoyed the harmless pleasures of social intercourse, and
cleared his mind of crime and mystery, and did not go back to his
office in Essex Street until the general holiday was over, and the
flavour of Christmas had faded out of the atmosphere.
It was on the day after his return to everyday life that Faunce
received a message from Scotland Yard, bidding him go there
immediately on important business, a summons that he made haste
to obey, since many of those cases which had afforded him
profitable occupation within the last few years had come to him by
the recommendation of his old chiefs in the Criminal Investigation
Department.
He found one of those chiefs seated in his private room, engaged in
conversation with a short, stout gentleman of middle age and
pleasing countenance, who looked like a soldier—fair-haired,
intelligent, and fussy.
"This is Mr. Faunce, Major Towgood," said the chief.
"Delighted to make your acquaintance, Faunce!" exclaimed the
Major, in a breathless way, bouncing up from his chair, wanting to
shake hands with Faunce, and suppressing the desire with a
backward jerk; "and if Mr. Faunce," turning to the chief, "can do
anything to set my poor mother-in-law's mind at rest about that
scapegrace of hers, I shall be very grateful—on my wife's account,
don't you know. Personally, I shouldn't be sorry to know he had gone
under for good."
"Major Towgood is interested in the fate of a Colonel Rannock, his
connection by marriage, who has not been heard of for some time."
"Not since last March, early in the month—sold his sticks—and
started for the Yukon River," interjected Major Towgood again
breathlessly, and with his eyes opened very wide.
"Colonel Rannock's disappearance—if it can be called a
disappearance—has caused considerable anxiety to his widowed
mother——"
"Women are such forgiving creatures, don't you know," interrupted
the Major. "Talk of seventy times seven! There ain't any combination
of figures that will express a mother's forgiveness of a prodigal son."
"And I have told Major Towgood," pursued the chief, with a shade of
weariness, "that I can highly recommend you for an inquiry of that
sort, and that if Colonel Rannock is to be found above ground—or
under ground—you will find him."
"I'll do my best, sir."
"And now, my dear Towgood, I don't think I can do any more for you."
Major Towgood jumped up and bustled towards the door. But he
wasn't gone yet. His gratitude was overpowering; and the chief had
to back him out of the room, politely, but decisively.
"You are just the man we want, Faunce," said the Major, as they
walked down a long corridor that led to the staircase. "Your Chief has
told me all about you—you were in the Bank of England case, he
said, and the Lady Kingsbury case—and—ever so many more
sensation trials—and now you're on your own hook—which just suits