Sans Continuous Diagnostics and Mitigation Evolving Federal Defenses With Cost Effective and Maintainable Data Integration Solutions

Continuous Diagnostics and Mitigation: Evolving
Federal Defenses with Cost-Effective and

Maintainable Data Integration Solutions
GIAC (GCDA) Gold Certification
Author: Andrew Davidow, andrew.davidow@student.sans.edu

Advisor: Dr. Tim Proffitt
Accepted: August 15, 2022
Abstract
Civilian federal agencies have struggled to implement the Continuous Diagnostics and
Mitigation program over the past decade. Billions of dollars have been spent, and the
cybersecurity tools have been deployed. Yet, there are challenges in getting the data from
those tools into the new CDM Dashboard powered by Elasticsearch and Kibana. Filebeat
and Logstash can solve this problem. The data from two CDM tools, HCL BigFix and
Tenable.sc were collected using Filebeat and Logstash and stored in Elasticsearch. This
approach is simple, maintainable, feasible, and cost-effective.
CDM: DON’T MISS A BEAT 2
1. Introduction
Americans have reportedly lost over six point nine billion dollars in the last
calendar year due to cybercrime (FBI, 2021). Over the past five years, the combined
losses have been $18.7 billion, with 2.76 million complaints to the Internet Crime
Complaint Center (FBI, 2021). Criminal attacks like fraud, scams, and theft have become
commonplace as “…the threats in the digital world mirror the threats in the physical
world” (Schneier, 2015). The real-world ramifications of events in cyberspace can be
devastating and far-reaching.
In May 2021, a ransomware attack shut down an oil pipeline that supplied 45% of
fuel to the U.S. East Coast (Committee on Homeland Security, 2021). Cyberspace has
even been recognized as the fifth domain of war, permeating land, sea, air, and space
(Welch, 2011). So, it should not be a surprise that people are waging cyber war on the
world’s biggest and most powerful organization, the U.S. Government.
The government impacts nearly all aspects of American lives, from ensuring the
food they eat is safe to the money they use to pay for it. Like all organizations, the
government depends on information systems to successfully carry out its missions and
functions at scale. The results could be dire if the confidentiality, integrity, or availability
of federal information systems are compromised. But just how do these attacks occur?
With the rapid innovation of information technologies came fundamental flaws in

their design and implementation. The internet wasn’t built with security in mind. A
vulnerability is “A flaw in a software, firmware, hardware, or service component
resulting from a weakness that can be exploited, causing a negative impact to the
confidentiality, integrity, or availability of an impacted component or components” (The
MITRE Corporation, n.d.). And the more complex information systems, software, and
networks become, the harder they are to secure.
A little over a year after 9/11 shook the nation, the E-Government Act of 2002
was signed into law (E-Government Act, 2002). It recognized that computer systems and
networks would play a critical part in the government’s public services and missions.
Moreover, it enacted the need to address cybersecurity risks at federal agencies not a
Andrew Davidow, andrew.davidow@student.sans.edu

moment too soon: “With our great reliance on computers and the numerous flaws found
in most systems, this is the Golden Age of Hacking” (Skoudis & Liston, 2006).
1.1. Federal Information Security Management Act

The Federal Information Security Management Act (FISMA) of 2002 was signed
into law along with the E-Government Act (E-Government Act, 2002). FISMA required
federal agencies to employ a comprehensive framework to ensure that the information
security controls on federal information systems and networks are effective.
Acknowledging the interconnectedness between federal operations, computers, and
networks, the need for a mechanism of government oversight and management of
information security programs was paramount. Under FISMA, the National Institute of
Standards and Technologies (NIST) became responsible for publishing information
security standards, guidelines, and minimum requirements for federal information
systems.
Fulfilling this responsibility, NIST Special Publication (SP) 800-53 provided

guidelines on recommended security controls for federal information systems (Ross et al.,
2005). A security control is defined as “A safeguard or countermeasure prescribed for an
information system or an organization designed to protect the confidentiality, integrity,
and availability of its information and to meet a set of defined security requirements”
(Joint Task Force, 2020).
1.2. Risk Management Framework

Five years after the original publication, SP 800-53 evolved into the risk
management framework (RMF). The RMF sought to enable a near real-time risk
management solution by integrating a robust continuous monitoring program within the
system development life cycle (Joint Task Force Transformation Initiative, 2010). The
continuous monitoring program would allow leadership to make risk-based, cost-
effective decisions on information systems that support core missions and functions.
In the RMF, information systems undergo a risk assessment where the potential
threats, known vulnerabilities, and the impact of an incident are measured. This informs
how the information system is categorized. Based on the risk assessment and

categorization, a baseline of security controls can be applied and tailored to the specific
risks. After the initial security control assessment, the system must be authorized to
operate. Finally, the system must be continually assessed since risks, the information
system, and its environment can change over time. These entire processes can be repeated
as necessary.
In summary, the RMF provides a holistic approach to managing information

system security risks by integrating it into every part of the organization. In a dynamic
and evolving information system environment, automated monitoring of the risk
management framework’s implementation and effectiveness provides a tactical and
strategic insight into the current security state.
1.3. Continuous Monitoring

Continuous monitoring (CM) of information systems enables detecting
compliance and risk issues at scale. “Information security continuous monitoring (ISCM)
is defined as maintaining ongoing awareness of information security, vulnerabilities, and
threats to support organizational risk management decisions” (Dempsey et al., 2011).
Manual audits are still necessary, but an automated approach is needed to measure the
implementation of the risk management framework at the government's scale.
The first iteration of a government-wide CM program was the Continuous Asset

Evaluation, Situational Awareness, and Risk Scoring (CAESARS) Reference
Architecture Report created by the Department of Homeland Security (DHS) in 2010.
This was expanded upon in the CAESARS Framework Extension (FE) in 2011:
“The end goal of CAESARS FE is to enable enterprise CM by presenting a

technical reference model that allows organizations to aggregate collected data
from across a diverse set of security tools, analyze that data, perform scoring,
enable user queries, and provide overall situational awareness” (Mell et al., 2012).
An example of how Mell et al. (2012) envisioned an enterprise-wide CM architecture is

shown in Appendix A. They present four subsystems: (1) data collection from data
sources; (2) an integration layer where that information can be aggregated, enriched, and
queried; (3) a reporting view that presents the data to a person and allows them to query

it; and (4) a risk scoring and analysis subsystem with predefined views that can drive
operational decisions. The CAESARS FE report also foresaw a challenge with
implementing a CM program: different federal civilian agencies had purchased various
products from various vendors, and not all those vendors had adopted standards for
shipping the data into another system like a CM repository. The paper advised working
closely with the vendor community to adopt specific standards that would enhance their
products and benefit their customers by integrating their tools with continuous
monitoring solutions.
1.4. Continuous Diagnostics and Mitigation

To address the challenges identified in the CAESARS FE report, the second
iteration of a government-wide CM program consolidated leadership under DHS for
civilian agencies in the federal executive branch (DHS, 2013). The project was called the
Continuous Diagnostics and Mitigation (CDM) program (DHS, 2013). DHS’s leadership
on the CDM program was codified into law in an update to FISMA, authorizing them “to
administer the implementation of information security policies for non-national security
Federal Executive Branch systems, including providing technical assistance and
deploying technologies to such systems” (FISMA Act, 2014). The strategic goal of CDM
was to strengthen federal networks against attacks by fixing the worst problems first. A
phased approach was adopted to answer the whos, whats, wheres, whens, whys, and hows
of events on the network (Quinn, 2016). Given the scale of the program, challenges were
bound to happen.
One of the biggest roadblocks was that the cybersecurity tools (CM data sources)
do not integrate easily with the integration layer components (CM repositories), where
risk scores were calculated: “[The] disjointed collection and use of data inhibits effective
governance, decision-making, and action” (Otto, 2017). This led to gaps in the
fundamental data source domain necessary for any continuous monitoring program. In
fact, they’re the same cautions expressed in the 2011 CAESARS FE report.
Corroborating these failures, the Government Accountability Office (GAO) found that:
“The 23 civilian agencies covered by the Chief Financial Officers Act of 1990
(CFO Act) have often not effectively implemented the federal government’s

approach and strategy for securing information systems...Until agencies more

effectively implement the government’s approach and strategy, federal systems
will remain at risk” (GAO, 2018).
A plan was already in motion to help address these failures. The CDM Dynamic
and Evolving Federal Enterprise Network Defense (DEFEND) contracts awarded
potentially billions of dollars to private sector companies to fix the gaps (Cordell, 2018).
Also, in 2018, a new child agency was formed under DHS, the Cybersecurity and
Infrastructure Security Agency (CISA; CISA Act, 2018). CISA assumed leadership over
the CDM program and decided to pivot the CDM dashboard to a technology called
Elastic Stack (Wiltshire, 2019). The plan to address the CDM program failures sounded
promising: new funding, new leadership, and new technology.
1.5. Elastic Stack

Elastic is a company that solves search problems. A search problem is the
challenge at the heart of the CDM program. The government has deployed a plethora of
cybersecurity tools amassing mountains of data. That data needs to be aggregated, sorted,
filtered, and searched to facilitate defenders in identifying and prioritizing risks. This can
be accomplished with four products in the stack: Beats, Logstash, Elasticsearch, and
Kibana.
The stack’s roots are in being free and open-source (Elastic, n.d.-g). In 2021,
Elastic moved the source code of Elasticsearch and Kibana from the Apache 2.0 license
to either the Elastic License or the Server Side Public License to protect the company’s
investments from non-reciprocal cloud service providers (Elastic, 2021a). “This license
change ensures our community and customers have free and open access to use, modify,
redistribute, and collaborate on the code” (Elastic, 2021a). To this day, the code remains
open, being housed in public repositories so that customers can adapt it for their own use
cases (Elastic, n.d.-g). This is key for solving the challenges of today and tomorrow in
the CDM program. Even though all the products are free, features required in the CDM
program cost money, like advanced security controls, advanced features, and service
level agreements (Elastic, n.d.-h).

1.5.1. Beats
Beats are free, open, agent-based data shippers. They’re lightweight and do not
require root privileges to run. One of the Beats, called Filebeat, has generic inputs that
can be configured to collect data (e.g., filestream, HTTP JSON, log, TCP, and UDP), and
it has 70 pre-built modules to simplify data retrieval from specific tools (Elastic, 2022b).
In a use case like CDM, Beats have tremendous potential to solve the problem of
retrieving data from the CDM tools.
1.5.2. Logstash
“Logstash is an open source data collection engine with real-time pipelining
capabilities” (Elastic, 2022c). It’s all about collecting, parsing, transforming, and
shipping data. First, Logstash input plugins are used to read from data sources like Beats,
shell commands, files, databases, and network protocols, to name a few (Elastic, n.d.-f).
Second, the data can be sent through numerous pipelines that can transform it from one
data type to another and parse the data’s content to normalize it. Filter plugins can be
conditionally applied in the pipeline to perform arbitrary data processing operations. Last,
the data can be shipped off to where it needs to go. In the CDM program, Logstash can
replace the integration layer components.
1.5.3. Elasticsearch
Elasticsearch is a free and open search engine that a Wired article once called
“Your own private Google” (Finley, 2012). It stores data as documents consisting of
field-value pairs that can have various data types. It is fast, scalable, and resilient. The
CDM Dashboard will require some of the paid features like document-level security for
granular access controls in multi-tenant environments or users scoped to assets in a
particular FISMA group (Wiltshire, 2019). The CDM Dashboard will also use cross-
cluster search (CCS; Harmon, 2021), which allows remote Elasticsearch clusters to be
queried. CCS is perfect for the FISMA reporting requirements in CDM and for quickly
responding to new threats or vulnerabilities.

1.5.4. Kibana
Kibana is the free and open front end to Elasticsearch that presents the data.
Whether it’s a dashboard, a map, a bar chart, or a search interface, Kibana has it covered.
Kibana makes it easy to craft search queries and visualize the data. This is where the
CDM Dashboard is built. The customizability of Kibana allows the CDM Dashboard to
implement the government’s Agency-Wide Adaptive Risk Enumeration (AWARE)
algorithm (CISA, 2020). The AWARE scoring methodology helps defenders prioritize
which problems to fix first.
1.6. CDM Current State and the Future

The potential of Beats, Logstash, Elasticsearch, and Kibana is extraordinary. And
yet, a year after the pivot to Elastic Stack was announced, the GAO sampled agencies on
CDM implementation progress and found that “…none of the three agencies had
effectively implemented all key CDM program requirements…” (GAO, 2020). Two
years after the pivot, even the DHS mothership was continuing to have challenges, as
“Efforts were still underway to automate and integrate the data collection process among
components so DHS could report additional data, as required” (DHS OIG, 2021). The
technology is sound, yet years have gone by, and many agencies have yet to achieve
success.
Since the CDM program’s inception in 2013, CDM contractors have developed
custom solutions to bring the data into the CDM Dashboard (U.S. Government, n.d.-b)
with varying degrees of success. Some solutions are riddled with unnecessary
complexities, redundancies, and inefficiencies. The Federal government has paid a total
estimated cost of $10.9 billion dollars (GAO, 2020) to get the CDM program up and
running. Take a moment to imagine what a cybersecurity tool integration might look like
if it was built by one group of contractors with hundreds of millions of dollars starting in
2013 (U.S. Government, n.d.-a) and then, in many cases, rebuilt by a new group of CDM
DEFEND contractors with hundreds of millions of dollars starting in 2018 (U.S.
Government, n.d.-b). It begs the question of whether anyone other than the contractors
that built the solution will be able to maintain the integration layer, compelling the
government to rely on them. The integration layer will inevitably break, need to be

patched, or need to be adapted for a new use case. Future funding for a legion of
developers is not guaranteed. Federal teams need an integration layer that is maintainable,
cost-effective, and feasible. And they need something they can implement in days, not
decades.
The researcher hypothesized that the capabilities already present in Elastic Stack
can be used to collect the data from CDM tools, perform arbitrary transformations on said
data, and ship the results off to Elasticsearch. Such a solution, if successful, would save
hundreds of millions of dollars in integration costs. Further cost savings would be
realized by eliminating redundant data aggregators, databases, or CM repositories in the
integration layer. This would reduce the overall complexity of the system to a single suite
of tools that are designed to work together (Elastic, n.d.-a), thereby providing a more
maintainable solution.
2. Research Method
The research began by identifying two CDM tools (GSA, 2022a, 2022b) that had
no existing Filebeat modules or Logstash plugins. Tenable.sc and HCL BigFix were
chosen, which have a combined federal investment of over a quarter of a billion dollars
(U.S. Government, n.d.-c, n.d.-d), indicating that if the experiment is successful, it would
have an immediate impact that matters.
2.1. Tenable.sc and BigFix

Tenable.sc (formerly called Security Center) allows an operator to manage
multiple vulnerability scanners from a single location, search scan data results, create
reports, and create dashboards (Tenable, n.d.-a). Vulnerabilities are discovered by active
scans, passive scans, or agent-based scans. An example of an active scan is when a
remote scanner enumerates all the open ports on a host to find out what services are
running, determine the version of those services, and report if they’re vulnerable to an
attack. For CDM, it can be used to report software vulnerabilities (VUL), configuration
settings management (CSM), and unauthorized hardware (Jensen, 2020).
BigFix is an endpoint management platform. BigFix clients are installed on each

computer and typically report through relays to the BigFix server. BigFix operators

interact with the server from the BigFix console or WebUI, which allows them to
perform system administration tasks. For example, an operator could create a fixlet (i.e.,
an action) that checks if a computer is missing a particular patch, then conditionally
download, install, and report if the patch was installed successfully. BigFix has an
application called Web Reports, which provides a read-only view of the console and
makes building reports very easy. Web Reports was used to create a hardware asset
management (HWAM) report and a CSM report based on the CIS checklists. Another
application called BigFix Inventory (BFI) was deployed to collect software asset
management (SWAM) data. All available software scan types were issued on the lab
hosts daily (HCL, n.d.-c).
2.2. Development Environment

2.2.1. Software
The lab machine operating systems were selected from the list of operating
systems supported by BigFix agent 10.0.4 (HCL, n.d.-b) since it’s an approved HWAM
tool in CDM (GSA, 2022b). Multiple versions of the CDM tools and Elastic Stack were
analyzed where possible. The Filebeat and Logstash configurations were built in Visual
Studio Code. The full list of software used in the research is documented in Appendix B.
This includes the selected CDM tools, operating systems, and software development
tools.
2.2.2. Lab environment

The lab environment was composed of physical, virtual, and containerized hosts
with a variety of operating systems and software configurations. For example, on the
RHEL 8.6 side, one virtual machine was installed with the environment group “Server
with GUI” with the CIS hardening profile and software from the “.NET Core
Development” and “Network Servers” groups. On the Windows side, a virtualized 2019-
member server with a GUI ran the MS SQL database for the BigFix server.
2.3. Experiments and Testing

Filebeat inputs, processors, and outputs, and Logstash input plugins, filter plugins,
and output plugins were experimented with to create the integrations for the CDM tools.

Experimenting and testing began by reading the vendor documentation on how data could
be retrieved from the CDM tool. This was then compared to the available inputs for
Filebeat and Logstash. Once data could be retrieved from the tool, Logstash filters were
applied to transform, parse, and ship the data to Elasticsearch.
2.3.1. Collect the data from the CDM tool

The research process involved experimenting with how data can be collected from
the CDM tools. Is the data logged to a file? Is it accessible over an API? Perhaps the
vendor product can be configured to send the data? The CDM tool options were then
matched to the Filebeat and Logstash input options.
The APIs were used to collect data from Tenable.sc and BigFix. This involved
reading the official documentation, crafting test queries, and then implementing the
queries in the input configurations. During initial testing, the output was sent to the
console’s standard output. This displays the events collected directly in the terminal
running Logstash or Filebeat.
Once the Filebeat HTTP JSON input filter was selected as the primary way to
retrieve the data from the Tenable.sc, the researcher iteratively built a Filebeat
configuration. The configuration leveraged the HTTP JSON capabilities for retrieving the
data at a configurable interval, retrieving the data in chunks (pagination), retries, rate
limiting, request transformations, and response transformations.
2.3.2. Transform and Parse

A Beats input plugin was configured in Logstash to receive the events collected
by Filebeat. Events were initially sent to standard output (i.e., printed to the console)
using the stdout plugin with the rubydebug codec. After confirming receipt in Logstash,
the event data were transformed into JSON format using the JSON plugin filter. The
fields in the console output were easier to read due to the transformation. The JSON
transformation is a necessary step for Elasticsearch, which indexes documents in JSON
format. This also facilitated sequential parsing of the fields and values.
Logstash filter plugins were used to process the event data into JSON field-value
pairs. The Elastic Common Schema Version 8.2 (Elastic, n.d.-b) was used for field

naming conventions. Commonly used filters include the mutate filter, the date filter, and
the JSON filter.
The Elasticsearch output plugin was used to send the data from Logstash to
Elasticsearch during the development process. Numerous test indices were used to
compare the results of Logstash configuration changes and conveniently view the data of
interest using queries in Kibana.
2.3.3. Map in Elasticsearch

When an event is stored in Elasticsearch, it is stored as a document; “Each
document is a collection of fields, which each have their own data type” (Elastic, n.d.-c).
Fields can have multiple data types. For example, the IP field in Appendix F is stored as a
keyword, which would require an exact match when searching for it. It is also stored as
an IP data type, allowing it to be queried using Classless Inter-Domain Routing (CIDR)
notation, as demonstrated in Appendix X. An Elasticsearch mapping was created to
define how the fields for each event are stored and indexed. The mapping definition
specified the data type for each field.
2.3.4. Validate in Kibana

The results were examined in Kibana throughout the development process. This
made filtering for the specific fields being worked on easy to review, as demonstrated in
Appendix Y. The Kibana “Dev Tools” were used to perform index operations (e.g.,
create, delete, search) and to apply the Elasticsearch mappings. Several hundred field
names and values were reviewed to verify that the Logstash permutations had the desired
effect.
2.3.5. Upgrade and Repeat

Upgrades of the CDM tools and Elastic Stack components were performed to test
maintainability across software versions. This involved determining if the same Filebeat
configuration could retrieve all of the data from the CDM tool. In addition, the Logstash
pipeline was examined for errors by reingesting datasets, documenting any required
changes, and analyzing the resulting documents.

3. Findings and Discussion

3.1. Findings
The Filebeat HTTP JSON input was able to retrieve all of the Tenable.sc fields
for asset, plugin, and vulnerability data. The complete list of fields brought in is available
in the Tenable API documentation (Tenable, n.d.-b) and in the example Elasticsearch
mappings shown in Appendices F, I, and L, respectively. The Tenable.sc Filebeat
configurations to retrieve the data are available in Appendices D, G, and J, respectively.
The Filebeat HTTP JSON input supports numerous authentication options, data
retrieval at configurable intervals, pagination, request transformations, response
transformations, and more (Elastic, n.d.-d). The configuration may be tuned to mitigate
the impact on a production environment or optimized for available resources. For
example, it could be configured to retrieve 5,000 records at a time, waiting 15 to 30
seconds between requests for records created in the last 24 hours. Settings like these and
many more may be tuned to meet reporting and performance requirements lending
credence to the feasibility of Filebeat data retrieval. Filebeat was configured to ship the
retrieved data over to Logstash.
Logstash was able to parse and transform the Tenable.sc data using filter plugins.
The default filter plugins were able to handle the vast majority of the data. Conditional if-
else statements were used to handle cases. Trickier situations like the vulnerability
priority rating context were handled using the ruby plugin filter, which gives free rein to
the Ruby programming language. See Appendix E, H, and K for the Tenable.sc Logstash
pipeline configurations for asset, plugin, and vulnerability data.
The Filebeat and Logstash configurations were built and tested on Elastic Stack
version 7.17.4, and no changes were needed when compared against version 7.17.5. This
was assessed by adding tags on plugin failures in the Logstash pipelines for errors.
Events with errors were sent to a separate Elasticsearch index for analysis and
remediation.
A one-month trial license for HCL BigFix was used for developing the
integration. BigFix Web Reports was used for creating an HWAM report, as depicted in

Figure 1. The 42 fields composing the HWAM report were selected from the default
content packaged with every BigFix deployment to be applicable to a wide audience. The
full list of fields is listed in Appendix S. A portable XML version of the report is
available in Appendix T that can be imported into Web Reports.
Figure 1. A cropped view of the HWAM report in Web Reports

Web Reports was also used for generating a CSM report. The BigFix server was
subscribed to all available CIS checklist external sites, such as the CIS Checklist for
RHEL 8 and the CIS Checklist for Windows Server 2019-MS. The available fields
remained the same across all CIS checklists. All fields were selected for the report, and a
subset is depicted in Figure 2.
Figure 2. A cropped view of the CSM report in Web Reports

After creating the HWAM and CSM reports, the Web Reports REST API (BigFix
Developer, n.d.-a) and SOAP API (BigFix Developer, n.d.-b) were examined for a means
to retrieve them. Both avenues of approach boiled down to requiring the Session
Relevance language (BigFix Developer, n.d.-c) to reconstruct the reports field by field.
The problem with session relevance is that it can be challenging to write, and statements
that seem to work fine in small deployments may cause a denial-of-service in large
BigFix deployments (Stewart 2015a, 2015b, 2016; Talbert, 2020). While it was as simple
as clicking checkboxes to generate the HWAM and CSM reports, it appeared that a

Relevance language subject matter expert was needed to create an integration, which
pulverized the prospects of a feasible and maintainable solution.
The researcher investigated how Web Reports was displaying its reports by using
the network panel in Google Chrome’s developer tools to inspect network activity. This
revealed a POST request to /post/shortentext with a URL encoded payload composed of
the column names, the number of results to retrieve, a start index, the column to sort by,
the sort direction, and the filters to apply. An example payload is shown in Figure 3.
Figure 3. A payload sent to /post/shortentext revealing URL parameters
These variables may be programmatically manipulated and sent as URL parameters in

GET requests to /json/computers to retrieve paginated results. Pagination is important for
large BigFix deployments, so the application isn’t overwhelmed. Moreover, the results
are sent in JSON format, which is ideal for ingestion into Logstash. An example request
to /json/computers is shown in Figure 4, which corresponds to the payload in Figure 3.
Figure 4. A GET request sent to /json/computers with URL parameters

As a proof of concept, Bash scripts using cURL (Stenberg, 2022) were created to
retrieve the reports. The scripts in Appendix U retrieve paginated results from the
HWAM and CSM reports in Web Reports and paginated SWAM results from BFI. These
scripts were run from Logstash using the exec input. Another researcher had previously
encountered the same challenges with retrieving data from Web Reports and developed a
Python script to retrieve from /json/computers (Scott, 2018a, 2018b).

Once in the Logstash pipeline, all retrieved results were parsed and transformed in
accordance with the Elastic Common Schema naming conventions. All fields in the
HWAM, CSM, and SWAM reports were successfully ingested into Elasticsearch. A
sample Elasticsearch document from each report is shown in Appendix W. The Logstash
configurations are shown in Appendices M, O, and Q for HWAM, CSM, and SWAM,
respectively, with corresponding Elasticsearch mappings in Appendices N, P, and R.
Exhaustive testing was performed to measure the level of effort involved in

maintaining the Logstash configurations across multiple versions of BigFix and Elastic
Stack. Five versions of Elastic Stack were tested for each version of BigFix. Three
versions of the BigFix platform were examined by retrieving the HWAM and CSM
reports from Web Reports. Two versions of BigFix Inventory were examined for the
SWAM report. Similar to the analysis performed on Tenable.sc, tags on filter plugin
failure were examined, and errors were sent to a test index to be resolved. In addition,
documents were exported from Elasticsearch and programmatically compared by using jq
(Dolan, 2018) to sort the document fields and Diffutils (Meyering & Eggert, 2021) to
compare differences. The only differences were expected changes to field values like the
BigFix agent version across upgrades, the document identification numbers, the ingestion
timestamps, and the available disk space. The compatibility matrix in Table 1 shows that
no changes were needed across the board.
BigFix BigFix BigFix BigFix BigFix

Elastic Platform Platform Platform Inventory Inventory
Stack 10.0.4 10.0.5 10.0.6 10.0.8 10.0.9
8.3.2 x x x x x
8.2.2 x x x x x
8.1.2 x x x x x
7.17.5 x x x x x
7.17.4 x x x x x
Table 1. Elastic Stack configuration compatibility matrix for BigFix
3.2. Discussion
Filebeat and Logstash have advanced data ingestion and transformation
capabilities. They are free, open, and extendable to add new features. When retrieving
and transforming data, there are myriad options and tool-specific integrations built by

Elastic and the community. The Filebeat and Logstash configurations created in this
research are easily adaptable for adding new fields to either CDM tool.
When used in the CDM program, Filebeat and Logstash should be configured for
continuous monitoring as opposed to interval monitoring. The current CDM requirement
is to retrieve data once every 72 hours (CISA, 2021). Logstash is capable of retrieving
mountains of data, and mitigations can be configured for spikes in events per second. In a
dynamic and evolving threat landscape, cyber defenders need today’s information today.
True continuous ingestion may be facilitated by the deployment architecture, which also
plays into how resilient the system is and how agency needs can be met. An example
deployment scenario is discussed in Appendix V.
An alternative to using Filebeat to collect data is Elastic Agent. Elastic Agent has
an HTTP JSON input and a generally available Tenable.sc version 5.18 integration
(Elastic, 2022b). Whether to use one or the other depends on numerous factors, given that
they have many of the same capabilities (Elastic, n.d.-e). At the moment, Filebeat has
specific features that may be favored by federal agencies, such as a locally managed
secrets keystore for secure settings (Elastic, n.d.-e).
Once the data is finally in Elasticsearch, Kibana may be used to build queries and
visualizations. Figure 5 demonstrates how fields are aggregable by computing the sums
of CPU cores and total RAM. The pie chart shows the distribution of operating systems
in the lab. A chart depicting the counts of hosts in various subnets is shown as well. If an
analyst clicked on the Linux Ubuntu 20.04 slice of the pie chart, a filter would be applied,
and all of the visualizations would update to reflect only that OS. The actual CDM
Dashboard would show the aggregated AWARE score and could be similarly filtered by
FISMA groups (CISA, 2020).

Figure 5. A hardware asset management dashboard

Figure 6 shows two CIS checklists retrieved from Web Reports. At the top is a
query bar that can be used to scope the dashboards, such as by querying for systems on a
public-facing subnet. Both charts display each check’s category and name. The RHEL 8
dashboard shows the number of applicable computers, remediated computers, and BigFix
open actions for each check. In the Windows Server 2019-MS dashboard, the hostname
and reference links to Web Reports are shown for each applicable check. A dashboard
like this could be used to prioritize the checks to remediate, given the set of security
controls applicable to the asset.

Figure 6. A configuration settings management dashboard

The software asset management dashboard shown in Figure 7 is composed of the
data retrieved from BFI. An OpenSSL dashboard shows the software version, the host it
is installed on, and the timestamp when it was last detected. The software heatmap
dashboard shows the top ten prevalent pieces of software across the lab environment. At
the bottom, a CVE tag cloud dashboard reflects the summed count of max CVEs per
software instance, where a larger font size indicates an increased frequency of
occurrences. While fun for demonstration purposes, a practical visualization would sort
the CVEs by their Common Vulnerability Score System (CVSS) scores or be based on a
catalog of known exploitable vulnerabilities (CISA, n.d.).

Figure 7. A software asset management dashboard
4. Recommendations and Implications

The data ingestion capabilities already present in Elastic Stack should be
leveraged in the Continuous Diagnostics and Mitigation program. They are cost-effective,

robust, and easy to maintain. If an agency has the data that can help make federal
networks safer but is struggling to get it in a risk management dashboard, they should use
the Elastic Stack’s capabilities to get the data. Filebeat has pre-built modules for many
tools on the CDM-approved product list (GSA, 2022a), such as Bluecoat, Check Point,
Cisco, Crowdstrike, and Cyberark (GSA, 2022b) chosen from the first five alphabetically
ascending. If a Filebeat module or Logstash input doesn’t exist, this research
demonstrates how one can be built. Filebeat and Logstash can fulfill the requirements of
the integration layer, and they are completely free.
4.1. Taking Ownership

Federal networks are under attack every single day. Ten years after the original
risk-based dashboard program started, the GAO reports that some implementations are
partial at best. There are inherent complexities in organizations as large as the
government, and the technical challenges of one agency differ from another. Things
inevitably go wrong. However, if the complexities of the solution can be significantly
reduced while meeting organizational and mission requirements, the simpler solution is
better. The CDM solution must be maintainable by cyber defenders now and in the
future. If the current CDM integration solution is so complex that it can only be
maintained by DEFEND contractors with hundreds of millions of dollars in funding (U.S.
Government, n.d.-a), that solution will inevitably fail.
This research shows that there is a simpler way that can be expanded upon to
meet the CDM mission requirements. Filebeat and Logstash are extendable to fulfill
needs not present in the hundreds of existing integrations. Filebeat, Logstash,
Elasticsearch, and Kibana are well documented. And they are cost-effective.
4.2. Implications for Future Research

One limitation in this study was that the researcher was unable to obtain a
Tenable.sc lab license from the vendor despite reaching out multiple times directly and to
resellers. This limited how rigorously maintainability could be tested across multiple
versions of Tenable.sc to a production instance running version 5.20.1. Due to the
sensitive nature of production data and the type of information that Tenable.sc collects,

images of the Tenable.sc data in Kibana were not shown in the findings. Future research
could test the Tenable.sc Filebeat and Logstash configurations on other versions.
For BigFix, cURL was used to retrieve the HWAM, CSM, and VUL reports.
Future research could retrieve these reports with the Filebeat HTTP JSON input by
adding (a) the ability to change the “request.method” from an initial POST to a GET for
BFI, and (b) the ability to store a session-based cookie for Web Reports.
Seneca once said, “Our lack of confidence is not the result of difficulty; the
difficulty comes from our lack of confidence” (Seneca, 65). As daunting as reshaping a
multi-billion dollar government cybersecurity program is, this research provides evidence
that Elastic Stack is a cost-effective, maintainable, and feasible solution to the integration
layer challenges. Others can expand upon this research by building Elastic Stack
integrations for CDM tools that don’t exist yet. Additional research can also show others
how to use some of the more advanced capabilities of Logstash.
5. Conclusion
Civilian federal agencies over the past decade have struggled to implement a risk-
based continuous monitoring dashboard with data sourced from cybersecurity tools.
Elastic Stack can solve this problem at a fraction of the cost. The research presented here
shows that it is entirely feasible to bring data from CDM tools into Elastic Stack without
the need for costly and complicated solutions. Since BigFix and Tenable.sc data can be
retrieved in JSON format, the Logstash pipelines essentially just rename the fields. While
events are in the pipeline, they can be enriched with data from numerous external
sources, such as a CSV plugin to append FISMA information, the DNS plugin to perform
reverse lookups on events that are missing a hostname, or the HTTP plugin to retrieve
asset information from an asset management REST API. Alternatively, post hoc
enrichment can occur by using Logstash to query Elasticsearch for missing fields.
The CDM program’s present level of funding will not last forever. The solutions
being built by DEFEND contractors must be maintainable by federal teams in the future.
The Filebeat and Logstash configurations developed in this research are compatible
across numerous versions of Elastic Stack and the CDM tools, meaning that no changes

were needed to retrieve all of the data. If a change is needed, making an adjustment is
easy. Say a new field is desired from BigFix. It’s as simple as adding the URL encoded
column or criteria to the input and then renaming that field in Logstash.
Elastic Stack looks very promising as a one-stack shop to bring in cybersecurity

data from numerous sources. There are currently 70 Filebeat modules (Elastic, 2022b), 92
Elastic Agent integrations (Elastic, 2021b), and 55 Logstash input plugins (Elastic, n.d.-
f). There’s also the ability to build a new input or modify an existing one since the code is
open and modular. Elastic Stack brings Ockham’s razor to the CDM solution by
eliminating unnecessary and costly systems while achieving the same results. The
Filebeat and Logstash configurations are easy to maintain and adapt as needed. Elastic
Stack can empower defenders by collecting, aggregating, storing, and displaying data
from cybersecurity tools to fulfill the CDM program’s mission.

References
BigFix Developer. (n.d.-c). Session Relevance Guide. Retrieved August 4, 2022, from
https://developer.bigfix.com/relevance/guide/session/
BigFix Developer. (n.d.-b). The BigFix SOAP API. Retrieved July 4, 2022, from
https://developer.bigfix.com/other/soap-api/
BigFix Developer. (n.d.-a). The BigFix REST API. Retrieved July 4, 2022, from
https://developer.bigfix.com/rest-api/
Committee on Homeland Security. (2021, June 9). Cyber threats in the pipeline: Using
lessons from the colonial ransomware attack to defend critical infrastructure.
GovInfo | U.S. Government Publishing Office.
https://www.govinfo.gov/content/pkg/CHRG-117hhrg45085/html/CHRG-
117hhrg45085.htm
Cordell, C. (2018, August 22). Booz Allen Hamilton snags $1b cdm defend task order.
FedScoop. https://www.fedscoop.com/booz-allen-hamilton-snags-1b-cdm-defend-
task-order/
Cybersecurity and Infrastructure Security Agency Act of 2018, Pub. L. No. 115-278
(2018). https://www.govinfo.gov/content/pkg/PLAW-115publ278/pdf/PLAW-
115publ278.pdf
Cybersecurity and Infrastructure Security Agency. (2020, September 20). Continuous
diagnostics and mitigation program: aware scoring.
https://www.cisa.gov/sites/default/files/publications/cdm-program-aware-scoring-
fact-sheet-092020-508.pdf
Cybersecurity and Infrastructure Security Agency. (2022). Continuous diagnostics and
mitigation program. Retrieved June 16, 2022, from https://www.cisa.gov/cdm
Cybersecurity and Infrastructure Security Agency. (2021, August 8). Continuous
diagnostics and mitigation program technical capabilities volume two:
requirements catalog version 2.4. General Services Administration.
https://www.gsa.gov/cdnstatic/Integrated_Technology_Services/CDM-PROG-
2021-CDM%20Technical%20Volume%202_v24.pdf
Page 13

Cybersecurity and Infrastructure Security Agency. (n.d.). Known exploited vulnerabilities

catalog. Retrieved August 15, 2022, from https://www.cisa.gov/known-exploited-
vulnerabilities-catalog
Dempsey, K., Chawla, N., Johnson, L., Johnston, R., Jones, A., Orebaugh, A., Scholl, M.,
& Stine, K. (2011, September). Information Security Continuous Monitoring
(ISCM) for Federal Information Systems and Organizations. NIST SP 800-137.
https://doi.org/10.6028/NIST.SP.800-137
Department of Homeland Security. (2013, August 13). A major step forward in better
protecting federal, state and local cyber networks.
https://www.dhs.gov/blog/2013/08/13/major-step-forward-better-protecting-
federal-state-and-local-cyber-networks
Dolan, S. (2018, November 1). jq. GitHub. Retrieved July 6, 2022, from
https://github.com/stedolan/jq/
E-Government Act of 2002, Pub. L. No. 107–347, title III. (2002).
https://www.congress.gov/107/plaws/publ347/PLAW-107publ347.pdf
Elastic. (2022a). Filebeat overview. Retrieved August 14, 2022, from
https://www.elastic.co/guide/en/beats/filebeat/8.3/filebeat-overview.html
Elastic. (2022b). Modules. Retrieved June 21, 2022, from
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html
Elastic. (2022c). Logstash introduction. Retrieved August 14, 2022, from
https://www.elastic.co/guide/en/logstash/8.3/introduction.html
Elastic. (n.d.-e). Beats and elastic agent capabilities. Retrieved July 24, 2022, from
https://www.elastic.co/guide/en/fleet/current/beats-agent-comparison.html
Elastic. (n.d.-b). Elastic common schema (ECS) reference [8.2]. Retrieved June 23, 2022,
from https://www.elastic.co/guide/en/ecs/current/index.html
Elastic. ( n.d.-h). Elastic pricing. Retrieved August 14, 2022, from
https://www.elastic.co/pricing/
Elastic. (2021a, December 21). FAQ on 2021 license change.
https://www.elastic.co/pricing/faq/licensing
Elastic. (2021b, November 3). Elastic Integrations. https://docs.elastic.co/integrations

Elastic. (n.d.-d). HTTP JSON input | Filebeat reference [7.17] | Elastic.

https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-HTTP
JSON.html
Elastic. ( n.d.-f). Input plugins. Retrieved June 6, 2022, from
https://www.elastic.co/guide/en/logstash/7.17/input-plugins.html
Elastic. (n.d.-c). Mapping | Elasticsearch guide [7.17] | Elastic. Retrieved June 23, 2022,
from https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping.html
Elastic. (n.d.-a). Overview. Retrieved August 14, 2022, from
https://www.elastic.co/guide/en/elastic-stack/current/overview.html
Elastic. (2022b, July 18). Tenable.sc. Retrieved August 2, 2022, from
https://docs.elastic.co/en/integrations/tenable_sc
Elastic. ( n.d.-g). Why free and open? Retrieved August 14, 2022, from
https://www.elastic.co/about/free-and-open
Federal Bureau of Investigation. (2021). Internet crime report 2021. Internet Crime
Complaint Center.
https://www.ic3.gov/Media/PDF/AnnualReport/2021_IC3Report.pdf
Federal Information Security Modernization Act of 2014, Pub. L. No. 113-283 (2014).
https://www.congress.gov/bill/113th-congress/senate-bill/2521
Finley, K. (2012, December 7). Your own private Google: The quest for an open source
search engine. Wired. https://www.wired.com/2012/12/solar-elasticsearch-
google/
General Services Administration. (2022a, July 15). Continuous diagnostics & mitigation
tools. Retrieved August 14, 2022, from
https://www.gsa.gov/technology/technology-products-services/it-
security/continuous-diagnostics-mitigation-tools
General Services Administration. (2022b, June 1). CDM Approved Products List (APL).
Retrieved June 16, 2022, from
https://www.gsa.gov/cdnstatic/May%202022%20APL_v2.xlsx
Government Accountability Office. (2018, December 18). Information security: agencies
need to improve implementation of federal approach to securing systems and
protecting against intrusions. https://www.gao.gov/assets/gao-19-105.pdf

Government Accountability Office. (2020, August). Cybersecurity: DHS and selected

agencies need to address shortcomings in implementation of network monitoring
program. https://www.gao.gov/assets/gao-20-598.pdf
Harmon, J. (2021, December 8). Why the U.S. government is scaling their cyber visibility
practices with elastic. Elastic Blog. https://www.elastic.co/blog/why-
usgovernment-scaling-cybervisibility-elastic-cdm-cybersecurity-analyze-data
HCL. (n.d.-b). BigFix Enterprise Suite Download Center 10.0.4.
https://support.bigfix.com/bes/release/10.0/patch4/
HCL. (n.d.-a). Preview: Checking common vulnerabilities and exposures (CVEs).
Retrieved August 14, 2022, from
https://help.hcltechsw.com/bigfix/10.0/inventory/Inventory/security/t_cve_checki
ng.html
HCL. (n.d.-c). Types of software scans.
https://help.hcltechsw.com/bigfix/10.0/inventory/Inventory/planinconf/c_types_so
ftware_scans.html
Jensen, C. (2020, February 26). CDM 2020: “Operationalizing CDM” through risk-
based vulnerability management. Tenable. https://www.tenable.com/blog/cdm-
2020-operationalizing-cdm-through-risk-based-vulnerability-management
Joint Task Force. (2020, September). Security and privacy controls for information
systems and organizations. NIST Special Publication (SP) 800-53, Rev 5.
https://doi.org/10.6028/NIST.SP.800-53r5
Joint Task Force Transformation Initiative. (2010, February). Guide for applying the risk
management framework to federal information systems: a security life cycle
approach. NIST Special Publication (SP) 800-37, Rev 1.
https://doi.org/10.6028/NIST.SP.800-37r2
Mell, P., Waltemire, D., Feldman, L., Booth, H., Ouyang, A., Ragland, Z., & McBride, T.
(2012, January). CAESARS framework extension: an enterprise continuous
monitoring technical reference model (second draft). NIST Computer Security
Resource Center.
https://csrc.nist.gov/csrc/media/publications/nistir/7756/draft/documents/draft-
nistir-7756_second-public-draft.pdf

Meyering, J., & Eggert, P. (2021, August 1). GNU Diffutils. The GNU Operating System
and the Free Software Movement. https://www.gnu.org/software/diffutils/
Microsoft. (n.d.). Maximum URL length is 2,083 characters in internet explorer.
Microsoft Support. https://support.microsoft.com/en-us/topic/maximum-url-
length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-
908b53c12246
Office of Inspector General. (2021, June 1). DHS has made limited progress
implementing the continuous diagnostics and mitigation program. Department of
Homeland Security. https://www.oig.dhs.gov/sites/default/files/assets/2021-
06/OIG-21-38-Jun21.pdf
Otto, D. (2017, October, 31). Managing risk against cyber uncertainties: using the
dashboard. DHS Federal Network Resilience Division.
https://www.cisa.gov/uscert/sites/default/files/cdm_files/FNR_CGB_TRNG_OCT
2017_Webinar_Slide_Presentation.pdf
Quinn, J. (2016, October 13). CDM: Transition to Phase Two and Beyond. CISA.
https://www.cisa.gov/uscert/sites/default/files/cdm_files/FNR_CPM_OTH_Webi
nar_October.pdf
Ross, R., Katzke, S., Johnson, L., Swanson, M., Stoneburner, G., Rogers, G., & Lee, A.
(2005, February 2). Recommended security controls for federal information
systems. NIST Special Publication (SP) 800-53.
https://csrc.nist.gov/publications/detail/sp/800-53/archive/2005-02-28
Robinson, A. (2018, April 30). Did Einstein really say that? Nature.
https://www.nature.com/articles/d41586-018-05004-4
Scott. (2018a, February 5). Interesting and useful JSON restful APIs in web reports.
BigFix Forum. Retrieved July 5, 2022, from
https://forum.bigfix.com/t/interesting-and-useful-json-restful-apis-in-web-
reports/24441
Scott. (2018b, February 5). DoubleDeuceBuddha/bfwr_api. GitHub. Retrieved July 5,
2022, from https://github.com/DoubleDeuceBuddha/bfwr_api

Seneca, L. A. (65). Moral letters to lucilius/Letter 104. Wikisource, the free library.
Retrieved August 15, 2022, from
https://en.wikisource.org/wiki/Moral_letters_to_Lucilius/Letter_104
Schneier, B. (2015). Secrets and lies: Digital security in a networked world. John Wiley
& Sons.
Skoudis, E., & Liston, T. (2006). Counter hack reloaded: A step-by-step guide to
computer attacks and effective defenses (2nd ed.). Pearson. p. 2
Stenberg, Daniel. (2022, June 27). curl. Retrieved July 12, 2022, from https://curl.se/
Stewart, J. (2015a, June 18). The way Bigfix session relevance statements are written
matters (part1). https://www.jgstew.com/bigfix/2015/06/18/The-way-BigFix-
Session-Relevance-statements-are-written-matters-(part1).html
Stewart, J. (2015b, September 14). The way Bigfix session relevance statements are
written matters (part2) computer names.
https://www.jgstew.com/bigfix/2015/09/14/The-way-BigFix-Session-Relevance-
statements-are-written-matters-(Part2)-Computer-Names.html
Stewart, J. (2016, October 11). The way Bigfix session relevance statements are written
matters (part3) whose. https://www.jgstew.com/bigfix/2016/10/11/The-way-
BigFix-Session-Relevance-statements-are-written-matters-(Part3)-Whose.html
Sullivan, T. (2017, June 21). The meaning of Kibana. Discuss the Elastic Stack.
https://discuss.elastic.co/t/the-meaning-of-kibana/89828
Talbert, J. (2020, January 1). Efficient session relevance query for computer properties.
BigFix Forum. Retrieved July 6, 2022, from https://forum.bigfix.com/t/efficient-
session-relevance-query-for-computer-properties/32820
Tenable. (n.d.-b). Tenable.sc API: Overview. Retrieved July 15, 2022, from
https://docs.tenable.com/tenablesc/api/index.htm
Tenable. (n.d.-a). Tenable.sc. Retrieved August 14, 2022, from
https://www.tenable.com/products/tenable-sc
The MITRE Corporation. (n.d.). Glossary. CVE. Retrieved June 12, 2022, from
https://www.cve.org/ResourcesSupport/Glossary?activeTerm=glossaryVulnerabil
ity

U.S. Government. (n.d.-b). USAspending.gov. Retrieved June 18, 2022, from

https://www.usaspending.gov/search/?hash=7ea619bd02788988d7cec1ef1dd8e5f
c
CDM awards in fiscal year 2013
U.S. Government. (n.d.-d). USAspending.gov. Retrieved June 20, 2022, from
https://www.usaspending.gov/search/?hash=6d5d4a92ca0d63c0642e7d91db585df
f
BigFix awards, all fiscal years
U.S. Government. (n.d.-c). USAspending.gov. Retrieved June 20, 2022, from
https://www.usaspending.gov/search/?hash=8b41902936fd8b20674ea6e573e4a7e
1
Tenable awards, all fiscal years
U.S. Government. (n.d.-a). USAspending.gov. Retrieved June 21, 2022 from
https://www.usaspending.gov/search/?hash=2ffa1d790d3bd9fdb249b5ffcebcbeba
CDM DEFEND awards
Welch, L. (2011, February 20). Cyberspace – the fifth operational domain. Institute for
Defense Analyses. https://www.ida.org/-/media/feature/publications/2/20/2011-
cyberspace---the-fifth-operational-domain/2011-cyberspace---the-fifth-
operational-domain.ashx
Wiltshire, D. (2019, August 29). ECS and elastic partner to deliver next-generation CDM
dashboard for the Department of Homeland Security. Elastic.
https://www.elastic.co/about/press/ecs-and-elastic-partner-to-deliver-next-
generation-cdm-dashboard-for-the-department-of-homeland-security

Appendix A: Enterprise continuous monitoring architecture
Appendix A. Enterprise continuous monitoring architecture (Mell et al., 2012)

Appendix B: Software
Author Name Version Link
Chance VS Code 0.0.4 https://github.com/randomchance/vscode-
Carroll extension: logstash-configuration-syntax
randomchance.l
ogstash
Meyering Diffutils https://www.gnu.org/software/diffutils/
& Eggert
Elastic Elasticsearch 7.17.4 https://www.elastic.co/downloads/past-
releases/elasticsearch-7-17-4
Elastic Filebeat 7.17.4 https://www.elastic.co/downloads/past-
releases/filebeat-7-17-4
Elastic Kibana 7.17.4 https://www.elastic.co/downloads/past-
releases/kibana-7-17-4
Elastic Logstash 7.17.4 https://www.elastic.co/downloads/past-
releases/logstash-7-17-4




Fabien VS Code 1.3.0 https://github.com/fbaligand/vscode-logstash-

Baligand extension: editor
fbaligand.vscod
e-logstash-
editor
Stephen jq 1.6 https://github.com/stedolan/jq/
Dolan
HCL BigFix 10.0.4 https://support.bigfix.com/bes/release/10.0/pat
Enterprise Suite ch4/
HCL BigFix 10.0.7 https://support.bigfix.com/bes/release/10.0/pat
Enterprise Suite ch7/
HCL BigFix 10.0.8 https://help.hcltechsw.com/bigfix/10.0/invent
Inventory ory/Inventory/planinconf/c_installing_bfi_co
mponents.html
James besclient docker https://github.com/jgstew
Stewart scripts
Microsoft SQL Server 2016 https://www.microsoft.com/en-
us/evalcenter/download-sql-server-2016
Microsoft Visual Studio 1.68.1 https://code.visualstudio.com/download
Code
Oracle VirtualBox 6.1.34 https://www.virtualbox.org/wiki/Downloads
Red Hat VS Code 1.8.0 https://github.com/redhat-developer/vscode-
extension: yaml
redhat.vscode-
yaml
Tenable Tenable.sc 5.20.1 https://www.tenable.com/products/tenable-sc
VMware ESXi 6.7.0 https://customerconnect.vmware.com/downlo
ads/info/slug/datacenter_cloud_infrastructure/
vmware_vsphere/6_7
Appendix B. Software

Appendix C: Lab Operating Systems

Author Name Version Link
https://apps.apple.com/us/app/macos-big-
Apple macOS Big Sur 11.6.7
sur/id1526878132?mt=12
https://apps.apple.com/us/app/macos-
Apple macOS Catalina 10.15.7
catalina/id1466841314?mt=12
https://apps.apple.com/us/app/macos-
Apple macOS Mojave 10.14
mojave/id1398502828?mt=12
https://www.microsoft.com/en-us/software-
Microsoft Windows 10
download/windows10
https://www.microsoft.com/en-
Microsoft Windows Server 2019
us/evalcenter/download-windows-server-2019
Oracle oraclelinux 8.6 https://hub.docker.com/_/oraclelinux

Red Hat CentOS Stream 8 https://www.centos.org/download/
Red Hat CentOS 7 https://www.centos.org/download/
Red Hat Enterprise Linux 7.9 https://access.redhat.com/downloads
Red Hat Enterprise Linux 8.6 https://access.redhat.com/downloads
https://www.debian.org/releases/stretch/debian-
SPI Debian 9.13
installer/
Ubuntu Desktop 18.04 https://releases.ubuntu.com/18.04.6/
Ubuntu Server 20.04 https://hub.docker.com/_/ubuntu
Appendix C: Lab operating systems

Appendix D: Tenable.sc Asset Filebeat Configuration

# Integration: Tenable.sc 5.20.1
# Config Version: 1
# Built for Filebeat 7.17.4 & 7.17.5
# Author: Andrew Davidow
# Release Date: August 6, 2022
# Purpose: Collects asset data from Tenable.sc
# References:
# https://github.com/elastic/integrations/tree/main/packages/tenable_sc
filebeat.inputs:
- type: httpjson
config_version: 2
interval: 60s
request.url: ${tenable_url}/rest/analysis
request.method: POST
request.ssl:
certificate_authorities:
- "${ca_cert}"
verification_mode: certificate
#request.retry.wait_min:
#request.retry.wait_max:
#request.retry.max_attempts:
request.transforms:
- delete:
target: header.User-Agent
- set:
value: 'Integration/1.0 (Elastic; Tenable.sc Beat; Build/7.17.4)'
- set:
target: header.x-apikey
value: "accesskey=${tenable_access_key}; secretkey=${tenable_secret_key}"
- set:
target: header.Accept
value: "application/json"
- set:
target: body.type
value: 'vuln'
- set:
target: body.query.type
value: 'vuln'
- set:
target: body.query.tool
value: 'sumip'

- append:
target: body.query.filters
value: |-
{
"filterName":"lastSeen",
"operator":"=",
"value": '[[.cursor.last_event_ts]]-[[(now).Unix]]'
}
default: |-
{
"operator":"=",
"value": '[[(parseDate now (parseDuration "-168h")).Unix]]-[[(now).Unix]]'
}
value_type: json
- set:
target: body.sourceType
value: 'cumulative'
- set:
target: body.startOffset
value: 0
- set:
target: body.endOffset
value: 2000
response.split:
target: body.response.results
response.pagination:
- set:
value: '[[toInt .last_response.body.response.endOffset]]'
fail_on_template_error: true
- set:
value: '[[add (toInt .last_response.body.response.endOffset) 5000]]'
cursor:
last_event_ts:
value: '[[if (ge (toInt .last_response.body.response.endOffset) (toInt
.last_response.body.response.totalRecords))]][[toInt
.last_response.body.timestamp]][[end]]'
output.logstash:
hosts: ["${logstash_url}"]

Appendix E: Tenable.sc Asset Logstash Configuration

# Integration: Tenable.sc 5.20.1 Asset
# Config Version: 1
# Built for Logstash 7.17.4 & 7.17.5
# Purpose: Transforms the data for ingestion into Elasticsearch
# References:
# https://docs.tenable.com/tenablesc/api/index.htm
input {
beats {
port => 5044
}
}
filter {
mutate {
rename => {
"[message]" => "[event][original]"
}
}
json {
source => "[event][original]"
target => "[json]"
skip_on_invalid_json => true
}
date {
match => ["[json][@timestamp]", "ISO8601"]
tag_on_failure => ["_dateparsefailure"]
}
mutate {
add_field => {
"[event][category]" => "host"
"[event][type]" => "info"
"[event][kind]" => "state"
}
}
if [json][uniqueness] {
mutate {
split => {
"[json][uniqueness]" => ","
}
tag_on_failure => "_mutate_error"

}
ruby {
code => '
a = event.get("[json][uniqueness]")
uniqueKey = ""
a.each { |a|
if a == "repositoryID"
uniqueKey += event.get("[json][repository][id]")
else
uniqueKey += event.get("[json][#{a}]")
end
}
uniqueKey += "_"
event.set("[tenable_sc][asset][custom_hash]", uniqueKey)
'
tag_on_exception => "_rubyexception"
}
}
# Fingerprint custom SHA256 hash. Concatenates the names and values of all fields
given in the source option into one string before doing the fingerprint computation.
fingerprint {
method => "SHA256"
concatenate_sources => true
source => ["[tenable_sc][asset][custom_hash]"]
target => ["[tenable_sc][asset][custom_hash]"]
}
if [json][ip] {
mutate {
add_field => {
"[related][ip]" => "%{[json][ip]}"
"[host][ip]" => "%{[json][ip]}"
}
}
mutate {
rename => {
"[json][ip]" => "[tenable_sc][asset][ip]"
}
}
}
if [json][dnsName] {
# Dissect the dnsName into two new fields if it contains a dot "." into host.name
(everything up to first period) and host.domain.name (everything after)
if [json][dnsName] =~ /\./ {
dissect {
mapping => {
"[json][dnsName]" => "%{[host][name]}.%{[host][domain][name]}"

}
tag_on_failure => ["_dissectfailure"]
}
mutate {
add_field => {
"[related][hosts]" => "%{[host][name]}"
}
}
mutate {
copy => {
"[json][dnsName]" => "[host][hostname]"
}
}
mutate {
rename => {
"[json][dnsName]" => "[tenable_sc][asset][dns][name]"
}
}
} else {
mutate {
add_field => {
"[related][hosts]" => "[json][dnsName]}"
}
}
mutate {
copy => {
}
}
# Technically only the host.name and not the host.name plus host.domain.name
mutate {
rename => {
"[json][dnsName]" => "[tenable_sc][asset][dns][name]"
}
}
}
}
mutate {
rename => {
"[json][uuid]" => "[tenable_sc][asset][uuid]"
"[json][score]" => "[tenable_sc][asset][score]"
"[json][total]" => "[tenable_sc][asset][total]"
"[json][severityInfo]" => "[tenable_sc][asset][severity][info]"
"[json][severityLow]" => "[tenable_sc][asset][severity][low]"
"[json][severityMedium]" => "[tenable_sc][asset][severity][medium]"
"[json][severityHigh]" => "[tenable_sc][asset][severity][high]"

"[json][severityCritical]" => "[tenable_sc][asset][severity][critical]"

"[json][policyName]" => "[tenable_sc][asset][policy][name]"
"[json][pluginSet]" => "[tenable_sc][asset][plugin_set]"
"[json][osCPE]" => "[tenable_sc][asset][os_cpe]"
"[json][biosGUID]" => "[tenable_sc][asset][bios][guid]"
"[json][tpmID]" => "[tenable_sc][asset][tpm][id]"
"[json][mcafeeGUID]" => "[tenable_sc][asset][mcafee][guid]"
"[json][lastAuthRun]" => "[tenable_sc][asset][last_auth_run]"
"[json][lastUnauthRun]" => "[tenable_sc][asset][last_unauth_run]"
"[json][hostUniqueness]" => "[tenable_sc][asset][host_uniqueness]"
"[json][uniqueness]" => "[tenable_sc][asset][uniqueness]"
"[json][repository][id]" => "[tenable_sc][asset][repository][id]"
"[json][repository][name]" => "[tenable_sc][asset][repository][name]"
"[json][repository][description]" => "[tenable_sc][asset][repository][description]"
"[json][repository][sci][id]" => "[tenable_sc][asset][repository][sci][id]"
"[json][repository][dataFormat]" => "[tenable_sc][asset][repository][dataFormat]"
}
}
if [json][macAddress] {
mutate {
gsub => ["[json][macAddress]", "[-:.]", "-"]
tag_on_failure => "_mutate_error"
}
mutate {
uppercase => ["[json][macAddress]"]
}
mutate {
add_field => {
"[host][mac]" => "%{[json][macAddress]}"
}
}
mutate {
rename => {
"[json][macAddress]" => "[tenable_sc][asset][mac]"
}
}
}
if [json][netbiosName] {
if [json][netbiosName] not in [related][hosts] {
mutate {
add_field => {
"[related][hosts]" => "%{[json][netbiosName]}"
}
}
}
mutate {

rename => {
"[json][netbiosName]" => "[tenable_sc][asset][netbios][name]"
}
}
}
mutate {
remove_field => ["[json]"]
}
if "preserve_original_event" not in [tags] {

mutate {
remove_field => ["[event][original]"]
}
}
}
output {
if "_dateparsefailure" in [tags] or "_mutate_error" in [tags] or "_rubyexception" in
[tags] or "_dissectfailure" in [tags] {
stdout {
codec => rubydebug {
metadata => true
}
}
}
elasticsearch {
hosts => ["${es_url}"]
index => "asset"
user => "${logstash_es_user}"
password => "${logstash_es_user_pw}"
cacert => "${cacert_path}"
# truststore => "${truststore_path}"
# truststore_password => "${truststore_pw}"
}
}

Appendix F: Tenable.sc Asset Example Elasticsearch

Mapping
{
"asset" : {
"mappings" : {
"dynamic_templates" : [
{
"strings" : {
"match_mapping_type" : "string",
"mapping" : {
"fields" : {
"text" : {
"type" : "text"
}
},
"ignore_above" : 256,
"type" : "keyword"
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"agent" : {
"properties" : {
"ephemeral_id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"hostname" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"name" : {
"type" : "keyword",

"fields" : {
"text" : {
"type" : "text"
}
}
},
"type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"version" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"ecs" : {
"properties" : {
"version" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"event" : {
"properties" : {
"category" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"created" : {
"type" : "date"
},
"kind" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}

}
}
}
},
"host" : {
"properties" : {
"domain" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"hostname" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"ip" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "ip"
}
}
},
"mac" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"input" : {
"properties" : {
"type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}

},
"related" : {
"properties" : {
"hosts" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"ip" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"tags" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"tenable_sc" : {
"properties" : {
"asset" : {
"properties" : {
"bios" : {
"properties" : {
"guid" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"custom_hash" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"dns" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}

}
},
"host_uniqueness" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"ip" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"last_auth_run" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"last_unauth_run" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"mac" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"mcafee" : {
"properties" : {
"guid" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"netbios" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}

}
}
}
},
"os_cpe" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plugin_set" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"policy" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"repository" : {
"properties" : {
"dataFormat" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"description" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"name" : {
"type" : "keyword",
"fields" : {
"text" : {

"type" : "text"
}
}
}
}
},
"score" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"severity" : {
"properties" : {
"critical" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"high" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"info" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"low" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"medium" : {

"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
}
}
},
"total" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"tpm" : {
"properties" : {
"id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"uniqueness" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"uuid" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
}
}
}
}
}
}
}

Appendix G: Tenable.sc Plugin Filebeat Configuration

# Integration: Tenable.sc 5.20.1 Plugin
# Config Version: 1
# Built for Filebeat 7.17.4 and 7.17.5
# Purpose: Collects plugin data from Tenable.sc
# References:
# https://docs.tenable.com/tenablesc/api/Plugin.htm
filebeat.inputs:
- type: httpjson
config_version: 2
interval: 15m
request.url: ${tenable_url}/rest/plugin
request.method: GET
request.timeout: 60s
request.ssl:
- "${ca_cert}"
#request.retry.wait_min:
#request.retry.wait_max:
#request.retry.max_attempts:
request.transforms:
- delete:
- set:
- set:
# Fields to test: agent
# The following parameters must be passed in as query string (as opposed to JSON) in
the format of: /plugin?filterField=id&op=eq&value=1&...
# family,vprContext,xrefs
- set:
target: url.params.fields
value:
id,name,description,type,copyright,version,sourceFile,dependencies,requiredPorts,require
dUDPPorts,cpe,srcPort,dstPort,protocol,riskFactor,solution,seeAlso,synopsis,checkType,
exploitEase,exploitAvailable,exploitFrameworks,cvssVector,cvssVectorBF,baseScore,te

mporalScore,cvssV3Vector,cvssV3VectorBF,cvssV3BaseScore,cvssV3TemporalScore,v
prScore,stigSeverity,pluginPubDate,pluginModDate,patchPubDate,patchModDate,vulnP
ubDate,modifiedTime,md5
- set:
target: url.params.filterField
value: pluginModDate
- set:
target: url.params.op
value: gt
- set:
target: url.params.value
value: '[[.cursor.last_event_ts]]'
default: '[[(now (parseDuration "-720h")).Unix]]'
- set:
target: url.params.sortField
value: modifiedTime
- set:
target: url.params.sortDirection
value: ASC
- set:
target: url.params.startOffset
value: 0
- set:
target: url.params.endOffset
value: 50
- set:
target: url.params.startOffset
value: '[[toInt (.last_response.url.params.Get "endOffset")]]'
- set:
target: url.params.endOffset
value: '[[add (toInt (.last_response.url.params.Get "endOffset")) 50]]'
response.split:
target: body.response
cursor:
last_event_ts:
value: '[[if (lt (len .last_response.body.response)
50)]][[.last_event.pluginModDate]][[end]]'
output.logstash:
hosts: ["localhost:5044"]

Appendix H: Tenable.sc Plugin Logstash Configuration

# Integration: Tenable.sc 5.20.1 Plugin
# Config Version: 1
# Built for Logstash 7.17.4 and 7.17.5
# References:
input {
beats {
port => 5044
}
}
filter {
mutate {
rename => {
}
}
json {
target => "[json]"
}
date {
match => ["[json][modifiedTime]", "UNIX"]
tag_on_failure => ["_dateparsefailure"]
}
# Fingerprint. Sets the indexed document's _id to a SHA-256 hash. Concatenates the
names and values of all fields given in the source option into one string before doing the
fingerprint computation.
fingerprint {
method => "SHA256"
source => [
"[json][modifiedTime]",
"[json][id]",
"[json][name]",
"[json][type]",
"[json][description]",
"[json][version]",

"[json][sourceFile]",
"[json][cpe]",
"[json][dependencies]",
"[json][riskFactor]",
"[json][vprContext]",
"[json][vprScore]",
"[json][baseScore]",
"[json][temporalScore]",
"[json][cvssVector]",
"[json][cvssV3Vector]",
"[json][pluginModDate]",
"[json][patchPubDate]",
"[json][vulnPubDate]",
"[json][family]"
]
target => "[@metadata][_id]"
}
# ECS event fields
mutate {
add_field => {
"[event][category]" => "configuration"
"[event][kind]" => "event"
}
}
# Rename fields
mutate {
rename => {
"[json][agent]" => "[tenable_sc][plugin][agent]"
"[json][id]" => "[tenable_sc][plugin][id]"
"[json][name]" => "[tenable_sc][plugin][name]"
"[json][description]" => "[tenable_sc][plugin][description]"
"[json][type]" => "[tenable_sc][plugin][type]"
"[json][copyright]" => "[tenable_sc][plugin][copyright]"
"[json][version]" => "[tenable_sc][plugin][version]"
"[json][sourceFile]" => "[tenable_sc][plugin][source_file]"
"[json][requiredPorts]" => "[tenable_sc][plugin][required_ports]"
"[json][requiredUDPPorts]" => "[tenable_sc][plugin][required_udp_ports]"
"[json][srcPort]" => "[tenable_sc][plugin][src_port]"
"[json][dstPort]" => "[tenable_sc][plugin][dst_port]"
"[json][protocol]" => "[tenable_sc][plugin][protocol]"
"[json][riskFactor]" => "[tenable_sc][plugin][risk_factor]"
"[json][solution]" => "[tenable_sc][plugin][solution]"
"[json][synopsis]" => "[tenable_sc][plugin][synopsis]"
"[json][checkType]" => "[tenable_sc][plugin][check_type]"
"[json][exploitEase]" => "[tenable_sc][plugin][exploit][ease]"

"[json][exploitAvailable]" => "[tenable_sc][plugin][exploit][is_available]"

"[json][exploitFrameworks]" => "[tenable_sc][plugin][exploit][frameworks]"
"[json][cvssVector]" => "[tenable_sc][plugin][cvss_vector]"
"[json][cvssVectorBF]" => "[tenable_sc][plugin][cvss_vector_bf]"
"[json][baseScore]" => "[tenable_sc][plugin][base_score]"
"[json][temporalScore]" => "[tenable_sc][plugin][temporal_score]"
"[json][cvssV3Vector]" => "[tenable_sc][plugin][cvssv3_vector]"
"[json][cvssV3VectorBF]" => "[tenable_sc][plugin][cvssv3_vector_bf]"
"[json][cvssV3BaseScore]" => "[tenable_sc][plugin][cvssv3_base_score]"
"[json][cvssV3TemporalScore]" =>
"[tenable_sc][plugin][cvssv3_temporal_score]"
"[json][stigSeverity]" => "[tenable_sc][plugin][stig_severity]"
"[json][md5]" => "[tenable_sc][plugin][md5]"
"[json][source]" => "[tenable_sc][plugin][source]"
"[json][family][id]" => "[tenable_sc][plugin][family][id]"
"[json][family][name]" => "[tenable_sc][plugin][family][name]"
"[json][family][type]" => "[tenable_sc][plugin][family][type]"
}
}
# dependencies. Split dependencies on comma and rename
mutate {
split => {
"[json][dependencies]" => ","
}
}
mutate {
rename => {
"[json][dependencies]" => "[tenable_sc][plugin][dependencies]"
}
}
# seeAlso. Change \n to a space then split into an array on the space
mutate {
gsub => ["[json][seeAlso]", "\n", " "]
tag_on_failure => "_mutate_error_seeAlso"
}
mutate {
split => {
"[json][seeAlso]" => " "
}
}
mutate {
rename => {
"[json][seeAlso]" => "[tenable_sc][plugin][see_also]"
}
}

# CPE. Change \n to a space then split into an array on the space

mutate {
gsub => ["[json][cpe]", "\n", " "]
tag_on_failure => "_mutate_error_cpe"
}
mutate {
split => {
"[json][cpe]" => " "
}
tag_on_failure => "_mutate_error_cpe"
}
mutate {
rename => {
"[json][cpe]" => "[tenable_sc][plugin][cpe]"
}
}
# Rename vprScore
if [json][vprScore] {
mutate {
rename => {
"[json][vprScore]" => "[tenable_sc][plugin][vpr][score]"
}
}
}
# Sets the vprContext "id" field's value to the new field's name
# Sets the vprContext "value" field's value to that the new field's value
if [json][vprContext] {
json {
source => "[json][vprContext]"
target => "[json][vprContext]"
tag_on_failure => ["_jsonparsefailure_vprContext"]
}
ruby {
code => '
a = event.get("[json][vprContext]")
a.each { |a|
event.set("[tenable_sc][plugin][vpr][context][#{a["id"]}]", a["value"])
}
'
tag_on_exception => "_rubyexception_vprContext"
}
}
# If the plugin published date is "-1" set it to false, else set it to true and parse the
modification date into a new field
if [json][pluginPubDate] {
if [json][pluginPubDate] == "-1" {

mutate {
add_field => {
"[tenable_sc][plugin][is_plugin_published]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][plugin][is_plugin_published]" => "true"
}
}
date {
match => ["[json][pluginPubDate]", "UNIX"]
target => "[tenable_sc][plugin][plugin_pub_date]"
tag_on_failure => ["_dateparsefailure_plugin_pub_date"]
}
}
}
# If the pluginModDate is "-1" set it to false, else set it to true and parse the
if [json][pluginModDate] {
if [json][pluginModDate] == "-1" {
mutate {
add_field => {
"[tenable_sc][plugin][is_plugin_modified]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][plugin][is_plugin_modified]" => "true"
}
}
date {
match => ["[json][pluginModDate]", "UNIX"]
target => "[tenable_sc][plugin][plugin_mod_date]"
tag_on_failure => ["_dateparsefailure_plugin_mod_date"]
}
}
}
# If the patchPubDate is "-1" set it to false, else set it to true and parse the modification
date into a new field
if [json][patchPubDate] {
if [json][patchPubDate] == "-1" {

mutate {
add_field => {
"[tenable_sc][plugin][is_patch_published]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][plugin][is_patch_published]" => "true"
}
}
date {
match => ["[json][patchPubDate]", "UNIX"]
target => "[tenable_sc][plugin][patch_pub_date]"
tag_on_failure => ["_dateparsefailure_patch_pub_date"]
}
}
}
# If the patchModDate is "-1" set it to false, else set it to true and parse the
if [json][patchModDate] {
if [json][patchModDate] == "-1" {
mutate {
add_field => {
"[tenable_sc][plugin][is_patch_published]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][plugin][is_patch_published]" => "true"
}
}
date {
match => ["[json][patchModDate]", "UNIX"]
target => "[tenable_sc][plugin][patch_mod_date]"
tag_on_failure => ["_dateparsefailure_patch_mod_date"]
}
}
}
# If the vulnPubDate is "-1" set it to false, else set it to true and parse the modification
if [json][vulnPubDate] {
if [json][vulnPubDate] == "-1" {

mutate {
add_field => {
"[tenable_sc][plugin][is_vulnerability_published]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][plugin][is_vulnerability_published]" => "true"
}
}
date {
match => ["[json][vulnPubDate]", "UNIX"]
target => "[tenable_sc][plugin][vuln_pub_date]"
tag_on_failure => ["_dateparsefailure_vuln_pub_date"]
}
}
}
date {
match => ["[json][modifiedTime]", "UNIX"]
target => "[tenable_sc][plugin][modified_time]"
tag_on_failure => ["_dateparsefailure_modified_time"]
}
# Split the xref field into an array using a comma delimiter
if [json][xrefs] {
mutate {
split => {
"[json][xrefs]" => ","
}
}
mutate {
rename => {
"[json][xrefs]" => "[tenable_sc][vulnerability][xrefs]"
}
}
}
mutate {
}
mutate {
}
}

output {
elasticsearch {
index => "plugin"
#document_id => "%{[@metadata][_id]}"
#truststore => "${truststore_path}"
#truststore_password => "${truststore_pw}"
}
}

Appendix I: Tenable.sc Plugin Example Elasticsearch

Mapping
{
"plugin" : {
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"agent" : {
"properties" : {
"ephemeral_id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"hostname" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"version" : {

"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"ecs" : {
"properties" : {
"version" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"event" : {
"properties" : {
"category" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"created" : {
"type" : "date"
},
"kind" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"input" : {
"properties" : {
"type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}

},
"tags" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"tenable_sc" : {
"properties" : {
"plugin" : {
"properties" : {
"agent" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"base_score" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"check_type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"copyright" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"cpe" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"cvss_vector" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}

}
},
"cvss_vector_bf" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"cvssv3_base_score" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"cvssv3_temporal_score" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"cvssv3_vector" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"cvssv3_vector_bf" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"dependencies" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"description" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}

}
},
"dst_port" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"exploit" : {
"properties" : {
"ease" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"frameworks" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"is_available" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"boolean" : {
"type" : "boolean"
}
}
}
}
},
"family" : {
"properties" : {
"id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"type" : {

"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"is_patch_published" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"boolean" : {
"type" : "boolean"
}
}
},
"is_patch_modified" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"boolean" : {
"type" : "boolean"
}
}
},
"is_plugin_modified" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"boolean" : {
"type" : "boolean"
}
}
},
"is_plugin_published" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"boolean" : {
"type" : "boolean"
}
}
},
"is_vulnerability_published" : {
"type" : "keyword",

"fields" : {
"text" : {
"type" : "text"
},
"boolean" : {
"type" : "boolean"
}
}
},
"md5" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"modified_time" : {
"type" : "date"
},
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"patch_pub_date" : {
"type" : "date"
},
"plugin_mod_date" : {
"type" : "date"
},
"plugin_pub_date" : {
"type" : "date"
},
"protocol" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"required_ports" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"required_udp_ports" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"risk_factor" : {
"type" : "keyword",

"fields" : {
"text" : {
"type" : "text"
}
}
},
"see_also" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"solution" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"source" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"source_file" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"src_port" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"stig_severity" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"synopsis" : {
"type" : "keyword",
"fields" : {
"text" : {

"type" : "text"
}
}
},
"temporal_score" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"version" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"vpr" : {
"properties" : {
"score" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
},
"long" : {
"type" : "long"
}
}
},
"context": {
"properties": {
"age_of_vuln": {
"type": "keyword",
"ignore_above": 256,
"fields": {
"text": {
"type": "text"
}
}
},
"cvssV3_impactScore": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"float" : {
"type" : "float"
}

}
},
"exploit_code_maturity": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"product_coverage": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"threat_intensity_last_28": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"threat_recency": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"threat_sources_last_28": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
}
}
},
"vuln_pub_date" : {
"type" : "date"
},
"xrefs" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
}
}
}
}
}

}
}

Appendix J: Tenable.sc Vulnerability Filebeat Configuration

# Config Version: 1
# Built for Filebeat 7.17.4 and 7.17.5
# Purpose: Collects vulnerability data from Tenable.sc in a given time range in given
chunks
# References:
filebeat.inputs:
- type: httpjson
config_version: 2
interval: 15m
request.url: ${tenable_url}/rest/analysis
request.method: POST
request.ssl:
- "${ca_cert}"
request.transforms:
- set:
- set:
- set:
target: header.Accept
value: "application/json"
- set:
target: body.type
value: 'vuln'
- set:
target: body.query.type
value: 'vuln'
- set:
target: body.query.tool
value: 'vulndetails'
- set:
target: body.query.filters
value: |-
{

"operator":"=",
"value": '[[.cursor.last_event_ts]]-[[(now).Unix]]'
}
default: |-
{
"operator":"=",
"value": '[[(now (parseDuration "-24h")).Unix]]-[[(now).Unix]]'
}
- set:
target: body.sortDir
value: 'DESC'
- set:
target: body.sourceType
value: 'cumulative'
- set:
target: body.sortField
value: 'lastSeen'
- set:
value: 0
- set:
value: 1000
response.split:
target: body.response.results
- set:
value: '[[toInt .last_response.body.response.endOffset]]'
- set:
value: '[[add (toInt .last_response.body.response.endOffset) 1000]]'
cursor:
last_event_ts:
value: '[[if (ge (toInt .last_response.body.response.endOffset) (toInt
.last_response.body.response.totalRecords))]][[.last_event.lastSeen]][[end]]'
output.logstash:
hosts: ["${logstash_url}"]

Appendix K: Tenable.sc Vulnerability Logstash Configuration

# Config Version: 1
# Built for Logstash 7.17.4 and 7.17.5
# References:
input {
beats {
port => 5044
}
}
filter {
mutate {
rename => {
}
}
# JSON conversion of the event
json {
target => "[json]"
}
# Fingerprint. Sets the indexed document's _id to a SHA-256 hash. Concatenates the
names and values of all fields given in the source option into one string before doing the
fingerprint computation.
fingerprint {
method => "SHA256"
source => [
"[json][pluginID]",
"[json][ip]",
"[json][uuid]",
"[json][firstSeen]",
"[json][lastSeen]",
"[json][exploitAvailable]",
"[json][vulnPubDate]",
"[json][patchPubDate]",
"[json][pluginPubDate]",

"[json][pluginModDate]",
"[json][pluginText]",
"[json][dnsName]",
"[json][macAddress]",
"[json][operatingSystem]",
"[json][pluginInfo]"
]
target => "[@metadata][_id]"
}
# Copy the @timestamp of the document back to the lastSeen value (when the scan
occurred)
date {
match => ["[json][lastSeen]", "UNIX"]
tag_on_failure => ["_dateparsefailure_timestamp"]
}
# Copy the formatted @timestamp into [tenable_sc][vulnerability][last_seen]
mutate {
copy => {
"[@timestamp]" => "[tenable_sc][vulnerability][last_seen]"
}
}
# Add fields for the event category, type, and kind
mutate {
add_field => {
"[event][category]" => "threat"
"[event][kind]" => "event"
}
}
# Vendor set to Tenable
mutate {
add_field => {
"[vulnerability][scanner][vendor]" => "Tenable"
}
}
# CVSS Version 3. If there's a value in the CVSS version 3 field [cvssV3*] fields, set
the vulnerability.score.version to 3
if [json][cvssV3BaseScore] or [json][cvssV3TemporalScore] {
mutate {
add_field => {
"[vulnerability][score][version]" => "3.0"
"[vulnerability][classification]" => "CVSS"
}
}
# CVSS Version 3. Rename the fields to ECS. The filter ignores if they don't exist.
mutate {

rename => {
"[json][cvssV3BaseScore]" => "[vulnerability][score][base]"
"[json][cvssV3TemporalScore]" => "[vulnerability][score][temporal]"
}
}
}
# Change \n to a space then split into an array on the space
if [json][seeAlso] {
mutate {
gsub => ["[json][seeAlso]", "\n", " "]
}
mutate {
split => {
"[json][seeAlso]" => " "
}
}
mutate {
rename => {
"[json][seeAlso]" => "[vulnerability][reference]"
}
}
}
# CVE. List of CVEs separated by a comma. If it's not empty, this builds an array of
CVE reference links and sets the appropriate fields
if [json][cve] {
mutate {
add_field => {
"[vulnerability][enumeration]" => "CVE"
}
}
mutate {
split => {
"[json][cve]" => ","
}
}
mutate {
copy => {
"[json][cve]" => "[vulnerability][id]"
}
}
mutate {
copy => {
"[json][cve]" => "[vulnerability][reference]"
}

}
# This builds the vulnerability.reference field by prepending the Mitre URL to the
list of CVEs
# The regex pattern (?<=^) identifies the start of a newline.
mutate {
gsub => ["[vulnerability][reference]", "(?<=^)", "https://cve.mitre.org/cgi-
bin/cvename.cgi?name="]
tag_on_failure => "_mutate_error_gsub_vulnref"
}
}
# Rename fields using ECS
mutate {
rename => {
"[json][description]" => "[vulnerability][description]"
"[json][severity][name]" => "[vulnerability][severity]"
"[json][severity][id]" => "[tenable_sc][vulnerability][id]"
}
}
# Set tenable_sc.vulnerability.id to the values of the fields listed in uniqueness
if [json][uniqueness] {
# Replaces the string "repositoryID" with the string value of the numeric repository
ID
# For example, if {[json][uniqueness] : "repositoryID,IP,dns"} and if
{[json][repository][id] : "5"} then {[json][uniqueness] : "5,IP,dns"}
mutate {
split => {
"[json][uniqueness]" => ","
}
}
ruby {
code => '
a = event.get("[json][uniqueness]")
uniqueKey = ""
a.each { |a|
if a == "repositoryID"
uniqueKey += event.get("[json][repository][id]")
else
uniqueKey += event.get("[json][#{a}]")
end
}
uniqueKey += "_"
event.set("[tenable_sc][vulnerability][id]", uniqueKey)
'
tag_on_exception => "_rubyexception"
}
}

# Fingerprint custom SHA256 hash. Concatenates the names and values of all fields
given in the source option into one string before doing the fingerprint computation.
fingerprint {
method => "SHA256"
source => [
"[json][pluginID]",
"[json][port]",
"[json][protocol]",
"[tenable_sc][vulnerability][id]"
]
target => "[tenable_sc][vulnerability][custom_hash]"
}
# Rename plugin fields
mutate {
rename => {
"[json][pluginID]" => "[tenable_sc][vulnerability][plugin][id]"
"[json][pluginName]" => "[tenable_sc][vulnerability][plugin][name]"
"[json][pluginText]" => "[tenable_sc][vulnerability][plugin][text]"
}
}
# Rename risk fields
mutate {
rename => {
"[json][severity][description]" =>
"[tenable_sc][vulnerability][severity][description]"
"[json][acceptRisk]" => "[tenable_sc][vulnerability][accept_risk]"
"[json][recastRisk]" => "[tenable_sc][vulnerability][recast_risk]"
}
}
# If hasBeenMitigated is 0, set the field to false; if it is 1 set the field to true
if [json][hasBeenMitigated] == "0" {
mutate {
add_field => {
"[tenable_sc][vulnerability][has_been_mitigated]" => "false"
}
}
}
else if [json][hasBeenMitigated] == "1" {
mutate {
add_field => {
"[tenable_sc][vulnerability][has_been_mitigated]" => "true"
}
}
}
else {

mutate {
add_tag => ["preserve_original_event", "_has_been_mitigated"]
}
}
# Rename vprScore
if [json][vprScore] {
mutate {
rename => {
"[json][vprScore]" => "[tenable_sc][vulnerability][vpr][score]"
}
}
}
# Sets the vprContext "id" field's value to the new field's name
# Sets the vprContext "value" field's value to that the new field's value
if [json][vprContext] {
json {
source => "[json][vprContext]"
target => "[json][vprContext]"
tag_on_failure => ["_jsonparsefailure_vprContext"]
}
ruby {
code => '
a = event.get("[json][vprContext]")
a.each { |a|
event.set("[tenable_sc][vulnerability][vpr][context][#{a["id"]}]", a["value"])
}
'
tag_on_exception => "_rubyexception_vprContext"
}
}
# IP fields
if [json][ip] {
mutate {
add_field => {
"[related][ip]" => "%{[json][ip]}"
"[host][ip]" => "%{[json][ip]}"
}
}
mutate {
rename => {
"[json][ip]" => "[tenable_sc][vulnerability][ip]"
}
}
}
# Port and Proto fields
mutate {

rename => {
"[json][port]" => "[tenable_sc][vulnerability][port]"
"[json][protocol]" => "[tenable_sc][vulnerability][protocol]"
}
}
# Parse the firstSeen date and rename
date {
match => ["[json][firstSeen]", "UNIX"]
target => "[tenable_sc][vulnerability][first_seen]"
}
# If an exploit is available set to true, else if an exploit is not available set to false
if [json][exploitAvailable] == "Yes" {
mutate {
add_field => {
"[tenable_sc][vulnerability][exploit][is_available]" => "true"
}
}
}
else if [json][exploitAvailable] == "No" {
mutate {
add_field => {
"[tenable_sc][vulnerability][exploit][is_available]" => "false"
}
}
}
else {
mutate {
add_tag => ["preserve_original_event", "_exploit_is_available"]
}
}
# Rename fields
mutate {
rename => {
"[json][exploitEase]" => "[tenable_sc][vulnerability][exploit][ease]"
"[json][exploitFrameworks]" =>
"[tenable_sc][vulnerability][exploit][frameworks]"
"[json][solution]" => "[tenable_sc][vulnerability][solution]"
"[json][synopsis]" => "[tenable_sc][vulnerability][synopsis]"
"[json][riskFactor]" => "[tenable_sc][vulnerability][risk_factor]"
"[json][stigSeverity]" => "[tenable_sc][vulnerability][stig_severity]"
"[json][baseScore]" => "[tenable_sc][vulnerability][base_score]"
"[json][temporalScore]" => "[tenable_sc][vulnerability][temporal_score]"
"[json][cvssVector]" => "[tenable_sc][vulnerability][cvss_vector]"
"[json][cvssV3Vector]" => "[tenable_sc][vulnerability][cvss_v3_vector]"
}
}

# Split the cpe field line breaks into an array

mutate {
split => {
"[json][cpe]" => '<br/>'
}
}
# Rename CPE field
mutate {
rename => {
"[json][cpe]" => "[tenable_sc][vulnerability][cpe]"
}
}
# If a vulnerability is not published set this to false, else set it to true and parse the
published date into a new field
if [json][vulnPubDate] == "-1" {
mutate {
add_field => {
"[tenable_sc][vulnerability][is_vulnerability_published]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][vulnerability][is_vulnerability_published]" => "true"
}
}
date {
match => ["[json][vulnPubDate]", "UNIX"]
target => "[tenable_sc][vulnerability][vuln_pub_date]"
tag_on_failure => ["_dateparsefailure_vuln_pub_date"]
}
}
# If a patch is not published set this to false, else set it to true and parse the published
if [json][patchPubDate] == "-1" {
mutate {
add_field => {
"[tenable_sc][vulnerability][patch][is_published]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][vulnerability][patch][is_published]" => "true"

}
}
date {
match => ["[json][patchPubDate]", "UNIX"]
target => "[tenable_sc][vulnerability][patch][pub_date]"
tag_on_failure => ["_dateparsefailure_patch_pub_date"]
}
}
# If the plugin is not modified set this to false, else set it to true and parse the
if [json][pluginModDate] == "-1" {
mutate {
add_field => {
"[tenable_sc][vulnerability][plugin][is_modified]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][vulnerability][plugin][is_modified]" => "true"
}
}
date {
match => ["[json][pluginModDate]", "UNIX"]
target => "[tenable_sc][vulnerability][plugin][mod_date]"
tag_on_failure => ["_dateparsefailure_plugin_mod_date"]
}
}
# If the plugin is not published set this to false, else set it to true and parse the publish
if [json][pluginPubDate] == "-1" {
mutate {
add_field => {
"[tenable_sc][vulnerability][plugin][is_published]" => "false"
}
}
}
else {
mutate {
add_field => {
"[tenable_sc][vulnerability][plugin][is_published]" => "true"
}
}
date {
match => ["[json][pluginPubDate]", "UNIX"]

target => "[tenable_sc][vulnerability][plugin][pub_date]"

tag_on_failure => ["_dateparsefailure_plugin_pub_date"]
}
}
# Rename fields
mutate {
rename => {
"[json][uuid]" => "[tenable_sc][vulnerability][uuid]"
"[json][checkType]" => "[tenable_sc][vulnerability][check_type]"
"[json][version]" => "[tenable_sc][vulnerability][version]"
"[json][bid]" => "[tenable_sc][vulnerability][bid]"
}
}
# Split the xref field into an array using a comma delimiter
if [json][xref] {
mutate {
split => {
"[json][xref]" => ","
}
}
mutate {
rename => {
"[json][xref]" => "[tenable_sc][vulnerability][xref]"
}
}
}
# If the dnsName is not null
if [json][dnsName] {
# if it contains a dot "." dissect the dnsName into host.name and host.domain.name
and add/append fields
if [json][dnsName] =~ /\./ {
dissect {
mapping => {
"[json][dnsName]" => "%{[host][name]}.%{[host][domain][name]}"
}
tag_on_failure => ["_dissectfailure"]
}
mutate {
add_field => {
}
}
mutate {
copy => {
}

}
mutate {
rename => {
"[json][dnsName]" => "[tenable_sc][vulnerability][dns][name]"
}
}
} else {
mutate {
add_field => {
"[related][hosts]" => "[json][dnsName]}"
}
}
mutate {
copy => {
}
}
mutate {
rename => {
"[json][dnsName]" => "[tenable_sc][vulnerability][dns][name]"
}
}
}
}
# netbios field
if [json][netbiosName] {
# If it's not in related.hosts, add it
if [json][netbiosName] not in [related][hosts] {
mutate {
add_field => {
"[related][hosts]" => "%{[json][netbiosName]}"
}
}
}
mutate {
rename => {
"[json][netbiosName]" => "[tenable_sc][vulnerability][netbios][name]"
}
}
}
# MAC address
if [json][macAddress] {
# substitute the regex set "[-:.]" with "-"
mutate {
gsub => ["[json][macAddress]", "[-:.]", "-"]
tag_on_failure => "_mutate_error_gsub_mac_address"

}
mutate {
uppercase => ["[json][macAddress]"]
}
mutate {
add_field => {
"[host][mac]" => "%{[json][macAddress]}"
}
}
mutate {
rename => {
"[json][macAddress]" => "[tenable_sc][vulnerability][mac]"
}
}
}
mutate {
add_field => {
"[host][os][full]" => "%{[json][operatingSystem]}"
}
}
mutate {
rename => {
"[json][operatingSystem]" => "[tenable_sc][vulnerability][operating_system]"
}
}
mutate {
rename => {
"[json][uniqueness]" => "[tenable_sc][vulnerability][uniqueness]"
}
}
mutate {
rename => {
"[json][hostUniqueness]" => "[tenable_sc][vulnerability][host_uniqueness]"
}
}
mutate {
rename => {
"[json][family][id]" => "[tenable_sc][vulnerability][family][id]"
}
}
if [json][family][name] {
mutate {
add_field => {
"[vulnerability][category]" => "%{[json][family][name]}"
}
}

mutate {
rename => {
"[json][family][name]" => "[tenable_sc][vulnerability][family][name]"
}
}
}
mutate {
rename => {
"[json][family][type]" => "[tenable_sc][vulnerability][family][type]"
}
}
# Rename fields
mutate {
rename => {
"[json][repository][id]" => "[tenable_sc][vulnerability][repository][id]"
"[json][repository][name]" => "[tenable_sc][vulnerability][repository][name]"
"[json][repository][description]" =>
"[tenable_sc][vulnerability][repository][description]"
"[json][repository][sciID]" => "[tenable_sc][vulnerability][repository][sci_id]"
"[json][repository][dataFormat]" =>
"[tenable_sc][vulnerability][repository][data_format]"
}
}
# Remove the left over JSON conversion of the event
mutate {
}
mutate {
}
}
}
output {
elasticsearch {
index => "vuln"
document_id => "%{[@metadata][_id]}"
#truststore => "${truststore_path}"
#truststore_password => "${truststore_pw}"
}
}

Appendix L: Tenable.sc Vulnerability Example Elasticsearch

Mapping
{
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
},
"event": {
"properties": {
"category": {
"type": "keyword"
},
"kind": {
"type": "keyword"
},
"type": {
"type": "keyword"
}
}
},
"host": {
"properties": {
"domain": {
"properties": {
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"hostname": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"ip" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "ip"
}
}
},
"mac": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"name": {
"type": "keyword",
"fields": {

"text": {
"type": "text"
}
}
},
"os": {
"properties": {
"full": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
}
}
},
"related": {
"properties": {
"hosts": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"ip" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "ip"
}
}
}
}
},
"tenable_sc": {
"properties": {
"vulnerability": {
"properties": {
"accept_risk": {
"type": "keyword"
},
"base_score": {
"type": "keyword"
},
"bid": {
"type": "keyword"
},
"check_type": {
"type": "keyword"
},
"cpe": {
"type": "keyword"
},
"custom_hash": {
"type": "keyword"
},
"cvss_v3_vector": {
"type": "keyword"
},
"cvss_vector": {
"type": "keyword"
},
"dns": {
"properties": {

"name": {
"type": "keyword"
}
}
},
"exploit": {
"properties": {
"ease": {
"type": "keyword"
},
"frameworks": {
"type": "keyword"
},
"is_available": {
"type": "boolean"
}
}
},
"family": {
"properties": {
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"type": {
"type": "keyword"
}
}
},
"first_seen": {
"type": "date"
},
"has_been_mitigated": {
"type": "boolean"
},
"host_uniqueness": {
"type": "keyword"
},
"id": {
"type": "keyword"
},
"ip" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "ip"
}
}
},
"is_vulnerability_published": {
"type": "boolean"
},
"last_seen": {
"type": "date"
},
"mac": {
"type": "keyword"
},
"netbios": {
"properties": {
"name": {
"type": "keyword"
}
}
},
"operating_system": {
"type": "keyword"

},
"patch": {
"properties": {
"is_published": {
"type": "boolean"
},
"pub_date": {
"type": "date"
}
}
},
"plugin": {
"properties": {
"id": {
"type": "keyword"
},
"is_published": {
"type": "boolean"
},
"is_modified": {
"type": "boolean"
},
"mod_date": {
"type": "date"
},
"name": {
"type": "keyword"
},
"pub_date": {
"type": "date"
},
"text": {
"type": "text"
}
}
},
"port": {
"type": "keyword",
"fields": {
"integer": {
"type": "integer"
}
}
},
"protocol": {
"type": "keyword"
},
"recast_risk": {
"type": "keyword"
},
"repository": {
"properties": {
"data_format": {
"type": "keyword"
},
"description": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"

},
"sci_id": {
"type": "keyword"
}
}
},
"risk_factor": {
"type": "keyword"
},
"severity": {
"properties": {
"description": {
"type": "keyword"
},
"id": {
"type": "keyword"
}
}
},
"solution": {
"type": "text"
},
"stig_severity": {
"type": "keyword"
},
"synopsis": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"temporal_score": {
"type": "keyword"
},
"uniqueness": {
"type": "keyword"
},
"uuid": {
"type": "keyword"
},
"version": {
"type": "keyword"
},
"vpr": {
"properties": {
"context": {
"properties": {
"age_of_vuln": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"cvssV3_impactScore": {
"type": "float"
},
"exploit_code_maturity": {
"type": "keyword"
},
"product_coverage": {
"type": "keyword"
},
"threat_intensity_last_28": {

"type": "keyword"
},
"threat_recency": {
"type": "keyword"
},
"threat_sources_last_28": {
"type": "keyword"
}
}
},
"score": {
"type": "float"
}
}
},
"vuln_pub_date": {
"type": "date"
},
"xref": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
}
}
},
"vulnerability": {
"properties": {
"category": {
"type": "keyword"
},
"classification": {
"type": "keyword"
},
"description": {
"type": "text"
},
"enumeration": {
"type": "keyword"
},
"id": {
"type": "keyword",
"ignore_above": 256
},
"reference": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"scanner": {
"properties": {
"vendor": {
"type": "keyword"
}
}
},
"score": {
"properties": {
"base": {
"type": "float"

},
"temporal": {
"type": "float"
},
"version": {
"type": "keyword"
}
}
},
"severity": {
"type": "keyword"
}
}
}
}
}
}

Appendix M: BigFix Web Reports Hardware Logstash

Configuration
# Integration: BigFix Platform 10.0.4, 10.0.5, 10.0.6
# Config Version: 1
# Built for Logstash 7.17.4, 7.17.5, 8.1.2, 8.2.2, 8.3.2
# Purpose: Transforms BigFix Web Reports hardware fields for ingestion into
Elasticsearch
input {
exec {
command => "sh -c /home/student/get-hwam.sh"
interval => 900
ecs_compatibility => "v8"
}
}
filter {
mutate {
rename => {
}
}
json {
target => "[json]"
tag_on_failure => ["preserve_original_event"]
}
mutate {
}
}
mutate {
remove_field => ["[process]", "[host]"]
}
split {
field => "[json][results]"
}
date {
match => ["[json][results][properties][R-Last Report Time][result]", "E, d MMM yyyy
HH:mm:ss Z",
"E, dd MMM yyyy HH:mm:ss Z"]

}
mutate {
add_field => {
"[agent][type]" => "bes client"
}
}
mutate {
add_field => {
"[ecs][version]" => "8.2"
"[event][provider]" => "BigFix"
}
}
mutate {
rename => {
"[json][results][computerHref]" => "[event][url]"
"[json][results][properties][R-Computer Name][result]" => "[host][name]"
"[json][results][properties][R-ID][result]" => "[host][id]"
"[json][results][properties][R-OS][result]" => "[host][os][full]"
"[json][results][properties][O-3093-35-18][result]" => "[host][cpu][cores]"
"[json][results][properties][O-3093-305-16][result]" => "[host][cpu][cores]"
"[json][results][properties][O-3093-35-6][result]" => "[host][cpu][processors]"
"[json][results][properties][O-3093-305-4][result]" => "[host][cpu][processors]"
"[json][results][properties][R-MAC Address][result]" => "[host][mac]"
"[json][results][properties][R-CPU][result]" => "[host][cpu][name]"
"[json][results][properties][O-13014-19-2][result]" => "[agent][scanner][version]"
"[json][results][properties][O-1-205-3][result]" => "[agent][relay][name]"
"[json][results][properties][D/Subnet%20Address/TjwTe57B3z9HXSvUiuNLLO8eV%2
bg][result]" => "[host][subnet]"
"[json][results][properties][O-3093-305-11][result]" => "[host][network][adapters]"
"[json][results][properties][O-3093-35-11][result]" => "[host][network][adapters]"
"[json][results][properties][O-3093-35-8][result]" => "[host][drive][count]"
"[json][results][properties][O-9307-34-2][result]" => "[host][model]"
"[json][results][properties][O-1-204-1][result]" => "[agent][version]"
"[json][results][properties][O-3093-33-1][result]" => "[host][dhcp_server]"
"[json][results][properties][O-3093-305-5][result]" => "[host][drive][names]"
"[json][results][properties][O-3093-35-9][result]" => "[host][drive][names]"
"[json][results][properties][O-3093-35-4][result]" => "[host][is_laptop]"
"[json][results][properties][O-3093-305-2][result]" => "[host][is_laptop]"
"[json][results][properties][O-3093-35-1][result]" => "[host][manufacturer]"

"[json][results][properties][O-3093-33-2][result]" => "[host][dns][servers]"

"[json][results][properties][O-9307-34-1][result]" => "[host][make]"
"[json][results][properties][O-3093-35-5][result]" => "[host][cpu][brand]"
"[json][results][properties][O-3093-305-3][result]" => "[host][cpu][brand]"
"[json][results][properties][R-Computer Type][result]" => "[host][type]"
}
tag_on_failure => "_rename_bulk"
}
#"[json][results][properties][][result]" => "[host][]"
if [json][results][properties][R-DNS Name][result] {
# Dissect the DNS Name into two new fields if it contains a dot
if [json][results][properties][R-DNS Name][result] =~ /\./ {
dissect {
mapping => {
"[json][results][properties][R-DNS Name][result]" =>
"%{[host][name]}.%{[host][domain][name]}"
}
tag_on_failure => ["_dissectfailure_DNS"]
}
mutate {
add_field => {
}
}
mutate {
rename => {
"[json][results][properties][R-DNS Name][result]" => "[host][hostname]"
}
}
}
}
else {
mutate {
add_field => {
"[related][hosts]" => "%{[json][results][properties][R-DNS Name][result]}"
}
}
mutate {
rename => {
"[json][results][properties][R-DNS Name][result]" => "[host][name]"
}
}
}
if [json][results][properties][R-IP Address][result] {

mutate {
rename => {
"[json][results][properties][R-IP Address][result]" => "[host][ip]"
}
}
mutate {
add_field => {
"[related][ip]" => "%{[host][ip]}"
}
}
}
if [json][results][properties][R-IPv6 Address][result] {
mutate {
rename => {
"[json][results][properties][R-IPv6 Address][result]" => "[host][ipv6]"
}
}
mutate {
add_field => {
"[related][ip]" => "%{[host][ipv6]}"
}
}
}
mutate {
copy => {
"[host][os][full]" => "[os][full]"
}
}
mutate {
lowercase => ["[host][is_laptop]"]
}
if
[json][results][properties][D/Total%20Size%20of%20System%20Drive/tmutWa7NR%2
bcxoW%2bcZJir9U7itaU][result] =~ /^[0-9]* MB$/ {
dissect {
mapping => {
"[json][results][properties][D/Total%20Size%20of%20System%20Drive/tmutWa7NR%2
bcxoW%2bcZJir9U7itaU][result]" => "%{[host][drive][system][size_megabytes]}
%{?MB}"
}
}
}
else {
mutate {
rename => {

"[json][results][properties][D/Total%20Size%20of%20System%20Drive/tmutWa7NR%2
bcxoW%2bcZJir9U7itaU][result]" => "[host][drive][system][size]"
}
}
}
if
[json][results][properties][D/Free%20Space%20on%20System%20Drive/8cqlsJBnBvC8
%2f%2f9KBo23Mzq6jts][result] =~ /^[0-9]* MB$/ {
dissect {
mapping => {
"[json][results][properties][D/Free%20Space%20on%20System%20Drive/8cqlsJBnBvC
8%2f%2f9KBo23Mzq6jts][result]" =>
"%{[host][drive][system][free_space_megabytes]} %{?MB}"
}
}
}
else {
mutate {
rename => {
"[json][results][properties][D/Free%20Space%20on%20System%20Drive/8cqlsJBnBvC
8%2f%2f9KBo23Mzq6jts][result]" => "[host][drive][system][free_space]"
}
}
}
if [json][results][properties][D/RAM/2pGqHiqzzkoNb6Lqh0XDPFnISwg][result] =~
/^[0-9]* MB$/ {
dissect {
mapping => {
"[json][results][properties][D/RAM/2pGqHiqzzkoNb6Lqh0XDPFnISwg][result]"
=> "%{[host][ram][total_megabytes]} %{?MB}"
}
}
}
else {
mutate {
rename => {
"[json][results][properties][D/RAM/2pGqHiqzzkoNb6Lqh0XDPFnISwg][result]"
=> "[host][ram][total]"
}
}
}
mutate {

}
}
output {
elasticsearch {
index => "bigfix-hwam"
#document_id => "%{[host][id]}"
}
}

Appendix N: BigFix Web Reports Hardware Elasticsearch

Mapping
{
"bigfix-hwam": {
"mappings": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"fields": {
"text": {
"type": "text"
}
},
"type": "keyword"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"agent": {
"properties": {
"relay": {
"properties": {
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"scanner": {
"properties": {
"version": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"version": {
"type": "version"
}
}
}
}
},
"type": {

"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"version": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"version": {
"type": "version"
}
}
}
}
},
"ecs": {
"properties": {
"version": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"event": {
"properties": {
"category": {
"type": "keyword"
},
"kind": {
"type": "keyword"
},
"provider": {
"type": "keyword"
},
"type": {
"type": "keyword"
},
"url": {
"type": "keyword"
}
}
},
"host": {
"properties": {
"cpu": {
"properties": {
"brand": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"cores": {
"type": "keyword",

"fields": {
"text": {
"type": "text"
},
"integer": {
"type": "integer"
}
}
},
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"processors": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"integer": {
"type": "integer"
}
}
}
}
},
"dhcp_server": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"ip": {
"type": "ip"
}
}
},
"dns": {
"properties": {
"servers": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"ip": {
"type": "ip"
}
}
}
}
},
"domain": {
"properties": {
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"

}
}
}
}
},
"drive": {
"properties": {
"count": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"names": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"system": {
"properties": {
"free_space": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"free_space_megabytes": {
"type": "keyword",
"fields": {
"long": {
"type": "long"
}
}
},
"size": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"size_megabytes": {
"type": "keyword",
"fields": {
"long": {
"type": "long"
}
}
}
}
}
}
},
"hostname": {
"type": "keyword",

"fields": {
"text": {
"type": "text"
}
}
},
"id": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"ip": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"ip": {
"type": "ip"
}
}
},
"ipv6": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"ip": {
"type": "ip"
}
}
},
"is_laptop": {
"type": "boolean"
},
"mac": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"make": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"manufacturer": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}

},
"model": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"network": {
"properties": {
"adapters": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"os": {
"properties": {
"full": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"ram": {
"properties": {
"total": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"total_megabytes": {
"type": "keyword",
"fields": {
"long": {
"type": "long"
}
}
}
}
},
"subnet": {
"type": "keyword",

"fields": {
"text": {
"type": "text"
},
"ip": {
"type": "ip"
}
}
},
"type": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"os": {
"properties": {
"full": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"process": {
"type": "object"
},
"related": {
"properties": {
"hosts": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"ip": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
}
}
}
}
}

Appendix O: BigFix Web Reports Compliance Logstash

Configuration
# Integration: BigFix Platform 10.0.4, 10.0.5, 10.0.6
# Config Version: 1
# Purpose: Transforms BigFix Web Reports CIS Microsoft Windows Server 2019
Benchmark fields for ingestion into Elasticsearch
input {
exec {
command => "sh -c /home/student/get-csm-win2019.sh"
interval => 900
}
}
filter {
mutate {
rename => {
}
}
json {
target => "[json]"
}
mutate {
}
}
mutate {
remove_field => ["[json][columns]", "[process]", "[host]"]
}
mutate {
add_field => {

}
}
split {
field => "[json][results]"
}
mutate {
rename => {
"[json][results][fixletHref]" => "[bigfix][csm][fixlet][reference]"
}
}
mutate {
copy => {
"[bigfix][csm][fixlet][reference]" => "[event][url]"
}
}
mutate {
rename => {
"[json][results][computerHref]" => "[bigfix][csm][computer][reference]"
}
}
mutate {
rename => {
"[json][results][dbid]" => "[bigfix][csm][dbid]"
}
}
if [json][results][properties][Relevance][result] == "Applicable" {
mutate {
add_field => {
"[bigfix][csm][is_applicable]" => "true"
}
}
}
else {
mutate {
add_field => {
"[bigfix][csm][is_applicable]" => "false"
}
}
}
# 11/5/2019, 5/18/2021, Month (no leading zero)/day (no leading zero)/YEAR
date {
match => ["[json][results][properties][Source Release Date][result]", "M/d/YYYY"]
target => "[bigfix][csm][source][release_date]"
}
mutate {

rename => {
"[json][results][properties][Source][result]" => "[bigfix][csm][source][name]"
}
}
mutate {
rename => {
"[json][results][properties][Source Severity][result]" =>
"[bigfix][csm][source][severity]"
}
}
mutate {
rename => {
"[json][results][properties][Source ID][result]" => "[bigfix][csm][source][id]"
}
}
if [json][results][properties][Computer][result] {
mutate {
rename => {
"[json][results][properties][Computer][result]" => "[bigfix][csm][computer][name]"
}
}
mutate {
copy => {
"[bigfix][csm][computer][name]" => "[host][name]"
}
}
}
mutate {
rename => {
"[json][results][properties][Type][result]" => "[bigfix][csm][type]"
}
}
mutate {
rename => {
"[json][results][properties][Applicable Computer Count][result]" =>
"[bigfix][csm][applicable_computer_count]"
}
}
mutate {
rename => {
"[json][results][properties][Remediated Computer Count][result]" =>
"[bigfix][csm][remediated_computer_count]"
}
}
mutate {
rename => {

"[json][results][properties][Category][result]" => "[bigfix][csm][category]"

}
}
mutate {
rename => {
"[json][results][properties][Name][result]" => "[bigfix][csm][name]"
}
}
mutate {
rename => {
"[json][results][properties][ID][result]" => "[bigfix][csm][id]"
}
}
mutate {
rename => {
"[json][results][properties][SANS][result]" => "[bigfix][csm][sans]"
}
}
mutate {
rename => {
"[json][results][properties][Open Action Count][result]" =>
"[bigfix][csm][open_action_count]"
}
}
mutate {
rename => {
"[json][results][properties][CVE][result]" => "[bigfix][csm][cve]"
}
}
mutate {
rename => {
"[json][results][properties][Sitename][result]" => "[bigfix][csm][sitename]"
}
}
mutate {
rename => {
"[json][results][properties][Download Size][result]" =>
"[bigfix][csm][download_size]"
}
}
mutate {
rename => {
"[json][results][properties][Progress][result]" => "[bigfix][csm][progress]"
}
}

if [json][results][properties][Visibility][result] == "Visible" {
mutate {
add_field => {
"[bigfix][csm][is_visible]" => "true"
}
}
}
else {
mutate {
add_field => {
"[bigfix][csm][is_visible]" => "false"
}
}
}
mutate {
rename => {
"[json][results][properties][Comments][result]" => "[bigfix][csm][comments]"
}
}
# TODO confirm date format matches, missing from lab sample
date {
match => ["[json][results][properties][Activation Time (Analysis)][result]",
"M/d/YYYY", "ISO8601"]
target => "[bigfix][csm][analysis][activated_time]"
tag_on_failure => ["_dateparsefailure_activated_time"]
}
mutate {
rename => {
"[json][results][properties][Activated By (Analysis)][result]" =>
"[bigfix][csm][analysis][activated_by]"
}
}
mutate {
rename => {
"[json][results][properties][Mean Time to Remediate][result]" =>
"[bigfix][csm][mean_time_to_remediate]"
}
}
mutate {
rename => {
"[json][results][properties][Unlocked Computer Count][result]" =>
"[bigfix][csm][unlocked_computer_count]"
}
}
mutate {

}
}
output {
elasticsearch {
index => "bigfix-csm"
}
}

Appendix P: BigFix Web Reports Compliance Elasticsearch

Mapping
{
"bigfix-csm": {
"mappings" : {
"dynamic_templates" : [
{
"strings" : {
"match_mapping_type" : "string",
"mapping" : {
"fields" : {
"text" : {
"type" : "text"
}
},
"type" : "keyword"
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"bigfix" : {
"properties" : {
"csm" : {
"properties" : {
"analysis" : {
"type" : "object"
},
"applicable_computer_count" : {
"type" : "long"
},
"category" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"computer" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"reference" : {
"type" : "keyword",

"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"dbid" : {
"type" : "keyword"
},
"download_size" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"fixlet" : {
"properties" : {
"reference" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"id" : {
"type" : "keyword"
},
"is_applicable" : {
"type" : "boolean"
},
"is_visible" : {
"type" : "boolean"
},
"mean_time_to_remediate" : {
"type" : "keyword",
"ignore_above" : 256
},
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"open_action_count" : {
"type" : "long"
},
"progress" : {
"type" : "keyword",
},
"remediated_computer_count" : {
"type" : "long"
},
"sitename" : {
"type" : "keyword",

"fields" : {
"text" : {
"type" : "text"
}
}
},
"source" : {
"properties" : {
"id" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"release_date" : {
"type" : "date"
},
"severity" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"type" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"unlocked_computer_count" : {
"type" : "long"
}
}
}
}
},
"ecs" : {
"properties" : {
"version" : {
"type" : "keyword",
}
}
},
"event" : {
"properties" : {
"category" : {
"type" : "keyword"
},

"kind" : {
"type" : "keyword"
},
"provider" : {
"type" : "keyword"
},
"type" : {
"type" : "keyword"
},
"url" : {
"type" : "keyword"
}
}
},
"json" : {
"properties" : {
"dir" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"pageSize" : {
"type" : "long"
},
"recordsReturned" : {
"type" : "long"
},
"results" : {
"properties" : {
"properties" : {
"properties" : {
"Activated By (Analysis)" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Activation Time (Analysis)" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Applicable Computer Count" : {
"properties" : {

"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"CVE" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Category" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Comments" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Deployed Action Count" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {

"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
},
"result" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"Download Size" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"ID" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Mean Time to Remediate" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Name" : {
"properties" : {
"name" : {

"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Open Action Count" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Progress" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Remediated Computer Count" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"SANS" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"

}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Sitename" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Source" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Source ID" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Source Release Date" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"

},
"result" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
},
"Source Severity" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Type" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Unlocked Computer Count" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"plural" : {
"type" : "boolean"
}
}
},
"Visibility" : {
"properties" : {
"name" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}

}
},
"plural" : {
"type" : "boolean"
},
"result" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
}
}
}
}
},
"sort" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
},
"startIndex" : {
"type" : "long"
},
"totalRecords" : {
"type" : "long"
},
"totalRecordsUnknown" : {
"type" : "boolean"
}
}
},
"tags" : {
"type" : "keyword",
"fields" : {
"text" : {
"type" : "text"
}
}
}
}
}
}
}
}

Appendix Q: BigFix Inventory Software Logstash

Configuration
# Integration: BigFix Inventory 10.0.8, 10.0.9
# Config Version: 1
# Purpose: Transforms the BFI Software Installations report for ingestion into
Elasticsearch
# References:
#
https://help.hcltechsw.com/bigfix/10.0/inventory/Inventory/integration/r_get_software_in
stances.html
input {
exec {
command => "sh -c /home/student/get-swam.sh"
interval => 3600
}
}
filter {
mutate {
rename => {
}
}
json {
target => "[json]"
}
mutate {
}
}
mutate {
remove_field => ["[process]", "[host]"]
}
mutate {
add_field => {


}
}
split {
field => "[json][rows]"
}
date {
match => ["[json][rows][computer][last_seen]", "ISO8601"]
}
mutate {
copy => {
"[json][rows][computer][name]" => "[bigfix][inventory][computer][name]"
}
}
mutate {
rename => {
"[json][rows][computer][name]" => "[host][name]"
}
}
mutate {
copy => {
"[@timestamp]" => "[bigfix][inventory][computer][last_seen]"
}
}
# id is the "Internal Computer ID" in BFI
mutate {
rename => {
"[json][rows][computer][id]" => "[bigfix][inventory][computer][id]"
}
}
date {
match => ["[json][rows][computer][valid_to]", "ISO8601"]
target => "[bigfix][inventory][computer][valid_to]"
}
mutate {
copy => {
"[json][rows][computer][remote_id]" => "[bigfix][inventory][computer][remote_id]"
}
}
# remote_id is the "Data Source Computer ID" in BFI. It matches the "id" in Web
Reports
mutate {
rename => {

"[json][rows][computer][remote_id]" => "[host][id]"

}
}
mutate {
rename => {
"[json][rows][is_current]" => "[bigfix][inventory][computer][is_current]"
}
}
mutate {
convert => {
"[bigfix][inventory][computer][is_current]" => "boolean"
}
}
mutate {
rename => {
"[json][rows][computer_health][agent_version]" => "[agent][version]"
}
}
mutate {
rename => {
"[json][rows][computer_health][last_scan_attempt]" =>
"[bigfix][inventory][computer][last_scan_attempt]"
}
}
mutate {
rename => {
"[json][rows][catalog_dimension][publisher_name]" =>
"[bigfix][inventory][publisher][name]"
}
}
mutate {
rename => {
"[json][rows][catalog_dimension][discoverable_name]" =>
"[bigfix][inventory][component][name]"
}
}
mutate {
rename => {
"[json][rows][catalog_dimension][version]" =>
"[bigfix][inventory][component][version]"
}
}
mutate {
rename => {
"[json][rows][component_version]" =>
"[bigfix][inventory][component][detailed_version]"

}
}
mutate {
rename => {
"[json][rows][path][path]" => "[bigfix][inventory][component][path]"
}
}
mutate {
rename => {
"[json][rows][component_id]" => "[bigfix][inventory][component][id]"
}
}
mutate {
rename => {
"[json][rows][max_cve][name]" => "[bigfix][inventory][component][max_cve]"
}
}
mutate {
rename => {
"[json][rows][catalog_dimension][created_at]" =>
"[bigfix][inventory][component][creation]"
}
}
mutate {
rename => {
"[json][rows][catalog_dimension][edited_at]" =>
"[bigfix][inventory][component][modification]"
}
}
mutate {
rename => {
"[json][rows][catalog_dimension][publisher_id]" =>
"[bigfix][inventory][component][publisher_id]"
}
}
mutate {
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {

index => "bigfix-swam"

}
}

Appendix R: BigFix Inventory Software Elasticsearch

Mapping
{
"bigfix-swam": {
"mappings": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"fields": {
"text": {
"type": "text"
}
},
"type": "keyword"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"agent": {
"properties": {
"version": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"version": {
"type": "version"
}
}
}
}
},
"bigfix": {
"properties": {
"inventory": {
"properties": {
"component": {
"properties": {
"creation": {
"type": "date"
},
"detailed_version": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},

"version": {
"type": "version"
}
}
},
"id": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"max_cve": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"modification": {
"type": "date"
},
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"path": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"publisher_id": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"version": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
},
"version": {
"type": "version"
}
}
}
}
},
"computer": {
"properties": {

"id": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"is_current": {
"type": "boolean"
},
"last_scan_attempt": {
"type": "date"
},
"last_seen": {
"type": "date"
},
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"remote_id": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"valid_to": {
"type": "date"
}
}
},
"publisher": {
"properties": {
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
}
}
}
}
},
"event": {
"properties": {
"category": {
"type": "keyword"
},
"kind": {
"type": "keyword"
},
"provider": {
"type": "keyword"
},

"type": {
"type": "keyword"
},
"url": {
"type": "keyword"
}
}
},
"host": {
"properties": {
"id": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
},
"name": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
},
"process": {
"properties": {
"command_line": {
"type": "keyword",
"fields": {
"text": {
"type": "text"
}
}
}
}
}
}
}
}
}

Appendix S: HCL BigFix Web Reports Hardware Asset

Management Report Fields
• Computer Name • Make
• Size of Volumes - Mac OS X • Model
• Number of Processor Cores - • Brand String of CPU - Windows
Mac OS X • Brand String of CPU - Linux
• Network Devices - Mac OS X • Hard Disks - Linux
• Machine Model - Mac OS X • Free Space on System Drive
• Laptop - Mac OS X • Total Size of System Drive
• Hostname - Mac OS X • Disk Drives - Windows
• Free Space of Volumes - Mac OS • Computer Type
X • Computer Model - Windows
• DNS Servers - Mac OS X • Computer Manufacturer -
• ID Windows
• BES Client Version • Domain/Workgroup - Windows
• BES Client's Parent Relay • Number of Fixed Drives -
• Scanner Version Windows
• DNS Name • CPU
• DNS Servers - Windows • Number of Processors -
• DHCP Servers - Windows Windows
• IP Address • Number of Processors - Linux
• IPv6 Address • Number of Processor Cores -
• Subnet Address Windows
• Network Adapters - Linux • Number of Processor Cores -
• Network Adapter - Windows Linux
• MAC Address • OS
• RAM • Last Report Time
• Laptop - Linux
• Laptop - Windows

Appendix T: HCL BigFix Web Reports Hardware Asset
Management Report (HWAM.beswrpt)
<?xml version="1.0" encoding="UTF-8"?>
<BESWebReport>
<Name>HWAM</Name>
<Description></Description>
<Type>TemplateReport</Type>
<Source></Source>
<Data></Data>
<Doctype></Doctype>
<URLParameters>#filterManager=%7b%22filterName%22:%22%22,%22filterID
%22:%22%22%7d&wr_computerTable=%22sort%3dR-
Computer%20Name%26dir%3dasc%26startIndex%3d0%26results%3d50%26q%3d%26
c%3dR-Computer%20Name%26c%3dO-3093-44-7%26c%3dO-3093-44-9%26c%3dO-
3093-45-7%26c%3dO-3093-44-5%26c%3dO-3093-44-6%26c%3dO-3093-45-
1%26c%3dO-3093-44-8%26c%3dO-3093-45-6%26c%3dR-ID%26c%3dO-1-204-
1%26c%3dO-1-205-3%26c%3dO-13014-19-2%26c%3dR-DNS%20Name%26c%3dO-
3093-33-2%26c%3dO-3093-33-1%26c%3dR-IP%20Address%26c%3dR-
IPv6%20Address%26c%3dD%2FSubnet%2520Address%2FTjwTe57B3z9HXSvUiuNL
LO8eV%252bg%26c%3dO-3093-305-11%26c%3dO-3093-35-11%26c%3dR-
MAC%20Address%26c%3dD%2FRAM%2F2pGqHiqzzkoNb6Lqh0XDPFnISwg%26c%
3dO-3093-305-2%26c%3dO-3093-35-4%26c%3dO-9307-34-1%26c%3dO-9307-34-
2%26c%3dO-3093-35-5%26c%3dO-3093-305-3%26c%3dO-3093-305-
5%26c%3dD%2FFree%2520Space%2520on%2520System%2520Drive%2F8cqlsJBnBv
C8%252f%252f9KBo23Mzq6jts%26c%3dD%2FTotal%2520Size%2520of%2520System
%2520Drive%2FtmutWa7NR%252bcxoW%252bcZJir9U7itaU%26c%3dO-3093-35-
9%26c%3dR-Computer%20Type%26c%3dO-3093-35-2%26c%3dO-3093-35-
1%26c%3dO-3093-33-3%26c%3dO-3093-35-8%26c%3dR-CPU%26c%3dO-3093-35-
6%26c%3dO-3093-305-4%26c%3dO-3093-35-18%26c%3dO-3093-305-16%26c%3dR-
OS%26c%3dR-Last%20Report%20Time%26columnOrder%3d%7B%22R-
Computer%20Name%22%3A0%2C%22O-9307-23-3%22%3A1%2C%22O-3093-44-
7%22%3A2%2C%22O-3093-44-9%22%3A3%2C%22O-3093-45-
7%22%3A4%2C%22O-3093-44-5%22%3A5%2C%22O-3093-44-
6%22%3A6%2C%22O-3093-45-1%22%3A7%2C%22O-3093-44-
8%22%3A8%2C%22O-3093-45-6%22%3A9%2C%22R-ID%22%3A10%2C%22O-1-
204-1%22%3A11%2C%22O-1-205-3%22%3A12%2C%22O-13014-19-
2%22%3A13%2C%22R-DNS%20Name%22%3A14%2C%22O-3093-33-
2%22%3A15%2C%22O-3093-33-1%22%3A16%2C%22R-
IP%20Address%22%3A17%2C%22R-
IPv6%20Address%22%3A18%2C%22D%2FSubnet%2520Address%2FTjwTe57B3z9H
XSvUiuNLLO8eV%252bg%22%3A19%2C%22O-3093-305-11%22%3A20%2C%22O-
3093-35-11%22%3A21%2C%22R-
MAC%20Address%22%3A22%2C%22D%2FRAM%2F2pGqHiqzzkoNb6Lqh0XDPFnI
Swg%22%3A23%2C%22O-3093-305-2%22%3A24%2C%22O-3093-35-
4%22%3A25%2C%22O-9307-34-1%22%3A26%2C%22O-9307-34-
2%22%3A27%2C%22O-3093-35-5%22%3A28%2C%22O-3093-305-
3%22%3A29%2C%22O-3093-305-
5%22%3A30%2C%22D%2FFree%2520Space%2520on%2520System%2520Drive%2F8
cqlsJBnBvC8%252f%252f9KBo23Mzq6jts%22%3A31%2C%22D%2FTotal%2520Size
%2520of%2520System%2520Drive%2FtmutWa7NR%252bcxoW%252bcZJir9U7itaU%
22%3A32%2C%22O-3093-35-9%22%3A33%2C%22R-
Computer%20Type%22%3A34%2C%22O-3093-35-2%22%3A35%2C%22O-3093-35-
1%22%3A36%2C%22O-3093-33-3%22%3A37%2C%22O-3093-35-
8%22%3A38%2C%22R-CPU%22%3A39%2C%22O-3093-35-6%22%3A40%2C%22O-
3093-305-4%22%3A41%2C%22O-3093-35-18%22%3A42%2C%22O-3093-305-
16%22%3A43%2C%22R-OS%22%3A44%2C%22R-
Last%20Report%20Time%22%3A45%7D%22</URLParameters>
<SourcePage>ExploreComputers</SourcePage>
</BESWebReport>

Appendix U: Web Reports Report Retrieval Scripts

These bash scripts retrieve data from Web Reports using cURL (Stenberg, 2022).
The number of results returned per request can be configured. Environmental variables
are used in place of values for usernames, passwords, the Web Reports URL and the
BigFix Inventory URL. First, a Bash script to retrieve a Web Reports HWAM report is
shown in Figure U1.
#!/bin/bash
# Filename: get-hwam.sh
# Purpose: Retrieve a custom hardware-based report.
COLUMNS="c%3DR-Computer%2520Name%26c%3DO-3093-44-7%26c%3DO-3093-
44-9%26c%3DO-3093-45-7%26c%3DO-3093-44-5%26c%3DO-3093-44-6%26c%3DO-
3093-45-1%26c%3DO-3093-44-8%26c%3DO-3093-45-6%26c%3DR-ID%26c%3DO-1-
204-1%26c%3DO-1-205-3%26c%3DO-13014-19-2%26c%3DR-
DNS%2520Name%26c%3DO-3093-33-2%26c%3DO-3093-33-1%26c%3DR-
IP%2520Address%26c%3DR-
IPv6%2520Address%26c%3DD%252FSubnet%252520Address%252FTjwTe57B3z9HX
SvUiuNLLO8eV%25252bg%26c%3DO-3093-305-11%26c%3DO-3093-35-
11%26c%3DR-
MAC%2520Address%26c%3DD%252FRAM%252F2pGqHiqzzkoNb6Lqh0XDPFnISw
g%26c%3DO-3093-305-2%26c%3DO-3093-35-4%26c%3DO-9307-34-1%26c%3DO-
9307-34-2%26c%3DO-3093-35-5%26c%3DO-3093-305-3%26c%3DO-3093-305-
5%26c%3DD%252FFree%252520Space%252520on%252520System%252520Drive%25
2F8cqlsJBnBvC8%25252f%25252f9KBo23Mzq6jts%26c%3DD%252FTotal%252520Si
ze%252520of%252520System%252520Drive%252FtmutWa7NR%25252bcxoW%25252
bcZJir9U7itaU%26c%3DO-3093-35-9%26c%3DR-Computer%2520Type%26c%3DO-
3093-35-2%26c%3DO-3093-35-1%26c%3DO-3093-33-3%26c%3DO-3093-35-
8%26c%3DR-CPU%26c%3DO-3093-35-6%26c%3DO-3093-305-4%26c%3DO-3093-
35-18%26c%3DO-3093-305-16%26c%3DR-OS%26c%3DR-
Last%2520Report%2520Time%26"
#Additional variables: columnOrder, sort, dir
# The following variables may be used to filter Web Reports: filterName, filterID,
filterDescription, matchType, conditionList, selectedContentTypeName, selectorList,
selectedOperatorName, selectedOperatorValue, selectedProperty, type, id, name,
displayName
# Log in and save the cookie token to cookies.txt

curl --X POST ${VAR_BF_WR_URL} --header 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode 'page=LoggingIn' --data-urlencode
'Username='${VAR_BF_USER}'' --data-urlencode
'Password='${VAR_BF_USER_PW}'' -k -s -c cookies.txt

# Get the report.

RESULTS=500
STARTINDEX=0
# The while loop demonstrates how paginated results can be retrieved be incrementing
start index and ending when the the number of RETURNED_RESULTS does not equal
RESULTS.
# RETURNED_RESULTS extracts the number of records returned, so it would be 10
when recordsReturned":10,
while : ; do
RESPONSE_PAGINATION=$(curl -k
"${VAR_BF_WR_URL}/json/computers?results=$RESULTS&startIndex=$STARTIND
EX&$COLUMNS" -s -b cookies.txt)
echo "$RESPONSE_PAGINATION"
RETURNED_RESULTS=$(echo "$RESPONSE_PAGINATION" | sed
's/.*recordsReturned"://' | cut -d ',' -f 1)
if [ $RETURNED_RESULTS != $RESULTS ]; then
break
fi
((STARTINDEX+=RESULTS))
done
Figure U1. A bash script used to retrieve the HWAM report during development.
Second, a Bash script to retrieve all of the CIS benchmarks from Web Reports is shown
in Figure U2. The script may be adjusted to only retrieve a subset of the benchmarks of
by adjusting the FILTER variable.
#!/bin/bash
# Filename: get-csm.sh
# Purpose: Retrieve all CIS benchmarks from BigFix Web Reports, with with all
available columns and applicable computers expanded.
FILTER="%7B%22matchType%22%3A%22all%22%2C%22conditionList%22%3A%5
B%7B%22selectedContentTypeName%22%3A%22Fixlet%22%2C%22selectorList%22
%3A%5B%7B%22selectedOperatorName%22%3A%22is%22%2C%22selectedOperator
Value%22%3A%22Visible%22%7D%5D%2C%22selectedProperty%22%3A%7B%22n
ame%22%3A%22Visibility%22%2C%22id%22%3A%22Visibility%22%7D%7D%2C%
7B%22selectedContentTypeName%22%3A%22Fixlet%22%2C%22selectorList%22%3
A%5B%7B%22selectedOperatorName%22%3A%22is%22%2C%22selectedOperatorVal
ue%22%3A%22%7B%5C%22BESSiteID%5C%22%3A%7B%5C%22DatabaseSiteID%
5C%22%3A15426%7D%7D%22%7D%2C%7B%22selectedOperatorName%22%3A%2
2is%22%2C%22selectedOperatorValue%22%3A%22%7B%5C%22BESSiteID%5C%22
%3A%7B%5C%22DatabaseSiteID%5C%22%3A15394%7D%7D%22%7D%2C%7B%2
2selectedOperatorName%22%3A%22is%22%2C%22selectedOperatorValue%22%3A%

22%7B%5C%22BESSiteID%5C%22%3A%7B%5C%22DatabaseSiteID%5C%22%3A1
5483%7D%7D%22%7D%2C%7B%22selectedOperatorName%22%3A%22is%22%2C
%22selectedOperatorValue%22%3A%22%7B%5C%22BESSiteID%5C%22%3A%7B%
5C%22DatabaseSiteID%5C%22%3A11366%7D%7D%22%7D%2C%7B%22selectedOp
eratorName%22%3A%22is%22%2C%22selectedOperatorValue%22%3A%22%7B%5C
%22BESSiteID%5C%22%3A%7B%5C%22DatabaseSiteID%5C%22%3A12181%7D%
7D%22%7D%2C%7B%22selectedOperatorName%22%3A%22is%22%2C%22selected
OperatorValue%22%3A%22%7B%5C%22BESSiteID%5C%22%3A%7B%5C%22Data
baseSiteID%5C%22%3A15515%7D%7D%22%7D%2C%7B%22selectedOperatorName
%22%3A%22is%22%2C%22selectedOperatorValue%22%3A%22%7B%5C%22BESSit
eID%5C%22%3A%7B%5C%22DatabaseSiteID%5C%22%3A15736%7D%7D%22%7D
%2C%7B%22selectedOperatorName%22%3A%22is%22%2C%22selectedOperatorValu
e%22%3A%22%7B%5C%22BESSiteID%5C%22%3A%7B%5C%22DatabaseSiteID%5
C%22%3A15464%7D%7D%22%7D%5D%2C%22selectedProperty%22%3A%7B%22n
ame%22%3A%22Site%22%2C%22id%22%3A%22Site%22%7D%7D%5D%7D"
COLUMNS="c=Name&c=Visibility&c=Unlocked%20Computer%20Count&c=Source
%20Severity&c=Source%20ID&c=Sitename&c=SANS&c=Progress&c=Open%20Actio
n%20Count&c=Mean%20Time%20to%20Remediate&c=Download%20Size&c=Deploy
ed%20Action%20Count&c=CVE&c=Comments&c=Activation%20Time%20(Analysis)
&c=Activated%20By%20(Analysis)&c=ID&c=Category&c=Applicable%20Computer%
20Count&c=Remediated%20Computer%20Count&c=Type&c=Source&c=Source%20R
elease%20Date"
#NEWFILTER="true"
# Log in and save the cookie token to cookies.txt

curl --X POST ${VAR_BF_WR_URL} --header 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode 'page=LoggingIn' --data-urlencode
'Username='${VAR_BF_USER}'' --data-urlencode
'Password='${VAR_BF_USER_PW}'' -k -s -c cookies.txt
# Get the report.

RESULTS=500
STARTINDEX=0
start index and ending when the the number of RETURNED_RESULTS does not equal
RESULTS.
# RETURNED_RESULTS extracts the number of records returned, so it would be 10
when recordsReturned":10,
while : ; do

RESPONSE_PAGINATION=$(curl -k
"${VAR_BF_WR_URL}/json/fixletproperties?results=$RESULTS&startIndex=$STAR
TINDEX&$COLUMNS" -s -b cookies.txt)
RETURNED_RESULTS=$(echo "$RESPONSE_PAGINATION" | sed
's/.*recordsReturned"://' | cut -d ',' -f 1)
if [ $RETURNED_RESULTS != $RESULTS ]; then
break
fi
((STARTINDEX+=RESULTS))
done
Figure U2. A bash script used to retrieve the CSM report during development.
Finally, a Bash script to retrieve a software instances report from BigFix Inventory is
shown in Figure U3. The number of rows to pull is configurable, and pagination will
occur until all results are retrieved.
#!/bin/bash
# Filename: get-swam.sh
# Purpose: Retrieve all software instances from BigFix Inventory
COLUMNS="columns%5B%5D=computer.id&columns%5B%5D=computer.name&col
umns%5B%5D=computer.valid_to&columns%5B%5D=prod_inv_id&columns%5B%5D
=is_shared&columns%5B%5D=is_suppressed&columns%5B%5D=component_id&colu
mns%5B%5D=product_metric_id&columns%5B%5D=catalog_dimension.discoverable_
name&columns%5B%5D=catalog_dimension.version&columns%5B%5D=bundle_name
&columns%5B%5D=processor.computer_type&columns%5B%5D=computer.last_seen
&columns%5B%5D=catalog_dimension.publisher_id&columns%5B%5D=catalog_dime
nsion.publisher_name&columns%5B%5D=component_version&columns%5B%5D=path
.path&columns%5B%5D=is_current&columns%5B%5D=max_cve.name&columns%5B
%5D=cve_rank.is_one&columns%5B%5D=catalog_dimension.created_at&columns%5
B%5D=catalog_dimension.edited_at&columns%5B%5D=computer.remote_id&columns
%5B%5D=computer_health.agent_version&columns%5B%5D=computer_health.last_sc
an_attempt"
LIMIT="50"
OFFSET="0"
CRITERIA="%7B%22and%22%3A+%5B%5B%22is_current%22%2C+%22%3D%22
%2C+%221%22%5D%2C+%5B%22is_suppressed%22%2C+%22%3D%22%2C+%220
%22%5D%5D%7D"
# Get cookie token.

TOKEN=$(curl -k -s -L -X POST "${VAR_BFI_URL}/api/get_token" -H 'Content-

Type: application/json' -H 'Accept-Language: en-US' --data-raw '{
"user": "'${VAR_BF_USER}'",
"password": "'${VAR_BF_USER_PW}'"
}' -o - | cut -d '"' -f 4)
# Get the total rows

TOTAL_ROWS=$(curl -k -s
"${VAR_BFI_URL}/sam/extended_software_facts.json?token=$TOKEN&limit=1&offs
et=$OFFSET&$COLUMNS&criteria=$CRITERIA" | head -1 | sed 's/{"total"://' | cut -d
',' -f 1)
# Get the report.

OFFSET
while : ; do
RESPONSE_PAGINATION=$(curl -k -s
"${VAR_BFI_URL}/sam/extended_software_facts.json?token=$TOKEN&limit=$LIMI
T&offset=$OFFSET&$COLUMNS&criteria=$CRITERIA")
((OFFSET+=LIMIT))
if ((TOTAL_ROWS-OFFSET < LIMIT)) ; then
LIMIT=$((TOTAL_ROWS-OFFSET))
curl -k -s
"${VAR_BFI_URL}/sam/extended_software_facts.json?token=$TOKEN&limit=$LIMI
T&offset=$OFFSET&$COLUMNS&criteria=$CRITERIA"
break
fi
done
Figure U3. Script to retrieve software instances from BigFix Inventory

Appendix V: Logstash Deployment in a Multi-tenant scenario

Let’s say there’s a parent agency with two child agencies as demonstrated in
Figure V1. The parent agency hosts the CDM Dashboard on-premise for itself and the
two child agencies. Here’s what a hypothetical deployment could look like.
Figure V1. Logstash deployment scenario for a hierarchical organizational model

For Child Agency A, Logstash is shown to collect events from two CDM tools.
Filebeat could be run on the same system as Logstash to remotely collect data from
Tenable.sc, or it could be run on the same host as Tenable.sc depending on operational

and performance requirements. The Logstash pipeline is configured to look at certain key
fields, and if they’re missing, it will query for them, as shown by the dashed line to DNS
and Asset Inventory. Child Agency A also has a requirement to send its data to a security
information and event management (SIEM) product, so Logstash is configured to
perform parsing on the data destined for the SIEM and sends the CDM data to the parent
agency.
Child Agency B is shown sending data from numerous systems in addition to the
CDM tools that may aid analysts. In this case, Logstash acts solely as an aggregator and
sends all the events to the parent for parsing. In this scenario, a message broker was
chosen by the parent agency as a temporary storage system. The advantage of the
message broker is that it allows the parent agency’s Logstash instance to retrieves the
logs as resources become available in the pipeline. It also adds data resiliency on top of
Logstash capabilities like the persistent queues and dead letter queues.

Appendix W: Examples of BigFix Documents in Kibana
Figure W1. Sample HWAM document (part one of two)

Figure W2. Sample HWAM document (part two of two)

Figure W3. Sample CSM document (part one of two)

Figure W4. Sample CSM document (part two of two)


Figure W5. Sample SWAM document (part one of two)
Figure W5. Sample SWAM document (part two of two)

Appendix X: Kibana Query Using CIDR Notation
Appendix X. A Kibana query using CIDR notation

Appendix Y: CVE Search in Kibana with Fields of Interest as

Columns
Appendix Y. A CVE search with fields of interest added as columns to the view

Sans Continuous Diagnostics and Mitigation Evolving Federal Defenses With Cost Effective and Maintainable Data Integration Solutions

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Sans Continuous Diagnostics and Mitigation Evolving Federal Defenses With Cost Effective and Maintainable Data Integration Solutions

Uploaded by

Copyright:

Available Formats

Continuous Diagnostics and Mitigation: Evolving

Federal Defenses with Cost-Effective and

GIAC (GCDA) Gold Certification

Author: Andrew Davidow, andrew.davidow@student.sans.edu

Accepted: August 15, 2022

With the rapid innovation of information technologies came fundamental flaws in

Andrew Davidow, andrew.davidow@student.sans.edu

1.1. Federal Information Security Management Act

Fulfilling this responsibility, NIST Special Publication (SP) 800-53 provided

1.2. Risk Management Framework

Andrew Davidow, andrew.davidow@student.sans.edu

In summary, the RMF provides a holistic approach to managing information

1.3. Continuous Monitoring

The first iteration of a government-wide CM program was the Continuous Asset

“The end goal of CAESARS FE is to enable enterprise CM by presenting a

An example of how Mell et al. (2012) envisioned an enterprise-wide CM architecture is

Andrew Davidow, andrew.davidow@student.sans.edu

1.4. Continuous Diagnostics and Mitigation

Andrew Davidow, andrew.davidow@student.sans.edu

approach and strategy for securing information systems...Until agencies more

1.5. Elastic Stack

Andrew Davidow, andrew.davidow@student.sans.edu

Andrew Davidow, andrew.davidow@student.sans.edu

1.6. CDM Current State and the Future

Andrew Davidow, andrew.davidow@student.sans.edu

2.1. Tenable.sc and BigFix

BigFix is an endpoint management platform. BigFix clients are installed on each

Andrew Davidow, andrew.davidow@student.sans.edu

2.2. Development Environment

2.2.2. Lab environment

2.3. Experiments and Testing

Andrew Davidow, andrew.davidow@student.sans.edu

2.3.1. Collect the data from the CDM tool

2.3.2. Transform and Parse

Andrew Davidow, andrew.davidow@student.sans.edu

2.3.3. Map in Elasticsearch

2.3.4. Validate in Kibana

2.3.5. Upgrade and Repeat

Andrew Davidow, andrew.davidow@student.sans.edu

3. Findings and Discussion

Andrew Davidow, andrew.davidow@student.sans.edu

Figure 1. A cropped view of the HWAM report in Web Reports

Figure 2. A cropped view of the CSM report in Web Reports

Andrew Davidow, andrew.davidow@student.sans.edu

Figure 3. A payload sent to /post/shortentext revealing URL parameters

These variables may be programmatically manipulated and sent as URL parameters in

Figure 4. A GET request sent to /json/computers with URL parameters

Andrew Davidow, andrew.davidow@student.sans.edu

Exhaustive testing was performed to measure the level of effort involved in

BigFix BigFix BigFix BigFix BigFix

Andrew Davidow, andrew.davidow@student.sans.edu

Andrew Davidow, andrew.davidow@student.sans.edu

Figure 5. A hardware asset management dashboard

Andrew Davidow, andrew.davidow@student.sans.edu

Figure 6. A configuration settings management dashboard

Andrew Davidow, andrew.davidow@student.sans.edu

Figure 7. A software asset management dashboard

4. Recommendations and Implications

Andrew Davidow, andrew.davidow@student.sans.edu

4.1. Taking Ownership

4.2. Implications for Future Research

Andrew Davidow, andrew.davidow@student.sans.edu

Andrew Davidow, andrew.davidow@student.sans.edu

Elastic Stack looks very promising as a one-stack shop to bring in cybersecurity