Automate the Creation of

ATT&CK Navigator Group

Layer Files with Python
Roberto Rodriguez

What do you do when you have a few hours of free time?

Well, besides working out or watching a favorite TV series
over and over, you create new API functions for one of
your favorite Python projects right? . That’s the case
for me with the ATT&CK Python Client project. A project
where I host a Python library called attackcti which was
developed to help others in the community to interact with
ATT&CK content through STIX/TAXII 2.0 APIs and some
Python code.

In this post, I will show you how you can programmatically

create ATT&CK Navigator threat actor layers in the form of
JavaScript Object Notation (JSON) files for every single
group available in ATT&CK.

Isn’t that already done?

If you had not noticed it yet, ATT&CK Navigator actor
layers are already provided for each group in the ATT&CK
site (i.e APT12 Group)

https://attack.mitre.org/groups/G0005/

if you click download, you get the following file:

ATT&CK Navigator Group Layer File

and if you click on view, you get to the following Navigator

view:
However, while creating a new function for the attackcti
library to export every single technique being used by
every single group documented by the ATT&CK team, I
figured I could take every single relationships between a
group and techniques and automate the creation of each
navigator layer file without going through every single
ATT&CK page and download every single file manually.

What is attackcti?
It is a python library developed as part of the ATTACK-
Python-Client project that I started last year (2018) and
that I use to access up to date ATT&CK content available
in STIX format via a public TAXII server. This project
leverages python classes and functions from the cti-
python-stix2 and cti-taxii-client libraries developed by
MITRE.
I had never heard of the STIX/TAXII 2.0 APIs before so I
decided to start the ATTACK-Python-Client project to
learn more about it. After writing a few scripts to interact
with ATT&CK’s public TAXII server, I decided to write my
own python library. For more information about the
development of the library, I highly recommend to read
the attackcti library documentation.

Install attackcti
You can install the library via pip as shown below

pip install attackcti

New attackcti function?

I wanted to create a function to loop through all the
documented groups in ATT&CK and extract the
techniques used by each group. The python library
already has a function called
get_techniques_used_by_group() which I use to retrieve
all techniques used by one specific group. However, when
I use the function to loop through every single group
(around 90), it takes a very long time to finish (Too many
API calls? maybe?).

Therefore, I decided to create a new function named

get_techniques_used_by_all_groups() using a more
efficient approach to retrieve techniques metadata from
more than 90 groups available in ATT&CK in a shorter
period of time.

What do you mean ?

In order for me to get techniques used by a specific
group, I need to retrieve STIX objects that contain those
relationships. However, making a request to ATT&CK’s
TAXII server with specific relationship filters is not as fast
as I would like it to be when performed in a loop through
more than 90 groups available in ATT&CK.

How does a Group-Technique

relationships work?
In STIX format, you need to find relationship STIX objects
of relationship type “uses” which define an intrusion-set
(Group) as the source and an attack-pattern
(Technique) as the target. For example, if I want to find
techniques used by the APT12 Group, I will need to find
relationship STIX objects like this one:

Relationship object
relationship — 5a49400c-2003–463c-8e6e-97b79f560675

The relationship object contains a source_ref field that

points to an intrusion-set and a target_ref field that
points to an attack-pattern object.
There is also a field named description for most of the
relationship objects. The one from the APT12 relationship
above states the following:

[APT12](https://attack.mitre.org/groups/G0005) has u

In STIX format, I would translate it to :

Group APT12 (intrusion-set — c47f937f-1022–4f42–8525

You can take the relationship object and use the

source_ref and target_ref values to retrieve specific
intrusion-set and attack-pattern objects.

Intrusion-set object
intrusion-set — c47f937f-1022–4f42–8525-e7a4779a14cb

Attack-pattern object
attack-pattern — 830c9528-df21–472c-8c14-a036bf17d665

Intrusion-set? Group? Attack-

pattern? Technique?
Before we continue, I believe it is important to understand
how ATT&CK objects and properties are represented in
STIX 2.0 format. I will be using intrusion-set and attack-
pattern a lot through this post.

https://github.com/mitre/cti/blob/master/USAGE.md#mapping-concepts

How do I query relationships in

ATT&CK?
Once again, I highly recommend to first read the docs I
put together about cti-taxii-client and cti-python-stix2
libraries. Those two summarize several of the concepts
that I had to read to understand how to perform a simple
query against ATT&CK’s TAXII server.

Import STIX and TAXII Libraries

from stix2 import TAXIICollectionSource, Filter, Com

from taxii2client import Collection

Set ATT&CK TAXII Collection ID

Variables
The public ATT&CK TAXII instance has three main
collections (Enterprise, PRE and Mobile). Every
collection has an ID which attackcti uses to retrieve
ATT&CK STIX objects from all those matrices.

ATTCK_STIX_COLLECTIONS = "https://cti-taxii.mitre.or
PRE_ATTCK = "062767bd-02d2-4b72-84ba-56caef0f8658"
MOBILE_ATTCK = "2f669986-b40b-4423-b720-4396ca6a462b

Initialize TAXII Collection Sources

According to STIX2 docs, the TAXIICollectionSource API
provides an interface for searching/retrieving STIX objects
from a local/remote TAXII Collection endpoint. In our case,
we are pointing to our ATT&CK TAXII Collection instances
( https://cti-taxii.mitre.org/stix/collections/<Collection
ID>)
ENTERPRISE_COLLECTION = Collection(ATTCK_STIX_COLLEC
TC_ENTERPRISE_SOURCE = TAXIICollectionSource(ENTERPR
TC_PRE_SOURCE = TAXIICollectionSource(PRE_COLLECTION
TC_MOBILE_SOURCE = TAXIICollectionSource(MOBILE_COLL

Initialize a Composite Data Source

According to STIX2 docs, a user can have a single
CompositeDataSource as an interface to a set of
DataSources. When an API call is made to the
CompositeDataSource, it is delegated to each of the (real)
DataSources that are attached to it. In our case, we have
three TAXIICollection sources (Enterprise, PRE and
Mobile) as defined in our previous step. Therefore, we
can use the CompositeDataSource class and the
add_data_sources method to attach every ATT&CK
TAXIICollection source and be able to query all of them at
the same time.

COMPOSITE_DS = CompositeDataSource()
COMPOSITE_DS.add_data_sources([TC_ENTERPRISE_SOURCE,

Retrieve All Relationships

Now that we can query all the ATT&CK TAXIICollection
sources at once, we can use the query method and a set
of filters to retrieve STIX objects of type relationship.
Remember the APT12 relationships file?
relationship — 5a49400c-2003–463c-8e6e-97b79f560675

Therefore, we can run the following query to retrieve all

relationships from every single ATT&CK TAXIICollection
source:

rels = COMPOSITE_DS.query(Filter("type", "=", "relat

I already automate all the code from above with the

already available function named get_relationships() .

from attackcti import attack_client

lift = attack_client()all_relationships = lift.get_r

What about all relationships from a

specific STIX object?
What if you want to be very specific and retrieve
relationships from a specific STIX object? You can use the
relationships method from the CompositeDataSource
class to retrieve relationships involving a given STIX
object. That is totally different from the query we used
before where we got all relationships available in ATT&CK:

rels = COMPOSITE_DS.query(Filter("type", "=", "relat

The query below will look for any relationship of

relationship type “uses” involving a specific STIX object
only.

rels = COMPOSITE_DS.relationships(stix_object, 'uses

I also automate this process with an already available

function named get_relationships_by_object() . You
still have to provide an STIX object for it to work. You can
use an intrusion-set object and retrieve all its
relationships.

from attackcti import attack_client

lift = attack_client()groups = lift.get_groups()
groups = lift.remove_revoked(groups)group_relationsh

You will get every relationship of relationship type “uses”

involving the specific intrusion-set. The function returns
relationship STIX objects similar to the one below:

The source_def points to an intrusion-set, but the

target_ref values could be a tool, malware or attack-
pattern objects.

How do I retrieve techniques used by

a group by leveraging the
relationships retrieved?
In that case, we want relationship objects that have
target_ref values of type attack-pattern. Following the
manual code I shared above, and the results from the
get_relationships_by_object() function, you can simply
query the ATT&CK Enterprise TAXIICollection source
with the filter below:

filter_objects = [
Filter('type', '=', 'attack-pattern'),
Filter('id', '=', [r.target_ref for r in group_rel
]techniques_used = TC_ENTERPRISE_SOURCE.query(filter

As I mentioned at the beginning of this post, I already

automate all the steps explained so far via the already
available function named
get_techniques_used_by_group() .

That function follows the following process:

Retrieves relationship STIX objects of relationship

type “uses” from a specific intrusion-set.
Retrieves attack-pattern objects looping through all
the relationships retrieved and uses the target_ref
values in each relationship to pass a filter to
ATT&CK’s TAXII public server.

You can simply run the following lines of code and you will
be able to retrieve all the attack-patterns used by one
specific intrusion-set:

from attackcti import attack_client

lift = attack_client()groups = lift.get_groups()
groups = lift.remove_revoked(groups)group_techniques

What happens when I try to retrieve

all techniques used by ALL groups?
You can apply the same get_techniques_used_by_group()
function, but against all the groups STIX objects that the
get_groups() function retrieves. You can do a simple for
loop over more than 90 groups as shown below:

from attackcti import attack_client

lift = attack_client()groups = lift.get_groups()
groups = lift.remove_revoked(groups)def get_techniqu
techniques_used = []
for group in groups:
techniques_used.append(lift.get_techniques_used_
return techniques_used%time techniques_used_by_gro

However, it takes longer than what I would like it to take. I

calculated the time by packing the code in a function and
used the Jupyter Notebook %time built-in magic
command as shown below:
I decided to measure how much time each loop was
taking so I added the Jupyter Notebook %time built-in
magic command to each
get_techniques_used_by_group() execution.

def get_techniques_used_by_all_groups():
techniques_used = []
for group in groups:
%time techniques_used.append(lift.get_techni
return techniques_used
With an average of 3-4 seconds per
get_techniques_used_by_group() execution times more
than 90 groups, it makes sense that the overall time is
around 4–5 minutes for the overall function. Once again,
it is fine when it is done to one group, but not as fast
when it is applied to more than 90 groups.

What if we collect all the

relationships at once?
Something that I am learning as I write scripts to interact
with ATT&CK’s TAXII server is that the less API requests
you provide to get the same results, the better. For
example, retrieving relationships for each individual
intrusion-set and getting information about each specific
attack-pattern involved with the intrusion-set takes
around 5 minutes. This is because I am running the
following two things around 90 times and waiting for the
server to handle each request individually one after the
other one:

all_relationships =
lift.get_relationships_by_object()

filter_objects = [
Filter('type', '=', 'attack-pattern'),
Filter('id', '=', [r.target_ref for r in all_relat
]
TC_ENTERPRISE_SOURCE.query(filter_objects)

Therefore, I decided to first collect all relationship objects

of relationship type “uses” from every STIX object
available in TAXII all in one API request. I measured how
much it would take to do that, and I was impressed by
how much less time it took. As you can see below, it took
5.56 seconds to get all the relationships at once, versus
4-5 seconds for each individual API request to get
relationships from only one STIX object.
from stix2 import TAXIICollectionSource, Filter, Com
from taxii2client import CollectionATTCK_STIX_COLLEC
ENTERPRISE_ATTCK = "95ecc380-afe9-11e4-9b6c-751b66dd
PRE_ATTCK = "062767bd-02d2-4b72-84ba-56caef0f8658"
MOBILE_ATTCK = "2f669986-b40b-4423-b720-4396ca6a462b
TC_ENTERPRISE_SOURCE = TAXIICollectionSource(ENTERPR
TC_PRE_SOURCE = TAXIICollectionSource(PRE_COLLECTION
TC_MOBILE_SOURCE = TAXIICollectionSource(MOBILE_COLL
COMPOSITE_DS.add_data_sources([TC_ENTERPRISE_SOURCE,
Filter("type", "=", "relationship"),
Filter('relationship_type','=','uses')
]
%time all_relationships = COMPOSITE_DS.query(filters

How do I find “intrusion-set uses

attack-pattern” relationships in our
results?
Once I collect all relationships, I filter them to only keep
relationships that have intrusion-set objects as the
source and attack-pattern objects as the target in every
relationship object. Remember?

In order to determine if source_ref and target_ref values

are of specific types, I use a function from the stix2.utils
library named get_type_from_id.

from stix2.utils import get_type_from_idgroup_relati

for rel in all_relationships:
if get_type_from_id(rel.source_ref) == 'intrusio
and get_type_from_id(rel.target_ref) == 'attack-
group_relationships.append(rel)len(group_rel
print(group_relationships[0])
one relationships result example:

Do we retrieve all groups and

techniques at once too? How do I get
all techniques used by all groups?
Now that we have all the relationships defining what
attack-pattern objects are used by every single
intrusion-set, I need to get more metadata about each
technique and group. All we have so far in each
relationship STIX object is the source_ref field pointing to
an intrusion-set ID and target_ref field pointing to an
attack-pattern ID. Instead of retrieving more metadata
about attack-pattern STIX objects defined on each
relationships that involves intrusion-set objects, I
decided to retrieve all the available intrusion-set and
attack-pattern STIX objects and do a basic match locally.

Get all groups and techniques

Using available attackcti functions, I can complement my
manual code and get all the groups and techniques
available in ATT&CK as STIX objects.

from attackcti import attack_client

lift = attack_client()groups = lift.get_groups()
techniques = lift.get_techniques()

Match Group -> Relationships

source_ref ID
Next, I take all the group_relationships results I got
before, and look for the specific intrusion-set ID in the
groups STIX objects list. Once there is a match, I create a
new python dictionary joining information about the
relationship and the intrusion-set objects. The most
important additional metadata is the target_ref field
which points to a specific attack-pattern ID involving the
intrusion-set. The results are then added to a new list
named group_techniques_ref as shown below:
import jsongroup_techniques_ref = []for g in groups:
for rel in group_relationships:
if g['id'] == rel['source_ref']:
gs = json.loads(g.serialize())
gs['technique_ref'] = rel['target_ref']
gs['relationship_description'] = rel['de
gs['relationship_id'] = rel['id']
group_techniques_ref.append(gs)

Match Attack-patterns -> New

Dictionary target_ref ID
I apply the same concept as before, and just loop through
all the attack-pattern objects and look for the specific
attack-pattern id in the new dictionary we created with
information about intrusion-set and relationships
objects. Once there is a match, I add additional
information from the attack-pattern itself to the python
dictionaries in the group_techniques_ref list. The results
then get added to a new list named
groups_use_techniques.

groups_use_techniques = []
for gt in group_techniques_ref:
for t in techniques:
if gt['technique_ref'] == t['id']:
tactic_list = list()
for phase in t['kill_chain_phases']:
tactic_list.append(phase['phase_name
gt['technique'] = t['name']
 gt['technique_description'] = t['des
gt['tactic'] = tactic_list
gt['technique_id'] = t['external_ref
gt['matrix'] = t['external_referenc
if 'x_mitre_platforms' in t.keys():
gt['platform'] = t['x_mitre_plat
if 'x_mitre_data_sources' in t.keys(
gt['data_sources'] = t['x_mitre_
if 'x_mitre_permissions_required' in
gt['permissions_required'] = t['
if 'x_mitre_effective_permissions' i
gt['effective_permissions'] = t[
groups_use_techniques.append(gt)
groups_use_techniques[0]

The result is a list of python dictionaries with

information about an intrusion-set and attack-pattern
based on a relationship of relationship type “uses”. In
other words, a group uses this technique . You can
see one example of the custom dictionary result below:

Finally, I took everything we have done so far and created

a new function named
get_techniques_used_by_all_groups(). All you have to
do now is:

from attackcti import attack_client

lift = attack_client()
%time techniques_used = lift.get_techniques_used_by_
How long does it take to run all that
now?
let’s test it! around12 seconds! from 5 mins to 12
seconds!

What does it all have to do with

ATT&CK Navigator?
Remember the Navigator group layer file that you could
download from the ATT&CK site? Well, with the new
function named get_techniques_used_by_all_groups()
you can easily retrieve all intrusion-set object and all the
attack-pattern objects related to it in a few seconds.
Therefore, it should be easy to build similar navigator layer
files with the results of the new function!

ATT&CK Navigator Group Layer File Example

We need a template!
You can take the Navigator group layer file from above
and use it as a template to loop through every singe
intrusion-set:

ATT&CK Navigator Group Layer Template Example

Automating creation of Navigator

group layers
All I need is the get_techniques_used_by_all_groups()
function, some additional lines of code and the template.

Get techniques used by all groups

from attackcti import attack_clientlift = attack_cli

Create a list of group dictionaries
To make things easier, I create a list of dictionaries where
each group name is the main key and the value is an
empty list where I would append every single technique
involving that group.

groups = lift.get_groups()
groups = lift.remove_revoked(groups)groups_list = []
for g in groups:
group_dict = dict()
group_dict[g['name']] = []
groups_list.append(group_dict)groups_list[89]
Group techniques by groups
We can then use the output of the
get_techniques_used_by_all_groups() function and start
appending techniques to the dictionaries with the key
name that matches the group name. If there is a match, I
create a new dictionary with specific information that I
could use for my Navigator files.

for group in groups_list:

for group_name,techniques_list in group.items():
for gut in techniques_used:
if group_name == gut['name']:
technique_dict = dict()
technique_dict['techniqueId'] = gut[
technique_dict['techniqueName'] = gu
technique_dict['comment'] = gut['rel
technique_dict['tactic'] = gut['tact
technique_dict['group_id'] = gut['ex
techniques_list.append(technique_dic
groups_list[89]
Make a dynamic template
We can take the template I shared before, and make it
dynamic one to loop through the dictionaries in the
groups_list list . This allows me to create a navigator
group layer file for each intrusion-set.

Dynamic Template:

Dynamic Navigator Group Layer Template

Run Dynamic Template

We can then use the template in a loop as shown below:

import jsonfor group in groups_list:

for k,v in group.items():
if v:
actor_layer = {
"description": ("Enterprise techniqu
"name": ("{0} ({1})".format(k,v[0]['
"domain": "mitre-enterprise",
"version": "2.2",
"techniques": [
{
"score": 1,
"techniqueID" : technique['t
"techniqueName" : technique[
"comment": technique['commen
} for technique in v
],
"gradient": {
"colors": [
"#ffffff",
"#ff6666"
],
"minValue": 0,
"maxValue": 1
},
"legendItems": [
{
"label": ("used by {}".forma
"color": "#ff6666"
}
]
}
with open(('{0}_{1}.json'.format(k,v[0][
f.write(json.dumps(actor_layer))
Output Templates
That’s it! As you can see on the left of my Jupyter lab
server, I was able to automatically create a Navigator layer
file for every group that had techniques mapped to them
(Group uses techniques).
Visualizing Navigator Group Layer
Files
All you have to do now is download one of the navigator
group layer files
Go to the ATT&CK Navigator site: https://mitre-
attack.github.io/attack-navigator/enterprise/

Click on the (+) icon to the right of the default “layer” tab
and select Open Existing Layer > upload from local

Once you select group layer file, you should get the
following view:
APT12 Automatic Generated Navigator Layer File

If I check the ATT&CK site, we can see the 4 techniques

being used by the specific group and I can also check the
ATT&CK Navigator view option to validate I get the same
results.
Can we create a new function and
export all groups navigator layer
files? 😆
Yes, I took all the final steps and created another new
function named export_groups_navigator_layers() and
made it available in the attackcti library. Therefore, all
you have to do now to replicate all the code in this post is
run the following 3 lines. You will get every single group
Navigator layer file in around 10–12 seconds! 😱 🍻

from attackcti import attack_client()lift = attacK_c

That’s it! Now, It
🍻is up to you how you can take those
concepts and the new available functions to improve the
creation of Navigator group layers. For example, you can
add more properties such as data sources to each
attack-pattern cell in the Navigator view.
If you are using a version of attackcti that does not have
these latest updates, just run the following:

pip install --upgrade attackcti

Also, I put together a Jupyter Notebook for you to use and

go through all the code presented in this post You
do not have to build your own Jupyter Notebook server to
run it, you can access it via the Binder link below :

Notebook name: export_groups_navigator_layers

Noteebook nbviewer:
https://nbviewer.jupyter.org/github/hunters-
forge/ATTACK-Python-
Client/blob/master/notebooks/export_groups_navigat
or_layers.ipynb
Notebook Binder:
https://mybinder.org/v2/gh/hunters-forge/ATTACK-
Python-Client/master
I hope you enjoyed this post! Working on the attackcti
library has helped me a lot to learn more about STIX/TAXII
2.0 APIs for ATT&CK. Therefore, I highly recommend to
start using it and explore the code behind it. I am sure
there is a lot that can be improved in attackcti, so if you
would like to continue the conversations and help
contribute to it with some other cool use cases, I
recommend to join a slack channel that I host for the
community:

Automatic free slack invitation:

https://launchpass.com/threathunting

References
https://pypi.org/project/attackcti/
https://attackcti.readthedocs.io/en/latest/

https://mitre-attack.github.io/attack-navigator/enterprise/

https://github.com/oasis-open/cti-python-stix2

https://github.com/oasis-open/cti-taxii-client

https://github.com/mitre/cti/blob/master/USAGE.md#map
ping-concepts

https://attack.mitre.org/groups/G0005/