6 Steps to a Games QA

“Test Intelligence” Dashboard

Prodipto Roy
Test Director, Lionbridge Game Services
#GameQualityForum

#GameQualityForum
Introduction: About Me

#GameQualityForum
 A Roadmap to a Dashboard

We have developed a step by step process to bring your

important business questions to a dashboard and
answer them through advanced visualization of data

#GameQualityForum
 The 6 Steps
How will you measure the performance of your team
and learn more about other areas of your business?

KPIs & Business Questions What data sources will you trust to answer your
questions, how and how often will you access it?
STEP 1
Data Discovery & Governance
STEP 2 How will you cull and transform your raw data to
STEP 3 Data Model & Measures yield structured and normalized data?

STEP 4
Visualization Prototyping How will you uniquely represent data for specific
STEP 5 questions?
Dashboard Construction
STEP 6
How will you cleanly lay out your visualizations and
Secure Sharing what sort of drill down or discovery UI is required?

To whom will you make your data visible, and what

measures will be used to ensure security?

#GameQualityForum
Step 1 • The first and most important step is defining the Key Performance Indicators
that will measure the performance of your teams
• These KPIs should focus on areas of the business that impact quality,
Development of efficiency, effectiveness, elasticity, and cost.
KPIs and Important  Bug trends

Business Questions  Play coverage

 Code coverage

??
 Test execution times
 Schedule slippage
 Automation yield
 Etc…
• An important part of this process is defining additional business questions
that give you more insight into other parts of the business
 Think about other data sources you might have access to like business
intelligence, or play instrumentation.
 Utilizing some of the above during public access periods can help inform a fuller
quality picture and also help you understand where players seem to have issues
and refocus your testing
 Or, this could simply mean some additional break down of the main KPIs, such as
quality by location, or productivity by team, or coverage by test case.

#GameQualityForum
Step 1 IMPORTANT: In your planning, you may determine that you don’t have all the
data you need to answer a question so be thorough and comprehensive in your
thinking. Your work in this step may drive changes or tweaks in your testing, or
Development of the codebase

KPIs and Important • For example, lets say that you want to investigate what is causing players to
Business Questions “give up” at a certain point in an experience as seen/reported using defect
clustering.

??
 Let’s say you have a map that you can overlay player progression data on, but
you don’t have a way to know what bugs exist along that play path
 In this case, you may ask your QA team to start including map coordinates as a
new element in their bugs.
 And it could also be the case that there is no good way for a tester to grab
coordinates, so there may be an ask for your dev team to expose that
information on screen in debug as well

• The sooner you can start planning your Test Intelligence Strategy, the better
prepared you and your production teams will be

#GameQualityForum
Step 2 • All defined KPIs and business questions need to be answered with trusted
data
• You need to define these authority sources and assess the security model
Data Discovery used for each
and Governance • You can then gain access and develop automated exports or direct
connections to the data sources that meet your security requirements, and
don’t adversely affect performance of production systems which may
consume data from a given source
• At the end of this step you will have connections or automated raw data
exports setup for each identified data source on a regular cadence

#GameQualityForum
Step 3 • During this step you will need to import all required data sources and create
a model using related fields and points of interest
• Next, you will cull the data down to the most important pieces and apply
Data Model transformation techniques
and Measures • At the end of this step you will have aggregate functions and normalized
Development data for presentation and filtering

#GameQualityForum
Step 4 • Each KPI and business question requires an appropriate visualization to
properly represent the data and offer it for consumption
• You need to carefully design visualizations which provide the most insight,
Visualization and answer questions in the most appropriate way for your viewers.
Development  Consider a CEO who has less intimate knowledge of your team’s work. They may
be better addressed via a report with explanatory text or an infographic with lots
of images
 On the flip side, a peer or direct manager would probably want more detail and
exploratory interactivity
• When it comes down to it, remember we all know what to expect from
certain chart/graph types.
 Don’t create a barrier to consumption by going way outside the box. It’s
paramount to easily locate and understand key information.
 Find a good balance, and when a common graph won’t cut it, then get creative.
 And if you are looking for some inspiration, here are 16 Captivating Data
Visualization Examples

#GameQualityForum
Step 5 • The individual data visualizations need to be organized and laid out to
deliver the information in a clean yet comprehensive way
 Before you start laying out in your tool of choice, make a sketch
Dashboard  Even with six or fewer elements on a report page, you still want to guide the
Construction viewer's eye to the critical metrics. Highlight important parts of your report with
a bright color or summary icon perhaps.
 Background matters. White (or light colors) will keep things clean and
professional, however black (or dark colors) will generally draw the viewer's eyes
to highlights more effectively.
 Slicers will need to be added allowing you to manipulate the data and provide
answers to questions we may not have included in the initial scope
• Above all, remember that everything should have purpose, and that no
amount of colorful and creative layout will disguise bad data
• On the next two slides we’ll look at a couple examples…

#GameQualityForum
Step 5
Dashboard Sample Lab Dashboard
Construction

[Redacted]\Art\Environment

#GameQualityForum
Step 5
Dashboard Sample Lead
Construction Dashboard

[Redacted]

#GameQualityForum
Step 6 • The final step is to securely share the dashboards with the appropriate
audience and open up all the necessary presentation platforms

Secure Sharing

#GameQualityForum
Selecting Visualization Tools
• There are a variety of industry standards, as well as
home grown tools available for the data visualization.
• Each has its own set of unique features, licensing
agreements, and the like
• You need to assess these technology options to
determine an optimal solution based on a variety of
factors

• How do you chose which is

the right one for you?

#GameQualityForum
 Selecting Visualization Tools
FAMILIARITY
 Complementary to technical features, a
tool’s fit is based on company culture,
direction, and currently supported
infrastructure
 For example, if Tableau is the standard at
executive meetings it can be difficult to
introduce a new look and feel

#GameQualityForum
 Selecting Visualization Tools
FAMILIARITY
 Complementary to technical features, a
tool’s fit is based on company culture,
direction, and currently supported
infrastructure
 For example, if Tableau is the standard at
executive meetings it can be difficult to
introduce a new look and feel
SECURITY
 Security is important to every organization,
and trade-offs are constantly being made
between data availability and IP protection
 But practical factors can come into play
regardless of specific security goals, ranging
from AD integration to access rights and
governance.
#GameQualityForum
 Selecting Visualization Tools
FAMILIARITY
 Complementary to technical features, a
tool’s fit is based on company culture,
direction, and currently supported
infrastructure
 For example, if Tableau is the standard at
executive meetings it can be difficult to
introduce a new look and feel
SECURITY
 Security is important to every organization,
and trade-offs are constantly being made
between data availability and IP protection
 But practical factors can come into play
regardless of specific security goals, ranging
from AD integration to access rights and
governance.
COST/LICENSING
 Licensing can play a large part in selecting a
technology, not only for the baseline toolset,
but for advanced features such as importing
or connecting to data, power user features,
and security

#GameQualityForum
 Selecting Visualization Tools
FAMILIARITY
 Complementary to technical features, a
tool’s fit is based on company culture,
direction, and currently supported
infrastructure
 For example, if Tableau is the standard at
executive meetings it can be difficult to
introduce a new look and feel
SECURITY
 Security is important to every organization,
and trade-offs are constantly being made
between data availability and IP protection
 But practical factors can come into play
regardless of specific security goals, ranging
from AD integration to access rights and
governance.
COST/LICENSING
 Licensing can play a large part in selecting a
technology, not only for the baseline toolset,
but for advanced features such as importing
or connecting to data, power user features,
and security

EXTENSIBILITY
 As businesses grow, software needs to be
ready to grow with them.
 When choosing visualization software, it
should align to the growing needs of your
business, and support necessary expansion

#GameQualityForum
 Selecting Visualization Tools
FAMILIARITY
 Complementary to technical features, a
tool’s fit is based on company culture,
direction, and currently supported
infrastructure
 For example, if Tableau is the standard at
executive meetings it can be difficult to
introduce a new look and feel
SECURITY
 Security is important to every organization,
and trade-offs are constantly being made
between data availability and IP protection
 But practical factors can come into play
regardless of specific security goals, ranging
from AD integration to access rights and
governance.
COST/LICENSING
 Licensing can play a large part in selecting a
technology, not only for the baseline toolset,
but for advanced features such as importing
or connecting to data, power user features,
and security

EXTENSIBILITY
 As businesses grow, software needs to be
ready to grow with them.
 When choosing visualization software, it
should align to the growing needs of your
business, and support necessary expansion
SHAREABILITY
 The most important part of data intelligence
is sharing the information and insights
gleaned and your tools need to be capable
of reaching all intended audiences.
 Ensure you think through your tooling, web
and mobile surface points when choosing a
technology stack

#GameQualityForum
 Selecting Visualization Tools
FAMILIARITY
 Complementary to technical features, a
tool’s fit is based on company culture,
direction, and currently supported
infrastructure
 For example, if Tableau is the standard at
executive meetings it can be difficult to
introduce a new look and feel
EXTENSIBILITY
 As businesses grow, software needs to be
ready to grow with them.
 When choosing visualization software, it
should align to the growing needs of your
business, and support necessary expansion
SECURITY
 Security is important to every organization,
and trade-offs are constantly being made
between data availability and IP protection
 But practical factors can come into play
regardless of specific security goals, ranging
from AD integration to access rights and
governance.
SHAREABILITY
 The most important part of data intelligence
is sharing the information and insights
gleaned and your tools need to be capable
of reaching all intended audiences.
 Ensure you think through your tooling, web
and mobile surface points when choosing a
technology stack
#GameQualityForum
COST/LICENSING
 Licensing can play a large part in selecting a
technology, not only for the baseline toolset,
but for advanced features such as importing
or connecting to data, power user features,
and security
USER EXPERIENCE
 Lastly, users need to be able to interact with
the software, dashboards, and other features
with ease
 An optimal solution is one that is intuitive,
responsive, accessible and cost effective to
both create and maintain.
Questions?
Thanks!

#GameQualityForum