X

Q1)

While an insufficiently experienced or understaffed team may impact velocity, these are not the
only factors that impact velocity. In scrum, velocity is relative only to the specific team and
there is no determining that the velocity is low or high in comparison to other team. There is
more appropriately a determination that the team would need to work at a higher velocity in
order to complete the priority work within the time available to the agile team.
If it is a matter of needing a skillset that is not available within the existing team, then replacing
individuals for staff that do have the requisite skills may improve velocity. Similarly, adding staff
so that work can be completed within the same timeframe may improve velocity. However, in
many situations replacing team members or adding additional staff to an agile team likely will
reduce velocity, at least initially, as new team members will have a learning curve to overcome
in terms of understanding the business problems to be solved, and how the team is organizing
and managing the work. If the team determines that additional or different skills are needed to
complete the work, then getting the existing staff training may be a viable alternative to
replacing team members. For Devis, our ideal team size is 7 +-2, with a diverse skillset of
developer, business analysis and testing skills. This size fosters an ability to collaborate and self-
manage more easily than a large team.
Other potential factors that impact velocity are the team is unprotected and being distracted by
other priority work, the team is not collaborating effectively to jointly develop understanding of
how to accomplish the work, the team may not be effectively using Story Points as a means to
scope the work and determine who is best suited to complete the work, or simply may not be
committing to as much work as they could be. In all cases, we encourage teams during
retrospectives to consider ways to improve their velocity and to incrementally implement new
tools, techniques and approaches to doing the work.
Q2)

Shortcomings: these 3 metrics alone do not give the ability to effectively evaluate usability,
security, quality or risk identification. There are relevant factors like security and usability, as
well as different kinds of software testing like load testing, that largely fall outside the
parameters of these metrics. If your system is not usable, with responsive load times, users may
abandon it. If security isn't addressed, risks emerge that can outweigh the entire potential
positive impact that your product might otherwise deliver.

Other ways of assessing quality include: performing formal code reviews within the team and
measuring quality in terms of impact upon business performance through retrospectives.

Effective ways to evaluate quality and risk: Devis combines metric data with other analysis on
the TEAMS, CMS and ProgramNet to inform management and evaluate quality and risk:

1. Tracking defects traditionally narrows the focus to bugs where code is not performing as
expected but misses the bigger picture that the overall system may not be meeting customer
expectations for functionality, serviceability and usability. Devis implements product reviews of
smaller chunks of code to reduce risk of missed stakeholder expectations and increase likelihood
of end product aligning to requirements – eg. Higher quality end result. End user satisfaction
with existing system components may be used to evaluate quality and identify areas for
improvement.

2. Code coverage by unit test lets you know that individual components are performing as
expected (or not) but does not address issues with system integration or potential impacts to
existing software components that are not part of the current development effort. Unit testing
also does not test for end user acceptance of features. . Devis stands up testing and staging
environments where the program is analyzed as it will work when implemented, and to ensure
the system can function under load.

3 static code analysis helps to eliminate unnecessary toil by automating repetitive tasks to catch
machine detectable errors within the code. The Devis testing process can thus focus more time
evaluate quality and detecting underlying risks through processes like peer code reviews; when
one developer is finished working on an issue, another developer looks over the code for
obvious logic errors, whether all use cases were fully implemented from the requirements,
whether automated tests are sufficient to review the code and other coding best practices such
as adhering to a style guide to avoid solutions that would otherwise burden the project with
unnecessary technical debt.
Q3)

Our first challenge was to coordinate previously siloed technical domains into a coherent
systems engineering approach inclusive of development and engineering, operations and
maintenance, security, disaster recovery and continuity of operations, quality assurance and
incident response. To do so we implemented Dev[Sec]Ops joint design-build-test efforts with all
the above disciplines participating, sharing joint responsibility and ownership for the success of
the system over the entire lifecycle, and allowing our engineers to resolve problems with shared
input from subject matter experts from across all disciplines in the team. We also reward team
members for cross training in secondary relevant disciplines.

A second challenge: coordinating previously fragmented patch management, monitoring,

logging, and other support services into a coherent approach. To do so we trained our entire
organization in ITIL Foundations and implemening ITIL principles of standardization and
improved service catalog delivery. We took a whole-network viewpoint for standardization,
including software defined services and security and conducted service delivery design sessions
using engineers and analysts from across the enterprise. We reviewed architectures, processes
and tools for potential improvements and implemented priority recommendation, and were
able to significantly increase compliance with standards, reliability of services and improve
forensic artifacts for security.

A third challenge: aligning Dev[Sec]Ops outputs with customer requirements for independent
governance and their SDLC stage-gate requirements. Devis adapted our Agile-Scrum planning,
product reviews and retrospectives to include client subject matter experts throughout each
software sprint. Incorporating customers into agile planning, combined with frequent
incremental product delivery for review and feedback, ensures alignment with stakeholder
needs, enhances compliance with governance requirements, leads to fewer misunderstandings
on how the system is being built,and improves acceptance of the end result.
Q4)
To effectively implement Continuous Integration/Continuous Delivery we had to resist the
desire to have everything implemented at once and we had to foster a culture for developing,
automating and implementing repeatable processes. To get the most benefit from CI/CD we
learned that we have to take disciplined approach to performing the work – automating bad
tests, or the wrong tests, won’t improve org. efficiency or effectiveness. Automating an
unnecessarily complex configuration management process similarly won’t improve matters.
Once we determined that a “big bang” approach to implement CI/CD was not going to be
practical, we decided to incrementally implement CI/CD as a continuous improvement effort
where our processes are incrementally automated, elaborated and improved over time as
opposed to all at once. We successfully implemented CI/CD by re-engineering our highest risk,
most complex processes first. We implemented tools for automation and integration only after
the underlying processes and procedures were understood, defined and documented and we
had the workflow modeled out.. We took this approach process by process and project by
project, automating standalone services such as system configuration, testing, code coverage,
then implementing orchestration tools and now more recently containerization.

A second challenge we overcame was the changing the mindset of the engineers, developers,
analysts and testers, whose jobs may appear to be threatened as more and more engineering
tasks are automated and integrated. The historical mindset of technical staff is to “stick build”
and test systems by hand. At Devis, we first train people to perform the higher order tasks
necessary for CI/CD, such as writing automated tests and test scripts and reward them for
applying the training to the development of repeatable, automated processes. We also reward
staff for reuse-that is not re-developing something where there is already a solution, but rather
taking an off-the shelf process and adapting it as needed for the current situation.

Q5)

a) Devis has designed, selected technologies, hosted, and delivered applications in public,
private, hybrid and gov cloud environments. On Federal contracts like TEAMS. Devis uses a
containerized design – a FEDRAMP compliant solution that runs Kubernetes on top of MS Azure
and stands up development, testing, staging and production environments in order to allow the
system to be fully vetted before any updates go live. TEAMS ties into Login.gov single sign on
(SSO) to validate users. We also document third party dependencies and track and identify
vulnerabilities in the 3rd party libraries that we're responsible for keeping patched and up to
date as a part of TEAMS support. We maintain constant assurance that controlled data stays
within controlled areas and never migrates to an uncontrolled area of the system.
The WRAPS II system being built for DOS has been designated a High Value Asset (HVA) by DOS,
so Devis is designing it to work with high value asset control overlays designed by DHS to
validate and continuously monitors WRAPS II activity as per security and government
organizational directives.

d) Protecting this information on systems such as WRAPS and TEAMS helps ensure that bad
actors in the cloud cannot uncover or reverse engineer a back door into those systems or
databases deemed too sensitive to be moved to the cloud. It helps protect confidentiality and
accuracy of data, which preserve the integrity of PTO's processes, safeguards its reputation and
gives patent seekers no reason to consider other options such as the European and Japanese
patent offices (EPO and JPO respectively). To better secure sensitive data as Government
regulations have evolved over time, Devis has grown from on-premise hosting at Government
and Devis locations to contractor provided cloud hosting, and now Devis implemented cloud
hosting leveraging Fed Clouds, using systems like Kubernetes and Heroku in both MSAzure and
AWS and achieving System Level ATO. These evolutions have evolved to match
user/stakeholder accessibility while balancing security risks available by Federal cloud providers
For example, on DOS WRAPS program Devis is migrating legacy infrastructure from on-premise
to a AWS Fed Cloud. On USAID TraiNet/VCS Devis has hosted, leveraged public cloud, and we
are now migrating/migrated that infrastructure to a gov cloud for TEAMS.
Q6)

A) Devis builds and plans its Scrum and schedules according to PMI PMBOK best practices to
ensure that personnel with the necessary skills abilities and knowledge stay within the team
throughout the lifecycle of the engagement. This continuity ensures that knowledgeable
personnel with necessary skillsets and critical information are not only working on their own
specific tasks, but are available to provide consistent, timely peer review of other work within
the Scrum, which is critical to the ability of the team to maintain velocity while delivering high
value code Devis uses daily standups, communication channels such as slack, confluence, and
JIRA to provide immediate feedback throughout the Sprint. Stand-ups and channel
communication occur on a constant basis as builds and tests and staging takes place and when
personnel representatives of the technical requirements are being addressed. These methods
contribute considerably to mission velocity, help avoid miscommunications and it provides a
pattern and forum for consistency and commitment for regularly scheduled and frequent
deployments into production from the Scrum Team.
B) Accuracy and consistency of estimation are critical elements of the planning process, and
failure to accurately estimate task requirements leads to disruptions in project velocity, which if
unchecked leads to delays in delivery of working code, delaying all work dependent upon that
new code. This is one reason Devis practices Story Points as a means of estimating scrum team
labor whenever possible. This Devis approach gives an enhanced means of assessing which
portions of the Scrum Team work are going to be the most difficult, needing the most time and
effort to produce high quality code, and it avoids some of the pitfalls of standard estimation
(where easy tasks are underestimated, and difficult tasks are over estimated). As a result, more
accurate estimation techniques help keep regularly scheduled deployments on schedule. Devis’
approach to story point estimation can be found on our website. See https://www.devis.com/?
portfolio=improving-project-completion-estimations-with-story-points for more details on the
Devis approach to story point estimation. Story point estimation process becomes more
accurate when the team has worked together already and understands how team decisions are
made, as per F).

G) The Definition of Done must be shared and understood evenly across the Scrum and the
client to define completion expectations, avoid waste and rework, and focus the Scrum Team on
a common goal to deliver high quality code on a scheduled and frequent basis. Ensuring that all
participants, Devis and client and any involved third parties, understand exactly what the team
is working toward helps us avoid needless rework and misunderstanding – one reason Devis
practices Acceptance Test Driven Development or ATDD establishing an up-front Definition of
Done. In ATDD, the product must pass established predefined acceptance tests that are
technically chosen to demonstrate that a system is capable of performing the contracted tasks.
Development proceeds with all parties understanding what 'done' means -- the system passes
acceptance testing -- and that acceptance testing is comprehensive enough that all parties can
place faith in its efficacy as a gateway to deployment within the enterprise.

Q7)

A) The ability to deal with Change has outsized impact on the ability of a team to succeed under
these circumstances. Devis staff have made more than 350 trips to 80 countries to gather user
requirements and provide training. We are also co-located with our DOS customer and we use a
number of change methods to enable success, particularly with complications such as dispersed
teams, customer co-location, and/or when the Government is the integrator. These include:
• Applying Adaptive Management methodology to maintain ongoing feedback loops with our
clients and other value stream members allowing us to fine tune changes before they result in
outsized consequences.
• Providing Teams with robust CMMI-3 and ISO 9001 processes for dealing with change and
unpredictability
• Using frequent reviews and retrospectives to uncover the trends before the risks are realized.
• Requiring close integration of development and operations staff early and often in the process,
using stand-ups, electronic Kanban boards like StoriesOnBoard, and communication tools like
Slack, JIRA and Confluence.
These techniques enable a better understanding of the team capabilities and they can also give
advanced warning to a government integrator when unforeseen changes occur or for features
that will require the most support as a system is deployed.
C) Devis utilizes various electronic tools such as JIRA, StoriesOnBoard, and Slack on our ongoing
work at TEAMS and WRAPS, to drive communications, and will establish the specific electronic
collaboration, and communication tools to be used (in collaboration with the government) to
facilitate collaboration and communication with dispersed teams. We Close communication is
critical throughout the process, while tools are not enough, stakeholders government and
contractor staff need processes that bring them into regular contact with one another and
encourages open sharing of information. Devis establishes early transparency eliminating
barriers that can arise between one viewpoint and another and allowing people to start sharing
concepts particularly when the government is more than just the product owner and is acting as
the System Ingrator. Devis fosters this activity by pairing developers and operators to align on
the value stream objectives . Development staff benefit from knowing Operations’ capabilities
and limitations and operators have early engagement on the solution outcome. Pairing
developers and operators builds trust and understanding across the disciplines.

G) Devis utilizes various electronic tools such as JIRA, StoriesOnBoard, and Slack on our ongoing
work at TEAMS and WRAPS, to drive communications, and will establish the specific electronic
collaboration, and communication tools to be used (in collaboration with the government) to
facilitate collaboration and communication with dispersed teams. We establish the paths,
protocols, and scheduling recurrent checkins or standup calls that will keep communications
flowing throughout the lifecycle of the program, and confirm them at the program kickoff.
Devis’ 25 years of global experience informs our approach to dealing with a dispersed teams.
Maintaining a daily scrum call of ten to fifteen minute has to be scheduled with an eye on
making sure all participants can join. Geographically separated teams can make it challenging to
find a good time for a daily call, which illustrates the necessity to establish a path for doing so up
front.

Q8)
In the case of ProgramNet, Devis took an existing knowledge management system used by staff
at USAID and redesigned it to address legibility and visibility issues with the existing ProgramNet
platform which prioritized aesthetics over legibility. Contrast was not properly taken into
account when selecting background and text colors, and the font size and spacing was lower
than it should have been in order to present as much information on the screen as possible, and
the way some data were structured on the existing system were problematic for content
administration. For example, pages were being tagged using a hierarchical taxonomy, but the
navigational structure was not generated dynamically from the tagged pages, so content
administrators had to manually order the pages to match the structure indicated by the
taxonomy.
With these goals to guide us, Devis began the process of gathering requirements for the new
system. Years of involvement with maintenance of the existing platform for years gave Devis an
extensive understanding of the types of users and their various needs, allowing us to concretely
define existing roles that could be assigned by system administrators (useful when writing user
stories and test cases, as we could address these roles which were already understood by the
development and design team.)
The requirements gathering phase primarily involved discussions with the product owners, who
themselves were longtime users of the current platform that had gathered relevant feedback
over the years from users via surveys and meetings – very useful in determining where we may
want to add or alter existing functionality. We heavily incorporated this feedback into our
written requirements.
As the requirements became more concrete, we also began working on the visual design of the
system. Once again, the knowledge of the product owners and the feedback they had gathered
was heavily utilized throughout this phase. The higher-level layout of pages was initially driven
by that of the existing system, but emphasis was made to address the issues of legibility and
visibility.
The visual design team incorporated larger font sizes and background colors that provided much
better contrast against the text colors used throughout the site. We created a more cohesive
style for headings and other textual elements throughout the site, going through multiple
iterations on page layout to gather feedback from the product owners and other stakeholders
on their side. As a result, we were able to produce a new visual design that was heavily guided
by the needs of the users.

When creating the new informational hierarchy, we reevaluated the structure based on
observations of the way users navigated throughout the existing site. We grouped related
functions that had once been reached from different areas of the site under a common heading,
and we provided additional means of reaching documents by exposing the taxonomy tags
associated with the content to users.
During system development, regular meetings were held with users and stakeholders to address
any unanswered questions or concerns that we may have had. The product owners were also
able to test the system throughout the process, as we regularly deployed updates to an
environment that they could access – allowing us to identify any underlying issues that may
have been overlooked during the earlier design phases.
POC for ProgramNet: Jessica Pomerantz jpomerantz@usaid.gov
Q9)

Over 26 years, Devis has created and delivered more than a dozen Federal solutions that align
with the NIST RFM and achieved ATO. Our systems directly support seven Federal customers
and connect indirectly to entities across the government, from the intelligence community to
the Library of Congress and we have never had an ATO revoked. Federal agencies supported
include USAID, DOS, DOJ, VA, DOL, and the US Access Board. Systems, ATO duration, details and
points of contact are listed below:

CMS.usaid.org; (2010 – Present) the Credit Management System used by USAID to track loan
guarantees. Custom software solution. POC: Karen Pak, kpak@usaid.gov

USAID-FFP.entellitrak.com; (2012-Present) the Food For Peace Management Information System

used by USAID to track stocks of prepositioned food aid. Based on the Entellitrak COTS product.
POC: Stephane Bright sbright@usaid.gov

GLAAS.usaid.gov; (2007-Present) the Global Acquisition and Assistance System used by USAID
for its global procurement. GLAAS integrates several systems, including the PRISM COTS
product, the Oracle based Operational Data Store, and Phoenix financial software. POC: Yvonne
Wilson, ywilson@usaid.gov

ProgramNet.usaid.gov; (2014 – Present) Drupal based web portal USAID uses to implement its
modern program lifecycle. POC: Jessica Pomerantz jpomerantz@usaid.gov

TraiNet-VCS.usaid.gov; (2003 – Present) Custom applications USAID uses to monitor training

and related visa compliance, currently being replaced by the TEAMS system under development
by Devis. POC: Barry Richardson, brichardson@usaid.gov

Disability.gov; 2004-2017 WordPress based portal for the Office of Disability Employment
Policy at DOL. POC: Chuck Conaty, cconaty@usaid.gov

NRD.gov; 2009-2015 The National Resources Directory, a WordPress based interagency effort
between DOL and the VA is a web portal for returning veterans. POC: Martin Tsai
mtsai@caci.com

DNA.gov: (2010-2015) a custom DOJ training portal to improve capabilities in using DNA
technology for forensics. POC: Lee Mockensturm, Lee.Mockensturm@usdoj.gov
ePolicyWorks.org –2009 – Present This Sharepoint and Ideascale based DOL platform everages
technology to address barriers to employment for persons with disabilities, and fosters real-time
collaboration around key issues. POC: Michael Reardon <reardon.michael@dol.gov>

Access-Board.gov – 2011- Present The Access Board is an independent Federal agency

responsible for maintaining and updating Section 508 guidelines. The Access Board website is
built in Joomla. POC: Susan Little Little@access-board.gov

NCD.gov 2010-2015 Custom built National Council on Disability website. POC: Anne Sommers
ASommers@ncd.gov

IAWG.gov 1999-2015 The Interagency Working Group at the Department of State that oversees
training exchange activity performed by the US government and tracks training activity across
each agency. We designed the FEDS tool used by IAWG using a MS Access backend. POC: Erik
Anderson andersonen@state.gov
Q10)

Team Size A: 21 days Our TEAMS program at USAID was initiated in 2017 and required 12 staff.
Devis was able to assign staff from its ranks who had the technical and SME knowledge and fill
many roles on TEAMS with technically experienced staff familiar with the domain,, so we had to
onboard fewer than 7 additional resources for the TEAMS contract at USAID. This onboarding
process took about three weeks to conclude. The challenge with TEAMS was to find personnel
with necessary experience and clearability, capable of working in a DevOps environment
delivering into the production environment at USAID, Covering all these requirements in a tight
federal labor market meant not just finding the candidates but ensuring we interviewed them
well enough to assess their technical skills as well as to predict their suitability for work on a
close knit team.

Allowance for teleworking, and Scrum processes that include virtual daily standups to integrate
input from the entire distributed team, meant Devis was able to fill these positions with
remotely based personnel that had the necessary skills, qualifications and clearance to work on
TEAMS.
TEAMS POC: Barry Richardson, brichardson@usaid.gov
Team Size C: 30 Days. For the $180M DOS WRAPS contract, Devis on-boarded approximately 85
Secret & TS staff in approximately 13 weeks in early 2018. These staff included application
developers, scrum masters, cloud architects, system administrators, engineers, cyber security,
testers, analysts and specialty SMEs. The average time to on-board staff from award to
providing the agency resources over this period was ~6-7 technical resources per week.

On the same DOS project, Devis was tasked to on-board 20 development/integration staff to
upgrade legacy applications, and infrastructure, enhance user experience and support security
compliance. Devis applied our internal and external staffing resources, staff referral bonuses as
well as our subcontractors to on board 20 Secret cleared technical staff in approximately 30 days
(~5 people/week). Challenges with this effort included staffing hard to find ServiceNow
developers, cyber and other development and integration staff; securing staff to work in Rosslyn
Virginia; and finding staff with the proper clearances, and within our T&M proposed rates. Devis
overcame these challenges by offering: Superior Benefits, competitive salaries, 15% (no match)
SEP IRA, immediate 100 % vesting, an incentive stock option plan, commuting reimbursement,
and employee bonus pools; we also offer $2500/Year training budget for every employee.
Further we applied Scrum tactics: daily stand-ups with Corporate and Program Staff; an on-line
candidate tracking system, JAZZHR, which allows internal and external hiring managers and
recruiters to manage candidate advancement, and identify any blockers.
WRAPS POC: Hilary Ingraham ingrahamHE@state.gov
Q11)

Devis maintains a distributed workforce, supporting our operations from remote locations
across the US and on assignment overseas. Our personnel are outfitted with the tools to work
remotely like a VPN, encrypted laptops, communications tools like Slack and conferencing
technology allowing shared screens and interaction., allowing us to offer seamless support to
our clients during COOP events and governmental closures like inclement weather day. These
tools are hosted in the cloud as well to avoid impacts from manmade disasters; they are
accessed from around the globe by Devis personnel on assignment, and a number of our staff
work every day from remote locations, so the technology our COOP planning depends upon is in
use every single day. We formally activate our COOP plans yearly as a tabletop exercise for every
system running under ATO. The COOP plans cover the systems and Devis’ plans to support the
system and customers with back up or remote resources when impacted by an outage or
localized event. Devis also leverages a geographically distributed workforce with staff in the
Washington DC area; Southwest Virginia; California, Texas and Colorado to support our
organization resiliency and ability to operate in a localized outage or event.

Devis works with different Federal cloud providers like MS Azure and AWS, and have moved
large systems like TEAMS from the AWS system to the MS Azure system upon client request,
using containerization that also contributes to the ease of COOP recovery. Devis maintains
COOP for our internal tools through Amazon Web Services, maintaining coverage across several
availability zones (AZs) via both East Coast and West Coast hubs – which provides for
tremendous resilience in the face of natural disasters. Devis uses cloud based automation tools
like Puppet and up to date configuration files, to quickly reprovision and stand up new virtual
infrastructure, minimize recovery time, data loss and any other impacts from localized events.

Devis takes COOP exercises as an opportunity to allow us to make sure that all the relevant
procedural documentation is accurate and up to date, and to cross train technical
administration staff to support our CI/CD pipeline environments with the necessary skills to
activate and maintain operations under our COOP planning. If we find an inconsistency, we
update it in real time during the exercise. One in particular was the discovery that only certain
datacenters in AWS allow for a certain type of database container that allows encryption at rest.
Since you cannot provision this type of encrypted database container automatically (it requires
a PKI key and the encryption feature to generate it is not yet available internally within AWS)
Devis manually encrypts the container and then upgrade the container type, which allows us to
manage the system and data in an encrypted state using our CI/CD Puppet/Hiera process. On
the first attempt we selected a random datacenter, created the smaller DB, tried to add the
feature, attempted to upgrade the size of the DB to what was needed, and it failed because the
AZ didn't support the larger encrypted size. The solution was to move to a different Availability
Zone within AWS where larger encrypted sizes are permissible.

In support of resilient operations and COOP/DR type events, Devis maintains remote work
policies for all staff wherever practicable or allowable by contract. Our personnel all have
encrypted laptops; in the event of emergency, we are provisioned for our personnel to telework
over a secure VPN that can be accessed by distributed staff across the globe via a cloud based
infrastructure, our systems are optimized for distributed access, and we have well defined
policies whenever the client’s worksite becomes inaccessible. These practices have been proven
to work in our deployments to over 350 global worksites in over 80 countries.