Professional Documents
Culture Documents
Automated Software Testing Using Open Source Testing Tools by Elfriede Dustin
Automated Software Testing Using Open Source Testing Tools by Elfriede Dustin
Automated Software Testing Using Open Source Testing Tools by Elfriede Dustin
Testing Tools
By Elfriede Dustin
Agenda
• Who is IDT
• State of Software Testing
• What is Automated Test and Re-test (ATRT)
• Approach to ATRT
• Advantage of open source
• ATRT and open source – Our IDE and
solution
• ATRT Challenges
2
http://www.idtus.com
Who is IDT?
• IDT specializes in the design, development, and
implementation of Automated Software Testing and
Re-Test (ATRT) and Quality Assurance (QA)
solutions
– Deliver turn key automated test suite
• Automated test strategy and identification of highest payoff areas to
apply automation for your project
• Selection of best automation tools for your project
• Using your existing test cases or develop test cases for you
• Test results documented in requirements traceability matrix
– Training and pilot project implementation with companies
3
http://www.idtus.com
Who is IDT?
• IDT specializes in the design, development, and
implementation of Automated Software Testing and
Re-Test (ATRT) and Quality Assurance (QA)
solutions
•We are currently hiring Java Developers
– Deliver turn key automated test suite
•Plus we’d
• Automated like toand
test strategy hire 2 or 3 grad
identification students
of highest payoff to
areas to
apply automation
work for us for your
part project
time ( i.e. 20 hours week )
• Selection of best automation tools for your project
•must be US citizen
• Using your existing test cases or develop test cases for you
• Test results documented in requirements traceability matrix
– Training and pilot project implementation with companies
4
http://www.idtus.com
State of Software Testing -
Why ATRT?
?
5
http://www.idtus.com
State of Software Testing -
Why ATRT?
50% or More of Overall Development Cost is Utilize Automated Testing Strategies and
Typically Spent on Testing Technology to Improve Productivity and Quality
Size & Complexity of SW Baseline
Spiral / Release 4
Test
Spiral / Release 1
Days
Test
Days
Test Manual vs. Automated
Days
7
http://www.idtus.com
ATRT Spans the Software Development
Life Cycle – Visual Modeling
Test
Use Components
cases
8
http://www.idtus.com
ATRT Spans the Software Development
Life Cycle – Requirements
Visual Modeling
Development Tools
Components
Visual Modeling
Components
Execution
AutomatedSoftware
Automated Testing Testing
Tools
Memory Leak
Detectors Components
Performance
Testing Tools Middleware
12
http://www.idtus.com
Types of Automated Test
Tools
Life-Cycle
Type of Tool Tool Description
Phase
Business Business Modeling Records definitions of user needs and automates rapid
Analysis construction of
Phase flexible, graphical, client-server applications
Configuration Baselines important data repositories
Management
Life-Cycle
Type of Tool Tool Description
Phase
Analysis and Database Design Develops second generation enterprise client-server systems
Design Phase
Structure Charts, Manage processes
Flowcharts, and
Sequence Diagrams
Test Procedure Generate test procedures from requirements, design, or data and object
Generators models
Syntax Checkers/ Perform syntax checking and have debugging capability; usually available
Debuggers with built-in programming language compiler
Programming Memory Leak and Detects runtime errors and memory leaks
Phase Runtime Error
Detection
Source Code Testing Verifies maintainability, portability, complexity, and standards compliance
Metrics Tools Code (Test) Coverage Identify untested code and support dynamic testing
Analyzers or Code
Instrumentors
Usability Measurements Provide usability testing as conducted in usability labs
14
http://www.idtus.com
Types of Tools (Cont’d)
Life-Cycle Type of Tool Tool Description
Phase
Other Testing Data Extraction Tool Extract Data from various formats into various formats
Life-Cycle
Test Data Generators Generate test data
Support Tools
File Compare Utilities Find discrepancies between files that should be identical in content
Network Testing Monitors, measures, tests, and diagnoses performance across the
entire network
GUI Testing Conducts automated GUI tests; capture/playback tools record user
(Capture/Playback) interactions with online systems so they may be replayed
automatically
Load/Performance Testing Conducts load/performance and stress testing
Collect/Create:
• Requirements
Test Results: • Test Cases
• Pass/Fail by Test Case • Test Procedures
• Pass/Fail Summary • Expected Results
• Test Time Required • Interface Specifications
• System/Component
Configuration Description
16
http://www.idtus.com
Why Open Source
17
http://www.idtus.com
Why Open Source?
• Advantage of Open Source:
– no licensing issues; i.e. license cost; maintenance;
etc.
– easily modifiable
– adaptable
– lightweight
– flexibility - not tied to one vendor
18
http://www.idtus.com
ATRTs IDE - Eclipse
Eclipse – this site lists all the plugins
http://www.eclipse-
plugins.info/eclipse/plugins.jsp?category=SCM&pager.offset=0&firstItem=1
22
http://www.idtus.com
Linux Capture/Playback Tools
IDT Findings
QF-Test http://www.qfs.de/en/qftestJUI/index.html
Froglogic http://www.froglogic.com/pg?id=Google&category=squishjava
KD Executor http://www.kdab.net/?page=products&sub=kdexecutor
Jameleon http://sourceforge.net/projects/jameleon/
Cantana http://www.ipl.com/pdf/p0003.uk.pdf
crontab Part of Unix
crontab, cruisecontrol, www.adminschoice.com/docs/crontab.htm
Dogtail http://people.redhat.com/zcerza/dogtail/about.html
http://www.cyberciti.biz/tips/linux-automated-gui-testing-with-python-based-
dogtail-tool.html
Expect http://expect.nist.gov/
23
http://www.idtus.com
Automated Test and Re-Test
(ATRT)
24
http://www.idtus.com
What is STAX?
STAX is an automation system
With STAX you can automate and monitor your entire test environment
ƒ System/product setup
ƒ Testcase distribution
ƒ Testcase execution
ƒ Testcase results collection and analysis
STAX consists of
ƒ A programming language designed for automation
–Programs written in this language are called jobs
ƒ An execution engine (similar to an interpreter for other programming/scripting
languages) which manages the execution and runtime behavior of the job
ƒ A GUI application, called the STAX Monitor, which provides a dynamically
updated view of your executing jobs
ƒ A tool (STAXDoc) that is used to generate documentation for your STAX jobs
25
http://www.idtus.com
STAF can run on:
* Windows 95
* Windows 98 (and 98 SE)
* Windows Millenium Edition
* Windows NT Version 4.0
* Windows 2000
* Windows XP
* Windows Server 2003 (IA32, IA64, AMD64)
* Windows Vista (IA32, AMD64)
* Linux (Intel32/64, AMD64, PPC32/64)
* Linux on zSeries (31-bit, 64-bit)
* AIX Version 4.3.3.0 or higher (32-bit, 64-bit)
* Solaris (Sparc) 2.6 and higher
* Solaris AMD Opteron 64-bit
* HP-UX 11.00 and higher (PA-RISC, IA64 32-bit and 64-bit)
* OS/400 V5R2 or higher
* z/OS UNIX V1.4 and higher
26
http://www.idtus.com
STAF Internal Services
• CRON Calls into STAF services at a specified time interval External (Java)
• EMAIL Allows you to send email messages External (Java)
• EVENT Provides a publish/subscribe notification system External (Java)
• EVENTMANAGER Allows you to call STAF services when a specified Event
occurs External (Java)
• HTTP Allows you to make HTTP requests which can be grouped together in a
session External (Java)
• LOG Provides a full-featured logging facility External (C++)
• MONITOR Allows a testcase to publish its current running execution status for
others to read External (C++)
• RESOURCE POOL Allows you to manage exclusive access to pools of elements,
e.g. VM UserIDs or Software Licenses External (C++)
• STAX Provides an XML-based execution engine External (Java)
• ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives
External (C++)
28
http://www.idtus.com
Technologies used in STAX
STAF
ƒ STAF provides the infrastructure on which STAX builds. The full power of STAF
and its services is exposed for use within STAX jobs.
XML
ƒ The STAX programming language is based on XML. This provides built-in
structure to your jobs, as well as providing a set of existing tools for use in
constructing your jobs, such as XML (aware) editors and XSLT.
Python
ƒ The STAX programming language builds on Python to provide a rich and
accessible data model
ƒ STAX's Python integration also allows you to access the wealth of existing
Python libraries
Java
ƒ The STAX programming language allows you to access existing Java
29
classes/libraries, providing another source for reuse
http://www.idtus.com
Automation Tasks
Automation Startup
System Setup
Execution
Monitoring
Testcase Execution
Synchronization
Testcase Output Analysis Resource Management
Testcase Cleanup
Results Notification
Automation Completion
30
http://www.idtus.com
End-to-End Automation with STAF and
STAX
31
http://www.idtus.com
Key ATRT Architecture
Requirements
• Support applications running on multiple computers
• Support applications developed in different languages
• Support applications running on different types of OS’s
• Support applications which have GUI and those which do not (for
example Interface testing)
• Support applications which use different types of network
protocols such as TCP/IP, DDS, etc
• Support integration of multiple commercial testing tools from
different vendors ( allowing as new or better products emerge
in the market they can be utilized )
• Support testing w/o having to install ATRT on the same
computers as the application under test and be able to be
distributed across computers 32
http://www.idtus.com
Prototype – Proof of Concept
Analysis & Display
Display Application Display Application
Application
Simulation of Application
Processing Messages
Simulation of Application
Processing Messages
ATRT
Test Manager
RTM
Bugzilla 33
Linux Based
STAF/STAX
http://www.idtus.com
ATRT Prototyping Effort
34
http://www.idtus.com
Additional Requirement
35
http://www.idtus.com
Automated Software
Testing
• Software testing paradox
• What is it?
36
http://www.idtus.com
Effective Automated
Software Testing
Current State:
Developing software in order to test software
Automate the test support development: automated IDL
to code generation; test data generation (using FireEye
from NIST); compare utilities; log file analyzers;
Our Automated Software Testing Goal:
• Minimize Human interaction
– Possibly get on par with Hardware Automated Testing?
37
http://www.idtus.com
Hardware Testing: current
state
• Many automated test generators for
manufacturing/ logic devices, circuit testing,
etc., available i.e. SEI/ATG, uMaster, etc.
• Automated testing without human interaction
is the norm for hardware testing
38
http://www.idtus.com
Hardware vs. Software
AT
How is hardware testing different?
• Software Challenge:
– Changing requirements
– GUI interfaces
– Code access/availability
– Defects uncovered earlier in the software
development lifecycle are cheaper to fix
• Solution flexibility required – problem
dependent
39
http://www.idtus.com
Proposed solutions
Focus on Reusable Components
– Test Data / Expected Results
– Interfaces
– Middleware
Assuming code and models are available:
• Model based testing; automated test case generators based on models
• Eclipse – effective ATRT IDE
• Probes – see Aprobe – Code instrumentation
• Building self-testable components
• Using standard language MOF to Text to generate automate test cases
GUI Testing tool ideas
New tools - other
• Working with NIST to produce additional tools
40
http://www.idtus.com
GUI Testing
• Jemmy - java library to test UI
• http://jemmy.netbeans.org/
• http://abbot.sourceforge.net/doc/overview.shtml
• Perl modules
41
http://www.idtus.com
Questions
We are hiring…….
Please come see me now if interested or send
email edustin@idtus.com with any follow-up
43
http://www.idtus.com