Professional Documents
Culture Documents
Generic Test Checklist
Generic Test Checklist
Generic Test Checklist
document bookmarks:
application: app x
version: 1.00.00
checklist items are based predominantly upon several texts (esp. cem kaner’s book - bug list appendix) + experience + other
i have found that when i miss bugs, it is convenient and effective to build a new check into the checklist to prevent a recurrence of the missed bug while
testing future builds or apps. (every test failure recognized is a lesson learned for which the solution is built into the master checklist—a mistake is not
necessarily a bad thing when you can build upon it.)
revision history
tester date comments
schedule
# milestone name sched’d actual
date date
checklist instructions
preparation
front end tools
tracking tools:
bug tracking: testing must have a bug tracking tool. use raid, access app, etc.
test case manager: used to track status of test cases, avoid duplication of efforts (re-use test cases with slight modification),
etc. use access .mdb, word .doc, etc.
configuration management: use visual source safe (or sim. prod.) to save test cases, test plans, etc. also use it to look for
code changes, churn rate, version control, etc.
diagnostic tools:
virus scan: use f-prot or similar virus scanner to check master disk sets. scan test machines periodically.
nt taskmanager: tracks resource usage, threads, performance, etc.
windiff/examdiff: compare file differences. registry dumps, dir c:\*.* /s, or ini file snapshot changes over time—take
snapshots both before and after broken state occurs…then compare. (or use fc.exe f/nt too).
wps: debug tool that lists all modules loaded into memory at a given point. (applet comes with windows nt.) save list and
windiff multiple lists over time, or across o.s.’s.
perfmon: nt performance monitor. check out processor, ram, disk access, network, server performance. (in nt4.0, type
‘diskperf’ at command prompt to activate disk stats after reboot.
sysinfo: msoffice app that lists all info about currently loaded modules in memory. (comes with msoffice.)
shotgun: compares differences in registry between different snapshots over time. does much more.
wmem: checks system resources (user & gdi) and memory used by loaded apps
dr.watson: debug tool that lists machine configuration, task list, module list, stack dump.
performance profiling: used to determine where bottlenecks occur in code, etc. built-into vb4.0 enterprise.
complexity analysis: cyclomatic complexity metric, mccabe’s metric, etc. to determine complexity of source code.
coverage analysis: ensure that testing hit all lines code, all functions, etc.
automation tools:
dialog check: runs standard battery of tests against dialog boxes.
smart monkey: ivan, koko, etc. excellent tools for quickly automating.
simulators: while testing modules, can use drivers or other simulators to provide inputs to functions / apps being tested. vb
code = good tool.
visual test: automation where appropriate (repeatable).
macro recorder: used to reproduce intermittent bugs easily without effort of visual test.
test data generator: used to populate tables with test data.
backend tools
db native gui tools: dig them up. enterprise manager, security manager, nt user manager for domains, etc.
db command line tools: dig them up. isql, etc.
vb / access apps: create these to test things out.
batch scripts: run long processes overnight. business rule enforcement, etc.
schema comparison tool: compare schemas between servers.
dba: consult local dba for other tools.
event viewer: look at nt event viewer for history of silent errors.
data elements list: list of databases, tables, stored procedures, etc. with descriptions.
computer:
execution time constraint: impossible objective given hardware platform? too loose, users angry?
main storage constraint: too much, unfeasible?
platform volatility: os too new? too old?
development code turnaround time: builds take too long?
…
…
personnel:
tester capability: any weak links?
automation experience: if automation is necessary, any weak links here? inexperienced? spend too much time automating
uselessly?
application experience: new to app? misunderstand or unaware of basic business rules / functionality / user scenarios?
…
…
project:
latest techniques: still using ancient techniques / tools that are inefficient? using techniques / tools that are brand new, kinks
not worked out, inefficient?
schedule tolerance: is float time and resource loading too inflexible?
…
…
initial research
existing documentation: dig up old manuals, test plans, test cases, materials in files, materials in visual source safe, etc.
screen shots: where appropriate (maintenance testing, existing products upgrades, etc.), take screen shots of all forms,
dialogs, etc. compact view app functionality.
fact gathering: after reviewing existing documentation, go on fact gathering mission to fill in holes. speak with developers
and support analysts. learn server names, server owners, passwords and logins, build version numbers, etc. that are pertinent
to your testing.
learn app - click test: click through all buttons, tabs, etc. to ensure they work properly. work recursively from upper left
corner through lower right of each form and dialog in app. (objective is to learn the new app.)
smoke test: does the new build pass the standard smoke test? (use vt4.0, smart monkey, macro recorder?, or manual tests.)
front end:
front end testing is concerned with testing through the application interface. this is standard black box testing.
please note that the items in this section are listed in order of priority. items at the top are higher priority than items at the bottom. the prioritization was based
upon marv parson’s observations given in a video lecture on the history of testing. his excellent observation is that first and foremost, a tester must be sure that
there are no data related bugs. second, the setup must work. third, there should be no ‘visible’ system crashes. fourth, everything else ‘should’ work (ui,
localization, etc.).
data testing
accuracy / integrity:
calculations - reports: calculation errors in reports? wrong data loaded? < link to backend testing section. >
calculations – app: bad underlying functions? overflow or underflow? outdated constants? impossible parenthesis? wrong
order of operators?
calculations – backend: can tester generate queries that show calculated values from query differ from actual app calc’d
values (in table)? (if yes, double check test query math.)
div by 0: can test force this error condition?
truncate: truncate instead of rounding precision of numbers? incorrect formula / approximation for conversion?
compatibility: work with existing data structures? (print reports or other summaries with old and new systems. compare.)
test data: generate sufficient amount to thoroughly test the app.
database connectivity:
save: does it fail? is all data saved?
retrieval: does it fail? is all data retrieved?
installation testing
install scenarios:
clean machine: does setup fail? does app fail when run? (first wipe clean machine and reinstall. write down setup specs.)
new bld over old bld: does setup fail? does app fail when run? when run old app, does it recognize need to upgrade install
(reinstall)?
new bld over new bld: does setup fail? does app fail?
uninstall: does start > settings > control panel > add/remove programs > uninstall app fail? does a fresh install following an
uninstall fail?
reinstall: does reinstall fail?
install path: do long filenames fail? do spaces in path fail? do other drives fail? does not default path fail?
data:
dataset: max / min size problems?
numeric: min’s / max’s / absurds problems?
alpha: problems with ascii 1-32? 128+? a-z? a-z? (remember to check 13-enter, 9-delete, tab, etc.)
numerosity: problem with number of elements in list?
field size: problems with field size (n chars, long in place of int, etc.)
error guessing: any inputs that will be most likely to break the system that actually break it?
data structures: failure within data structures? break at constraints for elements? (analyze data structures for boundary
conditions break constraints.)
files: file size problem?
timeouts: query timeouts?
app:
initial uses: does app fail or act peculiar at first run? anything strange at second run?
loops: boundary failure at loop counter (try mins, maxs, one less then max, etc.)?
repeats: if repeat same operations over and over 100,000 times, does app fail?
dates: problems with mins / maxs dates? year 2000? year 1899? time boundary problems (0, 12, 24, etc.)?
space: location boundary problems (too close, too far away)?
memory: boundary failure in memory (not stress test)?
syntax: any valid inputs that do not work? any invalid inputs that do work?
hardware:
monitors: problems with old monitors? too new monitors? too old/new drivers? color problems?
hard drive: problems with old drives? too new drives? too old/new drives? color problems?problem with number of drives?
with drive being full? with size of drive? combinations of drives with cdroms, etc.?
cpu: cpu too old? too new? too slow? too fast?
printers: problems with old printers? too new printers? too old/new drivers? color problems? shade problems at extremes?
miscellaneous: mouse/trackball/touchpad too old/new? keyboard too old/new? pens too old/new? drivers for each too
old/new?
timeouts: timeouts with communications?
ui testing
communication:
tool tips & status bar: missing command button help tips (yellow boxes) when mousepointer in proximity? missing status-
/command-line tip/descriptions (at bottom screen) when mousepointer in proximity?
missing info.: no instructions? cursor not present? cursor not switch to hourglass? no status box during long delays? states
which appear impossible to exit?
enable/disable toolbar buttons: should toolbar buttons be enabled / disabled to provide clarity for available operations?
wrong, misleading, confusing info.: factual or spelling errors? more than one name for same feature? information overload?
product support email address or phone number invalid (help about, splash screen)?
help text & error messages: inappropriate reading level? verbose? emotional? inconsistent? factual errors? truncated
message text? missing text? duplicate words?
training documents: factual or spelling errors? missing information? incorrect information? not properly translated? spot
check index…wrong page numbers? spot check table contents…wrong page numbers? wrong app name, version, or other
significant datum?
dialog boxes:
keyboard: any keys not work? (move thru dialog with tab, shift-tab and hotkeys.)
mouse: any mouse actions not work? (click, double-click, right-mouse click, etc.)
cancelling: any method unavailable? (escape key, close from ctrl-box menu, double-click ctrl-box, hit win95 x button.)
oking: any method unavailable (enter, double click listbox item, etc.)
default buttons: no default button on dialog (default at hit enter)? no cancel button on dialog (default at hit escape)?
layout error: wrong size? non-standard format?
modal: is window setup improperly (modal or not modal)? can user click behind dialog and cause problems?
window buttons: should control menu or min/max buttons be visible?
sizable: is window frame wrong (should dialog be fixed / sizable)?
title / icon: is dialog improperly titled? is dialog missing or using wrong icon?
tab order: incorrect tab order (jumps all over)?
display layout: poor aesthetics? obscured instructions? misuse of flash and/or color? heavy reliance on color? layout
inconsistent with environment? screen too busy (need tab controls, or toggle option buttons to reduce complexity/clutter)?
boundary conditions: any boundary problems? (reference boundary conditions section of this checklist for all text boxes in
every dialog.)
sorting: are drop down lists not sorted where should be? (check list boxes, combo boxes, menus, etc.)
active window: incoming mail kill app? help / quick preview / tip of day / cue cards / wizards / other apps running in
background or on top of app cause problem?
mdi forms: puke, is it unnecessary? i don’t like them (unnecessarily complex).
memory leak test: does app leak memory as user goes in and out of dialog boxes dozens of times? (run task manager or
other tool in conjunction with quick vt or macro recorder ‘script’ to check dialogs.)
command structure:
time wasters: garden paths (deeply nested command)? choices that do not work or are stubs, “are you really, really sure?”
menus: too complex? too simplistic? too many paths to same place? can not get there from here? hot key duplicates? hot
key idiosyncratic (not standard with other apps)? hot keys not work? unrelated commands tossed under same menu item?
popup menus: does right mouse button invoke popup menu? should it for efficiency?
command line parameters: forced distinction between upper/lower case? unable to reverse parameters (locate anywhere in
parm line)? abbreviations/names that are not allowed? no batch input (for testing, faster runs, etc.)? command line too
complex?
keyboard: failure to use cursor, edit or function keys? non standard use of cursor, dit or function keys? failure to filter
invalid keys at input? failure to indicate keyboard state changes? failure to scan for function and control keys?
state transitions: can not do nothing and leave? can not quit mid-program? can not stop mid-command? can not pause?
program rigidity:
user options: can not do what did last time? can not find out what did last time? can not execute customizable command?
are there side effects to the preference changes? is there infinite tailor-ability?
control: who is in control, comp. or user—is it appropriate? is the system novice friendly? is it experience-hostile?
artificial intelligence and automated stupidity? superfluous information requested? unnecessary repetition of steps?
output: limited to certain data or formats? can not redirect output? can not control layout? can not edit labeling
(tables/graphs)? can not scale graphs?
preferences:
user tailorability: can not toggle on’off noise, case sensitivity, hardware, automatic saves, etc.? can not change device
initializations, scrolling speed, etc.?
visual preferences: can not toggle: scroll bars? status bar (@ bottom screen)? tool tips? window maximize/minimize?
hidden windows? default view? dialog colors? complex/simple menu structure?
file: default with 4 most recent files? default directory for open/save as?
localization: defaults not match local? (date / time, currency, etc. should match system defaults.)
usability:
accessibility: can users enter, navigate and exit the app easily?
responsiveness: can user do what they want, when they want, easily?
efficiency: can users do what they want in a minimum amount of steps and that is clear? (wizard?)
comprehensibility: do users understand product structure, help system and documentation (or too complex, incomplete, etc.)?
user scenarios: be sure to write-up test cases that are user scenarios and simulate how a user will use the system.
ease of use: is the app easy to learn and use?
localization:
translation: mistranslated text? untranslated text? error messages not translated? text within bitmaps that need to be
translated (if so…oh shit)? macro language not translated?
english-only dependencies: dependencies on tools or technologies only available in english?
cultural dependencies: dependencies on concepts or metaphors only understood in english?
unicode: any issues here?
currency: not match locality (britsh pound, japanese yen, etc.)?
date/time: any problems with different formats for various countries?
constants: any constants that vary with locality (financial / accounting equations, tax rates, etc.)? if so, how handle?
dialog contingency: dialogs can not be resized to 125% if translation requires additional space?
performance testing
general: slow app? slow echoing? poor responsiveness? no type-ahead? no warning that operation will take a long time?
benchmarks: are there any discrepencies in performance among similar operations? are some operations too slow? (do
many. for reports, query times, etc. look for hooks in app to log out this info. develop automated processes if necessary to
carry out. write up results in table / matrix.)
profiling: are there any significant bottlenecks in performance? (tool to detect where in code cpu time is going.)
modem considerations: graphics and help at 300 baud? data transfers of large sets kill system?
load testing
volume test: fail at large size of input/output? (try batch processing where available to speed up testing.)
stress test: fail at rapid input? fail at preemptive input (starts before last input finished)?
execution limit test: fail at 2 instances of app running? 5 instances? 10 instances? 40 instances? 80 instances?
window limit test: fail if run several apps (esp. with lots of buttons=windows)? how does app respond (should be with low
memory error, and not be fatal.)
storage test: fail when hard-drive filled up.
memory issues: fail when limited ram exists? (eat ram until to check app response and survivability. limit other resources
and run app.)
resource not returned: doesn’t indicate when done with device? doesn’t erase old files from disk? doesn’t return unused
memory? wastes computer time?
prioritize tasks: is app failing to prioritize tasks? (should be doing lower priority items during down times. app might
prioritize, but never get to low priority tasks…need check to force action after xx days elapsed.)
error handling
error preventions:
disaster prevention: no backup facility? no undo? no “are you sure” confirmations? no incremental saves?
version control: inadequate version control (at startup)?
initial state: inadequate initial state validation? not check for missing (or outdated) components?
input: inadequate tests user input? inadequate protection against corrupt data? inadequate tests of passed parameters?
o.s. bugs: inadequate protection against o.s. bugs?
security: inadequate protection against malicious use?
coverage: errors that were not handled by programmer?
error response:
appropriateness: inappropriate messages? not easily understandable?
help: unable to hit f1, or help button to get further details? (big no-no.)
error detection: ignores overflow? ignores impossible value(s)? ignores error flag? ignores hardware fault or error
conditions? ignores data comparisons?
error recovery: lack of automatic error correction? failure to report an error? failure to set error flag? program flow returns
to wrong area? unable to abort errors easily? poor recovery from hardware problems? no escape from missing disk?
error logging: unavailable? not informative: lacks module name, or time/date stamp, or error number and name? unable to
toggle on/off easily?
review error log: any new errors entered into error log after your testing? (look for failed assertions and silent errors.)
audit trail: lack of audit trail? not informative audit trail (history of use): lacks date/time stamps, user name, etc? does it
slow down system excessively? if so, is there an option to toggle logging off?
race conditions
data: multiple simultaneous read/writes kill server? (multiple updates compete so that successor begins execution on top of
data that has not completed processing of predecessor operation.)
wrong assumptions: assume that one event or task finished before another begins? assume that input will not occur during a
brief processing interval? assume that interrupts won’t occur during brief interval? assume that a person, device, or process
will respond quickly?
prerequisite check: task starts before its prerequisites are met?
security
logins: do authorization rules exist? (they should to monitor violations, etc.)
passwords: are passwords forced to be changed every xx number of weeks? are the passwords forced to be more than 4
characters? p/w have mixed case? p/w have numerals and special characters?
encryption: should there be encryption at data outputs? data transmits? saved document/data repositories?
security violation plan: have procedures been established to report and punish violators?
off-hours: are users limited to specified hours during the day? should the limitation exist?
installation: are security measures temporarily suspended during installation? if so, can user stop install midday and access
confidential data, or alter priveleges, etc.?
security: device use forbidden by user/caller? specifies wrong privilege level for device.
automated processes
this section needs further development.
contingency plan: missing contingency plan in case automation doesn’t work? (for example, app may not be integrated
soon enough to build automation scripts, or takes too long.)
back end
back end testing is concerned with testing through the back end database system. this is also black box testing, and should be coordinated with front end testing.
please note that the items in this section are listed in order of priority. items at the top are higher priority than items at the bottom.
this section is in its infancy, and will require significant modification in the near future.
general
object dependency: run sql server dependency check on ojects that changed to determine scope of impacted objects. then
test all impacted objects (sp_depends). search front end for all references to changed stored procedures / tables / other
objects.
data integrity: ensure di is maintained, data validation is used, anticipate errors from manual investigation.
data tracking: all transactions validated (import and output sets manually verified via xl, etc.)
data cleanup: temp objects removed, etc.
data recovery: transaction failure should be recoverable (if rollbacks properly in place).
component maintenance: automated maintenance scripts for testing various scenarios.
component stress testing: subject test server to high loads, etc.
security: permissions, logins, etc.
performance
perfmon: tool to detect performance across all system resources. set it up and use it.
benchmarks: determine time required for scripts to execute, etc.
network failure
at login: enter bogus password or login id. how does system react? reset permissions to eliminate tester, then attempt login,
etc.
in process: pull out network connection in middle of process, login off network, etc.
high traffic: setup (or coordinate with busy part of day) app so run with high network traffic.
server failure
kill server: shut down server abruptly while client still running. compare to standard shutdown.
cpu activity: simulate high cpu consumption by having server execute many other tasks so that busy.
file i/o: simulate high activity by copying large batch files, or running test util designed for this.
services not started: run with services not started on server.
silent failures: go to server periodically, and review event viewer history.
maintenance
walk thru tables: any orphaned temp tables? any duplicate tables? any bogus data jumping out?
business rule enforcement script: any nulls in inappropriate fieds? any foreign key values in child table without matching
primary key values in parent table? any calculations that are not correct? (write up test script consisting of multiple queries
to check every business rule.)
integration testing
integration testing is concerned with testing the full application, from installation through all(or most) functionality. note that
drivers and stubs should be used in place of code that is not yet written. this is standard black box testing.
prerequisites
check-in: all modules should be checked in.
executable: the executable should be built. there should be a full setup too. confirm pull location and build number.
milestone checks
regression: regression test existing know bugs.
full functionality: cover as much functionality as possible. run the horizontal tests (higher priority) of the app, moving
vertically (deeper tests) where time permits. run installation tests, and compatability and configuration tests, etc.
objective: developers want zero defects—meaning that no new high priority bugs are found or high priority existing bugs
resurface. testers of course want to find these if they exist.
unit testing
unit testing is another name for white box or glass box testing. it is concerned with testing the app by walking through the source
code. note that drivers and stubs should be used in place of code that is not yet written. also note that white box testing is
typically performed by the developer, unless the project is large enough to accommodate separate white box testers.
code inspections
coupling: is the effort required to interconnect the components during integration (after all separate components written).
maintainability: is the effort required to locate and fix bugs efficiently, organized and logical code flow, not overly complex.
comments: thorough enough? or to excess to the point of redundancy of the code lines?
live inspections
case coverage: trace through code one line at a time in if…elseif…else…endif, and select case…case….end select
statements to check for failures. be sure to use debug window to alter condition variable/pointer values, and reset next
statement to repeatedly reset execution pointer to previously executed lines of code speeds up testing).
error handling coverage: force all errors in error handler to occur by enter them into debug window at appropriate lines of
code while tracing code (at debug enter error ### to force error to occur).
debug.print: use this to test values and assertions.
design testing
design testing occurs prior to the start of development. this section is located down here because it unfortunately gets less use
than the other sections above. (also, i perform maintenance testing, so there is negligible design work.)
this section is in its infancy and will require significant modification in the future.
data overview:
data flow diagrams: do they exist? are they detailed enough? without a fairly detailed plan, how in god’s green earth can all
of the individual programmers write their separate components and then have them seamlessly come together at the end of
the project? the first thing a sane person does on a road trip through new territory is to consult a road map…and a dfd is one
piece of the roadmap.
entity-relationship diagrams: do er diagrams exist showing the layout of the data (all tables, databases, etc.)?
data requirements: missing / incomplete list of data requirements for each of major systems? (should have system i/o
diagram including names, descriptions, and sizes of data elements.) any size problems with database?
data groupings: data groupings not in logical categories? (should be static, historical—no change likely, and transaction
related.)
standard naming: data names non-standard? spaces in names? names too long?
data relationships: primary keys and foreign keys not defined? (hierarchical relationships.)
source definition: source of data unclear? (dept., individual, server name, etc.)
test data requirements: has this been ignored?
spec review:
thorough: are there any incomplete or missing spec sections?
glossary: are there any amiguous terms (tla’s – three letter acronyms, etc.) in docs? (if yes, consider glossary.)
business rules: missing appendix which clearly defines business rules? (why should programmer / tester decipher designer’s
/ user’s intent when it can be so easily listed in specs?)
project baseline schedule: is project schedule unrealistic? is schedule incomplete? are there insufficient milestones? have
any tasks been left improperly hanging (not wrapped into fragnets of like tasks)? are any fragnets improperly left hanging
(not terminated into milestones)? are any milestones left hanging (not eventually rolled up into a single end of project
milestone)? have responsibilities not been assigned to some tasks? are there any unrealistic durations? are there any
nonsenscial predecessor / successor relationships? has contingency time not been built into the durations? did scheduler
ignore or fail to obtain input from staff?
acceptance criteria: missing / incomplete acceptance strategy? (needed to clearly define minimum scope of work -- when
work is complete.)
feature overview:
useless: is feature useless? if yes, why add complexity?
duplicate efforts: are we reinventing wheel? are there similar features, products (i.e.: ‘free’ code—snicker snicker) already at
ms?
competitor analysis: are there any similar features in competitors’ apps? if yes, did you fail to research it? (take the good
ideas, and take good notes of the bad ideas.)
priority: missing / unclear prioritization of all features? (very important f/crunch time when determine what features to nix.)
feature interactions: missing details? (for example: undo-able feature, repeatable feature, multi-user impacts, ill effects of
aborting feature, backward compatibility issues, relationship to other features, etc.)
other considerations:
user interface design: use front-end testing checklist section above to test ui design (menus, dialogs, etc.)
• undocumented features: not responsible for bugs appearing in undocumented or non-maintained areas of the specification.
• external product impact: not responsible for bugs dependent on released microsoft products. any bugs that are found will be
reported to the respective product group.
• backend data maintenance: sql server backup and recovery, etc..
bug reporting
• bug tracking system: use the bug tracking system provided (raid, etc.)
• sample embed report: here is a standard bug report. all bugs should contain this information at a minimum:
<bug report template here.>
user environment
server minimum requirements:
• p/166 machine
• 64mb or up memory
• 6gb hard disk space
• windows nt 4.0
• nt sql server 6.5
• 16mb or up memory
• 1200mb or up hard disk space
• super vga display
• win 3.11, win nt 3.1, win 95, win nt 4.0