TRUE BREW® Process Overview

QUALCOMM Proprietary
 QUALCOMM Incorporated
5775 Morehouse Drive
San Diego, CA. 92121-1714
U.S.A.
Copyright © 2003 QUALCOMM Incorporated
All Rights Reserved
Printed in the United States of America.
All data and information contained in or disclosed by this document are confidential and proprietary
information of QUALCOMM Incorporated, and all rights therein are expressly reserved. By accepting
this material, the recipient agrees that this material and the information contained therein are held in
confidence and in trust and will not be used, copied, reproduced in whole or in part, nor its contents
revealed in any manner to others without the express written permission of QUALCOMM
Incorporated.
Export of this technology may be controlled by the United States Government. Diversion contrary to
U.S. law prohibited.
BREW, BREW SDK, and MobileShop are trademarks of QUALCOMM Incorporated.
QUALCOMM, Binary Runtime Environment for Wireless, TRUE BREW, and The Grinder are
registered trademarks of QUALCOMM Incorporated.
All trademarks and registered trademarks referenced herein are the property of their respective
owners.

TRUE BREW® Process Overview

80-D4299-1 Rev. D
December 17, 2003
 Contents
1 Introduction 4
1.1 Abbreviations and acronyms 4

2 What is TRUE BREW Testing? 5

3 TRUE BREW Testing Overview 6

3.1 Documentation needed 6
3.2 Process overview 7
3.2.1 Preparing for application or extension submission 7
3.2.2 Submitting an application or extension 8
3.2.3 Testing overview 8
3.3 Phases of TRUE BREW Testing 10
3.3.1 Entrance Criteria 11
3.3.2 Exploratory 11
3.3.3 Handset/AUT Interaction 12
3.3.4 BREW Features 12
3.3.5 BREW User Interface 12
3.3.6 Adversarial 12
3.3.7 Application Download 13

4 Operator Testing 14

5 Next Steps 15

iii
QUALCOMM Proprietary
 1 Introduction
This document provides information on the TRUE BREW® Testing (TBT) process for a
Binary Runtime Environment for Wireless® (BREW) application. The objective of this
document is to identify the following:

The overall TBT process

All supporting information that may be of benefit to the reader

Note that this document should be considered as an introduction to the TRUE BREW Test
Guide and related documents that provides detailed testing requirements and procedures.
The TRUE BREW Test Guide can be found at the following address:
http://www.qualcomm.com/brew/developer/resources/ad/documentation.html

1.1 Abbreviations and acronyms

The following is a list of abbreviations and acronyms used in this document.

ADS Application Download Server.

AUT Application under test.
BREW™ Binary Runtime Environment for Wireless®. BREW is an application development
environment that is layered upon public services offered by mobile ASICs.
BREW SDKTM BREW Software Development Kit. This allows application developers to write and
test applets and modules in a familiar Windows desktop environment.
MIF Module Information File. The MIF Editor generates this binary file, which contains
information regarding the list of classes and applets supported by the modules.
TBT TRUE BREW® Testing.

4
QUALCOMM Proprietary
 2 What is TRUE BREW Testing?
TBT is a BREW application testing program. Its goal is to ensure that tested applications
meet a specified level of stability and compliance with the BREW platform. QUALCOMM
works with third party test organizations to implement the TBT program and assure
impartiality to each developer. An application must pass TBT before it can be
commercialized by operators participating in the TBT program.

The goal of TBT is not to evaluate the quality of an application in a subjective sense. For
example, TBT does not comment on the marketability or business case for an application. It
does not make any judgment on whether an application might be popular with a certain
audience, or whether it is “fun” to use. This is left to the discretion of BREW operators who
may choose to perform some additional testing once the application has passed TBT.

5
QUALCOMM Proprietary
 3 TRUE BREW Testing Overview
This document references two entities in the TBT process:

The third party test lab implements the TBT requirements, manages application
submission, and provides feedback to developers. The test lab grants TRUE
BREW status.

A QUALCOMM entity represents QUALCOMM on TRUE BREW issues.

QUALCOMM makes the TRUE BREW application available for distribution to the
operators. This entity also maintains the list of test requirements and collects more
general performance measures for the TBT process. This information is used to
improve the efficiency and effectiveness of the entire process, as well as providing
developers with information to improve their own development and testing efforts.

3.1 Documentation needed

The following documents are used by both the developer and the third party test lab to
evaluate and test a BREW application:

BREW™ User Interface Design Guidelines: Provides information on user

interface design considerations and usability principles especially targeted at
developers of applications for BREW devices.

TRUE BREW® Test Guide: Specifies the test requirements that a BREW
application must meet in order to acquire the TRUE BREW status.

TRUE BREW® Test Procedures: Provides a spreadsheet that presents

procedures for testing applications against the requirements. The spreadsheet
includes columns for noting test results, issues, and comments.

TRUE BREW® Application Specification Template: Provides the base for the
creation of the Application Specification (App. Spec.) document that is required
when submitting an application to TBT. It contains information that will assist the
testers in getting to know and evaluating the BREW application.

6
QUALCOMM Proprietary
 TRUE BREW Testing Overview

The above documents are available under the BREW Documentation section of the BREW
internet web site at:
http://www.qualcomm.com/brew/developer/resources/ad/documentation.html

Additional information is available under the Application and Extension Testing section of the
BREW internet web site at:
http://www.qualcomm.com/brew/developer/resources/bd/ext_test.html

3.2 Process overview

This section provides an overview of the TBT process. It also documents the steps
necessary to prepare an application for submission to TBT.

3.2.1 Preparing for application or extension submission

Before submitting an application for TBT, you must perform all of the following tasks:

Download the BREW Software Development Kit (BREW SDK™) and create a
BREW application.

Sign the application developer legal agreement and complete the questionnaire
providing access to the BREW Developer Extranet (Developer Extranet).
Download the BREW Testing and Commercialization package from the Developer
Extranet.

Develop the application. In addition, you should test the application both in the
BREW Emulator (Emulator) and on the handset by following the TRUE BREW
Test Guide.

The process is slightly different for BREW extensions:

Download the BREW SDK and create a BREW extension.

Sign the extension developer legal agreement and complete the questionnaire
providing access to the Developer Extranet. Download the BREW Testing and
Commercialization package from the Developer Extranet.

Develop a testing application that uses your extension. The third party test lab will
use this application to test your extension. Note that the testing application should
cover all interfaces provided by your extension. The sample application must also
provide testers with a user interface in order to interact with it. The user interface

7
QUALCOMM Proprietary
 TRUE BREW Testing Overview

does not have to be sophisticated, but it has to be functional and to conform to the
TBT requirements.

In addition, you should test the extension using the testing application both in the
Emulator and on the handset by following the TRUE BREW Test Guide.

3.2.2 Submitting an application or extension

Go to the third party test lab’s web site and submit your application. For example, NSTL
operates an approved TBT lab that can be accessed at the NSTL web site
(http://www.nstl.com/logoprogram/qualcomm_logoprogram.html). You must ensure that the
submitted application is signed using a VeriSign™ Class 3 Certificate. Note that TBT labs
may ask you to sign a legal agreement before you can submit your application.

You are required to submit the following materials:

A version of the application compiled for the target handset’s processor. BREW
currently runs on commercial devices using an ARM processor, and therefore an
ARM-compiled version is necessary.

A Windows-compiled version of the application designed to run on the Emulator.

The App. Spec. document.

For extensions, you must submit the following material:

A version of the extension and testing application compiled for the target handset’s
processor. BREW currently runs on commercial devices using an ARM processor,
and therefore ARM-compiled versions are necessary.

A Windows-compiled version of the extension and testing application to run on the

Emulator

The App. Spec. document for both the extension and sample application

For more detailed submission requirements, refer to the Developer Extranet at:
https://brewx.qualcomm.com/developer/operations/submitapp.jsp

3.2.3 Testing overview

Figure 3-1 is a high-level illustration of the TBT process.

8
QUALCOMM Proprietary
 TRUE BREW Testing Overview

Figure 3-1. TRUE BREW test process

Submission is canceled.
Status and testing report
sent to developer by email.
No
Rework required.

Developer submits Pass

Developer creates
application package to 3rd Entrance criteria executed entrance
BREW application
party lab submission site criteria?

YES

Full TRUE BREW testing

executed

Status and testing report Pass

sent by email to the YES
NO TRUE BREW
developer. Rework testing?
required

Minor Errors

Application receives Application receives

“Passed with Notes” TRUE BREW
status status

Status and testing report

sent by email to the
developer

Application enters BREW

distribution system.
Developer is notified
about next step.

The application developer submits applications to be tested through the third party test lab’s
application submission web page. After logging into the third party TBT program web page,
the application developer fills out a web-based form to proceed with testing. This form
includes a TRUE BREW-specific questionnaire that collects application-specific and testing-
specific information. Note that developers will incur TBT charges as described on the
following web page:

http://www.qualcomm.com/brew/developer/resources/bd/ext_test.html#fees

When an application is submitted, Entrance Criteria testing begins (see section 3.3.1
Entrance Criteria). If the application fails Entrance Criteria testing, the application
submission is canceled, and an email is sent to the developer. A testing report explaining the

9
QUALCOMM Proprietary
 TRUE BREW Testing Overview

issues is attached to the email. The developer has to fix these issues and resubmit to the
third party test lab.

If the application passes Entrance Criteria, the test lab runs the different phases of TBT as
described in sections 3.3.2 through 3.3.7. After the testing is completed, the test results are
evaluated; in particular, the errors detected during testing. Based upon this analysis, an
application will fall into one of the following three categories:

TRUE BREW status: The application meets all TBT requirements. No errors are
detected.

Pass-with-notes status: The application meets all TBT requirements with minor
errors that are noted in the testing report. These errors have a low probability of
occurrence and/or have minimal impact on the end-user experience. See TRUE
BREW Testing: Developer’s Guide to the TRUE BREW “Passed With Notes”
Process, which can be found at:
http://www.qualcomm.com/brew/developer/resources/ad/documentation.html

Fail: The application does not meet the TBT requirements. The application cannot
be commercialized. The developer has to fix the reported errors and resubmit to
the third party test lab.

In addition to errors that cause an application to fail or pass-with-notes, the testing of an

application may produce observations, made by the tester, that are not part of the TRUE
BREW pass/fail criteria. These observations are referred to as issues. Though not part of the
pass/fail criteria, these issues may be documented in the test report. Operators may
consider these issues when evaluating an application. It is important to note that the
identification and documentation of these issues is not the goal of TBT. They are
documented and identified as time permits.

3.3 Phases of TRUE BREW Testing

The TBT process is composed of several phases that can be run in a sequential or parallel
fashion. In the process of preparing your application for submission to TBT, it is strongly
recommended that you run these phases in order to ensure that your application will pass
upon first submission. You should not limit the time allocated to this phase, and you should
execute each test case for every screen display, function, and feature provided by your
application.

Each type of testing is described in Table 3-1 and in the following paragraphs.

10
QUALCOMM Proprietary
 TRUE BREW Testing Overview

Table 3-1. Application testing

Item Type of test Phase

1. Entrance Criteria Prior to starting TBT

2. Exploratory Testing TBT

3. Handset/AUT Interaction TBT

4. BREW Features TBT

5. BREW User Interface TBT

6. Adversarial TBT

7. Application Download TBT

3.3.1 Entrance Criteria

This portion of the test process is intended to catch easily discoverable errors so that they
can be corrected before more thorough testing begins.

Entrance Criteria testing is largely performed by inspection of the package submitted by the
developer, including the MIF (privileges, notifications, and so forth), directory structure, files
in the submission package, and the Application Specification. The last step of the entrance
criteria consists of downloading the application from an ADS and starting it up on the BREW
device. This cannot be performed by the developer. However, if the developer was able to
cable-load the application and start it up on the handset, it is expected that the application
will pass this phase of the testing process with no issues.

An application that does not meet these requirements receives the “cancelled” status and
not the “failed” status. The developer will not be charged a testing fee if this occurs.

3.3.2 Exploratory

Exploratory Testing is performed manually by using the application installed on the handset.
The tester uses the Application Specification as the basis for determining product
functionality and then verifies the existence and operation of the specified functionality. The
tester goes through the application screens and features just as the end-user would do.

11
QUALCOMM Proprietary
 TRUE BREW Testing Overview

A good introduction to exploratory testing can be found in the document General

Functionality and Stability Test Procedure (James Bach, Satisfice, Incorporated) at:
http://www.satisfice.com/tools/procedure.pdf.

3.3.3 Handset/AUT Interaction

These tests focus on evaluating the behavior of the application when suspended and the
ability of the application to resume activity at or near the point of suspension. A combination
of incoming calls, SMS messages, and alerts is used during this phase to suspend the
application. The application behavior should be independent of the particular type of
suspension (for example, SMS suspend versus incoming phone call suspend). The
suspend/resume behavior should be evaluated on all screens and all state transitions (for
example, setting up data calls, tearing down data calls, transitioning from one screen to
another, and so forth).

3.3.4 BREW Features

These tests are performed by evaluating specific capabilities or functions of the application
(for example, saving data, downloading ring tones, and setting up a network connection). All
of these capabilities would have been evaluated to some degree during Exploratory Testing.
The purpose of these tests is to focus in on significant BREW features used by the
application as advertised in the Application Specification document.

3.3.5 BREW User Interface

These tests focus on how the application reacts to Clear and Send key inputs. The Clear
key, for instance, has a special meaning in terms of navigation and the user interface: It
should take the user back to the previous screen. This phase also focuses on response time
of the application as the user navigates through the screens and functions of the application.

3.3.6 Adversarial

These tests are performed using both manual and automatic procedures.

Manual procedures include placing the handset in a shielded box to ensure that the
application can handle loss-of-service events while performing certain functions. They also
include using the MaxFileCount tool and the Shaker tool to create low memory and low file
handle conditions on the handset and see how the application reacts to those conditions.

12
QUALCOMM Proprietary
 TRUE BREW Testing Overview

Automatic procedures consist of running the application in the Emulator and on the handset
in conjunction with The Grinder® tool, which sends random key events to the application.

3.3.7 Application Download

These tests are performed through interaction between the handset and the ADS. They
include disabling and then restoring the application, and having the ADS recall the
application. Note that it is not possible for the developer to perform this phase.

13
QUALCOMM Proprietary
 Operator Testing

4 Operator Testing
After an application has been made available to the operators, an operator may choose to
test the application as part of its acceptance program. The operator test is conducted solely
at the discretion of the operator. The operator typically interacts directly with the developer at
this point to identify and resolve any issues. QUALCOMM encourages operators to disclose
all operator-specific application requirements and test cases to developers.

If the application fails this last stage of the testing process, the following scenarios may
occur:

The application fails for a specific operator but is available to other operators.

If the application developer is required to modify the application in response to

operator testing, the application is required to repeat the TBT application submittal
process.

If QUALCOMM determines that the reason for failure may be applicable to other operators,
the TRUE BREW status may be revoked for the application.

14
QUALCOMM Proprietary
 5 Next Steps
After your application passes TBT, you will be notified by email and receive instructions
about what the next step should be. You may have to complete an export compliance
questionnaire and submit it to QUALCOMM before your application can be uploaded into the
BREW Distribution System (BDS). Once your application is uploaded into the BDS, you will
have to complete the distribution and pricing information found on the Developer Extranet.

A complete guide to application distribution can be found under the Documents section of
the following web page: https://brewx.qualcomm.com/developer/operations/operations.jsp

15
QUALCOMM Proprietary