3.Why 90th percentile and not average?

Percentile’ – a common but most important term, especially in performance testing. From the day first
when we started analysing any performance testing report, we heard about 90th percentile. Even our
mentor gave more stress on 90th percentile figures for response time. So, what is this 90th percentile
exactly?

Let’s try to understand with an example. If you had 10 sheep and each sheep eat some KGs of grass on a
daily basis. One day you weighted the grass and noted the figures of each sheep’s intake. Refer to the
below table:

Sheep# S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

Grass(kg) 3 3.2 4 4.8 3.6 2.9 3.4 3 3.8 3.9

Now, you need to find out what amount of grass has been consumed by 90% of sheep? So simply you
need to sort the number with respect to consumed grass and ignore the last value.

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

Sheep# S6 S1 S8 S2 S7 S5 S9 S10 S3 S4

Grass(kg) 2.9 3 3 3.2 3.4 3.6 3.8 3.9 4 4.8

90th percentile value in 10 entries is a 9th value which is 4, so just ignore S4 with 4.8 (keep it hungry for
some days, it eats so much).

The conclusion is 90% of total sheep either eat 4 KGs grass or below, so you got an upper limit of grass
consumption. In terms of performance testing, you need to sort response time of a particular transaction
or request in increasing order and then ignore 10% of the total count having high values. The last highest
number in the remaining values will be 90th percentile.

Example:

A performance test script is executed for 25 iterations. The response time of the login transaction of each
iteration is:

S. No. Iteration No. Login (Response Time (in sec))

1 1 1.5

2 2 1.6

3 3 1.1

4 4 0.9

5 5 2.1

6 6 1.9
7 7 1.4

8 8 1

9 9 0.8

10 10 1.5

11 11 1.8

12 12 1.1

13 13 1.6

14 14 1.7

15 15 1.3

16 16 0.9

17 17 1

18 18 1.5

19 19 2.3

20 20 1.9

21 21 1.8

22 22 1.2

23 23 1.4

24 24 0.9

25 25 1.5

Now, sort the list in increasing order with respect to response time.

S. No. Iteration No. Login (Response Time)

1 9 0.8

2 4 0.9

3 16 0.9

4 24 0.9

5 8 1
6 17 1

7 3 1.1

8 12 1.1

9 22 1.2

10 15 1.3

11 7 1.4

12 23 1.4

13 1 1.5

14 10 1.5

15 18 1.5

16 25 1.5

17 2 1.6

18 13 1.6

19 14 1.7

20 11 1.8

21 21 1.8

22 6 1.9

23 20 1.9

24 5 2.1

25 19 2.3

Now, 22.5 is the 90% of the number of transactions i.e. 25.

=> 25 x (90/100) = 22.5

Round-off to 23. So the 23rd value will be 90th percentile which is 1.9 seconds. It means 90% of total
iterations having response time 1.9 seconds or less than it. Similarly, you can calculate other percentile
values like 70th, 80th or 95th percentile.

How 90th percentile calculated in MS Excel?

MS Excel uses below formula to calculate 90th percentile:

90th Percentile = 0.9 * (Number of Values – 1) + 1

The same formula was used by LoadRunner version 6.5. The method to find out 90th percentile has been
changed in LoadRunner version 7 and above.

Why we need 90th percentile in Performance Testing?

Percentile is often considered as a performance goal. If the given SLA has 90th percentile NFR and it
meets during the test then it shows that 90% of the users have an experience that matches your
performance goals. It gives additional confidence to the client over his application.

Sometimes average response time appears extremely high and individual datasets seem normal. Even a
couple of peaks in response times, skew the average response time numbers and impact the test. In such
scenarios, 90th percentile (or other percentile values) eliminate the unusual spike data from the result.

In reality, most of the applications have very few high spikes in the graph; a statistician would say that the
curve has a long tail. A long-tail does not imply many slow transactions, but few that are magnitudes
slower than the norm. In that case, 90th Percentile helpful because it ignores 10% of the request having
the spike (this can be ignored).

If the 50th percentile (median) of response time is 5 seconds that means that 50% of the transactions are
either as fast or faster than 5 seconds. If the 90th percentile of the same transaction is at 8 seconds it
means that 90% are as fast or faster and only 10% are slower. The average, in this case, could either be
lower than 5 seconds or somewhere in between. A percentile gives a much better sense of real-world
performance because it shows a slice of response time curve.

If we calculate the difference of the 90th percentile value and the average response time value and divide
this difference with the average response time value then it gives an idea of the spread of different data
points. If the ratio is extremely small, it means that average and 90th percentile values are very close to
each other and will indicate good and constant performance of the application. However, if the ratio is
large, it shows high deviation in response time and non-uniform performance of the application. This is
one of the methods where 90th percentile is useful, although I would recommend to draw your conclusion
using standard deviation only.

Percentiles are a really great and easy way of understanding the real performance characteristics of your
application. They also provide a great basis for automatic base-lining, application behavioural learning
and optimizing your application with a proper focus. However, averages are ineffective because they are
too simplistic and one-dimensional. In short, percentile (90th, 95th, 99th) is great in performance testing
world!

5. Regular Expression

Regular expressions are a tool used to extract a required part of the text by using advanced
manipulations. ... In JMeter, the Regular Expression Extractor is useful for extracting information from
the response. 
8. What are there in the NFRs and how to design a test plan

What are the Non-Functional Requirements?

Non-functional requirements are the testing goals which are created especially for performance testing,
security testing, usability testing, etc. It is a combined requirement for all types of non-functional test.
Since PerfMatrix is a core performance testing site, so non-functional word will be specific to
Performance Testing only. Some simple examples of Non-Functional Requirements are:

 The number of users handled by the application

 Page Response Time
 The number of requests processed by the application per unit time
 CPU Utilization
 Memory Utilization
 Error rate etc.
In non-functional requirement, some of the goals are time-bound like the response time of a page, request
per second, resource utilization etc. whereas some of the goals are load bound like real-world user load,
throughput etc.

Difference between functional and non-functional requirement:

The main difference in functional and non-functional requirement is that functional requirements are end-
user result oriented and stressed over the correct output. For example: If a user uses a banking application
and clicks ‘Account Balance’ link then the application must display the correct balance available in his
account. On the other hand, the non-functional requirements are oriented to the performance of the system
in terms of responsiveness and load-bearing capacity. For example: If 1000 users hit ‘Account Balance’
link at the same time then the application must respond to all the users within a defined time (say 3
seconds).

Purpose of NFR Gathering:

The project team or client sets the expectation for performance testing in the form of non-functional
requirements. To collect the requirement, analyse them from performance testing perspective and finalise
the quantitative NFRs; all these steps fall under the NFR gathering phase of PTLC (Performance Test
Life Cycle). All the requirements are documented, categorized and concluded in the Non-Functional
Requirement Document. The end result of this phase provides quantitative NFRs which helps to prepare a
correct workload model during performance testing.

A Test Plan is a detailed document that describes the test strategy, objectives, schedule, estimation,
deliverables, and resources required to perform testing for a software product. Test Plan helps us
determine the effort needed to validate the quality of the application under test. The test plan serves as a
blueprint to conduct software testing activities as a defined process, which is minutely monitored and
controlled by the test manager.
9. PT lifecycle and your roles and responsibilities

What is Performance Testing Life Cycle (PTLC)?

Performance Testing Life Cycle is a systematic process to carry out the non-functional testing of a
software system or application. Mostly all the software industries follow this process to plan the
performance testing activities and find out the performance bottleneck in the software system.

What is the need for Performance Testing Life Cycle (PTLC)?

A decade ago, performance testing was not so important and it could be a reason that a small part of the
world was using the technology and the existing hardware and software were meeting their requirements.
Now, there is a change in the scenario. When a start-up starts or launches a website or a mobile
application, the questions come in their mind are whether the application can handle X amount of users or
not? whether the application is so responsive or not? Even the end users expect a quick response to their
clicks. Day-by-day company and consumer requirements knee down the developer to think about a more
tuned application which can handle multiple users load with quick response. Now, the questions are how
they can measure the performance of an application? What would be the performance metrics? Which
components should be involved in performance testing and on what basis?

To answer all these questions a process had been evolved in which performance test activities are divided
into the number of phases. The purpose was to set-up a smooth and channelized process to achieve the
performance testing goal. Each phase of this process is interdependent and has its own set of activities.
All the phases are bind to a cycle called “Performance Testing Life Cycle or PTLC”.

Advantage of Performance Testing Life Cycle:

Actually, Performance Testing Life Cycle (or PTLC) is not only made for software industries to set-up a
process for performance testing, but it also provides benefit to the client. A client or project team sets the
expectation about the application performance at the initial phase of PTLC and then compare the expected
result with the actual result at the time of delivery (in the last phase of PTLC). Performance Testing Life
Cycle helps the application development companies to get the proper requirement from the client, develop
a tuned application, boost-up the performance of the application and meet the client’s expectation.

Phases of Performance Testing Life Cycle:

PTLC defines different phases of performance testing from risk assessment to test result analysis. It
bridges the gaps between the expected result and actual result (only related to performance testing) which
helps to deliver a responsive application with better performance.

A quick summary of PTLC phases:

1. Risk Assessment
 Purpose: To check the eligibility of the components for performance testing on the basis of a risk
score.
 Accountability: Performance Test Lead/Manager
 Deliverable: Performance Testing Risk Assessment Document
 Read the full article to get more detail: Risk Assessment in Performance Testing
2. Requirement Gathering & Analysis
 Purpose: To understand the client’s expectation and conclude the requirements
 Accountability: Performance Test Lead/Manager
 Deliverable: Non-Functional Requirement Document (Performance Testing Requirement
Document)
 Read the full article to get more detail: Non-functional Requirement Gathering
3. Performance Test Planning
 Purpose: To prepare test strategy as per Non-Functional Requirement
 Accountability: Performance Test Lead/Manager
 Deliverable: Performance Test Plan Document
 Read the article to get full detail: Performance Test Planning
4. Performance Test Design (Scripting)
 Purpose: To create the test scripts as per the approved Performance Test Plan
 Accountability: Performance Test Team Members (Engineer (Create)/Analyst (Create)/Lead
(Create/Review)/Manager (Review))
 Deliverable: Performance Test Scripts
 Read the article to get full detail: Performance Test Scripting
5. Workload Modelling
 Purpose: To create the workload scenario as per the approved Performance Test Plan using test
scripts
 Accountability: Performance Test Team Members (Engineer/Analyst)
 Deliverable: Performance Test Scenarios
 Read the article to get full detail: Workload Modelling
6. Performance Test Execution & Result Analysis
 Purpose: To execute the performance tests as per timelines documented in the approved
Performance Test Plan and analyse the test results
 Accountability: Performance Test Team Members (Engineer/Analyst)
 Deliverable: Individual Test Results
 Read the article to get full detail: Performance Test Execution & Result Analysis
7. Reporting and Recommendation
 Purpose: To prepare and publish the test report and provide the recommendations
 Accountability: Performance Test Lead/Manager
 Deliverable: Final Performance Test Report
 Read the article to get full detail: Performance Test Reporting

Responsibilities
 Like coming to my roles & responsibilies I involved in almost all stages of the P.T life cycle
starting from the NFR gathering to test reporting.
10.meaning of () . * + ?

(  ):these enclose the portion of the match string to be returned

.:match any character
+: match one or more times
?:don't be greedy, i.e. stop when first match succeeds
*: match zero or more times

13) Setup and Teardown Thread Group

Setup Thread Group

setUp Thread Group is a special type of JMeter Thread Group that can be utilized to perform Pre-Test
actions. The behaviour of setUp Thread Group is as same as the normal Thread Group element. The only
difference is that setUp Thread Group executes before the test proceeds to execute normal Thread Group.

What is the use of ‘setUp Thread Group’?

setUp Thread Group is very much helpful to perform pre-test activities like:

 To create new users by doing the sign-up activity

 It is used to get a large dataset from a database into a JMeter variable that can be used during the
test
 To get and log the version number from the system under tests version number.
 To run a javascript to set JMeter properties based on more simple input parameters/properties.
Let’s say you want to configure the selection of target host a simple true/false value, but in your
test, you need to expand it to different strings, and you do not want to have logic spread out all
over your test plan.5. To apply some pre-test condition based on that Thread Group (Test Case)
will be selected.
Teardown Thread Group

tearDown Thread Group is a special type of JMeter Thread Group that can be utilized to perform Post-
Test actions. The behaviour of tearDown Thread Group is as same as normal Thread Group element. The
only difference is that tearDown Thread Group executes after the completion of normal Thread Group
execution.

What is the use of ‘tearDown Thread Group’?

tearDown Thread Group is very much helpful to perform post-test activities like:

 To delete test users after completion of the test

 To delete the bulk of data/order/pending request after completion of the test
 It is used to clear/delete dataset from a database for next run

19) Gaussian and Poisson timer

As the name suggested, JMeter Gaussian Random Timer is used to generate and add the random delay
before the execution of a sampler. This timer is based on the Normal or Gaussian Distribution Function.
The delay (think) time is the sum of the Gaussian distributed value (with mean 0.0 and standard deviation
1.0) times the deviation value you specify and the offset value.
How to add ‘Gaussian Random Timer’?
Follow the below steps to add the Gaussian Random Timer in the script:

1. Select any appropriate element from the below list where you want to add the timer
a. Test Plan
b. Thread Group
c. Controller
d. Sampler
Right-click on the element
Hover the mouse on ‘Add’
Hover the mouse on ‘Timers’
Click ‘Gaussian Random Timer’

What are the input fields ‘Gaussian Random Timer’?

‘Gaussian Random Timer’ has following input fields:

1. Name: To provide the name of the timer

2. Comments: To provide arbitrary comments (if any)
3. Deviation (in milliseconds): The number shows how much the delay can deviate from the given
offset towards the higher and lower range. 
4. Constant Delay Offset (in milliseconds): The number shows a constant delay which will be
added in random number generated by Gaussian Function in the range of given deviation value.

Figure 01: Gaussian

Random Timer

Example:
If Constant Delay Offset is 1000 and deviation is 200 then Gaussian Random Timer will generate the
random number in the range of 800 to 1200 milliseconds:

Gaussian Random Timer Delay = 1000 + GF*200  

where GF value could be from -1 to 1

As the name suggested, JMeter Poisson Random Timer is used to generate and add the random delay
before the execution of a sampler. This timer is based on the Poisson Distribution Function. The delay
(think) time is the sum of the Poisson distributed value multiplied by defined lambda value and the offset
value.

How to add ‘Poisson Random Timer’?

Follow the below steps:

1. Select the appropriate element where you want to add the timer
a. Test Plan
 b. Thread Group
c. Controller
d. Sampler
Right-click on the element
Hover the mouse on ‘Add’
Hover the mouse on ‘Timers’
Click ‘Poisson Random Timer’

What are the inputs for ‘Poisson Random Timer’?

‘Poisson Random Timer’ has the following input fields:

1. Name: To provide the name of the timer

2. Comments: To provide arbitrary comments (if any)
3. Lambda (in milliseconds): The number shows how much the delay can deviate from the given
offset towards the higher and lower range. 
4. Constant Delay Offset (in milliseconds): The number shows a constant delay which will be
added to the random number generated by Poisson Function in the range of given deviation value.

Example:
If Constant Delay Offset is given 1000 and Lambda is given 200 then Poisson Random Timer will
generate the random number in the range of 800 to 1200 milliseconds:

Poisson Random Timer Delay = 1000 + PF*200   

where PF value could be from -1 to 1

Figure 01

Additional Information:
 The Poisson Random timer is processed before each sampler in the scope in which it is found; if
there are several timers in the same scope, all the timers will be processed before each sampler.
 Timers are only processed in conjunction with a sampler. A timer which is not in the same scope
as a sampler will not be processed at all.
 To apply the timer to a single sampler, add the Poisson Random Timer as a child element of the
sampler.
 To apply the timer after a sampler, either add the Poisson Random Timer as a child element of the
next sampler or as a child element of Flow Control Action.
20) Synchronisation timer

In JMeter, the purpose of the Synchronizing Timer is to hold the threads until X number of threads have
arrived, and then they are all released at once.  It adds delays between requests such that all (defined)
threads fire at the same time thus creating heavy load bursts on the application. A Synchronizing Timer
can thus create large instant loads at various points of the test plan. This timer is very useful to make users
(threads) to do the simultaneous task (simultaneous users).

Figure 01: Synchronizing Timer

How to add ‘Synchronizing Timer’?

Follow the below steps to add Synchronizing Timer:

1. Select the appropriate element where you want to add the timer
a. Test Plan
b. Thread Group
c. Controller
d. Sampler
Right-click on the element
Hover the mouse over ‘Add’
Hover the mouse over ‘Timers’
Click ‘Synchronizing Timer’

What are the input fields of ‘Synchronizing Timer’?

‘Synchronizing Timer’ has the following input fields:

1. Name: To provide the name of the timer

2. Comments: To provide arbitrary comments (if any)
3. Number of Simulated Users to Group by: This attribute is used to define the number of threads
to release at a time. If 0 is given then JMeter will release the same number of users which is
defined in the thread group. 
4. Timeout in milliseconds: This value helps to control the timeout of the requests during the test. If
0 (zero) is set then the timer will wait for all the threads to reach the value which is defined
in “Number of Simultaneous Users to Group”. If superior to 0 (zero), then the timer will wait till
the number given in “Timeout in milliseconds” irrespective of thread count. If wait time exceeds
the given timeout value then the users are released without meeting “Number of Simultaneous
Users to Group” criteria. The default value is 0 (zero).
Example

Figure 02

21) What are controllers and its types and uses?

Logic Controllers in Jmeter

To understand the use of different logic controllers in JMeter. Let take an example in which we have
following configurations 1 Thread Group, 3 Samplers in Http Request form and 2 Listeners in table and
tree form.

Now we implement following controllers on this above configuration & check what results we will get.
Simple Controller

Simple Controller doesn’t provide any functionality its just a container that contains user request.
In this example we set Thread Properties as number of threads 1, ramp-up period 1, loop count 5 and 3 http
request. So, a total request hit as 15.
Loop Controller
Loop Controller provides functionality to run user request to specific number of times. Add loop controller
to thread group by using below steps.
Right click on Thread Group >> Add >> Logic Controller >> Loop Controller
In this example we set Thread Properties as number of threads 1, ramp-up period 1, loop count 5 and 3 http
request. So, a total request hit as 30.
Random Controller
Random Controller provides functionality to run user request in random order for each loop i.e. One
random user request in each loop. Add random controller to thread group by using below steps.
Right click on Thread Group >> Add >> Logic Controller >> Random Controller
In this example we set Thread Properties as number of threads 1, ramp-up period 1, loop count 5 and 3 http
request. So, a total request hit as 5.
Random Order Controller
Random Order Controller provides functionality to run all user request in random order for each loop. Add
random order controller to thread group by using below steps.

Right click on Thread Group >> Add >> Logic Controller >> Random Order Controller

In this example we set Thread Properties as number of threads 1, ramp-up period 1, loop count 5 and 3 http
request. So, a total request hit as 15.
Interleave Controller

Interleave Controller provides functionality to run all user request in each loop and request in each loop
will be in sequential order. Add interleave controller to thread group by using below steps.

Right click on Thread Group >> Add >> Logic Controller >> Interleave Controller

In this example we set Thread Properties as number of threads 1, ramp-up period 1, loop count 5 and 3 http
request. So, a total request hit as 5.
Once Only Controller
Once Only Controller provides functionality to run any user request to run only once per loop. Add once
only controller to thread group by using below steps.
Right click on Thread Group >> Add >> Logic Controller >> Once Only Controller

In this example we set Thread Properties as number of threads 1, ramp-up period 1, loop count 5 and 3 http
request on which we applied once only controller in one request. So, a total request hit as 11.
Module Controller
Module Controller provides functionality to run which module needs to run. Add module controller to
thread group by using below steps.
Right click on Thread Group >> Add >> Logic Controller >> Module Controller

In this example we set Thread Properties as number of threads 1, ramp-up period 1, loop count 5 and 2
controllers that contains http requests. Now add module controller & click on dropdown link. You will find
created modules.

Now you are ready and set to implement the controllers. If you have any questions regarding the same or
anything else to add-on please let me know in the comment section below.

31) What is a template and Match no in a Regular Expression Extractor?

Template - choose the group you would like to extract from the regular expression
 $1$ to refers to group 1, $2$ to refers to group 2, etc. $0$ refers to whatever the entire expression
matches.
Match no means that the regular expression matched at least some part of the target
34) Beanshell related questions

How to Use BeanShell in JMeter

Beanshell is one of the most advanced JMeter built-in components. It supports Java syntax and extends it
with scripting features like loose types, commands, and method closures. If your test case is uncommon
and implementation via embedded JMeter components becomes tricky or even impossible, BeanShell can
be an excellent option to achieve your objectives.

BeanShell entities in JMeter have access to both internal JMeter APIs and any external classes that are
loaded into the JMeter classpath (be sure to drop necessary jars into /lib/ext folder of your JMeter
installation and place all necessary “import” statements at the top of your BeanShell scripts).

All JMeter versions provide the following BeanShell-enabled components:

 Beanshell Sampler – A standalone sampler

 Beanshell PreProcessor – A pre-processor to another sampler that is executed before the sampler and
can be used for prerequisite setup (i.e. to generate some input). 
 Beanshell PostProcessor –  A post-processor that is executed after the sampler and can be used for
recovery or clean-up.
 Beanshell Assertion – An advanced assertion with full access to JMeter API. Java conditional logic can
be used to set the assertion result.
 __Beanshell Function – A JMeter Function that allows execution of custom BeanShell code during a
sampler run.

36) What all values in Jmeter need correlated and what all we need to associate explicitly.

What is Correlation?
Correlation is the process of extracting some value from the response of one step into the request of
another step. It captures and stores the dynamic response from the server and passes it on to subsequent
requests. So, What is a Dynamic Response?

 
A response is considered dynamic when it returns different data for each iterating request, occasionally
affecting successive requests. Correlation is a crucial process during performance load test scripting
because if we don’t handle it carefully, the script will become useless.

Why do we Need Correlation?

Correlation is one of the most important aspects of scripting. It fetches dynamic data from preceding
requests and posts it to the subsequent requests. Let’s take an example to find out why exactly we need
correlation.

39) What does a test report contain?

The Report Summary contains the ollowing information:

 Test start time and end time,

 APDEX scores for every single request and container,
 A pie chart named Requests Summary which gives the proportion of Successful / Failed samples.