How to Leverage Automation Scripts for Performance Testing

Early performance testing is one of the key aspects in the agile testing and using existing automation scripts for performance testing is one of the ways which early performance testing could be achieved with minimum effort. The whole idea of this endeavor is to run the automation scripts in parallel so that you can simulate a load towards your server.

This will not be a one to one replacement for performance testing, but doing this early in the life cycle helps the DevOps teams to uncover the potential performance issues early before the release or code under change is given to the performance engineer to be tested. Performance engineers have become a scarce resource and their bandwidth has become a real problem in many organizations. On the other hand automation scripts are ready before it undergoes the performance testing so this whole approach can be adopted quite easily. Automation frameworks have become well established now thus automation scripts typically becomes available well ahead now than they used to be in sometime back.

The advantages of this approach are;

1. Identify the performance issues early in the life cycle. This could with regards to response times, concurrency issues, database issues, server resource health issues or client side rendering issues, etc
2. Performance test scenarios usually covers only the highly used workflows. Running automation scripts as a load test would find hidden performance issues in low usage scenarios
3. Typically performance test covers positive scenarios, where as there could be hidden performance issues in negative scenarios. Since automation scripts usually covers the negative test cases this will help us to identify performance issues in the the negative functional workflows.
4. Sometime this could help us to uncover function issues under a low load. For example session handling issues, race condition, threads synchronization issues, etc

When it comes to leverage selenium automation scripts for perf testing there are few ways or variations where you can adopt;

1. Using jmeter with Junit plugin
2. Using TestNG with modified TestNG.xml
3. Using TestNG with DataProvider
4. Running the scripts in Headless mode using either HTMLUnitDriver
5. Using Selenium Grid to run the existing scripts in multiple agents

Now we will evaluate the each option in detail;

Using jmeter with Junit plugin

You can save your existing selenium scripts which has used junit as the test runner into a jar file. (In Eclipse -> Right click on ‘Test Class’ –> Export –> Java –> JAR File –> Finish )

Then copy that particular jar file to lib\junit folder in the junit installation.

Create a Thread Group and then add ‘JUnit Request’ Sampler to Thread Group.

Now you can select your test class and corresponding test method which you would like to expose to a load. When specify the number of threads in the Jmeter Thread Group, this particular method will be executed in parallel with the given level of concurrency.

However the major problem that we have with this approach is that you do not have a way to pass arguments into the test, so that the test will be executed with the same data set which you have specified in the test. Even if you have jmeter plugin for TestNG, you will be obstructed by the same problem of not being able to use different set of test data in the load test. Hence this cannot be taken as a realistic load test even though it created a load simply because when you use same data over and over again, the subsequent transactions will load the data from the cache.

Using TestNG with modified TestNG.xml

This was basically done by adjusting the testng.xml file in a way such it has multiple tests with different set of data as Parameters. Then you can make the testsuite run as parallel. Simply you can duplicate a Test and change the parameter values, so that each test will be exeucted with different data. This will resolve the problem which we discussed in the above approach.

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Suite"parallel="tests">

<test name="Test">

<parameter name="serachtext" value="cricket"/>

<test name="Test2">

<parameter name="serachtext" value=" football" />

And in the selenium test we will have to use @Parameters annotation to pass the input data into the test

@Parameters("serachtext")

public void testGoogleSearch(String searchtext) throws InterruptedException{

driver.findElement(By.id("gbqfq")).sendKeys(searchtext);

Using TestNG with DataProvider

This is by far the best approach which we could came up with. This will execute the test methods defined in the selenium scripts in parallel mode, so that it will send the http requests to the web server concurrently creating a load like it does in a typical load testing tool like HP LoadRunner or Jmeter.

a) Create a property file (data.properties) with the necessary data (data.properties) for all the users with the variables. This should have all the test data needed to run the test for multiple users and each variable should have the postfix with incremented number (e.g : pwd0, pwd1, etc).

ep0=corin

userName0=perfteststudd1

password0=perfteststudd1

text0=text

ep1=corin

userName1=perfteststudd2

password1=perfteststudd2

text1=text

ep2=corin

userName2=perfteststudd3

password2=perfteststudd3

text2=text

b) Create a 2D array in the selenium automation script to get all the variables/parameters in to it from the property file.

public Object[][] createDataArray() throws FileNotFoundException, IOException{

              Properties prop = new Properties();

              prop.load(new FileInputStream("data.properties"));

              int dataSize = prop.size();

              System.out.println(dataSize);

              Object[][] obj = new Object[dataSize/4][4];

              for(int i=0;i<(dataSize/4);i++){

                     obj[i][0] = prop.getProperty("ep"+i);

                     obj[i][1] = prop.getProperty("userName"+i);

                     obj[i][2] = prop.getProperty("password"+i);

                     obj[i][3] = prop.getProperty("text"+i);

              }

              return obj;

       }

c) Define a testNG DataProvider and Pass that 2D array

@DataProvider(name="ep",parallel=true)

       public Object[][] epData() throws FileNotFoundException, IOException{

              return createDataArray();

       }

d) Make it as Parallel=True

@DataProvider(name="ep",parallel=true)

e) Associate the dataProvider with the test method which you need to run concurrently using @Test testNG annotation

@Test(dataProvider = "ep",threadPoolSize=10)

       public void verifyEP(String ep,String userName,String password,String text) throws Exception{

              try{

              driver = new FirefoxDriver();

              driver.get("http://"+ep+".ecollege-labs.com");

              System.out.println("Login to application");

              login(userName, password);

              System.out.println("Logged successfully");

              Thread.sleep(4000);

              }

              catch (Exception e) {

                     throw new Exception("Issue in verifying EP "+e.getLocalizedMessage());                  

              }     

       }

       public void login(String userName,String password){

              fieldUserName = driver.findElement(By.name("Username"));

              fieldPassword = driver.findElement(By.name("Password"));

              btnLogin = driver.findElement(By.xpath("//input[@value='Go to Class']"));

              fieldUserName.sendKeys(userName);

              fieldPassword.sendKeys(password);

              btnLogin.click();

       }

This will open multiple Firefox browser sessions and execute the same test concurrently.

You can use the same approach for the API level tests as well (Ziggy Framework);

    @Test(dataProvider = "ep",threadPoolSize=200)

    public void LoadTest(String ep,String userName,String password,String text)throws Exception

    {

       long sum=0;

       long avg=0;

       int i;

       for(i=1;i<=5;i++)

       {

           long startTime = System.currentTimeMillis();

              verifyInstruactorsTest(ep,userName,password,text);

                     long endTime = System.currentTimeMillis();

              long duration=endTime-startTime;

              Reporter.log("Iteration:"+i+",User:"+userName+",Response time:"+duration);

              sum=sum+duration;

             

       }

       avg=sum/i;

       allSum.add(avg);

       Reporter.log("User:"+userName+",Response time:"+avg);

    }

    @AfterClass//(dependsOnMethods="LoadTest")

    public void GetOverallAverage()

    {

       long overallsum = 0,overallavg;

        for(int j=0; j < allSum.size(); j++){

              overallsum = overallsum + allSum.get(j);

         }

     

       overallavg=overallsum/allSum.size();

       System.out.println("Average Response time:"+overallavg);

       Reporter.log("Average Response time is:"+overallavg);

    }

      

      

       public void verifyInstruactorsTest(String ep,String userName,String password,String text) throws Exception

       {

              String strReqAccessToken = ContextBuilder.getAccessToken(

                           ep,userName,

                           password);

              String strPath = Utils.constructReqPath("2026908",

                           CourseListContext.INSTRUCTORS_DETAILS);

              CourseListContext objCourseListContext = CourseListContextBuilder

                           .getParams(strReqAccessToken, strPath,

                                         HttpURLConnection.HTTP_OK,

                                         CourseListContext.INSTRUCTORS_DETAILS,

                                         ZiggyConstant.POSITIVE);

              CourseListExecutionDrivers.getInstance().execute(objCourseListContext);

             

       }

             

}     

The above piece of code calculate the average response time as well. What it does is that it starts a timer before the test method is executed and it ends the timer once the method is successfully completed. Then it calculates the time difference which equivalent to transaction response time. In order to calculate the average transaction response time all the response times for each method execution kept in an array and in the AfterClass method it calculates the average response time by dividing the total response time by total number of iterations.

Running the scripts in Headless mode using either HTMLUnitDriver

Instead of open multiple browser sessions you can run the test in headless mode by using HTMLUnitDriver as oppose to using WebDriver or rather FireFox driver.

driver = new HtmlUnitDriver();

However HTMLUnitDriver has its own limitations when it comes to identifying UI elements, hence using this will be a challenge specially for Automation frameworks like eSelenium.

Using Selenium Grid to run the existing scripts in multiple agents

One of the limitation of running the test in UI mode is that it opens multiple browser sessions, so that the client machine eats up resources and it does not allow to run more and more threads. In order to meet this challenge what you can probably do is you can run limited number of threads (may be 5 to 10 maximum, which means 5 to 10 browser instances) in one agent and distribute the test in to multiple agents using Selenium Grid. Since the end objective is to create a load towards your servers, this will avoid the limitation of the client being stressed by distributing the workload to multiple agents.

For any clarification on any of these approaches feel free to contact me on;

malindamapatuna@gmail.com