A new test report has been developed by the Automation Team.

Alt text

We have different sets of tests, the ones that you will notice in this article are the UATs and the Nightlies; the UATs are all of the User Acceptance Tests and we run them when we are approaching the release date, the Nightlies are a small set of the UATs that run every night.
To manage our tests and start running them, we use a continuous integration tool called Jenkins, and to automate them, we use tools like Cucumber, which generates an excellent report with all the test scenarios that failed and passed.
Apart from having the test reports on Jenkins the QA team maintained three different spreadsheets where we kept all the test results for the User Acceptance Tests and the Nightlies. The main reason we have those spreadsheets is that we wanted a single place where we could go and see the test results for all of the different cities and types of test.
Those spreadsheets were manually updated by having someone looking at the results on Jenkins and then writing them on the spreadsheets.
Filling out all those spreadsheets every day took a lot of time that could be used more efficiently doing more relevant work.


That time is over now! We love to automate things,
so that was automated!



How the results are accurately generated

First things first, before talking about the script and the SpreadSheet, let's look at how we generate the test results.
Besides Cucumber, we also use Cuyabra, which is essentially an internal Ruby gem that allows us to run our tests against two different applications (the passenger app and the driver app) at the same time. As we are using Cuyabra, it's this gem that is responsible for deciding whether if the scenario passed or failed and not Cucumber. Also, our tests can have more than one runtime; each time a scenario fails due to a reason that is not related to the applications, we will rerun that scenario again after all the other scenarios have run. Let's say, for example, the Wi-Fi connection was lost, we rerun that scenario again because it didn't fail due to a bug in the application. Cuyabra generates the final test results, and there is one file for each runtime. This means that the first runtime results could be:

19 scenarios ; 17 passed ; 2 failed

Then, if those 2 scenarios failed due to an external reason to the app, they would rerun and the results could be:

2 scenarios ; 2 passed

In the end, the final result would be:

19 scenarios ; 19 passed.

In this situation, we would have two reports (and we could have more, depending on the number of runtimes). One possible solution could have been looking at all the reports, getting the total number of scenarios, all the passed and failed ones and then figuring it out what the final result was. Instead, we implemented another solution. We created a JSON file with the following fields

{ "date":"", "total_scenarios":0,
                   "failed_scenarios":0, "failed_scenarios_names":"" }

and we added some lines of code into the hooks file!

Alt text

Also, for each runtime we create a new workspace, so we need to move this file to that new workspace! At the end of all the runtimes, we keep this JSON file, and we also print it to the console.


The Ruby Script

The script is simple. It uses the jenkins_api_client which is a Ruby Client Library for communicating with a Jenkins CI server and manages its jobs; it also uses another Ruby Gem to read from/write to spreadsheets in Google Drive, google_drive. It gets the results from Jenkins and puts them into the correct worksheet on the Google SpreadSheet.

Alt text

Step 1 - Connection to Jenkins

In order to access the jobs in Jenkins we have to build a connection, we just need to call the method that creates a new connection given parameters like Jenkins URL, the user e-mail and password.


Step 2 - Construct a Google Drive Session

session = GoogleDrive.saved_session('google_config.json')
@file = session.file_by_title('Hailo ATs Report')

The JSON file contains a client id and a token. If you want to know more about Google Drive Authorization you can go to this link. The second line is to get the spreadsheet that we want, in this case, we just want the Reports file.


Step 3 - Getting the Results

For each job, before trying to get the test results we check for the build status. If it is running we just skip that iteration; there won’t be results if it is running, right? If it isn't, we will get the test results from the current build, and from any previous builds that we may have missed for some reason.
It is just a simple for loop which will go through all the jobs that we want to get the results.
The jobs stored in a JSON file are loaded into a ruby hash which besides having the name of the job, and some other attributes, has the last build that the script got the results from. The last build attribute is quite important so we don’t lose any results and also don’t get any duplicated ones.


Step 4 - Write the results on the worksheet

In each iteration of the for loop mentioned above, the method fill_worksheet is called. It is responsible for writing the results into the proper worksheet; it starts by finding the particular sheet where that job results should be written, then it automatically fills in the new results into the next available row.


The Google Spreadsheet

Alt text

The first worksheet is a panel containing a monthly chart for each combination of job, platform and actor (passenger or driver). For each one of those charts, we can select the city that we want to see the results.
There is one worksheet per each combination mentioned above. It is where the script puts the test results. In those worksheets, we have some more information about the builds and we have some filters which are quite helpful to filter the results based on dates and cities for example.
Those charts in the Dashboard update automatically each time a new result is added! In the right side of the Dashboard, there is some 'stuff', as you can see in the animation above.
That's where the magic happens again!
We have all the possible cities to select there, as well as the start and end of the current month (also automatically generated). There are some queries running which get the results from the worksheets filtering them by the current month date range and the city selected.
Then those queries results feed into the right chart!



What's next?

  • Improve the presentation layer (The Google SpreadSheet)
    Add the other kind of tests that we have to the reports, add more charts to analyse the results.
    Add more information about each test result, i.e., the scenarios that failed.
    Eventually, move from GoogleSpreadSheets to a simple WebApp.

  • Unify the way we generate the results.
    At the moment, as you can see we have the results being generated by Cuyabra / Cucumber and also in the hooks file.
    One objective is to find a way to generate the results in one single place.

  • Instead of having the script running 'isolated' every 10 minutes, add a post job in Jenkins to upload the results as soon as the test has finished.

  • Do you want to know more information about the implementation? We can be reached at automation@hailocab.com.

  • Any ideas? Share them with us! Email us or leave a comment!