Running webpagetest from CLI

/ jankka / Leave a comment

Sometimes you have a need to run web page load speed test without an UI. And of course, as per usual, there’s a node module for that.

I’ve been checking out https://www.webpagetest.org/ and they do provide a web interface, and an api, as well https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis.

Besides those two sources, there’s also a node module for running the tests:

https://github.com/marcelduran/webpagetest-api

Here’s how to get it working (running on macOS at the moment).

  1. git clone git@github.com:marcelduran/webpagetest-api.git
  2. cd webpagetest-api
  3. docker build -t webpagetest-api .“
  4. Get the API Key from webpagetest.org/getkey.php
  5. Run the test: docker run -it –rm webpagetest-api -k YOURAPIKEY test https://twitter.com/marcelduran
{
  "statusCode": 200,
  "statusText": "Ok",
  "data": {
    "testId": "190206_19_f4ed62b215aa61ed44085cdf0cac1779",
    "ownerKey": "c65d1fd5da5233aada1c4b1064bf39d05b4a81c4",
    "jsonUrl": "https://www.webpagetest.org/jsonResult.php?test=190206_19_f4ed62b215aa61ed44085cdf0cac1779",
    "xmlUrl": "https://www.webpagetest.org/xmlResult/190206_19_f4ed62b215aa61ed44085cdf0cac1779/",
    "userUrl": "https://www.webpagetest.org/result/190206_19_f4ed62b215aa61ed44085cdf0cac1779/",
    "summaryCSV": "https://www.webpagetest.org/result/190206_19_f4ed62b215aa61ed44085cdf0cac1779/page_data.csv",
    "detailCSV": "https://www.webpagetest.org/result/190206_19_f4ed62b215aa61ed44085cdf0cac1779/requests.csv"
  }
Advertisement

Report Portal Installation

/ jankka / 3 Comments

Introduction

We do have a need to analyse the stability and usability of our tests, too. We do monitor our systems from here and there, check the code coverage of /development code/. Funny when you think about it, but test automation is part of development process and should be addressed as such. Even more if you’re developer who runs the code.
I have been running all sort of analytics based on test executions in Jenkins, written parsers in python and building dashboards on ELK -stack. And boy it has been fun 😀
Then I heard about this tool, Report Portal (http://reportportal.io) from a podcast from Joe Colantonio and decided to take a look. I am nowhere near to make any conclusions, but after a brief usage it does look promising.

Running locally

Report Portal has a demo environment you can try with your tests, but it really comes in handy if you host it yourself. And they provide instructions to that in http://reportportal.io/download. I have followed the instructions and listing them down in here. To be honest, I do think they do brilliant job already and my notes here are more for myself. To recall the process in a better way and sure, and to get some clicks on my blog, too 😀
First of all, you’ll need to have `docker`installed. After that everything should be easy.
NOTE:  In MacOs (at least) you should allocate at least 5 GB memory to the docker process. I had only 2 GB and everything was looking good from the process side, except that I could not log in. I do recall running the same in Linux without changing anything and it just worked.
  •  mkdir ~/reportportal
  • cd ~/reportportal
  • Download the docker-compose -file:wget https://github.com/reportportal/reportportal/raw/master/docker-compose.yml
  • mkdir -p data/elasticsearch 
  • chmod g+rwx data/elasticsearch
  • sudo chgrp 1000 data/elasticsearch (here’s a difference. I had to run the chgrp in MacOS as sudo)
  • docker-compose -p reportportal up -d --force-recreate
When the compose is ready, you should be able to browse to the Report Portal UI: http://localhost:8080/ui/
Log in using default/1q2w3e.

Robot Framework integration

In order to get the Robot Framework results to report portal, you need to use a separate library.  GitHub – reportportal/agent-Python-RobotFramework. I will create a new blog post on this and add more information there.

jmeter/Blazemeter integration

Basically all we have to do is to generate a junit.xml -report from the test run, zip it and then send it to the report portal API. I will create a new blog post on this and add more information there.

jest-puppeteer integration

GitHub – reportportal/agent-js-jest: ReportPortal agent for the Javascript Jest unit test framework. I will create a new blog post on this and add more information there.

Day 4 (of 30 Days of Automation in Testing)

/ jankka / Leave a comment

Day 4: What kind of testing can automation support you with?

Unit testing comes to my mind as the first one.It helps me as a tester in a way that I let developers do the deeds and learn to trust they do things right. Even reviewing the unit test code is actually a neat learning experience, once you get to understand the unit test logic.

Functional testing during or right after the build, can also be done by same unit testing approaches, which is also a neat way to verify the functionality. The benefits on that one is that the code is readable by the developer who wrote it and (in case it’s not made with Perl) even with other members of the team or company. To have the tests usable and readable in the same IDE developers use, leads also to easier debugging. What usually lacks is decent reporting, but then again, all we need to know that it fails when it should and passes when everything really is OK.

Mocking the test environment and harnessing the SUT. They walk hand in hand. There’s plenty of nice mocking tools, wiremock for example. One might also re-invent the wheel here. I mean the mock can be just a file you fetch from another system or write yourself. Does not sound too much automating, though 😀

Load testing is also one of the standard ones to be used with automation. Given the powers of the servers, new container technologies and distributed load testing services, you can build your own solution or use a ready made service, like blazemeter or octoperf.

Repetitive functionality tests, regression tests and release tests in any step of the development process are good to automate, given that you are going to use the automation more than few times.

Then the whole world of Continuous Delivery and Continuous Integration. It will not happen properly if you do not have the right tests put in right places. Well it might be OK to let the users test the product, but that would mean that you should have an automated monitoring system that gathers the data you will need. That’s also one way of automation that will help.

And of course all of the test data generating tools, self-made or otherwise, bash- or python -scripts for setting up the environments together with containers and their managing tools (docker, docker-compose and kubernetes to start with).

I most likely am missing something here, still. I mean where do you draw the line when it comes to automation in testing? Is it only the testing tools (like jMeter, Robot Framework etc.) or the tools that let you yourself automate mundane tasks to ease your testing process.

 

 

30 Days of Automation in Testing

/ jankka / Leave a comment

Once again it is summer, once again Ministry of Testing comes up with interesting 30 days challenge.
This year I spent my vacation already in May, so that excuse for not doing this won’t do; I’ll have to tackle this one.
The challenge is about test automation, sponsored by SmartBear (I do not like their products, but what can you do except not to use them 😉 ).

Day 1: Look up some definitions for ‘Automation’, compare them against definitions for ‘Test Automation’

Whereas automation definition in Wikipedia (the source of the sources) is actually handling a wide range of different mechanical and programmatical approaches, Test Automation is considered only to be software related and listed as part of software development. I do consider that one could test things mechanically and control the checkers, tools and results by machine, at least when it comes to simple checks. Wherever you’ll need human interpretation, it is actually easier to get a fellow human to make a judgement. Machines are good to make repetitive tasks easy, but they are also well capable to repeat an erroneus step several times automatically, unless someone controls the situation.
But anyhow, the biggest difference between automation and test automation according to wikipedia is the narrowing of test automation on something that’s done with software.

Day 2: Begin reading an automation related book and share something you’ve learnt by day 30.

Agile Automation and Unified Functional Testing by Rajeev Gupta

I’m also trying to make it through this course:
REST API Automation testing from scratch-(REST Assured java)

Day 3: Explore the automation thread on The Club and contribute to the conversation

Added my contribution to https://club.ministryoftesting.com/t/when-to-start-automation/11743/10

JMeter and HiDPI -displays

/ jankka / Leave a comment

It does bug the hell out of me that in Windows 10 jMeter 4 seems to be working just nicely, regardless on the display -resolution. That’s just plain wrong. I mean, Linux and MacOs are superior and advanced operating systems, or?
Anyhow, when I run jMeter 4 (or the earlier ones, too) in Linux – haven’t touched that in MacOs in ages, but I assume the situation has not changed a bit – the display and the fonts are so tiny that it makes the back of my head hurt from the tension. It is no good.

Apparently, there’s a bug (or 2) in jMeter:
https://bz.apache.org/bugzilla/show_bug.cgi?id=58426
https://bz.apache.org/bugzilla/show_bug.cgi?id=59168

Luckily there is stackoverflow. I managed to find the cure from there and from jMeter manual:
https://jmeter.apache.org/usermanual/hints_and_tips.html#hidpi

How to do it?

  1. Open the apache-jmeter-4.0/bin/user.properties
  2. Add the following lines in the end of the file
jmeter.hidpi.mode=true
 jmeter.hidpi.scale.factor=2.0
 jmeter.toolbar.icons.size=48x48
 jmeter.tree.icons.size=32x32
 jsyntaxtextarea.font.family=Hack
 jsyntaxtextarea.font.size=28

Voilá. You’re done. All you need to do now is to start jmeter.

Sourcetree Update in Windows 10

/ jankka / Leave a comment
  1.  Update has hung for few months (i usually work in Linux anyhow)
  2. I choose to accept
  3. New update for version 1.9.0 is downloaded and installed, not more glitches than normal Windows installer brings
  4. After update I get a pop-up that this version will not be updated/maintained anymore and I can choose to ignore or get to the downloads page
  5. I get to downloads, I’m curious
  6. a new version 2.3.5.0 is downloaded, so I install it
  7. Installation is smooth, and I am bugged about the earlier version once again when I start the app.

Now, really? I have to update to a non-supported version first to get the latest, separate, completely new version? How smooth is this?

 

There has been Robots (TM)

/ jankka / Leave a comment

 

We did have the (I suppose it was) first Robot Framework -related MeetUp in Sweden yesterday evening. It was held at the premises of Fareoffice Car Rental Solutions Ab in Kungsholmen.

There was not plenty of participants, but there was enough. The company and the office are small, so it was actually a good thing to have a ‘lagom’ crowd. Which in this case was 9 from outside the company, me, Pekka Klärck and 4 from Fareoffice.

First Pekka gave an introduction and background talk about Robot Framework. It was actually good to watch (for me), for I learned a few new things, once again. Even though he had held almost the same speech during the day before, while having a Robot Framework Workshop/training to Fareoffice, there still was few new issues to cover. Besides that we were talking about general usage of plug-ins on IDEs, RoboCon and running the tests with different setups.

The second was my trial of fire. I had a presentation about Robots in Containers. It went surprisingly smoothly. I had few technical glitches, but I knew they were there so the ghost of a demo-god did not ruin my presentation.

Of course there were few things I’d do differently. First of all, it was too quick. Secondly, I could’ve concentrated more on practical execution of the steps I was describing with  pictures; a live work is always better than preserved slide. Even if the deck of slides are be done with Prezi.

So it goes. I was tense, nervous and did all the typical flaws a Finn can do when representing and reflecting my MeetUp arrangements and presentation; I picked up all the mistakes I thought I had made. That is pretty much a built-in feature for us grown up in (the 80’s at least) Finland. Luckily, you can always trust a Swede to be there and comfort you. Empathy in Sweden is a strong and positive thing. Thank you for all participants for being there and for the support.

The best part of the MeetUp was the people and the discussions we had. It was great to see that there were others using Robot Framework here in Sweden.

I was also asked about running Selenium tests on IE/Edge, and we ended up showing the tests the way we do it; running them from Jenkins and in Browserstack. But that is not running on premise. Which meant I could not give a straight answer, which bugged me a bit, as usually. It started to feel kind of a challenge and I might want to pick it up on next creative Friday (once a month tradition at Fareoffice). So to say, spin up a Windows server and install Zalenium in it. Working with Windows would be a worthy challenge for me, an avid Linux -user as I am, and could serve as a good reminder on the fact that even the operating systems should be seen as tools. And every tool has its purpose.

In the end, we decided to create a MeetUp group and have the next meeting at Eficode’s premises in Stockholm. There was also few ideas about where to host RoboCon in 2019. All in all I am really happy I decided to push this one through, believe me, I had my doubts beforehand 😀

 

Logstash filter for Robot Framework

/ jankka / Leave a comment

We are currently working on CI/CD -setup at work. As part of that, the tests need to be able to be implemented as a part of the pipeline.
Generally, the pipeline consists of steps/stages done with jenkins pipeline. The benefit on this is that the whole process and definition of the stages (Deploy, test etc) will be done by the developer team and stored in the teams own repository and is therefore controlled by the team also. Which is definitely a great step towards for the teams having more freedom and more responsibility when it comes to deliver the applications/solutions to the production. Needless to say it will also affect to the visibility of the quality and to the need of tests.

Plus that it will definitely keep the test team on their toes. Keeping ahead becomes a really neat challenge 😀

Now that does add more requirements also on the testing tools. First of all, the tools we use should be able to be used from containers. Which means that everything is dockerized. Well, the test code itself is in the repository, but the engines running the tests are in the containers.
We use, whenever we can, a general docker images from dockerhub.
Sometimes it won’t work like that. So we end up re-inventing the wheel.

That was the case with logstash. We will need to be able to filter the Robot Framework’s output.xml and send it to elasticsearch. There was two possibilities to do that; logstash filtering or xml parsing. The xml-parsing remains to be done still (I am going to do it), but I did manage to create the logstash -filter. It is not completely flawless, not even the most elegant, but at the moment it seems to be working as it should. To be honest, I was aiming to have a one more blunt instrument for our test needs.

The filter:

robot-results.conf

input {
 file {
 path => [ "/output.xml"]
 }
}

filter {
 xml
 {
 source => "message"
 store_xml => true
 target => "doc"
 xpath =>
 [

"msg", "doc.msg",
 "arguments", "doc.args",
 "kw", "doc.keyword",
 "status", "doc.status",
 "status/@status", "doc.test.status",
 "robot", "doc.robot",
 "errors", "doc.errors",
 "statistics", "doc.statistics",
 "suite", "doc.suite",
 "tag", "doc.tag",
 "total", "doc.total",
 "/kw", "leftovers",
 "/arguments", "leftovers"

]



}
}



output {
 elasticsearch {
 hosts => ["elastic"]
 index => "logstash-%{+YYYY.MM.dd}"
 }
}

Dockerfile:

FROM logstash

ADD robot-results.conf /etc/logstash/conf.d/robot/results.conf
CMD logstash -f /etc/logstash/conf.d/robot/

Running the container:

docker run --add-host=elastic:127.0.0.1 janmat/logstash-robot

 

Install Secure Lab Tools in Fedora 26

/ jankka / 1 Comment

I am working my way to dive in to the world of security testing. We have been going through the tasks on 30 days of security testing from Ministry Of Testing Dojo. The themed tasks is actually a really good way to keep up in learning new topics and deepening your knowledge on different issues at hand. Ministry Of Testing has a nice series of themed months on the catalogue and I warmly recommend to check them out.

We have been doing the themed months a bit differently. First of all, we accept the fact that there is weekends and people do not have to live, breathe and urinate testing. Even though it does help from time to time. So, our approach has been mainly to do 30 days of testing during the weekdays. Which means that instead of 4 weeks, we’ll accomplish it in approximately 6 weeks.

Anyhow, one of the things beside the security testing challenge has been us having a course on Ethical Hacking. The course is available in Udemy and it is reasonably priced, so I recommend that, at least if you’re not familiar with penetration testing and hacking techniques in general.

So, we go through tools and techniques and use Kali Linux for that. Which seems to be powerful to use. As I am running Fedora 26 on my workstation, I am running the penetration test stuff on Fedora Boxes (more stable than VirtualBox), but I noticed that it would actually be nice to have the tools on my actual workstation, too.

So I went and googled a bit and as I knew, someone had already solved my issue.  As I am using Finnish language on the laptop, my installation command was like this:

# sudo dnf groupinstall Turvallisuuslaboratorio

For most of the people who do not have the capability to understand Finnish, it would make sense to use something more, how to put it, understandable language, like English.

So, in that case I suppose the command should work like this:

# sudo dnf groupinstall security-lab

By the way, while writing this, I did write the Kali Linux on a USB disk. It actually feels better to have it there than fooling around with virtual machines (in this case). Even though I’ll have to reboot the computer if I want to run it.

Install jMeter in Fedora

/ jankka / Leave a comment

I was a bit surprised for Fedora not having jmeter in the repository, but then I thought I could as well do the thingy in the old way 😀

First of all, we need to get the release:

wget http://apache.mirrors.spacedump.net//jmeter/binaries/apache-jmeter-3.1.tgz

Then we need to extract it:

sudo tar -xzf apache-jmeter-3.1.tgz -C /opt/

After that, we create the symlink to the jmeter -binary:

sudo ln -s /opt/apache-jmeter-3.1/bin/jmeter /usr/local/bin/jmeter

And as last, we create a desktop shortcut:

vi ~/.local/share/applications/jmeter.desktop

[Desktop Entry]
Type=Application
Encoding=UTF-8
Name=JMeter
Comment=JMeter
Exec=/usr/local/bin/jmeter
Icon=/opt/apache-jmeter-3.1/docs/images/jmeter_square.png
Terminal=false

 

That’s it. We’re done 😀