robot framework

Report Portal Installation

/ jankka / 3 Comments

Introduction

We do have a need to analyse the stability and usability of our tests, too. We do monitor our systems from here and there, check the code coverage of /development code/. Funny when you think about it, but test automation is part of development process and should be addressed as such. Even more if you’re developer who runs the code.
I have been running all sort of analytics based on test executions in Jenkins, written parsers in python and building dashboards on ELK -stack. And boy it has been fun 😀
Then I heard about this tool, Report Portal (http://reportportal.io) from a podcast from Joe Colantonio and decided to take a look. I am nowhere near to make any conclusions, but after a brief usage it does look promising.

Running locally

Report Portal has a demo environment you can try with your tests, but it really comes in handy if you host it yourself. And they provide instructions to that in http://reportportal.io/download. I have followed the instructions and listing them down in here. To be honest, I do think they do brilliant job already and my notes here are more for myself. To recall the process in a better way and sure, and to get some clicks on my blog, too 😀
First of all, you’ll need to have `docker`installed. After that everything should be easy.
NOTE:  In MacOs (at least) you should allocate at least 5 GB memory to the docker process. I had only 2 GB and everything was looking good from the process side, except that I could not log in. I do recall running the same in Linux without changing anything and it just worked.
  •  mkdir ~/reportportal
  • cd ~/reportportal
  • Download the docker-compose -file:wget https://github.com/reportportal/reportportal/raw/master/docker-compose.yml
  • mkdir -p data/elasticsearch 
  • chmod g+rwx data/elasticsearch
  • sudo chgrp 1000 data/elasticsearch (here’s a difference. I had to run the chgrp in MacOS as sudo)
  • docker-compose -p reportportal up -d --force-recreate
When the compose is ready, you should be able to browse to the Report Portal UI: http://localhost:8080/ui/
Log in using default/1q2w3e.

Robot Framework integration

In order to get the Robot Framework results to report portal, you need to use a separate library.  GitHub – reportportal/agent-Python-RobotFramework. I will create a new blog post on this and add more information there.

jmeter/Blazemeter integration

Basically all we have to do is to generate a junit.xml -report from the test run, zip it and then send it to the report portal API. I will create a new blog post on this and add more information there.

jest-puppeteer integration

GitHub – reportportal/agent-js-jest: ReportPortal agent for the Javascript Jest unit test framework. I will create a new blog post on this and add more information there.
Advertisement

Day 4 (of 30 Days of Automation in Testing)

/ jankka / Leave a comment

Day 4: What kind of testing can automation support you with?

Unit testing comes to my mind as the first one.It helps me as a tester in a way that I let developers do the deeds and learn to trust they do things right. Even reviewing the unit test code is actually a neat learning experience, once you get to understand the unit test logic.

Functional testing during or right after the build, can also be done by same unit testing approaches, which is also a neat way to verify the functionality. The benefits on that one is that the code is readable by the developer who wrote it and (in case it’s not made with Perl) even with other members of the team or company. To have the tests usable and readable in the same IDE developers use, leads also to easier debugging. What usually lacks is decent reporting, but then again, all we need to know that it fails when it should and passes when everything really is OK.

Mocking the test environment and harnessing the SUT. They walk hand in hand. There’s plenty of nice mocking tools, wiremock for example. One might also re-invent the wheel here. I mean the mock can be just a file you fetch from another system or write yourself. Does not sound too much automating, though 😀

Load testing is also one of the standard ones to be used with automation. Given the powers of the servers, new container technologies and distributed load testing services, you can build your own solution or use a ready made service, like blazemeter or octoperf.

Repetitive functionality tests, regression tests and release tests in any step of the development process are good to automate, given that you are going to use the automation more than few times.

Then the whole world of Continuous Delivery and Continuous Integration. It will not happen properly if you do not have the right tests put in right places. Well it might be OK to let the users test the product, but that would mean that you should have an automated monitoring system that gathers the data you will need. That’s also one way of automation that will help.

And of course all of the test data generating tools, self-made or otherwise, bash- or python -scripts for setting up the environments together with containers and their managing tools (docker, docker-compose and kubernetes to start with).

I most likely am missing something here, still. I mean where do you draw the line when it comes to automation in testing? Is it only the testing tools (like jMeter, Robot Framework etc.) or the tools that let you yourself automate mundane tasks to ease your testing process.

 

 

There has been Robots (TM)

/ jankka / Leave a comment

 

We did have the (I suppose it was) first Robot Framework -related MeetUp in Sweden yesterday evening. It was held at the premises of Fareoffice Car Rental Solutions Ab in Kungsholmen.

There was not plenty of participants, but there was enough. The company and the office are small, so it was actually a good thing to have a ‘lagom’ crowd. Which in this case was 9 from outside the company, me, Pekka Klärck and 4 from Fareoffice.

First Pekka gave an introduction and background talk about Robot Framework. It was actually good to watch (for me), for I learned a few new things, once again. Even though he had held almost the same speech during the day before, while having a Robot Framework Workshop/training to Fareoffice, there still was few new issues to cover. Besides that we were talking about general usage of plug-ins on IDEs, RoboCon and running the tests with different setups.

The second was my trial of fire. I had a presentation about Robots in Containers. It went surprisingly smoothly. I had few technical glitches, but I knew they were there so the ghost of a demo-god did not ruin my presentation.

Of course there were few things I’d do differently. First of all, it was too quick. Secondly, I could’ve concentrated more on practical execution of the steps I was describing with  pictures; a live work is always better than preserved slide. Even if the deck of slides are be done with Prezi.

So it goes. I was tense, nervous and did all the typical flaws a Finn can do when representing and reflecting my MeetUp arrangements and presentation; I picked up all the mistakes I thought I had made. That is pretty much a built-in feature for us grown up in (the 80’s at least) Finland. Luckily, you can always trust a Swede to be there and comfort you. Empathy in Sweden is a strong and positive thing. Thank you for all participants for being there and for the support.

The best part of the MeetUp was the people and the discussions we had. It was great to see that there were others using Robot Framework here in Sweden.

I was also asked about running Selenium tests on IE/Edge, and we ended up showing the tests the way we do it; running them from Jenkins and in Browserstack. But that is not running on premise. Which meant I could not give a straight answer, which bugged me a bit, as usually. It started to feel kind of a challenge and I might want to pick it up on next creative Friday (once a month tradition at Fareoffice). So to say, spin up a Windows server and install Zalenium in it. Working with Windows would be a worthy challenge for me, an avid Linux -user as I am, and could serve as a good reminder on the fact that even the operating systems should be seen as tools. And every tool has its purpose.

In the end, we decided to create a MeetUp group and have the next meeting at Eficode’s premises in Stockholm. There was also few ideas about where to host RoboCon in 2019. All in all I am really happy I decided to push this one through, believe me, I had my doubts beforehand 😀

 

Logstash filter for Robot Framework

/ jankka / Leave a comment

We are currently working on CI/CD -setup at work. As part of that, the tests need to be able to be implemented as a part of the pipeline.
Generally, the pipeline consists of steps/stages done with jenkins pipeline. The benefit on this is that the whole process and definition of the stages (Deploy, test etc) will be done by the developer team and stored in the teams own repository and is therefore controlled by the team also. Which is definitely a great step towards for the teams having more freedom and more responsibility when it comes to deliver the applications/solutions to the production. Needless to say it will also affect to the visibility of the quality and to the need of tests.

Plus that it will definitely keep the test team on their toes. Keeping ahead becomes a really neat challenge 😀

Now that does add more requirements also on the testing tools. First of all, the tools we use should be able to be used from containers. Which means that everything is dockerized. Well, the test code itself is in the repository, but the engines running the tests are in the containers.
We use, whenever we can, a general docker images from dockerhub.
Sometimes it won’t work like that. So we end up re-inventing the wheel.

That was the case with logstash. We will need to be able to filter the Robot Framework’s output.xml and send it to elasticsearch. There was two possibilities to do that; logstash filtering or xml parsing. The xml-parsing remains to be done still (I am going to do it), but I did manage to create the logstash -filter. It is not completely flawless, not even the most elegant, but at the moment it seems to be working as it should. To be honest, I was aiming to have a one more blunt instrument for our test needs.

The filter:

robot-results.conf

input {
 file {
 path => [ "/output.xml"]
 }
}

filter {
 xml
 {
 source => "message"
 store_xml => true
 target => "doc"
 xpath =>
 [

"msg", "doc.msg",
 "arguments", "doc.args",
 "kw", "doc.keyword",
 "status", "doc.status",
 "status/@status", "doc.test.status",
 "robot", "doc.robot",
 "errors", "doc.errors",
 "statistics", "doc.statistics",
 "suite", "doc.suite",
 "tag", "doc.tag",
 "total", "doc.total",
 "/kw", "leftovers",
 "/arguments", "leftovers"

]



}
}



output {
 elasticsearch {
 hosts => ["elastic"]
 index => "logstash-%{+YYYY.MM.dd}"
 }
}

Dockerfile:

FROM logstash

ADD robot-results.conf /etc/logstash/conf.d/robot/results.conf
CMD logstash -f /etc/logstash/conf.d/robot/

Running the container:

docker run --add-host=elastic:127.0.0.1 janmat/logstash-robot

 

Excuses get in the way

/ jankka / Leave a comment

I know, every excuse is just an excuse on failing to prioritise, but sometimes the prioritising actually gets you nailed down to something where you just have to concentrate and work on.  This week has been one of those.

So to say, releases flowing in from doors and windows and I find myself testing (or wanting to test) them all.

Which of course has meant that I haven’t been able to fulfil the 30 Days of Testing assignments. Currently I am lagging behind 1½ – 2 days. My plan is to get back on the track during this week, anyhow, meaning that I’ll do something during the weekend.

This is just to inform that I am aware of the situation.

Besides that, I ended up going through this tutorial yesterday and realised that this mochaJs-thing seems to be a neat way to learn JavaScript and some test development 😀 I might even give it a more thorough run later on. I also discussed with the author (Viktor Johansson) on collaborating and creating some neat tutorial with BDD & Robot Framework. Oh, and managed to install Skype on the Fedora, which is always an accomplishment 😉

We’ll see what tomorrow brings.

Re-evaluating and bouncing

/ jankka / Leave a comment

I managed to actually re-evaluate my plans and combine them to my skill-set. There is a slight cap there. I can now see, that the skills of my programming are not near the needs of the project. Now that means that I’ll have to build the resources.

This means, of course, that the Algorithm -training is going to be continued, but that it is not enough. In fact I decided to get back to basics and go through two topics.

  1. Programming in Python 3
  2. Twisted tutorial (and everything related)

So getting those 2 things done will, eventually, get me closer to my ultimate goal: to become an efficient technical tester (A Test Toolsmith, some might say).

Now earlier, in this kind of projects, I’ve let myself sidetrack myself. Pretty easily also. So in this case, I’ll have to learn myself some discipline and perseverance. What I mean is that no matter if I get the ideas on how to develop the project I have in mind (being the messaging answering machine or my Crossfit whiteboard), I should only write the ideas down and get back to them after I’ve finished the ‘course’.

Now the learning should not take too long. Honestly put, I have had programming education in Java and Python. Java -education took place long time ago, just before I was hired at my first Test Engineer -job. The Python training, in the other hand was only few years ago. So in that way re-learning the stuff (and adding more, the course was just a basic one) should not take too much time. And this time I just have to make it all the way. The benefit on having this done at home, as my private project, is that I can also implement my skills right away after I’ve done the learning. Lately the usage of my learned skills has been more or less nothing. I know it is matter of determination at work, too, but previous tasks at work has kept me so busy otherwise that the usage of any coding capabilities has been more or less closer to the zero. Now that also seems to be changing, hopefully for good.

Besides Python, I do have a sneaky feeling that I should learn a bit more groovy. Which actually means that I should refresh also the java -skills. Not the worst idea either. Maybe after the Python is done I can concentrate more on groovy. Besides, we do use groovy -scripting when using SoapUI at work, so that one comes also naturally by the work, too. The other day I managed to even fetch a file from FTP to my desktop. Woohoo 😀

Last, but not least is that I do still have my project plans. To create something useful by using Python, Robot Framework and ATDD -methods to test and create everything. I just need to build my wheels first. Slow and steady should eventually win the race.

Install Robot Framework to Ubuntu 14.04

/ jankka / 3 Comments

By following these fairly easy steps you can get the Robot Framework with Selenium2Library and SSHLibrary installed on Ubuntu 14.04. Now you could go and fetch the installation packages and for example compile everything from the scratch. That’s all just fine with ,e. Anyhow, you could also use these instructions and slip all the hassle.

First of all, to install everything, you will need to have pip. Pip is a Python package installer and it helps you out a lot in case you need to install anything related to Python.

  • Install pip 
    sudo apt-get install python-pip
  • Install Robot Framework
sudo pip install robotframework
  • Install Selenium2Library
sudo pip install robotframework-selenium2library
  • Install SSHLibrary
sudo pip install robotframework-sshlibrary

Now that was all I needed for getting started. However modifying the Robot Framework Test Cases from a text editor is a tricky business. So it would be wise to either use LibreOffice Calc or some other software that can read Tab Separated -files. There’s also a bunch of plugins for vim, Emacs, Sublime and Eclipse (or LiClipse) to get the TSV -format highlighted in the editor. The other way to do it is to use Robot Framework IDE, called RIDE. RIDE is indeed a decent tool for handling the keywords and variables in a correct format. The installation procedure, however, was not at all that straightforward as you might think. While you can install the RIDE with pip, you still need to have wxPython in order to run it.

  • Install RIDE
sudo pip install robotframework-ride
sudo apt-get install libwxgtk2.8-dev libwxgtk2.8-dbg
sudo apt-get install build-essential
sudo apt-get install python-wxtools python-wxgtk2.8-dbg

Now you can get the ride started on command line by easily writing ride.py on the terminal. Happy trails 😀 Next article should be about installing the whole stuff to Windows. Which seems to be a whole lot harder than it should be.

Cross platform rant

/ jankka / Leave a comment

I just can’t get it; How come it seems to be too difficult for developers provide a cross platform functionality that actually works? I mean come on you there. I’ve been learning to use this Robot Framework as the starting point for my own private ATDD -project (Called Marvin, due to Hitchhikers Guide to Galaxy, of course). This morning I started to create more test cases in order to start (later on) the development of the new log in -features. And I thought it would be a good use for my work laptop. Really, it has 16 GB of RAM, a processor and a hard drive. The only unfortunate thing about it is that it uses Windows 7. Due to reasons not quite clear to me, to be honest. It seems to have something to do with the IT -departments capabilities on monitoring and updating the end -users laptops and most likely the ability to remote -reset the hard drive in case the laptop is stolen. Or lost. Apparently the solution used there, in the wide America, is not flexible enough to be used in the real world. Well, now I’m just being nasty here, but still, we do develop stuff that runs above Linux and the development is done on windows. This of course applies to testing, too. Sigh.

Anyhow, everything worked just fine in the beginning. I did install the RobotFramework -eclipse -plugin and all. Ended up installing Robot Framework and its Selenium2Webdriver -library, too. Like I did last week on the Linux- & OSX- laptops. Like said, everything was fine. Until I had to actually execute the tests I had created.

I just can’t get the ancient profile -thinking Firefox keeps having on windows. Really. There’s no point of that. At the moment it actually really just slows down the development. And of course gains my frustration and gets me writing this blog -entry (which is not that bad thin, though). I tried to follow the instructions, too. Really guys, you who develop the Robot Framework, you could create a decent entry on how to configure Firefox and webdriver in Windows in order to get the test cases executed. All I found was Python Webdriver -instructions, which I have been using previously myself. Those do work. But there was no single entry anywhere that pointed to a working version of the setup.

So here I am; Browsing the web on my Firefox running on my ancient laptop that runs Ubuntu 14.04 LTS (With encrypted HardDrive, by the way) and getting ready to install the RobotFramework Eclipse -plugin to LiClipse pulling the latest changes from GitHub and executing the actual tests. In windows that was impossible (well ok, just NOT worth the effort), in Linux, it should not take more than 30 minutes for me to get a decent FAIL on the first test cases.

Learning curves

/ jankka / Leave a comment

It seems I have a road ahead. Not sure what is going to happen, but at least I know that I’ll enjoy the ride. The ride will be long and it will be hard at times, but all I have to do is to remember that I have all the time in the world for this and the target is not to create a perfect application, but to learn how to automate tests and develop with Python and Robot Framework. Both skills I see as needed in the future.

First of all, after everything was installed yesterday, I sat down and went through some tutorials. Both git & Robot Framework are getting more familiar now. Far from being perfect, but at least I’ve been using both. Git is, by the way, a lot more intuitive with the commands than CVS. And it really neat to use a service like GitHub to store my stuff in. At the moment the solution I started to lab with is stored as private in GitHub. In the future I plan to create a public variant on that in order to get it shared properly.

I also managed to start the Algorithms -project and created the first excercise. It is a simple insertion sort I managed to get together by help from stackoverflow (a really good site to search for information;)) and a slight use of my brain. I actually needed the stackoverflow -solution more or less as interpreter to the algorithm -books pseudo -code.

And last, but not least, I actually have now done my first time ever ATDD -approach on developing. The test case was created and it failed in the beginning. After some coding I managed to get it also go through and got a PASS as result. And the test case is of course using selenium webdriver -library to verify that the web -page is up and running.

As a bonus I actually got an idea about a solution on how to parse some input strings properly with groovy -script in SoapUI. A thing I will need to do at work next week anyhow.

So to say, it has been a proper weekend.