There has been Robots (TM)

 

We did have the (I suppose it was) first Robot Framework -related MeetUp in Sweden yesterday evening. It was held at the premises of Fareoffice Car Rental Solutions Ab in Kungsholmen.

There was not plenty of participants, but there was enough. The company and the office are small, so it was actually a good thing to have a ‘lagom’ crowd. Which in this case was 9 from outside the company, me, Pekka Klärck and 4 from Fareoffice.

First Pekka gave an introduction and background talk about Robot Framework. It was actually good to watch (for me), for I learned a few new things, once again. Even though he had held almost the same speech during the day before, while having a Robot Framework Workshop/training to Fareoffice, there still was few new issues to cover. Besides that we were talking about general usage of plug-ins on IDEs, RoboCon and running the tests with different setups.

The second was my trial of fire. I had a presentation about Robots in Containers. It went surprisingly smoothly. I had few technical glitches, but I knew they were there so the ghost of a demo-god did not ruin my presentation.

Of course there were few things I’d do differently. First of all, it was too quick. Secondly, I could’ve concentrated more on practical execution of the steps I was describing with  pictures; a live work is always better than preserved slide. Even if the deck of slides are be done with Prezi.

So it goes. I was tense, nervous and did all the typical flaws a Finn can do when representing and reflecting my MeetUp arrangements and presentation; I picked up all the mistakes I thought I had made. That is pretty much a built-in feature for us grown up in (the 80’s at least) Finland. Luckily, you can always trust a Swede to be there and comfort you. Empathy in Sweden is a strong and positive thing. Thank you for all participants for being there and for the support.

The best part of the MeetUp was the people and the discussions we had. It was great to see that there were others using Robot Framework here in Sweden.

I was also asked about running Selenium tests on IE/Edge, and we ended up showing the tests the way we do it; running them from Jenkins and in Browserstack. But that is not running on premise. Which meant I could not give a straight answer, which bugged me a bit, as usually. It started to feel kind of a challenge and I might want to pick it up on next creative Friday (once a month tradition at Fareoffice). So to say, spin up a Windows server and install Zalenium in it. Working with Windows would be a worthy challenge for me, an avid Linux -user as I am, and could serve as a good reminder on the fact that even the operating systems should be seen as tools. And every tool has its purpose.

In the end, we decided to create a MeetUp group and have the next meeting at Eficode’s premises in Stockholm. There was also few ideas about where to host RoboCon in 2019. All in all I am really happy I decided to push this one through, believe me, I had my doubts beforehand 😀

 

Logstash filter for Robot Framework

We are currently working on CI/CD -setup at work. As part of that, the tests need to be able to be implemented as a part of the pipeline.
Generally, the pipeline consists of steps/stages done with jenkins pipeline. The benefit on this is that the whole process and definition of the stages (Deploy, test etc) will be done by the developer team and stored in the teams own repository and is therefore controlled by the team also. Which is definitely a great step towards for the teams having more freedom and more responsibility when it comes to deliver the applications/solutions to the production. Needless to say it will also affect to the visibility of the quality and to the need of tests.

Plus that it will definitely keep the test team on their toes. Keeping ahead becomes a really neat challenge 😀

Now that does add more requirements also on the testing tools. First of all, the tools we use should be able to be used from containers. Which means that everything is dockerized. Well, the test code itself is in the repository, but the engines running the tests are in the containers.
We use, whenever we can, a general docker images from dockerhub.
Sometimes it won’t work like that. So we end up re-inventing the wheel.

That was the case with logstash. We will need to be able to filter the Robot Framework’s output.xml and send it to elasticsearch. There was two possibilities to do that; logstash filtering or xml parsing. The xml-parsing remains to be done still (I am going to do it), but I did manage to create the logstash -filter. It is not completely flawless, not even the most elegant, but at the moment it seems to be working as it should. To be honest, I was aiming to have a one more blunt instrument for our test needs.

The filter:

robot-results.conf

input {
 file {
 path => [ "/output.xml"]
 }
}

filter {
 xml
 {
 source => "message"
 store_xml => true
 target => "doc"
 xpath =>
 [

"msg", "doc.msg",
 "arguments", "doc.args",
 "kw", "doc.keyword",
 "status", "doc.status",
 "status/@status", "doc.test.status",
 "robot", "doc.robot",
 "errors", "doc.errors",
 "statistics", "doc.statistics",
 "suite", "doc.suite",
 "tag", "doc.tag",
 "total", "doc.total",
 "/kw", "leftovers",
 "/arguments", "leftovers"

]



}
}



output {
 elasticsearch {
 hosts => ["elastic"]
 index => "logstash-%{+YYYY.MM.dd}"
 }
}

Dockerfile:

FROM logstash

ADD robot-results.conf /etc/logstash/conf.d/robot/results.conf
CMD logstash -f /etc/logstash/conf.d/robot/

Running the container:

docker run --add-host=elastic:127.0.0.1 janmat/logstash-robot

 

You pick the right tool

 

As you can see, it is sometimes crucial to select a right tool. Even to stick to it, regardless on how awkward and painful it might feel. I just had to open with this scene, it is anyhow from one of my all time favorite movies. The book it is based on, is definitely one of my top 5:s.

We’ve all been there. It’s late night, we’re out in the park, having fun and all we have left is bottle of wine. And of course the corkscrew is nowhere to be found. So you start using your imagination. You might have a normal screw and a screwdriver, maybe a pair of tongs, too, you might be carrying a multi-tool, for what I know. Or not. The cork stays in the bottle.

You go through your pockets again, ask your friends, someone says that all he’g got is a pen, other one offers you a rock.

You see where I’m going here? A pen is mightier than a rock? Well, at least now it is.

Now you could open the bottle by smashing the head to a rock. Or smashing the rock to the head of the bottle. That might work, too. In the other hand, there’s always risks of getting bottle broken so, that the glass gets inside the bottle. Besides, that will create a mess in the grass and I a devoted dog owner and animal lover hate when people break bottles or other glass and leave the stuff there. Besides, people can get hurt too, for real.

So that leaves you the pen, right. You could write a letter – I know, it’s old school, but it’s nice sometimes, for real – to someone with a corkscrew to drop by. What’s there to loose? You got the whole weekend ahead of you (Did I mention it is a Friday night?) Or you could rob a corkscrew store with it, not the world’s best idea, though, giving it to be late night and the stores are all closed.

You could also take the pen and push the cork inside the bottle. That actually does work. You might get to spill some, but then again, at least there isn’t going to be shattered glass anywhere for the paws of the two to four legged friends. Notice I left a place for three legged dogs or cats, even squirrels.

So, you take the pen and grab the bottle. Your hands are getting a bit sweaty so you what your sleeve around the bottle. It keeps it in its place. Not completely, but firm enough. You take a breath.

You push the pen against the cork and push. Nothing happens, except the veins in your temples seems to be exploding, your face turns to red and the pen hurts your palm. A lot. You let go for awhile, and try again. No change to the situation. Cork is till on and you seem to be screwed.

A friend of yours, the one that has taken one more than you, mumbles something and offers you the stone from his hand. You look at him and smile and are about to shake your head, but change your mind and take the stone.

It is smooth on the other side, the side agains your palms has some edges and it feels, if not cold, then at least cool. You take the stone, push the bottle against the ground and hold it between your feet and hold the pen on your other hand against the cork. Then you slowly but firmly hammer the pen inside the bottle with the stone and finally, you’re done. Everything’s fine again, you drink the wine, get in to the night and wake up next morning with a hangover and some blurry memories. You might even end up having fun, who knows.

What I mean here is that you should choose your tools, for real. First of all you need to know that you have a need for a tool, then you need to check what requirements you have for it, then you need to find it. Thanks to the interwebs, it is fairly easy nowadays. After finding the tools, take several, and use them for awhile in the situation needed. So to say, evaluate them.

If you run into problems with the tool (you should, for real, even a sledgehammer needs some maintenance), try to find out if anybody has had the same issues. Most likely you’re not alone.

Check the maintenance costs. If you use more time maintaining the tool than the flaky tests, you probably have the wrong tool. Regardless on how good it looks, sounds or feels.

And once and for all; don’t get stuck with the first evaluation, don’t get stuck with your evaluation choice, either. In case the tool loses its focus and usability, make sure you can move away from it.

I myself are at the moment in that kind of situation: using a multitool with a gentle learning curve, but the maintenance and the license is starting to feel bad. It was a tool I was familiar with, a tool I’ve used for years in the previous companies, and it used to be an open source tool. They ended the open source path last year (if I remember correctly) and otherwise turned the usability a bit more worse, too.

So I’m considering moving the tests to another multitool, an open source based tool with steeper learning curve, but a considerably larger user group. Actually, I’ve done my consideration, all I need to do now is to transfer and modify the current tests from the first tool to the second one.

Some days of testing

Well, of course everything always goes as planned. Or then again, of course not. And when they don’t go the way you thought, planned or wanted, you might get a temptation to just let it go. But letting go would let you only quit, not finish. Determination or stubbornness – call it what you will – you need it. Especially when handling things that are voluntary and not always easy to squeeze in to your day.

That being said, I’ve been struggling. Anyhow, here’s a small report on things I’ve managed to accomplish so far.

Day 7: Find an accessibility bug.

I suppose it was this one that broke my flow. First of all, I came in a day late, dollar short. Second thing was that I’ve never done any accessibility testing. That, me being in the case, leads some  issues.

I have always been ‘try first, stumble and read the manual to get to know what did I do wrong’ -type of guy. This time was no exception. I ended up reading few short sentences on the  W3 accessibility testing wiki and rushed my way to turn on accessibility features on my notebook. After fooling around for awhile I got nowhere, if not to the land of frustration. The laptop is old and slow and it had difficulties to read the web pages in Finnish so that I would actually understand. That was due to the lack of memory (most likely) and bad/lacking support of the chipset. I’m running CloudReady/Chromebook OS on Asus EeePC with Intel Atom chipset. It fails to work on most of the OS I’ve tried.

So I gave up.

You see, testing is my profession and therefore it actually is my passion. But not on my spare time. I have divided my days between work and non-work, and it suits me fine. It just don’t help doing test related tasks (regardless how fun they are :D) on my spare time. It’s not that I wouldn’t like, but as I have my priorities at work, I also have them when I’m at home. And in that kind of issues, it actually is really easy to quit. Which I did.

But now I went and read a bit more on accessibility testing and noticed that there is few semi-automated tools listed on the page. I suppose I’ll give them a try at work on some of our interfaces. Semi automation is still partly automation (which I do like a lot) and I might also find something worthwhile. You never know where you end up when you start testing without a decent specification. And even with a spec you might find yourself in between rock and a weird place.

 

My plan was to write about the other tests I did (but did not write about), but this post starts to get long enough, I’ll have to fill the other blanks later on.

Features of a (good) test environment

Easy to deliver

The test environment should be able to be delivered by request with as little input as humanly possible. A click of a button, a SMS sent, a commit to VCS should be able to act as a trigger.

Steady

The test environment should be as stead and as reliable as possible so that we can rely on that the failing tests are not caused by faults in the test environment

Open Source

I do see that the benefits of open source tools and projects are way more beneficiary than what you get from plain commercial tools. Even though you might end up paying for the support and building in the knowledge, it still pays off in the end.

Easy to Set-up

The test environment should be able to be set up by just pressing a button. If external information is needed a set of variables should be able to entered automatically.

Easy to Reset

You should be able to reset the test environment to a desired level of functionality at any point of time in order to re-execute the failed tests. Furthermore, it would be great to have a possibility to automatically run deeper analytical tests automatically in case of failure.

Easy to Monitor

Test environment should provide an interface where you can easily see the status of the tests, historical metrics and also the status of the test environment.

Provides test data

Test environment should contain test data generator which could by request fill in the databases with relevant test data. It should contain a simple interface in order to receive and response with the requested test data. The test data generator should not affect to the performance of the actually tested item.

Provides test results and metrics

Test environment should be able to provide the status of itself. And since it consists of the whole system plus the underlying parts, it should be able to deliver the status as effortless to the users and stackholders as possible.

You could, for example have a status interface. An interface that provides the status of the test environment in a single view.

There should be an interface – a web interface would nowadays be enough – that gives you all the relevant information about the tested items. And all this in a glimpse.

For the event monitoring, you could use ELK (Elasticsearch, Logstash & Kibana).

For the test monitoring, you could use the wall display on Jenkins. That at least shows the status of the executed tests. What I am missing from there is the status of individual test cases instead. One way to get them would be to build the tests in Jenkins per test case and add separate views per test suite.

But even that would tell the status only per test suite. What if you had several test suites in the project and 4/5 of them would be executed, 1/5 would be skipped and 10% of the executed test cases would have failed. How to display that?

Above those you would need to have to know the other metrics; load test results, combined result of different test suites, status of the all available test suites and test cases combined with the execution of the test cases today. The list seems to be endless here.

And all in a single look.

Or would it be enough to just have one page that indicates the status of the test environment together with the status of the test results. With two different indicators.

But what should the indicators then be? Traffic lights? Metrics? Curves? Flowcharts? Or combination of them?

What I’d like to know here is if there already exists such a system, or should I start to create that myself? Not that I haven’t been reinventing the wheel before (trust me, I have created my share of 3-sided ones), but it might help if I’d knew I wouldn’t have to.

Delivering a Hippo

When delivering software through the delivery chain, I’d like to refer to the software as a Hippopotamus. Previously I have heard Michael Bolton (Not the singer, or the guy from Office Space) refer to software (and the delivery process) and especially requirement handling by telling a story about blind men and elephant. I do have my reasons to switch the elephant to a hippo, which I then explain later on.
If you like elephants more than hippos, too bad.

The Story

There is a group of blind men. They get a task to describe what a hippopotamus looks like. They need to give the description to a man that will build a crate to transport the hippo to the wild.
The group is first gathered around the head. They describe the beast as a big headed creature that has a huge mouth and sharp, long teeth, relatively small eyes and small, but alert ears. The creature seems to also have a slightly bad temper. They measure the size of the head and give the description to the transport team.
After the description, the transport team builds a crate and the delivery starts.
They get the whole head to the crate, it fits perfectly. But then there comes something else.
It seems they’ve forgotten something.
At this point it’s about the time for me to reveal a bit on how do I see this. First of all the group of men, the blind ones, I saw them first as bunch of software developers delivering the latest features on maintenance project. But to be fair, the group should also contain some test engineers, requirement responsible (product owner or whatever the title is), project manager and so forth. For as far as I know, regardless on the role on the project, in this kind of scenario we people tend to get blind. And it is not related on this scenario. When we as humans concentrate on the problem at hand we have great capabilities on concentrating on just that and to forget everything else. Which is somewhat crucial. It is crucial for the delivery to be delivered as it should’ve been. It should also be crucial to focus on other things than just the delivery itself.
What the group of blind men has forgotten is that the hippo has it’s body. The body is bigger and wider than the head and therefore it has no place in the crate. However, if they want to deliver the head, they should also deliver the body. So they turn back to the hippo and feel the sheer size of it’s body.
They describe the quite huge creature to the  transport team. The transport team, for they are clever, will now build a crate to contain both the head and the body. Not just the body.
So everything should now be fine, or what do you think?
The team has now delivered the complete hippo to the transport team and managed to get it inside the crate. Sure, and everybody is happy. The main functionality is delivered with the maintenance package without huge amount of flaws in it. As it usually is.
But did they forget something? No, everything is neatly packaged and ready to be delivered.
Well, the hippo has the rear end. And as you know, hippos tend to mark their territory with their rear end. They are clever in that way that they spread their manure (aka hippo -shit) around by wagging their tail vigorously. It’s an effective way to spread the message. And there we have the slip through. To be honest It wouldn’t matter if the group of men were blind or not; slip through will eventually happen every now and then to the teams that are or are not visually impaired. Really, we’ve all been there.
What we can learn from here is few things. First of all, the roles of the team should be spread a bit. There should be at least one person taking care of the maintenance and regression. Whether that is a test engineer or a developer does not actually make any greater difference. It usually is enough that you verify that he regression should not happen and that the old stuff is delivered also. The other thing to consider would be to take one of the transport team to check the delivery.
And for the love of God (or whatever deity your world goes around – money, consuming, stock exchange, sex, rock n’ roll etc), don’t get me wrong here. I do not mean that we need more people to the team. I do think that more blind men see just as much as less of them. What I mean with the role is that we, humans, are capable of switching to different roles. Even during the workday. Regardless what I think about multi-tasking (I am strongly against, really, it does not work for me), I do think that same person that handles the delivery of the main functionality can also handle te delivery and verification of the maintenance delivery. And the verification of regression tests. Does not even have to be a developer to do that. Or QA, depending on from which side you look at it. There’s plenty of things we QA -engineers can learn from the developers and developers can also learn a thing or two from testers. Some of the test engineers are even capable of writing code themselves and most of the developers are capable of testing their (or someone else’s code) during the day. And after this is done, they can continue concentrating to deliver the main functionality. It really is as simple as that. Nothing fancy here.
So, now we’ve covered the main functionality and the regression. The head and the body are happily delivered. What to do with the tail?
First of all. We will always have slip throughs. Really, there is no way we can get rid of them. There is no way we can test and code in a way that will lead to a delivery and simultaneously not have anything weird hidden in the delivery. But of course we should do anything we can to prevent them happen.
What could be helping here was that there is a role in the team for experimental testing. Do something unconventional. Someone from the team should get to bash the hell out from the application, tease the boundaries, break it, create chaos. Do things users won’t do (that list is actually shorter than you think ;)). That will, eventually reveal some of the bugs that the more conventional testing won’t do. If possible, automate that and do it all the time. Just keep the chaos running.

Re-evaluating and bouncing

I managed to actually re-evaluate my plans and combine them to my skill-set. There is a slight cap there. I can now see, that the skills of my programming are not near the needs of the project. Now that means that I’ll have to build the resources.

This means, of course, that the Algorithm -training is going to be continued, but that it is not enough. In fact I decided to get back to basics and go through two topics.

  1. Programming in Python 3
  2. Twisted tutorial (and everything related)

So getting those 2 things done will, eventually, get me closer to my ultimate goal: to become an efficient technical tester (A Test Toolsmith, some might say).

Now earlier, in this kind of projects, I’ve let myself sidetrack myself. Pretty easily also. So in this case, I’ll have to learn myself some discipline and perseverance. What I mean is that no matter if I get the ideas on how to develop the project I have in mind (being the messaging answering machine or my Crossfit whiteboard), I should only write the ideas down and get back to them after I’ve finished the ‘course’.

Now the learning should not take too long. Honestly put, I have had programming education in Java and Python. Java -education took place long time ago, just before I was hired at my first Test Engineer -job. The Python training, in the other hand was only few years ago. So in that way re-learning the stuff (and adding more, the course was just a basic one) should not take too much time. And this time I just have to make it all the way. The benefit on having this done at home, as my private project, is that I can also implement my skills right away after I’ve done the learning. Lately the usage of my learned skills has been more or less nothing. I know it is matter of determination at work, too, but previous tasks at work has kept me so busy otherwise that the usage of any coding capabilities has been more or less closer to the zero. Now that also seems to be changing, hopefully for good.

Besides Python, I do have a sneaky feeling that I should learn a bit more groovy. Which actually means that I should refresh also the java -skills. Not the worst idea either. Maybe after the Python is done I can concentrate more on groovy. Besides, we do use groovy -scripting when using SoapUI at work, so that one comes also naturally by the work, too. The other day I managed to even fetch a file from FTP to my desktop. Woohoo 😀

Last, but not least is that I do still have my project plans. To create something useful by using Python, Robot Framework and ATDD -methods to test and create everything. I just need to build my wheels first. Slow and steady should eventually win the race.