test

Features of a (good) test environment

/ jankka / Leave a comment

Easy to deliver

The test environment should be able to be delivered by request with as little input as humanly possible. A click of a button, a SMS sent, a commit to VCS should be able to act as a trigger.

Steady

The test environment should be as stead and as reliable as possible so that we can rely on that the failing tests are not caused by faults in the test environment

Open Source

I do see that the benefits of open source tools and projects are way more beneficiary than what you get from plain commercial tools. Even though you might end up paying for the support and building in the knowledge, it still pays off in the end.

Easy to Set-up

The test environment should be able to be set up by just pressing a button. If external information is needed a set of variables should be able to entered automatically.

Easy to Reset

You should be able to reset the test environment to a desired level of functionality at any point of time in order to re-execute the failed tests. Furthermore, it would be great to have a possibility to automatically run deeper analytical tests automatically in case of failure.

Easy to Monitor

Test environment should provide an interface where you can easily see the status of the tests, historical metrics and also the status of the test environment.

Provides test data

Test environment should contain test data generator which could by request fill in the databases with relevant test data. It should contain a simple interface in order to receive and response with the requested test data. The test data generator should not affect to the performance of the actually tested item.

Provides test results and metrics

Test environment should be able to provide the status of itself. And since it consists of the whole system plus the underlying parts, it should be able to deliver the status as effortless to the users and stackholders as possible.

You could, for example have a status interface. An interface that provides the status of the test environment in a single view.

There should be an interface – a web interface would nowadays be enough – that gives you all the relevant information about the tested items. And all this in a glimpse.

For the event monitoring, you could use ELK (Elasticsearch, Logstash & Kibana).

For the test monitoring, you could use the wall display on Jenkins. That at least shows the status of the executed tests. What I am missing from there is the status of individual test cases instead. One way to get them would be to build the tests in Jenkins per test case and add separate views per test suite.

But even that would tell the status only per test suite. What if you had several test suites in the project and 4/5 of them would be executed, 1/5 would be skipped and 10% of the executed test cases would have failed. How to display that?

Above those you would need to have to know the other metrics; load test results, combined result of different test suites, status of the all available test suites and test cases combined with the execution of the test cases today. The list seems to be endless here.

And all in a single look.

Or would it be enough to just have one page that indicates the status of the test environment together with the status of the test results. With two different indicators.

But what should the indicators then be? Traffic lights? Metrics? Curves? Flowcharts? Or combination of them?

What I’d like to know here is if there already exists such a system, or should I start to create that myself? Not that I haven’t been reinventing the wheel before (trust me, I have created my share of 3-sided ones), but it might help if I’d knew I wouldn’t have to.

Advertisement

Sidekick

/ jankka / Leave a comment

Apparently it never is enough. Nothing is. Man born a hoarder is a hoarder, even during the nights or weekends. The last 2-3 weeks have been kind of challenging mayhem at work. I have to admit that my capability in adopting to varying situations has been tested. Several times. Long days, tough days. No time to reset, besides in the evenings at home or at the gym. Mostly both. Luckily training the unknown when following the CrossFit helps me mentally to get used to attacking the unknown. That’s a blessing. But even though it has been rough, it surely seems I haven’t been capable to do enough on the creative side.

I’ve been bouncing this idea of web -service that could help me as a tester. And since nobody is going to do it for me – partly due to my lack of capability to describe it properly and inthusiastically enough, partly due to the fact that every single other one at the work is evenly loaded – I’ll need to do it myself.

What I have planned is to do the whole stuff with Open Source SW, using Python, Django and git. Cucumber has been fetched and installed, as well as RobotFrameWork. Those are the tools for me to use.

Developing is a creative process and I’m pretty sure I’ll learn a lot on the way. I’m going to tackle this from the ATDD (Acceptance Test Driven Development) point of view. Hopefully I learn new stuff for testing as well as the crucial stuff for automating my tests and using development skills as tools for my work as a tester.

Now, since this is going to be part of my work, it is going to be a system test support tool, I cannot release anything (at least as it is) on my GitHub. I do have plans, though, to put something there, but as stripped version.

So far the project has been setting up stuff: Yesterday, a fresh install of Ubuntu 14.04 to my Dell Latitude, today getting the tools installed and reading the first introduction chapter for Git -usage. Tomorrow I should be good to go.

And for the record, I don’t except things being too ready too soon. This is going to be done for my education and amusement only. Status updates might follow every now and then anyhow 😀

Vagrant recommendation

/ jankka / Leave a comment

For testing server -side solutions with VirtualBox, I highly recommend to use Vagrant (http://www.vagrantup.com/). Works perfectly on testing purposes, the environment is set up by entering following at the terminal prompt:

$ vagrant box add base http://files.vagrantup.com/lucid32.box

$ vagrant init

$ vagrant up

That initialises, fetches and starts up one Ubuntu Lucid Lynx -server. To connect to connect to the server after this:

$ vagrant ssh

After working for awhile you can either halt, suspend or just plain exit from the server and then get back to it by writing ‘vagrant ssh’ again. The most genius part is that you can re-set the server back to defaults after everything is done. That’s just plain perfect for testing 😀

To re-set the Vagrant -server:

$ vagrant destroy

Of course there is drawbacks, as always, but to be honest, it’s a neat tool.

In order to get to see even more usable Vagrant Boxes, browse to: http://www.vagrantbox.es/

Building up the competence

/ jankka / Leave a comment

After being back from vacation has been neat. At least what came to the office atmosphere. The first week I worked almost completely from home, week after that there was almost nobody at the office, so I could really work easily by myself. Now I notice that the sounds of the keyboards rattling and mice buttons clicking keeps bugging me off. Not sure why, though. Usually I get the feeling when I’m tired. And now it’s monday and I won’t admit being tired after nice and refreshing weekend. Luckily there’s Spotify and earphones. 😀

During the holiday I was more or less online all the time, which can be seen as a good or bad. But as my friend has told me, everything has its good and bad drawbacks. I myself found the holiday really good and refreshing. Both on personal and professional level.

As you might know, I’m working as Test Engineer in QA/Test consultant company. The thing is that I got a phone call and an email during my first holiday week telling me I got an interview with a potential (and really interesting) client next monday, if it just suited my plans. Well, it did. I went to the interview and ended up having a test exam delivered to me the next day. Not completely unexpected, for they were looking for test developers.

I’ve been in  a similar situation before, last year at Spotify and that did not go too well. Or at least I found something I was not capable of doing. Regardless on the training I’ve taken and the education I have, my development capabilities were far from perfect. I did my best ( at the time) and came up with solution made with bash -scripts.

This time, though, I decided to really give it a shot with a decent language. I did have my Python -training   last year, for test automation purposes the language feels still to be really good one. Sadly I did not have any possibility to start using it at my previous work. Well, I might have could, but I never did even try. Besides that I did stick up with my old habits (and bash -scripts), which really rarely leads to anything new and exiting. 

So I went and jump into Python. Ended up doing some Python -related stuff few hours per day for around 10 days. With some breaks, for it still was my holiday 😉 It basically meant I did work around 3 days with Python and ended up with one solution. Then I got back at work and noticed I could make it a little bit better before delivering to the client. Which I did.

Now that the version is delivered I can see it’s not perfect. They rarely are. So at least I’ve got something to work with still.

This all has been really interesting. Getting inside the development, even for test purposes, has been an eye opener. More or less it seems that this has been the one tool missing from my case. So in that way I really do understand using my own time on learning and getting inside Python. For now on it will definitely be easier for me to adapt to that kind of situation. Besides that, now I can keep up learning myself more and more. I suppose that is the best way to do it. To build up and maintain the competence while I still can. My target is now to keep on developing something with Python daily. The same thing I’ve had my writing; I’ll need to create a new habit out of it. By that way I’ll keep up my touch to the language and hopefully will get my competence built up more and more every day. Well at least weekdays. Weekends are still worth to be used with family 😀

Now I’m not saying people should use their own time doing job -related stuff, and I myself don’t see I was doing it like that. I just used the work as an excuse for getting an uplift for my own competence. And (you might argue with this) my competence is not made only for work purposes, it is something I can use in my everyday life, somehow, too. Even if I won’t sit down and code I can learn myself how to get to know new stuff and that won’t hurt in the real life situations either.

Now, few tips:

  1. Eclipse & pyDev is a brilliant couple. I did myself use ADT Bundle for Eclipse for we need Android development environment for test automation purposes.
  2.  Stackoverflow has been a brilliant source of information when in need. And I’ve been using it a lot 😀
  3. In case you need to add plot graph on your web site (I need to) it seems matplotlib could do the trick.  Here’s how to get it installed in OS X Mountain Lion
$ ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)"
$ brew doctor
$ brew install python
$ export PATH=/usr/local/bin:/usr/local/share/python:$PATH
$ easy_install pip
$ brew install gfortran
$ pip install numpy
$ pip install scipy
$ brew install pkg-config
$ brew install freetype
$ brew install libpng
$ pip install matplotlib
$ python
>>> import numpy
>>> import scipy
>>> import matplotlib

 Â