Sunday, January 14, 2018

Testing Tour Stop #1: Pair Exploring with Maaret

My testing tour has officially started! Wonder what's a testing tour? Then head over to my post about my personal development challenge for this year first.

For the first stop on my tour I consider myself really lucky and honored that Maaret Pyhäjärvi agreed to pair test with me. I admire her work in the community and was glad to meet her personally at last year's Agile Testing Days, where I could see her testing and critical thinking, coaching and feedback skills in action; to name only a few. With every day of our session coming closer, I grew more nervous about it; even though rationally I knew that this was an awesome opportunity to learn even more from her, and that I have nothing to lose. In hindsight, our session was absolutely worth it as I indeed learned a lot in a short period of time.


The Session

We both wanted to dive deeper on exploratory testing and practice our skills. Maaret proposed to tackle the Freemind version for Mac; according to her it's "buggy as hell and therefore really rewarding for newbies". This makes it a great system under test when teaching about testing (idea noted ;). To reduce the scope of our session we focused on the find & replace feature of the application.

As soon as we started testing, we discovered so many different issues together, it was really hard to keep track. All in all we felt the user experience was really poor and the features rather glued on top of each other without having a consistent vision in mind. We came to that result really fast together; and our findings proved it when we went further. Overall, having two testers explore a feature together produced great results in a short amount of time and thus proved really efficient.

The identified issues were spread all of the place. Some questions we asked ourselves several times:
  • What does this mean? What does this message try to tell us?
  • Why does this feature behave in a completely different way than similar features in comparable applications?
  • Why are things so inconsistent? Why do labels or icons imply different functionality than they actually offer? How come even a simple form shows different spacing between input fields, just a few pixels below each other?
  • Why is this data displayed in this location? It does not have anything to do with the feature. Why not put it in a different place where it might actually provide value?
  • Why would we need this data or feature in the first place?
  • Where is the functionality I am looking for? Why is it so hard to find things and learn this tool?
The funniest issue we found was that we could copy nodes of an older mindmap to a new file and the creation timestamp would not get updated for those nodes, thus showing a timestamp way before the birth of the current mindmap. This made me feel not only as testing traveler but also as time traveler! :D

What I Learned about Testing

Reflecting together on our testing session, we addressed several important topics. Here are my lessons learned about testing.
  • Where's our box? Who says what's in scope and what not? In this session we defined our box fluently on our way, calling each other out in case we started to divert from it. This was totally fine for us, but in the end we would have loved to have an overall picture of our box to see what we intentionally left out and postponed for further sessions.
  • You discover there are too many combinations requiring a lot of time effort? Check if you can automate your test data setup. We didn't go this way in this session, but this was one of the things we could do in a future one.
  • Whatever you document - always chose a notation that lets you quickly identify how many issues, questions, follow-up charters there are and what your feeling about the level of quality is.
  • During the session we talked a lot about different aspects of exploratory testing - without even mentioning the approach itself. Observation as a skill, heuristics, and more. It's quite easy to start with exploring, but it's always experience that brings you further.
  • Last but not least: testing and finding bugs is fun! I really love hands-on testing and close collaboration with people to get the best out of us.

What I Learned about Myself

No matter how I talk about things, I probably will be caught doing things differently than I said. Certain situations might trigger habits I am not aware of. Having someone else observing things was so helpful to point that out.

In this session this fact became obvious when it came to note taking. In the beginning of the session I told Maaret I'm just shortly noting down what we learn. To do so, I had a physical notebook right beside me and scribbled down on it as we proceeded. When reflecting on our session, Maaret asked me whether I had used a certain notation to quickly get a summary from my testing notes. I had not. Which stunned me. When I do testing solo, I always use a digital form for my notes, be it a comment on a ticket, a mind map, or anything; but always including a certain notation to quickly let me know how many issues I found, which questions are still open, what other things I postponed for further sessions, what should be automated, and so on. But this time I totally diverted from it.

So Maaret asked me why I had chosen a physical notebook in the first place this time? The best answer coming to my mind was that maybe I fell into a habit. As I only use a physical notebook in meetings, be them at the same location or online. Like in interview calls. Did I fell into a habit because of the situation I was in? Why did I divert from my normal testing behavior? After reflecting a bit further, my intentions were the following:
  • I told Maaret I was taking notes as I wanted to let her know what I'm currently doing as we couldn't see each other (we turned off video to improve call quality).
  • I wanted to roughly keep track of what we were doing to help our testing.
  • I wanted to have notes of our session to make it easier for me to blog about it later on; as this was part of my experiment.
Maaret pointed out how hard it was for her to not be able to see my notes, or see me changing my focus away from the application in order to take them. She tried to be mindful and not rush ahead and put me under pressure, but it was not an easy thing to do.

What I Learned about Pairing

My most important lesson here was: Be mindful! And keep the focus of both synchronized. The note taking example made that obvious. We could have just taken notes together, in the same moment, seeing the same screen, agreeing on the notes to take and the notation to chose - and everything would have been easy for both of us. The same would apply to draw models, or our box of what's in scope.

Also, we didn't really talk about how to do pairing. We both assumed we would do strong-style pairing, with one being the navigator and one the driver on the keyboard. We both share this knowledge and are used to this approach (probably Maaret more than me, but I made great experience with this style over the past year). However, in future sessions with other testers I have to set the stage first so we have a shared understanding how to pair.

This session was quite interesting with regards to pairing dynamics as the remote access tool to allow switching roles did not work. So we had to stick with our roles for the whole session, with me being the navigator and Maaret the driver, sharing her screen. It would really be interesting to observe our pairing dynamics further when we are able to switch roles, or with her being the only navigator.

What I Learned about This Experiment

It's absolutely worth it. My gut feeling is that I am on the right track here and it will actually help me become a better tester. In any case I'll be wiser than before; or let's say, more experienced.

There are also some things I have to improve. First of all, I have to prepare better for the next sessions. This first time I could rely on Maaret, as she chose the system under test and thought about how to collaborate during the session. Next time I should drive the technical setup myself, especially for the next online session. I really have to get a proper remote access tool allowing us to switch roles. To let the other person chose the topic to pair on, however, turned out to be a good thing; this way serendipity might be on my side and teach me more. I would dive into areas, applications, approaches I might not have chosen myself.

Where to Next?

Maaret gave me great food for thought. And not only that, she agreed to further pair testing sessions with me; a great chance to improve things and learn more from each other!


My next testing tour stops are still to be scheduled, but a few testers from inside and outside my company already agreed to pair with me. I am looking forward to each and every session with them. I'm really curious what I will learn next.

No comments:

Post a Comment