Testing Tour Stop #23: Pair Automating with Marianne

When Marianne Duijst scheduled a pair testing session with me, I was super excited. The last time we met each other in real life was at CAST, and it was a real pleasure. Just as this testing tour stop!

Preparation Phase

Both Marianne and I love to explore, however, for our learning session we chose to practice automation together so we both can improve in this area. We wanted to have a basic setup to start from so we both looked for projects that suited our needs. We considered especially the following options.

• The recently introduced Automation in Testing Online application, based on the restful booker platform provided by Mark Winteringham, as practice app for automation of all kinds.
• The BDD Trader application provided by John Ferguson Smart, as demonstration for BDD and test automation techniques for Rest APIs with Cucumber, Serenity BDD and the Screenplay pattern, including exercises.
• Another test automation tutorial with Cucumber, Serenity BDD and the Screenplay pattern, this time using Selenium WebDriver to automate scenarios through the UI, provided by Vamsi Krishna Darbhamulla.

Before our session, I had a closer look on each of those options. I got all running and checked whether I could extend them without too much hassle in an easy way. Getting back to Marianne with my results, we decided to give the BDD Trader application a try, as testing against an API was something that appealed to both of us. Marianne found that all the courses and workshops usually work on UI level, not on API. As she is coming from data warehouse testing she felt this would be fun and educational to do.

This was so cool. Such a great way to learn & experiment together. Getting it to finally work & understand why felt awesome! Combining technical skills & domain knowledge for the win. #lisiisfabulous #learningtogether #challengeyourself

— Marianne Duijst (@marianneduijst) October 16, 2018

Combining Technical Skills and Domain Knowledge for the Win

At the beginning of our session I introduced Marianne to what I already knew about the BDD Trader application. Our starting point was the project's readme which included explanations about the application domain. In a nutshell, the product allowed users to buy and sell shares using real market data.

As we had agreed in advance, we paired the strong-style way, using my usual suspect of a mob timer. We started the application and verified it was running. We decided to follow the provided exercises, allowing us to get to know the application and already existing tests.

The first part of our session was mainly taking instructions, copying and pasting code. This task was not challenging, however, it gave us time to familiarize ourselves with a few things.

• We got an overview of the application and a first set of its capabilities.
• We gained a shared understanding of the diverse knowledge and experience we bring in to the pair as persons. I could introduce Marianne to some technical and tooling topics like IntelliJ as IDE or Cucumber. Marianne, on the other hand, had worked in the financial sector for banks and could introduce me to the domain of trading stocks I knew so little about.
• We found how to setup our working environment to level our playing field. Marianne shared that she was color-blind, so the IDE coloring problems like missing class imports in red font was not helpful. Therefore we adjusted IntelliJ's colors for red-green color vision deficiency which at least improved the situation. It's easy to forget we don't all share the same access to technology while it's super important to make it as easy as possible for all of us.
• We learned how the scenarios had been put together, how to make them pass and how to make them fail as well, which was just as crucial as getting them running.

While following the exercises, we found two issues with the provided documentation.

1. One code snippet contained numbers like "(1)" that were used as references to explain what was happening at each step further on in the documentation. However, these should have been commented out to allow easier copying and pasting.
2. Two other code snippets included a method expecting a parameter of type Long, but when the method was called an Integer was passed.

Both issues easy to fix, however, they made us realize we had fallen into the copy and paste trap. We had not given one thought whether we should automate the provided scenarios at all, where the risks of the product really were, and so on; things that you would normally start with. While noting this, we accepted it for the sake of our learning purpose for this session.

When we tackled the task of implementing variations on our current scenarios, it got really interesting on a completely different level. Now our thinking skills were in high demand. By adding our own scenarios and adapting their parameters we explored not only how the tests were implemented, but also how the product behaved. For example, we found that we were not allowed to spend more than we had in our account; and kept in mind that we would add a test for this.

One scenario caused us quite some headaches. We wanted to verify the product behavior when a registered user made losses on multiple shares. At first we questioned what was really meant by profit here. We tried different parameter values, and the test failed. We noticed we had forgotten to adapt further parameters and changed those as well. However, the scenario kept failing. We commented out all other scenarios of the feature just to be sure there was no dependency. This way our focus fell on the background for all scenarios, providing us details about the test data setup we had missed before. We struggled with identifying the correct data in the data table we used, so we provided the expected values from the error message output, without understanding why - and the test passed. Only then we finally saw which pieces of the jigsaw puzzle we had missed and realized why they were - of course - needed!

Ending Happy

We decided to stop here in a green state and reflect. I felt this whole setup mimicked the case that you join a new team and look at an already existing product with already existing test automation. We were trying to figure out what we saw. We were sharing knowledge about the tooling used. We were aiming to get on common ground regarding the domain knowledge required. And in the end we knew why! Alone we would have sat a lot longer, asking ourselves this question again and again, from different angles. Combining our knowledge was the key here.

Marianne shared it was great to experiment in this way. She felt she was still at a position where coping with all new things at once would have been too overwhelming for her: the IDE, the provided code, the libraries, etc. But she really liked thinking about the logical problem. We made a perfect match here, as from time to time I felt so dumb, having a hard time understanding financial domain topics although I felt I should easily grasp those concepts. Marianne caught me here before I got too frustrated with my own lacking. Where Marianne was happy to get navigated through technical details and tools and having someone else generating ideas from this perspective, I was super happy to have someone being able to make informed assumptions about the domain and the related product behavior that we could verify.

Sometimes people having expertise in certain areas, be it technical or domain oriented or anything else, are tempted to rush ahead and inevitably lose pairing partners that "don't get it"(yet). This is where strong-style pairing can make such a difference. Using this approach we have to express our thoughts and ideas, we navigate each other through parts we don't know yet, and we have to explain concepts otherwise the other one would not be able to navigate next. Doing it together, we both got farther than we would have on our own and learned from each other at the same time. To say it with Marianne's words: "This was so cool."

We ended happy, and I'm even happier to meet Marianne again in person at Agile Testing Days in just a few weeks. Her awesome "Sketchnoting Adventures" workshop is definitely on my list of favorites, and I heard there's still a chance to get yourself one step out of your own comfort zone and give an agile unicorn talk!

So excited for Agile Testing Days @AgileTD in less than a month! Made sure my #sketchnote pens still work and that I'm ready to meet all you fabulous people there. The programme is MASSIVE. Which sessions are you eagerly awaiting? #AgileTD #tiredhands #funchallenges #stepbystep pic.twitter.com/EJi8KQz4T2

— Marianne Duijst (@marianneduijst) October 14, 2018

Testing Tour Stop #15: Pair Evolving a Test Strategy with Toyer

The testing tour continues. Today I had the honor to pair test with Toyer Mamoojee. Since end of 2016, when we agreed on our first pact to try ourselves as conference speakers, we are having a call once every two weeks. We talk about all things testing, exchange experience, trigger thoughts, provide feedback and support. Still, we haven't ever tested a product hands-on together. You can imagine how happy I was when Toyer took the chance and scheduled a pair testing session with me on my testing tour!

It's only a pleasure Lisi, and glad I could be part of you amazing Testing Tour. I learned many aspects today which I will use going forward regarding approach and things to keep in mind. I really enjoyed pairing with you on this 1. #accountabilityBuddy #TestingTour

— Toyer M (@tottiLFC) 17. Juli 2018

What topic to pair on?

That's one of the common first questions that come up after scheduling a session. In this case, Toyer said he would love to do something from scratch, gather test ideas together, align our thinking. We know each other and our viewpoints quite well, but we haven't yet practiced testing together. He wanted to see how we really go about testing certain things, ask questions and see each other's thought processes when actually testing.

Based on his input, I thought let's tackle an application we both don't know, explore it and come up with a very first test strategy based on the gathered knowledge. As I knew that Toyer discovered mind maps for his work some time ago and learned to love them for many purposes, I thought that this could be our way to document our very first strategy draft, keeping it lightweight, easily editable, visual.

Having a list of potential applications to tackle at hand, I reached out to Toyer and asked whether he wanted to learn more details about my idea before our session, or rather not. He answered "I'm tempted to say share more information.. but I would like to be surprised too, as I want to see how I can tackle something without preparing". So he chose the latter option to which I can really relate. Over the last years I am continuously learning to tackle things without over-thinking; and I'm not done with learning this yet.

Evolving a Strategy While Exploring

At the beginning of our session I presented my topic idea to come up with a test strategy for a new product, and Toyer agreed to go for it. Lucky me, otherwise we both would have had to cope with an unprepared situation! ;-) So I provided different options as our system under test of which Toyer chose Blender, an open source 3D creation tool. I had some rare encounters with this application back at my first company when we developed an AI middleware for game developers, but had hardly touched it ever since. Toyer thought it looked really promising as we normally don't get to test these kinds of applications.

Toyer shared that first of all, he would ask which kind of need this application wants to fulfill and do related research upfront. For the limited time box of our session, however, we decided to skip this and explore ahead. Toyer accepted my suggestion to draft our strategy in a mind map, so we created it and continuously grew it according to our findings. He also agreed to do strong-style pairing while exploring, so I started up my favorite mob timer, set the rotation to four minutes, and off we went. It quickly became clear that we knew each other well. Collaboration was easy and communication fluent. We could fully focus on exploring Blender from a high level point of view, trying to grasp its purpose and main capabilities, identifying limitations and potential risks. We were actually doing a recon session, just as Elisabeth Hendrickson describes in her awesome book Explore It! Reduce Risk and Increase Confidence with Exploratory Testing.

Throughout our session we gathered lots and lots of findings and discoveries, adding more and more important points to our test strategy.

• Learnability. The application is not intuitive at all. It's a real expert tool. Still, everybody is a first-time user once, and even if you know the domain the user experience this product offers is not that great.
• Functional scope. The more we explored, the more functionality we discovered. The whole tool seems really powerful, but again, is not easy to understand.
• Screen resolution. The GUI is cluttered with many UI elements, sidebars, popups and more. On our laptop screen that was already a challenge, and it will still be one on larger screens.
• Usability.
• Menus, popups and tooltips looked very similar which made it hard to distinguish the purpose of each.
• Feedback on actions was often missing or confusing.
• Some sidebars displayed content related to views we previously visited, not getting updated with information of the current view. This way, they sometimes obscured needed information.
• Consistency.
• Some actions worked in one area but not in the same way in another.
• Some sidebars were named x but the header label said y.
• Some delete actions asked for confirmation, others just instantly deleted the item.
• Portability. We tested Blender on macOS. The product is also offered for Windows and Linux. At several points we found strange unexpected behavior and made the assumption that it might have been due to porting issues for the macOS version. For some points, I could even confirm that assumption when writing this blog post and checking the Windows version of Blender.
• Maintainability and reusability. The GUI offered many sidebars, popups and views that shared similar layouts and menus. We noted to investigate whether they were duplicated or re-used components.
• Robustness. We encountered error messages on invalid input that was not caught or prevented.
• Automatability and testability. The application offers a Python API. We found Python commands offered in tooltips, the API reference in the help menu and an integrated Python console. The console behaved differently than we knew it form other terminals, but still it was very interesting that you could automate operations; which would also increase the product's testability.
• Discoverability and documentation. The help menu offered an operator cheat sheet; clicking on it triggered a temporary message to look at the OperatorList.txt which we could not find. Only later I learned that we had not come across the text editor where you could open named text file. What a hidden support feature. Also, we found the linked release notes page to be empty. We didn't dive deeper into the manual, but all available documentation would have to be tested as well, especially for an expert tool like this.

All in all, we made our way through many parts of the application. We made quite some assumptions. And we found we still haven't seen a lot yet. In the end, we didn't have a final test strategy, but a good starting point to be iterated over.

Time to Reflect

We covered a lot in the limited time. We gathered lots of insights, ideas, assumptions to verify. We tested a product in a domain we both don't know much about, being a desktop application instead of our usual web applications. We tried to gather information and keep a holistic view on things, not diving deep yet, focusing on uncovering the different aspects to test for such kind of a tool. All the way mapping out our world and the points to tackle in our test strategy. As we learned more, our strategy evolved. We didn't reach an end by far. If this would be our product to test, we would iterate over it while learning more.

The unknown domain had its own charm. We approached the product as a black box, not looking under the hood in the first place. We brought lots of testing knowledge, but quickly saw we lacked the domain knowledge. Toyer made an important point here: when hiring a tester for such kind of a product, it would be best to look for someone who was already exposed to these kind of tools or related areas of expertise. We could still provide lots of value and ask questions which might go unasked otherwise, but we would quickly pair up with a product person or business analyst to model the product from domain point of view. And also sit with developers to model the product's architecture.

Pairing up helped once again a lot. To see different things at the same time, by looking at different parts of the screen. To grow the mind map way faster than any of us would have done on their own. And to include different thoughts and viewpoints.

Enrich Your Experience

This was the fifteenth stop on my testing tour so far. In the beginning I had only planned ten sessions, one per month from beginning of this year until end of October. Although I already overachieved my initial goal, each further session enriched my personal experience and brought me in contact with different approaches to learn from; all that while practicing my skills hands-on. Right now, I am reflecting on my whole journey so far as I am crafting a talk about my experiences on this tour which I have the honor to give at CAST and SwanseaCon this year. And just while doing so, another tour stop was scheduled with me, further people indicated interest to pair up or listen to my lessons, and I'm having further test sessions with awesome people. I'm curious where else this will lead me. What a wonderful time.