#SecurityStories: Ethical Hacking Courses Revisited

My first contact with security testing was back in 2016. My company offered us a Pluralsight account so we could benefit from their vast course catalog. As I had been inspired to learn more about security, this felt like the perfect match. I watched several of the security related courses offered on Pluralsight back then.

Four years later, Pluralsight granted everyone free access to their offer throughout April. This made me wonder: what if I revisited those courses with the security knowledge I have today? This felt like too good a chance to let go, and led me to the following hypothesis.

I believe that following parts of Pluralsight's ethical hacking courses will result in surprising knowledge and deepened understanding.
I know I'll have succeeded when I made a new connection of existing knowledge and realized that pieces of the puzzle were falling together.

What I remembered from 2016 was that these courses were worth it. Even though I had limited knowledge back then, they helped my gain a lot more awareness and insights into this vast area of expertise. Rewatching these courses now four years later, having a lot more security knowledge than before, was absolutely worth it as well. I found I had a better understanding these days, and I rediscovered aspects, techniques and tools I simply didn't memorize back then. If you have any chance to get a Pluralsight account (or make good use of the ten days trial) and you're in for learning more about security, these courses are top-notch in my eyes. Very informational, very well explained, able to follow also with limited previous knowledge - and you can also follow along hands-on if you want. This time I managed to watch the following courses, which represent about a third of the ones available.

• Ethical Hacking: Understanding Ethical Hacking by Dale Meredith
• Ethical Hacking: Reconnaissance/Footprinting by Dale Meredith
• Ethical Hacking: Scanning Networks by Dale Meredith
• Ethical Hacking: Hacking Web Applications by Troy Hunt
• Ethical Hacking: Hacking Web Servers by Troy Hunt
• Ethical Hacking: Penetration Testing by Dale Meredith

While I can't and don't want to spoil all the course content, there are several points that frequently came up. Pieces of knowledge that I (re-)learned, that re-established or created new connections in my brain, and that are now (hopefully) etched on my memory.

• It's hard to be an ethical hacker. 
• To be able to review systems and infrastructure from a security standpoint, to test the current solution, create better solutions, and retest them, you need a lot of knowledge and skills. You basically have to be an expert with operating systems, programs and networks, proficient with vulnerability research, mastering hacking techniques, have a lot of software knowledge in general, be able to think outside the box, have great communication and management skills, lots of patience - and more. This quote from Dale Meredith really fits well: "Practice builds knowledge, knowledge builds confidence."
• You have to follow a strict code of conduct. You need explicit permissions in writing before you can do anything. This includes your own employer! For practice, there are lots of intentionally vulnerable apps whose purpose it is to hone your skills. Yet whatever you find in real life, even by coincidence - report it. In addition, when it comes to penetration testing, a major part of the work consists of documentation. So document everything, report everything. Yet make sure to choose a secure medium to store findings, and a secure channel to report these findings. It's way too easy to do the job for the attacker and deliver all information on a silver plate.
• You can't stop attackers, so the job is not to stop them but to discourage them, misdirect them, and slow them down. Time is on the attacker's side, not the ethical hacker's. An attacker only needs to find one opening, while being on the ethical side of things you have to find all of them and also make sure they're covered.
• It's a lot about information gathering. Really, a lot.
• The so called reconnaissance phase is probably the biggest and most important in the endeavor to penetrate a system. There's so much to find out about applications, infrastructures, organizations, individuals, and more. Much of the information is just freely and publicly shared, completely legal to retrieve, and easily accessible for everyone. Just using a search engine like Google can reveal lots of vulnerabilities; especially when you know what to look for and how to feed the advanced search options. So many places can give valuable information to attackers, among them also your own website (job offers are a great source!) or what employees share on social networks. The horrifying thing: this is just the tip of the iceberg, and you can find a lot without investing much effort. 
• If attackers find interesting information, they might go further and start scanning your networks, i.e. looking for "live" systems and identifying them. Using a bunch of different scanning techniques they can discover what ports are open or closed, whether those systems are running any services, and more. They basically probe the target network to find out as much information as they can about the system. All this adds to what they already found during reconnaissance. Oh, and - we are probably being continuously scanned. Remember, time is on the side of attackers. Drawing out a network can help detect holes and remember them on the long run.
• Fingerprinting helps as well to identify further information. Operating systems usually behave in certain ways that let you make conclusions about the system. You can determine the host via sending well-crafted packets, or use banner grabbing to check for welcoming messages that already reveal information about the target system.
• When it comes to web applications - well, they reveal way too much information by nature already. You can see the whole frontend source code, all the JavaScript executed. If client-side security constructs are in place (which you shouldn't have by any means!), like password constraints, they are very easy to discover and work around. Browsers nowadays offer protection for several attacks. Still, there's a lot they simply cannot defend against, like parameter tampering (any input from client side is untrusted data!) or persistent cross-site scripting as then the malicious data is already in the database.
• Ignorance, laziness and misconfiguration are way too common and make things way too easy. How many times have we just copied over a solution we found on the internet? How many times have we just made use of a new framework without a thorough security review of its source code? How many times have we even considered that this could be exactly the reason for its existence? How many times have we just kept the default configuration for applications, frameworks or servers; not to mention default passwords? Well, we all know the answer. It's hard to accept the truth - and frightening at the same time, as we can assume how many other people building products probably are sharing these feelings.
• There is a plethora of tools out there to help all sides. As "plethora" is one of Dale Meredith's most favorite words, I simply had to include it in this post. But seriously, there's a tool for everything. Most of them are completely legal, as they also help for many other absolutely ethical and valid purposes. Yet as it is with any tool, they can be used for good and evil and all the shades of gray in between. Let's list some examples, yet be aware that they are not even scratching the tip of the iceberg. There are proxies like Burp Suite, OWASP ZAP or Fiddler. There are network tools like Nmap or netcat. There are website crawlers or copying tools like HTTrack or Netsparker. There is the Google Hacking Database or MetaGoofil for reconnaissance. When it comes to web apps, the browser's development tools might already be your best friend. To quote Dale Meredith once more: for each purpose, "pick a tool and learn it, love it, use it."
• Social engineering is way too easy. People are usually the weakest link. Convincing them to reveal information does require social skills, yet with enough confidence these kind of attacks are scarily often successful. From looking over someone's shoulder to following someone holding the door into the building. From searching your trash (yes they do) to impersonating internal IT. From phishing attacks to distressed calls for support. This makes you think of your own behavior a lot. I haven't even re-watched the whole course on social engineering, yet in all the other courses this technique was referred to at least once. In the end - it's still all about the people, and our education is crucial.
• Seemingly minor risks can be turned into full blown exploits. It's all about the context and how things can be connected. One information can help you to another, one exploit can lead to another. Again, time is on the side of the attacker. It's way too easy to discard an issue as too minor, not important, not revealing interesting information, simply not posing much risk. But - is it really? Let's not make this too easy.

(By the way, when reading all of the above - do you also see the similarities to testing in general?)

There's so much more I learned watching these courses. If you have the chance to check them out, I can only highly recommend them. I've only watched 26 hours of currently overall 79 hours of course material on the Ethical Hacking (CEH Prep 2018) path. I am eager to watch them all at some point. Some day I will.

All this really made me think even more about security in all areas. Not only when developing our application or interacting within an organization, yet also as an individual. In my eyes it's not about getting paranoid, but about stopping being careless. I wouldn't leave the door to my apartment wide open, either. That being said - I just revealed I'm living in an apartment. You never know what piece of information can help attackers. For example, I got a lot more cautious around sharing photos from my living areas on the internet; I wouldn't want to reveal my address there as well, and it's probably way to easy to conclude to it anyway. Well, doing a thorough check on my own behavior a well as the applications and infrastructure I'm using - that's definitely on my list as another experiment.

As always in this series of #SecurityStories: if you learned something new in the area of information security from this post, please let me know by leaving a comment or sending me a direct message on Twitter. Your feedback is much appreciated.

#SecurityStories: Threat Modeling

It's time to start writing about my personal challenge this year: telling #SecurityStories. My goal is to help people gain new insights when it comes to all things security; a very essential topic that's unfortunately often dismissed in favor of our own convenience. I have to admit, I dismissed security concerns way too often myself, and unfortunately I still catch myself doing so. I want to change this. Behavior change starts with awareness, and that's a big part of this story as well.

Let me share my experiences creating a threat model for the very first time on my own. It was Dan Billing who introduced me to threat modeling in his "Web Application Security" tutorial at Agile Testing Days 2018. The next time I heard about this approach was only last year at TestBash Manchester. Saskia Coplans gave a great talk about it (check it out my sketchnote or even better the full recording on the Dojo if you have a pro license by any chance) and we also did an example together during open space.

"Threat Modelling: How Software Survives in a Hacker's Universe" by @ms__chief #TestBash #SketchNotes pic.twitter.com/yajcQsOTj6

— Elisabeth Hocke (@lisihocke) October 4, 2019

The idea to start my challenge with a story about threat modeling came from one of the security testing sessions I had together with Peter Kofler beginning of January. He asked me if I knew anything about threat modeling and I shared with him what I learned at the conferences. To paraphrase him: "I already learned something from you: threat modeling and why it's important, why testers would like to learn it." Remember, the desired outcome for my personal challenge is that ten people have confirmed that they learned something new from me in the area of information security. You can imagine how happy I was to hear that feedback from Peter even before really starting out on this journey.

Today:
🎉 achieved a first step on my #SecurityStories challenge without even aiming for it
😊 learned that giving personal feedback can be rewarding
😍 just loved working with 2 amazing developers, sharing our core beliefs regarding testing & quality and crafting our common set

— Elisabeth Hocke (@lisihocke) January 10, 2020

All this gave me the idea for my first #SecurityStories experiment. It took me some time until I could finally start it in March, yet it was a perfect match with an opportunity I had at work.

I believe that creating a threat model for our own product will result in applied knowledge and surprising findings.
I know I'll have succeeded when I discovered an unknown attack vector for our own product.

For a long time developing my team's product, security was not our biggest concern. After all, we are building an internal application and the little security testing we did was mainly focused around access control and permissions. Now I took the chance to do a more structured risk analysis when it comes to security by creating a threat model. Better late than never!

Building on the knowledge I gathered at conferences, I started with research and reading up on several resources. Here are the ones that helped me most when trying to understand the main steps of creating a threat model.

• Threat Model (Wikipedia)
• Threat Modeling (OWASP)
• Advanced Threat Modeling
• Threat Modeling: Getting from None to Done
• Threat Modeling with STRIDE
• Pushing Left, Like a Boss -Part 6: Threat Modelling
• Microsoft Threat Modeling Tool
• Elevation of Privilege (EoP) Threat Modeling Card Game

According to these resources, there are two important things to consider when starting out. First, it's strongly suggested to create the threat model with a group of diverse people to end up with a holistic picture: the whole development team, a business analyst, an architect, whoever adds a new perspective. My team, however, was suddenly finding themselves in crisis mode and fully focused on other topics. Therefore, I decided to start this learning journey on my own, creating a first model version, then involving a few people to refine it, presenting our findings to the whole team and continue refining as we go. Better starting imperfectly than not at all.

Second, the resources agreed that no tool itself is required to create a threat model, and a whiteboard might be the best medium for this group discussion. Over the last weeks, my team also had to learn how to work fully remote, full time. Therefore, I considered creating a digital version already in the first step. This would also help me adapt the model as I learned more without having to redraw it each time. I was curious if there would be tools specifically designed for threat modeling. I found several, yet most of them are not maintained anymore. Besides OWASP's Threat Dragon, which is still under development yet already offers a native client for both MacOS and Windows. I decided to give it a try and see if it would fulfill my needs. I wasn't disappointed. Using Threat Dragon did prove worth it. This application is great as an idea and already provides lots of features. Granted, it's still under development, and I could feel it. For example, usability is not yet too great. I'd love to have the option to add comments or other descriptive text fields to the model. I'd love to be able to select multiple diagram items and move their position all together. Still, I haven't regretted this, and I especially appreciate the nice report this tool provides.

Let me be clear about it. I created a threat model for the very first time, and I very well might have gotten things wrong, or done in a way that's not recommended. As it still proved to be an exercise very well worth its time, I'm just going to tell you what I did and how things went. Due to the nature of the thing, I cannot show you my results - yet OWASP provides several examples in their Threat Model Cookbook so you can get an idea of what I'm talking about.

Why threat modeling and how does it work?

Threat modeling is a structured way to brainstorm about threats. It's important to consider as many factors as we can think of from diverse perspectives to get a holistic view. For example, we also need to consider malicious acts from within the company, or simple human error. Keep in mind that a model is never correct as it does not represent the full reality, yet models help us think - in this case about security.

Security is a quality attribute that needs to be built into the system from the start. We want to improve and include security concerns early on into how we develop, test and run our services. On the one hand it's about making our systems less prone to human error, on the other hand it's about not leaving the door wide open and make things hard enough for attackers to become less attractive as target. When it comes to the latter, some might think that our internal product does not provide valuable information, that there are more attractive targets in the company; yet we simply cannot tell for sure what is valuable and what not. For example, we could grant people access to other systems in the company through our services without being aware of it, our server resources could be misused for other activities, or we could lose all our data by mistake.

From what I learned, the threat modeling process is basically about the following questions and steps.

1. What are we working on?
2. What can go wrong?
3. What are we going to do about it?
4. Did we do a good job?

What are we working on?

As we cannot easily tell how attackers think or what's valuable to protect, the most common approach is to have a look at what we are building and identify the data flow between the different parts. To create such a data flow diagram, we used the following common notation.

• Process (circle): Our services and code.
• Store (two parallel horizontal lines): Data stores (e.g. files, databases, shared memory, message queues).
• Actor (rectangle): External entities, everything but our code and data. This includes people and cloud software.
• Data flow (pointed arrow): Connects processes to other elements.
• Trust boundary (dotted line): Indicates where the trust level changes (e.g. from internet to intranet through a firewall, from web server to database, from our application to an external third party service). There is always a trust boundary when your code goes to someone else’s, or their data comes to your code. We perform security checks only inside our trust boundary.

Before starting my first version of the diagram, I looked up earlier architecture drawings we had created to get inspired. I then started listing assets like stores, actors and processes. Doing so, I had to become clear about above standard notation. What is used for what again, which category would fall that one in? Looking at the threat model examples helped. I quickly noticed that this is very iterative work. I jotted down everything that came into my mind: the services we own, the integrations we have to other systems, the trust boundaries we operate in.

As with all kind of visualization techniques, I realized once more the power of modeling. It really helps you think! Just doing this was a great exercise also for testing and quality in general, not only security focused. It really helped to get the overall picture again; especially as our product landscape and its complexity grew heavily over the last five years. Just a little internal application? Far from that with all infrastructure included and so many things to consider. I realized a thorough model would really take time, yet each step on this endeavor was so much worth it. I got to know our system better again, realized more clearly how things are connected and where they could break, and created a better understanding of our architecture components and how traffic actually goes through them. It was great to have that model not only in my head, but visually "on paper" so I could align it with my teammates and discover any discrepancies or unknown risks.

With more and more components added, the diagram grew and grew. It slowly felt like an overwhelming task. I wondered how much detail should be added to the data flow? Where was it okay to simplify the model, where would exactness help? It's a model after all and it's main purpose is not to depict absolute reality but help us think. Still, it's not easy to decide whether I should add all kind of requests and the used protocols, or abstract this. Should I add the kind of data flowing, down to single entities? That would let the diagram explode. Maybe rather keep the flow generic? In the end, I just decided on one way as a proposal and left the rest for future iterations.

What can go wrong?

Based on the diagram, I started to identify threats. To do so, STRIDE is the most common approach for threat modeling. I've found a good overview on STRIDE that I'll copy here to get us on a shared page.

• Spoofing
• Property Violated: Authentication
• Definition: Impersonating something or someone else.
• Example: Pretending to be any of Bill Gates, Paypal.com or ntdll.dll
• Tampering
• Property Violated: Integrity
• Definition: Modifying data or code
• Example: Modifying a DLL on disk or DVD, or a packet as it traverses the network
• Repudiation
• Property Violated: Non-repudiation
• Definition: Claiming to have not performed an action.
• Example: “I didn’t send that email,” “I didn’t modify that file,” “I certainly didn’t visit that web site, dear!”
• Information disclosure
• Property Violated: Confidentiality
• Definition: Exposing information to someone not authorized to see it
• Example: Allowing someone to read the Windows source code; publishing a list of customers to a web site.
• Denial of service
• Property Violated: Availability
• Definition: Deny or degrade service to users
• Example: Crashing Windows or a web site, sending a packet and absorbing seconds of CPU time, or routing packets into a black hole.
• Elevation of privilege
• Property Violated: Authorization
• Definition: Gain capabilities without proper authorization
• Example: Allowing a remote internet user to run commands is the classic example, but going from a limited user to admin is also EoP.

While I was still extending the data flow diagram, I was already taking note of any threats that came to my mind. All this was quite an iterative process, while continuously learning more. I heard several people describe threat modeling as dull - yet I found it quite interesting, there's a lot to discover using this structured way to think about these kinds of risks! When pondering on the diverse potential threats and ways to mitigate them I learned a lot about them. My biggest question in this step was how many threats to list? Only those I know are critical, or all I can think of? In the end I decided to go for the latter in favor of a more holistic picture, while being well aware there would be a lot more that I am even not aware of yet.

Now was the time to involve my team and get initial feedback on the model. I invited my colleague who knows our infrastructure best for a call and walked him through what I've done so far. I asked him to double-check the diagram and the initially brainstormed threats. Admittedly, I was quite anxious to hear his feedback. You can imagine my relief when he confirmed that I had created a model that fit to both our shared mental models, with only a few minor adjustments that I gladly worked into the diagram. He shared with me that he has never seen a threat model before and found it very useful to think about risks. What a great entry point for risk discussions indeed!

Step by step I went through all of the resources again and continuously extended the model, adding anything I missed. With each iteration I discovered something new, something else to consider, another potential threat. I have a lot more ideas what to check for and I am far from finished refining the model. There's so much more to look into, like checking the tech stack we use for known vulnerabilities, cross-checking with the OWASP Top 10 security risks, considering social engineering attacks, and more.

What are we going to do about it?

Identifying threats is not enough. We also need to decide what to do about them. For each individual threat, we have the following options at hand. 

• Remove the threat (e.g. removing the respective functionality)
• Mitigate the threat (e.g. through standard practices like encryption)
• Accept the threat (be careful about “accepting” risk for your customers)
• Transfer the threat (e.g. via license agreements or terms of service)

For each threat I was aware of, I made a first assessment, or rather educated guess. I was happy to involve our product owner in this step, presenting him the whole picture. He was intrigued seeing our whole service landscape visualized and recognized that it grew a lot more over the years than he had realized. He asked further valuable questions, and also helped assess the threats from his point of view.

At this point I decided it was time to document the current state in our wiki and invite the team for a presentation. I wanted to get everyone introduced to threat modeling and our current model version, including all assumptions it was built on. We had a short session, and promptly I got further invaluable feedback! More pairs of eyes instantly caught what we missed before, and also detected a flaw in the visualized data flow. Perfect input to refine our model further.

The more I learn, the more I know what I don't know yet. There are so many more things to think about, yet having this model is a great discussion base. We're far from done - yet a big step further.

Did we do a good job?

Finally, we need to validate that the identified threats have actually been handled.

We haven't done this yet for all identified threats. There's still a lot more to do indeed. Yet again, the effort already proved worth it. I know a lot better what to look for, also with each new change we're implementing. My hope is, that my team does know that better now as well, and we all use this increased awareness to find good solutions together.

A Living Model

In the end, our threat model is supposed to be a living document. As our socio-technical system changes, this model will change as well. There are several triggers for a revision, like the following examples.

• We develop a new service or remove a previous one.
• The architecture of one of our services changes.
• We introduce a new technology.
• The infrastructure conditions change.
• The knowledge and skills of our development team grow.
• External actors and their interactions with our product change.

I'm curious how threat modeling will help us in the future, as it already helped us in the present. Our awareness increased, we can make more informed decisions together when it comes to security. That's a big step for us indeed. When it comes to anything else, only time will tell.

All this was based on an experiment. Could I prove my hypothesis and identify a previously unknown attack vector for our product through threat modeling? The answer is clear: yes, and more than one. We have work to do.

One more question remains. Having read this story, have you learned something new around information security that you weren't aware of before? If so, please leave a comment or write me a direct message on Twitter. Have fun with threat modeling!

Testing Tour Stop #25: Pair Penetration Testing with Peter

Just a few days before my testing tour is coming to an official end, I had my final stop with Peter Kofler.

Just had the final stop on my #TestingTour together with @codecopkofler. We had our third session on penetration and security testing and it was a great ending to a fantastic tour! 🎉

— Elisabeth Hocke (@lisihocke) October 29, 2018

We had two sessions on my tour already together which makes Peter the only one I had the pleasure of pairing with three times. In all three sessions we tackled different security testing challenges, each time using Dan Billing’s Ticket Magpie as our target. The first time we focused on exploring the application using SQL injection. The second time we tried automating SQL injections. This time we planned to have a closer look at cross-site scripting (XSS) and see what fake content we could place.

Starting Out

In the beginning of our session, we looked for potential input fields where we could give cross-site scripting a try. We registered as new user and found a comment field on each offered concert. We added a comment containing an HTML tag to see whether this was allowed, and it was. We went ahead and added a simple script that should show up an alert. Interestingly, the Chrome browser considered this a potential cross-site scripting attempt and blocked it showing the error code "ERR_BLOCKED_BY_XSS_AUDITOR". Navigating again to the concerned page, however, indeed showed us the desired alert.

As we had already proven that the application was vulnerable to cross-site scripting, we considered to try out actual scenarios. What if we changed the link of the button to book a concert to lure the user to a different site? We used our JavaScript knowledge and tried out how to target the desired elements in Chrome's Dev Tools console until we had the right command. We added the script in our comment and this worked just fine!

What about posting comments in the name of another user? This was even easier than considered, we found the user name to be a hidden field of the form so we could simply change it using DevTools before submitting the comment. Tampering with web parameters was not our goal, so we decided to inject a script to change already existing user names. We found out how to select the desired elements to change, submitted our comment including the script, and all targeted user names got replaced. However, we found it did not work when adding a new comment in the name of the targeted user. Right, the script had already been executed at that point in time. So we changed the implementation to be executed only when the DOM content had been fully loaded. It still didn't work, but why? The console showed us that an error was thrown. What was wrong with our script? After spending some time on debugging we realized it might just be a copy and paste error, we might have copied some non-ASCII text from a website sample. We sanitized our code - and now it worked!

All this was surprisingly easy, so we still had some time left which we spent on brainstorming what else we might try on our current target practice application or what we in general would want to learn more about.

• Cross-site scripting attack tools like XSS-Proxy
• Denial of service (DOS)
• Remote code execution
• XML External Entity (XXE) Processing (for which would need an interface)
• Session hijacking
• Cross-site request forgery (CSRF)

One of the best resources to get an overview on the most critical web application security risks alongside further resources are still the OWASP Top 10.

Looking Back

We were really successful with our attempts. Frustration was kept at a minimum, not everything worked at once but we made it work together. This was great for learning. However, it felt quite easy as well. This might have been due to the fact that both Peter and me complemented each other very well regarding our knowledge. Peter contributed most of the JavaScript knowledge where I lacked a lot of practice, and I was fluent with the Chrome DevTools that he never used as pure backend developer.

Time flew by and collaboration was smooth. The only thing we noticed: it was not easy for me to not see Peter's screen. I could not see when he shifted his focus away from the shared screen to his screen in order to research useful resources. Though this style had worked in previous sessions for us, we considered we would try researching together and sharing screen control in a future session. Would we have a future session? Well, the whole area of security testing is still really interesting for both of us so we might go deeper together on this topic outside of my testing tour.

This was the last stop on my testing tour in 2018. I aimed for ten pair testing sessions in ten months and ended up with 25. I am still amazed by this awesome learning journey together with so many amazing people. My final task to conclude the tour is to take time and reflect on it as whole, so stay tuned!

Testing Tour Stop #15: Pair Evolving a Test Strategy with Toyer

The testing tour continues. Today I had the honor to pair test with Toyer Mamoojee. Since end of 2016, when we agreed on our first pact to try ourselves as conference speakers, we are having a call once every two weeks. We talk about all things testing, exchange experience, trigger thoughts, provide feedback and support. Still, we haven't ever tested a product hands-on together. You can imagine how happy I was when Toyer took the chance and scheduled a pair testing session with me on my testing tour!

It's only a pleasure Lisi, and glad I could be part of you amazing Testing Tour. I learned many aspects today which I will use going forward regarding approach and things to keep in mind. I really enjoyed pairing with you on this 1. #accountabilityBuddy #TestingTour

— Toyer M (@tottiLFC) 17. Juli 2018

What topic to pair on?

That's one of the common first questions that come up after scheduling a session. In this case, Toyer said he would love to do something from scratch, gather test ideas together, align our thinking. We know each other and our viewpoints quite well, but we haven't yet practiced testing together. He wanted to see how we really go about testing certain things, ask questions and see each other's thought processes when actually testing.

Based on his input, I thought let's tackle an application we both don't know, explore it and come up with a very first test strategy based on the gathered knowledge. As I knew that Toyer discovered mind maps for his work some time ago and learned to love them for many purposes, I thought that this could be our way to document our very first strategy draft, keeping it lightweight, easily editable, visual.

Having a list of potential applications to tackle at hand, I reached out to Toyer and asked whether he wanted to learn more details about my idea before our session, or rather not. He answered "I'm tempted to say share more information.. but I would like to be surprised too, as I want to see how I can tackle something without preparing". So he chose the latter option to which I can really relate. Over the last years I am continuously learning to tackle things without over-thinking; and I'm not done with learning this yet.

Evolving a Strategy While Exploring

At the beginning of our session I presented my topic idea to come up with a test strategy for a new product, and Toyer agreed to go for it. Lucky me, otherwise we both would have had to cope with an unprepared situation! ;-) So I provided different options as our system under test of which Toyer chose Blender, an open source 3D creation tool. I had some rare encounters with this application back at my first company when we developed an AI middleware for game developers, but had hardly touched it ever since. Toyer thought it looked really promising as we normally don't get to test these kinds of applications.

Toyer shared that first of all, he would ask which kind of need this application wants to fulfill and do related research upfront. For the limited time box of our session, however, we decided to skip this and explore ahead. Toyer accepted my suggestion to draft our strategy in a mind map, so we created it and continuously grew it according to our findings. He also agreed to do strong-style pairing while exploring, so I started up my favorite mob timer, set the rotation to four minutes, and off we went. It quickly became clear that we knew each other well. Collaboration was easy and communication fluent. We could fully focus on exploring Blender from a high level point of view, trying to grasp its purpose and main capabilities, identifying limitations and potential risks. We were actually doing a recon session, just as Elisabeth Hendrickson describes in her awesome book Explore It! Reduce Risk and Increase Confidence with Exploratory Testing.

Throughout our session we gathered lots and lots of findings and discoveries, adding more and more important points to our test strategy.

• Learnability. The application is not intuitive at all. It's a real expert tool. Still, everybody is a first-time user once, and even if you know the domain the user experience this product offers is not that great.
• Functional scope. The more we explored, the more functionality we discovered. The whole tool seems really powerful, but again, is not easy to understand.
• Screen resolution. The GUI is cluttered with many UI elements, sidebars, popups and more. On our laptop screen that was already a challenge, and it will still be one on larger screens.
• Usability.
• Menus, popups and tooltips looked very similar which made it hard to distinguish the purpose of each.
• Feedback on actions was often missing or confusing.
• Some sidebars displayed content related to views we previously visited, not getting updated with information of the current view. This way, they sometimes obscured needed information.
• Consistency.
• Some actions worked in one area but not in the same way in another.
• Some sidebars were named x but the header label said y.
• Some delete actions asked for confirmation, others just instantly deleted the item.
• Portability. We tested Blender on macOS. The product is also offered for Windows and Linux. At several points we found strange unexpected behavior and made the assumption that it might have been due to porting issues for the macOS version. For some points, I could even confirm that assumption when writing this blog post and checking the Windows version of Blender.
• Maintainability and reusability. The GUI offered many sidebars, popups and views that shared similar layouts and menus. We noted to investigate whether they were duplicated or re-used components.
• Robustness. We encountered error messages on invalid input that was not caught or prevented.
• Automatability and testability. The application offers a Python API. We found Python commands offered in tooltips, the API reference in the help menu and an integrated Python console. The console behaved differently than we knew it form other terminals, but still it was very interesting that you could automate operations; which would also increase the product's testability.
• Discoverability and documentation. The help menu offered an operator cheat sheet; clicking on it triggered a temporary message to look at the OperatorList.txt which we could not find. Only later I learned that we had not come across the text editor where you could open named text file. What a hidden support feature. Also, we found the linked release notes page to be empty. We didn't dive deeper into the manual, but all available documentation would have to be tested as well, especially for an expert tool like this.

All in all, we made our way through many parts of the application. We made quite some assumptions. And we found we still haven't seen a lot yet. In the end, we didn't have a final test strategy, but a good starting point to be iterated over.

Time to Reflect

We covered a lot in the limited time. We gathered lots of insights, ideas, assumptions to verify. We tested a product in a domain we both don't know much about, being a desktop application instead of our usual web applications. We tried to gather information and keep a holistic view on things, not diving deep yet, focusing on uncovering the different aspects to test for such kind of a tool. All the way mapping out our world and the points to tackle in our test strategy. As we learned more, our strategy evolved. We didn't reach an end by far. If this would be our product to test, we would iterate over it while learning more.

The unknown domain had its own charm. We approached the product as a black box, not looking under the hood in the first place. We brought lots of testing knowledge, but quickly saw we lacked the domain knowledge. Toyer made an important point here: when hiring a tester for such kind of a product, it would be best to look for someone who was already exposed to these kind of tools or related areas of expertise. We could still provide lots of value and ask questions which might go unasked otherwise, but we would quickly pair up with a product person or business analyst to model the product from domain point of view. And also sit with developers to model the product's architecture.

Pairing up helped once again a lot. To see different things at the same time, by looking at different parts of the screen. To grow the mind map way faster than any of us would have done on their own. And to include different thoughts and viewpoints.

Enrich Your Experience

This was the fifteenth stop on my testing tour so far. In the beginning I had only planned ten sessions, one per month from beginning of this year until end of October. Although I already overachieved my initial goal, each further session enriched my personal experience and brought me in contact with different approaches to learn from; all that while practicing my skills hands-on. Right now, I am reflecting on my whole journey so far as I am crafting a talk about my experiences on this tour which I have the honor to give at CAST and SwanseaCon this year. And just while doing so, another tour stop was scheduled with me, further people indicated interest to pair up or listen to my lessons, and I'm having further test sessions with awesome people. I'm curious where else this will lead me. What a wonderful time.