Monday, June 18, 2018

Testing Tour Stop #13: Pair Penetration Testing with Peter

Today I stopped by Peter Kofler again on my testing tour, picking up our penetration testing endeavors from our first pairing session. We agreed to stay with Dan Billing’s Ticket Magpie as our target system. Last time we left with multiple options how to continue: use tools for automation, or try another attack like cross-site scripting, or explore the source code for vulnerabilities. Peter had come across sqlmap, an "automatic SQL injection and database takeover tool". It looked promising so we decided to give it a try and focused our pairing session on it.

We started out by getting sqlmap to work, installing the required Python version and cloning the GitHub repository. This went nice and easy, without any issues.

Next step was getting to know the tool and figuring out the many options it provides. We already learned last time that the shop's login form was vulnerable to SQL injection, so we used this as our target. From this starting point, we learned step by step what we needed to provide and what not, what single parameters do, how to see what the tool is exactly trying out.

It started out easy - and then it stopped being easy. The nice learning curve we had in the beginning started to flatten out. Whatever we tried, the tool always told us that the login form does not seem to be prone to SQL injection. But why? As humans we could see that our target provided us the information needed to identify existing user accounts. Somehow the tool did not recognize this. Or rather: we did not find out what the tool was missing to be able to recognize it.

In the end, we closed our session with the given result and mixed feelings. We were annoyed we couldn't finish successfully, it was a real pity. We still enjoyed our session, however, and learned a bunch again. It looked easy in the beginning, but then we stumbled and left the session with lots of question marks, already thinking about how to continue. The route of going deeper and investigating more might lead us to the solution, or not any further at all. We won't know before we try it.

What went really smooth was our collaboration. We used the Pomodoros technique this time, breaking down our 90 minutes session into short intervals of 25 minutes, skipping the breaks. This way we had several checkpoints to decide how to proceed, making sure we always stayed aligned. Also, we instantly applied our own pairing style we defined in our last session. I shared my screen, mostly we worked together on it, sometimes we separated our focus to research simultaneously. We used our own shared Google document from the beginning this time to take notes we both could see.

Compared to our first session, I felt that collaboration went easier as we already had worked together, got to know each other a bit and had those basic working agreements to build on. Although this second session was not as successful as the first, we both shared the opinion that pairing itself is invaluable to generate ideas what the problem could be and what to try next to solve it. Pairing challenges our own understanding, creating a shared one. Pairing is the learning time we don't want to miss.

Friday, June 8, 2018

Testing Tour Stop #12: Pair Exploring App Development with João

Recently, I've been at a lot of stops on my testing tour. Here's the summary of another one, and there are more still to come! This time, I had the honor to pair with João Proença. João was one of those people who got inspired by Toyer's and my pact as learning partners at Agile Testing Days 2017. A few months later at European Testing Conference 2018 we had several long conversations and found that perfect mixture of similarities and differences in each other where both can learn a lot from. Toyer and I are really happy that he decided to join our extended pact group along with Dianë Xhymshiti! All in all, it's a pleasure to exchange knowledge with him, so I was really happy he decided to become part of my testing tour.

Preparation? Already done!

Now, normally it was me who prepared a system under test or even a topic to pair on, as most people didn't have a clear notion of what to tackle. This time, however, I did not have to do anything besides looking forward to the session. The reason was that João managed to enable us to have a look at one of his company's products: OutSystems' development environment. How great is that?! Here's part of the email he wrote in advance of our session.
What I'll be proposing we do tomorrow is a bit of paired-up exploratory testing over the "first experience" one has with the product my company, OutSystems, offers. The fact that you know little about it is great!
I'll give you a bit more detail tomorrow when we begin, but we will most likely be using the currently available features of the OutSystems platform, but also some new unreleased ones our teams have been working on!
A couple of teams here that work on product design and UI development are very interested on the outcome of our session and have asked me if we can record it, so that they can watch it later and collect feedback.
Would you be ok with us recording our session for our internal use (a video of our interaction with the platform)?
Recording? I even discussed that idea with Cassandra on our common stop, so yes I really wanted to give it a try. And all this to help people improve their product? Of course, that's exactly what I love to do!

Getting Our Session Started

OutSystems claims to enable you to "Build Enterprise-Grade Apps Fast". On their website you can find the following definition of their product.
OutSystems is a low-code platform that lets you visually develop your entire application, easily integrate with existing systems, and add your own custom code when you need it.
Crafted by engineers with an obsessive attention to detail, every aspect of our platform is designed to help you build and deliver better apps faster.
In the beginning of our session, João explained that OutSystems' development environment, service studio, has major release cycles of a few months just like other IDEs as Microsoft's Visual Studio or Apple's Xcode. The goal was to tackle the software from a newbie perspective. Product design people were really curious about the experience a new user would have, so João asked me to really speak my mind during testing. Sure thing!

We agreed to split the testing session into the following two steps.
  1. We do the "Build a Web App in 5 min" tutorial for the current version 10.
  2. We create a new app in the upcoming service studio version 11 which is currently under test and not yet released.
João does not work on the team developing the UI, so the whole topic was also new for him. Still, of course he's an advanced user already and knows his way around the software. Therefore, we decided that I would be the navigator for step one, and for step two we can switch roles as he also did not see anything about it yet.

Then it was about time. We started recording, and we started testing.

The Joy of Exploring a New Applications

Now, I won't share the video recording, and I won't share any details about the second step testing the beta version, trying to create a web app for a real use case. What I can share, however, are the highlights of the first step, doing the tutorial of the current version to get a first feeling for the IDE. Why? Because the great thing is, you can simply try it for yourself! Personal environments are completely free and available in the cloud as long as you keep them running. For our session, João had prepared everything up to the point that we started the five minutes tutorial to develop a new web app.

One of the major topics we came across all the time during testing was usability.
  • Issues with the tutorial itself. Even recognizing the tutorial as tutorial. Same navigation button in the tutorial sometimes referring to the help, sometimes to the next step. Not seeing helper arrows pointing to the area of interest. Not displaying these arrows when viewing the tutorial steps. Not having it obvious that you don't need Microsoft Excel. Confusing ordering of explanations what to do. Missing steps to take. Constant re-positioning of the tutorial dialog; at one point even misinterpreting this behavior as the next step, triggering me to execute it twice. Having not reached 100% when finishing the tutorial.
  • Inconsistent or misleading styling. Surprisingly different styling for the tutorial, the tutorial steps, as well as any other application dialog. A new application button that looks inactive and does not draw attention. Triangles used as navigation arrows which rather look like running an app.
  • And more issues. Superfluous scrolling not providing more options. A wizard step only providing one option, so why having it at all. Doing the same thing twice leading to two different results on purpose. Viewing your new web application first in the smartphone view. Unexpected field validation in the resulting demo application.
I talked a lot about the feelings I had regarding the application. As a new user, it's very important for me that I don't get lost but find my way through the first steps easily enough, that I get aha moments of what's happening so I learn, and that I don't get annoyed or frustrated by the software. At several points, however, I got those feelings. I had to learn the tutorial's way how to lead me through first, and then could successfully complete it.

I tried to really slip into the shoes of a new user. From a technical point of view I understand how things are built and why certain decisions had been made, but from a pure user point of view I would have asked many questions, so I raised them. Especially as I understood the tutorial trying to take the user by the hand and to make it as easy as possible for them to create a first app. Furthermore, new users will always compare the system at hand to applications they already know, using those as oracles for their expectations.

In our second step, we switched roles and learned about the new version with several things revised. We found again quite interesting things which I cannot disclose here, but João took them with him to forward them to the team.

All in all, all this is of course only part of the story, from a tester perspective. Trying to find those things that can still be improved. The product looked really nice and easy and I would love to dive deeper into it!

In Retrospect

It was a really nice experience to have everything so well-prepared by João - my thanks to him for that! It was awesome to explore yet another completely unknown application. It was fun to be an actual new user as a tester and imagining the perspective of a real new user at the same time who wants to start developing an app, trying to learn what they would learn. It was great we collaborated well and our results were really productive. It's absolutely awesome we have a video recording of our session! This way we could focus on learning and providing feedback. The downside, of course, was that I now had to go through nearly two hours of video material to be able to extract the goodies for this blog. For real-life sessions I would not skip note taking as we did so I would still get that quick overview of issues, questions, or further areas to explore about. Still, video recording can be invaluable there, too. It's proof of what actually happened! Awesome to reproduce and report bugs, or even to investigate them.
Sometimes there are bits of information that are crucial but you only acknowledge that afterwards when you had to go back. ~ João
When watching the recording, I saw lots of things I did not realize during testing, even though João drove for me and I could focus on observing and navigating. Exploring a video recording provides lots of feedback as well! About the application as well as about our own procedure when testing, our thoughts and ideas. Lots to learn from.

We set off exploring with certain goals in mind, and we reached our destinations. However, we did not structure too much how to get there. This is the sort of freedom I love when exploring, with every step we learn more about potential ways ahead of us and can decide which to take. We don't have to follow the one and only route strictly but can discover so much more on our way, even if it's not in our focus.

On a meta level, I noticed that with more practice it's getting indeed easier for me to pair with other people. It still helps me if I had met them already in real life. I feel the fear to look silly is getting less so I can focus more on the actual task at hand and freely express my thoughts. Sessions were constructive from the beginning, but were mostly covered by a layer of doubt about myself to deal with on top.

There's another thing that only came to my mind when writing this blog post: it felt awesome exploring an IDE. Again. I only realized now that I have way more experience in that area than I might have told even myself. As a user of IDEs like Eclipse or IntelliJ of course. But even more importantly, as a tester. I spent my first years as a tester testing an AI middleware for computer game developers. We developed our own engine, IDE and server, and we also integrated in Unity. I did lots of exploring back then already, I even wrote those kind of beginner tutorials myself! :D And I loved this awesome domain.

Finally, the very best thing: João said he took a lot of value out of the session himself. He can now bring the issues we found to the teams. It was great to hear that they do lots of usability tests and really care about the feedback, using it to improve the product!

Saturday, June 2, 2018

Testing Tour Stop #11: Pair Exploring Test Ideas with Viki

The first time I met Viktorija Manevska was back at Agile Testing Days 2015, the very first conference I've ever attended. Before the first conference day started, I joined the lean coffee session hosted by Lisa Crispin and Janet Gregory. Both Viki and I found ourselves at Lisa's table, sitting next to each other. This is where we started to exchange experiences. Since then, we met each other each year at the same lean coffee table on the first conference day! :D Viki helped me a lot by sharing her story as first-time conference speaker and providing feedback on my own first steps towards public speaking. We had several Skype calls over the last year which proved invaluable regarding knowledge exchange, just like all my sessions with Toyer Mamoojee. I really love how I have met up with more and more other awesome testers over the last months! I signed up for a virtual coffee to get to know other testers like Milanka Matic, Amber Race or Kasia Morawska. I really enjoyed the calls I had with Mirjana Kolarov. And I loved meeting Anna Chicu finally for the first time! All those meetings were invaluable, full of knowledge sharing, story telling and having a good time together.
Now, back to Viki. She just recently moved to Germany to work for a consultancy. Having worked in product companies before, her current project provided new opportunities as well as posed new challenges for her. We decided to focus our pair testing session on one of those, on a part of testing that might not come to mind as first: generating and challenging test ideas as input for exploratory testing sessions.

A Session of a Different Kind

This time, there was no software under test - instead, we tested and challenged ideas. Test ideas that Viki had already brainstormed and provided as input for our testing. The result? We discovered more ideas. We restructured current ideas. We removed duplicates. We challenged procedure and more. All with the goal to improve and make it better, like we testers usually do with everything that gets into our hands.

While brainstorming and discussing, we talked a lot about testing itself. Have you used tours as a method to explore a product? I normally use cheat sheets heavily to generate further test ideas, identify further risks, come up with further questions about things we haven't talked about yet. Cheat sheets like the infamous test heuristics cheat sheet by Elisabeth Hendrickson, among other great resources. This way, wee started talking about exploratory testing itself. How structured should it be? Sometimes I feel my own sessions would benefit from a bit more structure, but I'm still happy with starting out to look for risks in the area of A and ending up with discovering a lot in section S. However, exploring is not unstructured and seldom goes without a clear mission or note taking. What I really love is Elisabeth Hendrickson's definition of exploratory testing, as stated in her most awesome book Explore It!.
Simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next
In my opinion, too much structure or being too strict about it removes the part of adapting your way according to what you learned. Having the product show you the way, or better multiple ways, following them to explore areas of unknowns. At times I wonder whether Maaret Pyhäjärvi was thinking of that when saying that "the software as my external imagination speaks to me"; a notion I often pondered over.

Now, the system to brainstorm test ideas for was a typical untypical web application. Something known like signing up for a user account, something unknown due to the specific domain. We started with ideas around negative testing, UI testing, smoke testing, and workflow testing. Going forward, we produced a lot more ideas, asking so many questions about the product, and also about testing itself.
  • Would you consider different browser settings like turning JavaScript off as negative testing? It could be a normal use case for certain users.
  • What about high security settings or incognito modes? Some users just care more about privacy than others, still they don't mean any harm to your application.
  • What about popup blockers or ad blockers? They are quite frequently used.
  • How to best limit the input for a name field? This made me think of Gojko Adzic's recent book Humans vs Computers as well as his famous Chrome extension Bug Magnet, a really useful tool to quickly test out valid or invalid input values.
  • Or the Big List of Naughty Strings! I've known it for a long time but haven't used that one much yet. There's definitely more to it.
  • This brought up the topic of SQL injection, a common vulnerability to check for. In general, user data must not be at risk, so let's see if we can get login information, take over user sessions, or simply can view other user's unsecured pages.
  • What about tampering with system settings like your local time, could you trick the application to provide you better offers that are based on dates?
  • You send emails? Really important to test those properly then. In one of my previous teams, formatted emails were displayed very differently using different email clients. Are the parameters of the email template filled with the correct values depending on given conditions? Are potential images attached that need to be downloaded? Are the links correct? Could the email be considered as junk and filtered out? How long does it take to receive the email?
  • What happens if you bookmark the URL while being in the middle of a process to share it with someone else?
  • What about the browser back and also forward buttons, could they lead to unexpected, inconsistent, or simply unpleasant behavior?
  • What about consistency regarding styling? Is the UI accessible, e.g. for color-blind people?
  • Which kind of browsers, browser versions, operating systems/devices, screen resolutions do we really want to support?
  • Would you have manual smoke tests? Why not automate those sanity checks instead of repeating them over and over again?
  • What about alternative paths through the application, mixing up the order in which you can do things or skipping steps - would you still reach your goal as a user? Should you be able to do so or is there a prerequisite for critical things?
  • Have you ever tried the nightmare headline game to discover what would be the worst to go wrong?
  • Oh and: although we received so many emails about it lately - have you really considered GDPR compliance?

Looking Back

Both of us loved the session! Viki said she was really satisfied with the outcome. It was great to hear a different opinion, especially from another fresh tester point of view coming from the outside. Sharing experiences, experiments and stories provided lots of value for both of us. I was really glad my input helped her further, especially regarding testing emails, SQL injection and GDPR. I personally really enjoyed to have a completely different pair testing session, exploring mind maps, challenging ideas and thoughts, discussing structures, procedures, and strategies. Exchanging information about different positions, roles, expectations, and ways of collaboration - which is so important as well when testing. Talking about exploratory testing, what we understand, what we do, how we give feedback and create transparency, how we provide value. Just as Viki said, quoted freely: "Name it like you want, but it's important to deliver value." I so much agree.