Showing posts with label cast. Show all posts
Showing posts with label cast. Show all posts

Saturday, October 13, 2018

Testing Tour Stop #22: Pair Exploring an Online Drawing Application with Rachel

In the original experiment design for my testing tour, I aimed for doing ten stops from beginning of 2018 until end of October. Now, getting closer to this deadline, I realized the community reception was a lot better than expected so that I even had to limit my number of sessions. And suddenly, only one slot was left, one more potential day I could make time. A few more stops would come afterwards, but this was a potential gap in my calendar. I considered asking either specific persons or in general asking around on Twitter if anyone wanted to grab that slot; but then decided to test this out, to wait and see if someone would go for it themselves. Turned out this was a great decision, as Rachel Kibler reached out and filled the gap!
I met Rachel at CAST earlier this year. I attended her mobile and chatbot testing workshop which raised my awareness to a lot of things to look out for, what can go wrong, what are specific problems in the those areas. She also attended my talk about this testing tour, and wrote some very kind words about it, concluding with "Now I want to do the same. There’s so much to learn!"

I'm really happy Rachel seized the opportunity, we had a great pair testing session together!

Getting Started

In the beginning of our session, Rachel shared she was pair testing with Maaret Pyhäjärvi only two days ago and it got her hyped up! I could really relate to that as I had my first testing tour stop with Maaret. If you're reading this, and you haven't tried it yourself so far: Maaret is wonderful and pair testing with her amazingly enlightening. She really has the gift of feedback.

Rachel and I decided to go for Sketchpad, an online drawing application. I had offered it as potential test target to several of my previous pair testing partners before, however, they always ended up choosing a different option. Therefore I was happy Rachel now picked up this app! None of us knew it, so we started from a level playing field.

Rachel was happy with strong-style pairing, so I set my usual mob timer to the usual four minutes and off we went.

What a surprise!

We decided to first see what the application offers. We started broad to get an overview first and then to make an informed decision where to dive deeper into. Here's how the session went.
  • We opened the application and started to draw something on the empty canvas. The calligraphy brush was active by default and we used it to create a closed form. We wanted to fill it and found nice gradient fill options. How to use those was not completely obvious, but after realizing we had to increase opacity and these settings were only used for new elements to instantly fill them, we created a nice new gradient-filled form.
  • We went further and found we could also create stars, in different sizes and rotations. The application made it easy to resize elements or rotate them, providing live information about width, height, angle and offering guiding lines for positioning in relation to the other elements. Our first impression a few minutes in: this product looks quite nifty!
  • What about adding a text element? Here we go. Let's add a new line - hm, why is the cursor blinking in the middle of our first line? Let's write something more - the new text appeared in a new line, having the font size automatically decreased to fit all text into the box. Well, you can like it or not.
  • Let's see what the emoji tab offers. Oh, it's about adding cliparts. By the way, the smiley icon for this tab looked quite scary. Okay, we can add a cartoon cow to our canvas. Let's add a different one, the penguin looks cute! Choosing it not only selected it as our clipart, but also instantly placed a huge instance of it on our canvas. Didn't like that so much. We placed more of them - and one appeared as a tiny splotch. Strange.
  • There's another tab, offering a filling option. Now we assumed to be able to fill single elements. However, it seemed we hit the background instead of a single element, as everything got filled! (Going back to this later, it seemed something went wrong when initially trying to fill an element, as this is indeed feasible.) We discovered a nice feature: We had selected another gradient filling option, a rainbow pattern, and now could even define where which color started.
  • Now what about those elements on our canvas. Do they offer a context menu? Indeed! But why does it offer to copy all? Trying it out, we found all elements got selected. Only later we realized we had triggered the context menu for the background which offered different options than the menu for single elements. We re-encountered this issue several times later on in our session, which proved to be quite annoying. It occurred when we already hovered over the element and clicked without that the element frame was displayed already.
  • Next to the context menu options, the related shortcut was displayed. However, it was the shortcut for macOS, although we tested on a Windows laptop! Did they only implement the Mac version? Or maybe the detection of the operating system failed?
  • We had noticed we could move elements over each other. The element context menu offered to change the layer, sending elements completely to the back or front, or doing so stepwise. We found that sending an element one layer backwards did not have any impact. Maybe each element represented one layer so we just could not see the effect? We checked the layer view and found this assumption to be true, but still the sending backwards feature was broken. Checking the console later it showed a warning from raven.js was thrown: "Unhandled rejection (<{"message":"Tool not found."}>, no stack trace)". Not nice.
  • The element context menu also offered to create a stamp out of the given element. Doing so, displayed the form as new stamp brush in the toolbox - however, with inverted colors! This came really unexpected. Later I found this was due to the dark theme set by default.
  • We added our new custom stamp to the canvas. The next surprise! Not our form was created, but a gray rectangle - why? We tried to create another custom stamp - and suddenly this worked as expected. Hm. What was different? We came up with several theories and assumptions, trying out combinations of what we had done before, but could not reproduce the issue. And yet, we had the evidence on the canvas that something went wrong here.
  • Maybe only the very first custom stamp did not work? Let's start a new drawing from scratch. To do so, what if we simply reloaded the page? To our surprise, our previous drawing was not lost but still displayed! But the URL did not show any identifier and we were not logged in. We checked the application tab in the Chrome DevTools and found that a cookie had been stored as well as several values in the local storage. We cleared them all, reloaded the page and found ourselves on a clean canvas. We created some new elements as well as a new custom stamp from them, and now the stamp brush worked as expected from the beginning.
  • Keeping the DevTools open showing the console, we found several warnings to be thrown; something to look into in a separate session.
  • We decided to go deeper on stamps and tried different configurations. What surprised from user point of view was that when dragging the stamps brush, the single stamp instances were of different size. We found that the brush was set to a range of potential sizes by default. Maybe to make this feature discoverable? Would be interesting to learn about the algorithm behind how the size is calculated.
  • We tried further stamp settings. The offset setting behaved quite understandable. Each of those settings offered a bar to set values in a visual way. Dragging the handle, we got a live preview of the effect the new value would have. Besides for the spacing setting! We found that only for smaller ranges the preview got triggered, but failed to understand why not for larger ones.
  • The rotation setting was even better. Suddenly we had dragged the handle for the minimum value farther to the right than the handle for the maximum value, inverting the displayed range! Okay, for rotation this might make sense, but we would not have expected this and the other settings did not allow it either.
  • The bar of the size setting offered a fixed minimum and maximum value but also provided a field to enter a specific one, which simply asked for trying values higher than maximum, lower than minimum, zero and negative values. The application dealt with it without producing errors, using only the limits. But the interface still accepted the incorrect value when clicking outside the field. A very tiny red bar appeared to show the value was not valid, but that was easy to get ignored.
  • We felt we spent enough time around stamps and set off to something different: the history tab. All our actions were listed and enumerated, the latest one on top. Hovering over the entries, we found the tooltip say "The past". The current entry was "Now". Clicking on a past record, the ones farther on top got faded out and said "The future". What a nice way to communicate this!
  • When moving through time, we realized that the redo button in the left sidebar appeared and vanished, seemingly without good reason. We went to the past, the redo appeared. We went a few steps back in the future, not to the beginning, and the redo disappeared. But why? We could have still stepped through all future items by using redo until reaching again our latest status. We went back further into the past, and the redo button got displayed. Clicking on the same entry again, the "Now", the button vanished again!
  • Going back in time, we suddenly had a canvas state where some of our custom stamp brush lines got turned into lines of those gray rectangles instead! Moving through history, we could not make them convert again to our actual ones. We tried to export the drawing to an image, and the gray boxes were exported! We decided to reload the page - and suddenly we had our custom stamp brush line back. Really weird.
  • Back to the history, we wanted to see what happened when we were at a past state and made a change. We wanted to move an element, and a text element got created. Right, the text tool was still selected. Why didn't we see this? We realized that this tool did not change the cursor to provide a hint which tool you currently had selected - like other tools we had tried. Inconsistent and not helpful.
  • We browsed a bit further. Checking the settings dialog we found that an autosave feature was active. Something to look into separately.
  • We went further to open a drawing, expecting to open files we had first exported - but no, to our surprise it offered us the drawings we had previously worked on! Even the one we had created before clearing the local storage. Seemed we missed something. So we went back to the DevTools application tab and found that the drawing information had been stored in the IndexedDB. Nice that users don't lose their drawings; not nice that they are not informed that their drawings fill up the browser's storage. We deleted all entries for one of our images and reloaded the page, and the image was indeed not found, instead opening a new canvas.
  • We opened one of the other images - and got caught by surprise again. The drawing showed several elements in a strange way, cutting them off. And: our wonderful default gray box stamps had reappeared.
We felt this was a good point to stop our session and take a few minutes to reflect.

Looking Back

Rachel found that pairing was awesome for generating ideas. The four minute timer felt just perfect. At one point towards the end she shared she was getting tired and running out of ideas. However, she knew she had to push herself just a bit further so the other one could take over navigating.

She addressed she found it hard to take notes during the session, as even navigating the other one to note something down would have interrupted the flow. I offered the option to take notes together in the beginning but then did not bring it up again. In my recent pair testing sessions I was trying to not enforce my ways on others too much and instead learn about new approaches.

Rachel said the session was fun, and it was great that the navigator could really take the time and focus to observe, having their eyes out. This helped a lot when facing weird behavior.

I found myself hearing us saying lots of things like "wait, what was that, did you see that?", "that was weird", "hmmm" (the sound of frowning), "strange", "ohhh!", "that was unexpected". We made lots of explicit assumptions and also tried to verify them which I really liked.

In general I found it harder to come up with great test ideas for this kind of application that was so different from what I normally tests. I had used lots of drawing applications already, but not having the focus on testing. It was great to try testing something completely different, and it was even better to do it together to generate more and better ideas.

Overall, this product really surprised me. In my previous sessions, the test target quickly showed issues or quirky behavior we frowned upon. This time, the product looked indeed quite nice in the beginning! A positive surprise. Then we started to find issues. More of them. When it finally came to opening existing drawings, the surprise had turned from pleasant to unpleasant. Still, this stop was yet again a fun learning session and I am already looking forward to the last scheduled ones!
Posted by at No comments:
Labels: , , , , , , , ,

Friday, October 5, 2018

Testing Tour Stop #21: Pair Accessibility Testing with Alex

Alex? Now, this Alex is not the Alex I already paired with. This Alex is Alex de los Reyes, whom I met in person at CAST 2018. He had listened to my talk "Cross-team Pair Testing: Lessons of a Testing Traveler" and therefore learned about my testing tour. And was intrigued! Inspired from the moment, he agreed to pair test with Amit Wertheimer.
But how to get started? Alex and I had a call where I explained in more detail how I found partners to test with, how I prepare sessions, how to run them and how to follow-up.
Even before we had this first call, Alex scheduled a pair testing session with me. Since then I was looking forward to this one and it turned out to be awesome!

Testing for Accessibility: Why, What, How

As with most partners, we had several topics to pair on in mind. In the end we decided to go for accessibility as we both wanted to get better in this area and for Alex it's becoming really relevant in his current project. Best time to practice together!

In the beginning of our session we decided to first try our skills on a practice application and afterwards tackle a real productive website to apply what we learned. The selected demo we chose was Accessible University 3.0.
"Accessible University (AU) is a fictional university home page designed to demonstrate a variety of common web design problems that result in visitors with disabilities being unable to access content or features. [...] Use the AU site to
  1. demonstrate common web accessibility principles at trainings, presentations, and workshops on accessible web design.
  2. learn common web accessibility problems and solutions in an easy-to-understand way." (Accessible University 3.0)
The demo offers an inaccessible site where you can detect at least 18 accessibility issues, a list of all issues including detailed explanations and the accessible version of the site having all issues fixed.

What can I say? We had a blast while discussing our experiences, sharing our accessibility knowledge and finding issue over issue! First, we looked for quick wins using our eyes (we're biased beings), things that instantly stood out that we knew could cause trouble for people who have to rely on a different access to technology. Second, we inspected the page structure and found even more issues using the experience and knowledge we had. And third, we used tooling to discover further issues and get inspired to explore more. Alex does not like the WAVE Chrome extension too much and rather prefers Axe. However, for this session we decided to go for another option, letting me get to know another feature of the wonderfully useful Chrome DevTools: Lighthouse, allowing you to run audits for accessibility. The report provided an overall score, automatically detected issues and their instances, as well as a useful checklist of what to explore for as it could not be detected manually. Super easy, fast and really worth it.

As I can really recommend to try this demo app on your own, I won't list our findings directly here in this blog post. If you'd like to compare your results with ours then check out our mind map of potential issues. In the end, we compared them with the provided list of 18 accessibility issues and found that we had identified a lot of them! At the same time, we learned about many more problems we had not been aware of at all before and solutions we even didn't know existed. Awesome!

Having still 15 minutes to go, we decided to follow our original plan and tackle a productive site as well. We selected agoda, a travel booking platform. Having only a very limited time box, we decided to instantly use Lighthouse and discovered the page's score was not much higher than the one of the inaccessible practice app. This way we found so many issues very quickly already. We also stumbled across other usability issues like changing navigation or not responsive design. Last but not least we tried to tab through the homepage which offered a navigation menu and a form to book a hotel room - and here's what happened. Just try it yourself!
  • For the first few tabs we saw where we were as navigation menu elements showed a blue highlighting frame.
  • The next few tabs suddenly did not show anymore any focus, so we got lost.
  • The next tab opened the calendar for the arrival date at the hotel - strange, as we had skipped the first form element to enter a destination.
  • We tabbed to the second calendar for the departure date, tabbed again and landed in a sort of drop-down / dialog / sub-form / whatever element to provide the number of rooms, adults and children.
  • We tabbed on. And found ourselves trapped in this very UI element! We could neither tab forward nor backward.
When reproducing the issue for this blog, I tried to see whether we really skipped the destination field, trying to tab backwards from the calendars - and noticed Shift+Tab did not allow me to tab backwards but had the same effect as Tab itself, both tabbing forward! I'm speechless.

Looking Back

As for each pairing session, we took some time in the end to reflect on what we learned and share our thoughts. First of all: The demo app was a really valuable target to explore, it raised awareness of the multiple traps to fall into when it comes to accessibility. Interestingly, we noticed we didn't even look at the accessible version of the page to see an example how to do it well! This is definitely still worth looking into. In any case, we both loved our approach of first practicing with the demo app, getting our brains prepared, and then switching to a productive application, being instantly able to spot issues there with the knowledge gained. We identified a lot within only 15 minutes - time worth spent.

It turned out again that bad accessibility design is often also bad user experience for everyone. Designing accessible sites improves the experience for everyone else as well. However, accessibility has to be considered already during design. Nowadays there are so many tools to support us, like checking for sufficient color contrast already in mockups.

Talking about our collaboration, it was once again a very easy session for both of us. We had agreed to use strong-style pairing and a mob timer to frequently switch roles. Alex also had made some experience with this approach already and felt comfortable with it. Our flow was very natural. Especially one fact caught our eye: we followed up on each other's ideas, using the "yes, and ..." rule from improvisational theater. Alex had already noticed that ignoring this is a common problem when pairing. In all the sessions I did on my testing tour, I found this to be one of the most important key points to pair successfully with strangers, being productive from the first minute on.

The session was thoughtful. It was super productive. It was so relevant for Alex' current project, where accessibility is a big part of the testing strategy. And the session was fun! Like several others before him, also Alex advised to "do it with someone else" as this makes it so much easier. "Don't test alone, for your own safety." I have nothing to add.
Posted by at No comments:
Labels: , , , , , , , , , ,

Monday, August 20, 2018

CAST 2018 - A Community Adventure

Two weeks ago, I had the great opportunity to be part of CAST 2018, taking place in Cocoa Beach, Florida. I had heard great things about last years' events, and indeed, it turned out to be yet another wonderful conference!

The theme was "Bridging Between Communities", a theme I really loved. It is great to exchange experience with other testers working in a similar context. It is also great to share knowledge with testers working in different ones. And it is great to learn from people who do not identify as tester as well! Maria Kedemo was program chair and created a wonderful offer. It was really hard to pick a session and I would have loved to go to each and every one of them. Thanks a lot for the diversity provided.

Organizers in general were great and really supportive. Together with their volunteers they did a great job! I felt very welcome, communication was easy, and facilitation and support during the conference was awesome. Especially as a speaker this part is nowadays really important for me. Though we had some short-term changes in the schedule or regarding rooms, everything went smooth and without major problems.

Getting to a conference as speaker is a great thing for me in any case. It's not only that I get the chance to share my experience and give back to the community from which I received so much myself, or that I receive the opportunity to level up my presentation and training skills, but also that I learn so much from these conferences myself. I learn as an attendee, I can reach out to fellow speakers way more easily, and I get approached myself by other participants myself. As I identify as introvert at heart, this helps me so much to connect with people and learn from them. Thank you for making it so much easier for me.

At CAST 2018, there were so many wonderful people with whom I enjoyed many insightful conversations.
  • Lisa Crispin. I consider myself really lucky that we had so many opportunities over the last year to meet and exchange thoughts. I enjoy all our conversations and am already really happy that we are going to meet again at three more conferences within the next half year!
  • Marianne Duijst. Thank you for listening, for providing constructive feedback, for facilitating the questions for my talk, for your most wonderful sketch notes, and last but not least for spending a full day with me after the conference doing awesome touristic stuff! I am so glad I got to know you.
  • Ashley Hunsberger. It was such a great coincidence that we had a first call just a few weeks before CAST! It was even more lovely meeting you in real life, and just picking up things from where we left them. Thanks for the evening spent talking about anything and everything.
  • Lena Wiberg & Tomas Rosenqvist. It was fabulous to meet you in person, and that even already on the plane to Orlando. Thank you for sharing your experience. You gave me so much food for thought!
  • Angie Jones. Finally I had the opportunity to join one of your tutorials! It was great. Even better was the opportunity to talk with you. Thanks for making it easy, as well as for your feedback! :-)
  • Amit Wertheimer. It was great meeting you again, I really enjoyed our deep conversations on so many topics! Already looking forward to the next chance.
  • Jan Eumann. There are still not too many people who already experienced mob programming, mob testing, or mob anything. I really enjoy sharing thoughts with those who did! Even greater considering the fact that we now submitted a proposal for a mob session at another conference together.
And there were so many more lovely people! Maria Kedemo. Louise Perold. Anne-Marie Charrett. Jenny Bramble. Richard Bradshaw. Alex de los Reyes. Bailey Hanna. Many more great persons with whom I loved exchanging knowledge during tutorial, workshops, talks, lunch, or a bus ride.
If you came that far in my post you might think this conference was all about people. Well, I think conferences are indeed about the people, just like software development is. Still, those people also shared great content. I learned a bunch again and took several ideas with me back to work. Here are my highlights of each conference day I attended.
  • Tutorial day. The tutorial "Advanced Automation for Agile: UI, Web Services, and BDD" by Angie Jones. She taught everyone how to create a well designed automation framework, shared her experience, demoed what she told, and made sure everyone shared the same page. At times I perceived the pace as slow, yet "slow allows for thoughtful thinking" to quote Maaret Pyhäjärvi. It felt good to learn that I know more than I thought I would. That I could easily follow and still had time to help others, and was able to do so.
  • Conference day 1. This day is tricky, as I had two favorite sessions. The workshop "An Exploration of Observing - Creating system awareness in our quest for quality" by Louise Perold was great to practice our observations skills hands-on and especially debrief what happened and what we perceived. And the talk "Risk Based Testing: Communicating Why You Can't Test EVERYTHING" by Jenny Bramble was so entertaining that I even forgot that I was next! It also made obvious how important communication, talking about risk to guide our testing, and team morale really is.
  • Conference day 2. Finally I could hear Marianne Duijst's talk "Wearing Hermione’s Hat: Narratology for Testers". Learned a lot about biases, perspectives, trust when it comes to information, as well as enabling others. It was simply awesome. If you get a chance to hear it, take it. It should have been a keynote for everybody to hear.
Well, and I also had two sessions myself. I gave my brand-new talk "Cross-team Pair Testing: Lessons of a Testing Traveler" for the first time at a conference, speaking about my testing tour. Although my timing was not working out well enough, it seems people did not notice and still felt it to be consistent. After the talk, there was a facilitated discussion which I really feared in the beginning - but it went well and people asked many great questions so we could dive a bit deeper on the topic. And the best thing: Alex de los Reyes came up to me after my talk and told me that he really related to it, shared my fears and wanted to pair with other testers for some time himself already. Now my talk was finally the trigger for him to actually do so! Amit Wertheimer instantly joined in, and both agreed to have a pair testing session. How awesome is that?! It seems more people came out of my talk inspired. That was my goal and I am really happy about this kind of feedback. What more could one want?
In addition to my talk, I gave my workshop "Mobservations - The Power of Observation in a Mob". I had less time to prepare myself for it as expected as schedule slots were swapped, so I felt less energized and the session less organized as it could have been. Still it went out okay, people got value out of it, and I also received constructive feedback how to improve it further. As Louise Perold shared with me, workshops are never perfect, they can always be improved. And they depend also on the people you have and your skills to adapt to them and the context.
In conclusion, the people were great. The conference was great. And then also the social activities around were great! For example, the sponsors enabled most of the people to visit Kennedy Space Center where we all had dinner together. What a location for that!
As a bonus: there were two rocket launches around the conference dates - who can say they offer that?!
Last but not least: I have the most awesome team ever.
Posted by at 4 comments:
Labels: , , , , , , , , , , , , , ,
Subscribe to: Posts (Atom)