I met Rachel at CAST earlier this year. I attended her mobile and chatbot testing workshop which raised my awareness to a lot of things to look out for, what can go wrong, what are specific problems in the those areas. She also attended my talk about this testing tour, and wrote some very kind words about it, concluding with "Now I want to do the same. There’s so much to learn!"Yesterday the last slot on my #TestingTour 2018 got booked - I'm speechless! Many thanks to everyone who joined me already and to those who are still going to join me until end of October! ๐๐๐— Elisabeth Hocke (@lisihocke) October 4, 2018
I'm really happy Rachel seized the opportunity, we had a great pair testing session together!
Getting Started
In the beginning of our session, Rachel shared she was pair testing with Maaret Pyhรคjรคrvi only two days ago and it got her hyped up! I could really relate to that as I had my first testing tour stop with Maaret. If you're reading this, and you haven't tried it yourself so far: Maaret is wonderful and pair testing with her amazingly enlightening. She really has the gift of feedback.Rachel and I decided to go for Sketchpad, an online drawing application. I had offered it as potential test target to several of my previous pair testing partners before, however, they always ended up choosing a different option. Therefore I was happy Rachel now picked up this app! None of us knew it, so we started from a level playing field.
Rachel was happy with strong-style pairing, so I set my usual mob timer to the usual four minutes and off we went.
What a surprise!
We decided to first see what the application offers. We started broad to get an overview first and then to make an informed decision where to dive deeper into. Here's how the session went.
- We opened the application and started to draw something on the empty canvas. The calligraphy brush was active by default and we used it to create a closed form. We wanted to fill it and found nice gradient fill options. How to use those was not completely obvious, but after realizing we had to increase opacity and these settings were only used for new elements to instantly fill them, we created a nice new gradient-filled form.
- We went further and found we could also create stars, in different sizes and rotations. The application made it easy to resize elements or rotate them, providing live information about width, height, angle and offering guiding lines for positioning in relation to the other elements. Our first impression a few minutes in: this product looks quite nifty!
- What about adding a text element? Here we go. Let's add a new line - hm, why is the cursor blinking in the middle of our first line? Let's write something more - the new text appeared in a new line, having the font size automatically decreased to fit all text into the box. Well, you can like it or not.
- Let's see what the emoji tab offers. Oh, it's about adding cliparts. By the way, the smiley icon for this tab looked quite scary. Okay, we can add a cartoon cow to our canvas. Let's add a different one, the penguin looks cute! Choosing it not only selected it as our clipart, but also instantly placed a huge instance of it on our canvas. Didn't like that so much. We placed more of them - and one appeared as a tiny splotch. Strange.
- There's another tab, offering a filling option. Now we assumed to be able to fill single elements. However, it seemed we hit the background instead of a single element, as everything got filled! (Going back to this later, it seemed something went wrong when initially trying to fill an element, as this is indeed feasible.) We discovered a nice feature: We had selected another gradient filling option, a rainbow pattern, and now could even define where which color started.
- Now what about those elements on our canvas. Do they offer a context menu? Indeed! But why does it offer to copy all? Trying it out, we found all elements got selected. Only later we realized we had triggered the context menu for the background which offered different options than the menu for single elements. We re-encountered this issue several times later on in our session, which proved to be quite annoying. It occurred when we already hovered over the element and clicked without that the element frame was displayed already.
- Next to the context menu options, the related shortcut was displayed. However, it was the shortcut for macOS, although we tested on a Windows laptop! Did they only implement the Mac version? Or maybe the detection of the operating system failed?
- We had noticed we could move elements over each other. The element context menu offered to change the layer, sending elements completely to the back or front, or doing so stepwise. We found that sending an element one layer backwards did not have any impact. Maybe each element represented one layer so we just could not see the effect? We checked the layer view and found this assumption to be true, but still the sending backwards feature was broken. Checking the console later it showed a warning from raven.js was thrown: "Unhandled rejection (<{"message":"Tool not found."}>, no stack trace)". Not nice.
- The element context menu also offered to create a stamp out of the given element. Doing so, displayed the form as new stamp brush in the toolbox - however, with inverted colors! This came really unexpected. Later I found this was due to the dark theme set by default.
- We added our new custom stamp to the canvas. The next surprise! Not our form was created, but a gray rectangle - why? We tried to create another custom stamp - and suddenly this worked as expected. Hm. What was different? We came up with several theories and assumptions, trying out combinations of what we had done before, but could not reproduce the issue. And yet, we had the evidence on the canvas that something went wrong here.
- Maybe only the very first custom stamp did not work? Let's start a new drawing from scratch. To do so, what if we simply reloaded the page? To our surprise, our previous drawing was not lost but still displayed! But the URL did not show any identifier and we were not logged in. We checked the application tab in the Chrome DevTools and found that a cookie had been stored as well as several values in the local storage. We cleared them all, reloaded the page and found ourselves on a clean canvas. We created some new elements as well as a new custom stamp from them, and now the stamp brush worked as expected from the beginning.
- Keeping the DevTools open showing the console, we found several warnings to be thrown; something to look into in a separate session.
- We decided to go deeper on stamps and tried different configurations. What surprised from user point of view was that when dragging the stamps brush, the single stamp instances were of different size. We found that the brush was set to a range of potential sizes by default. Maybe to make this feature discoverable? Would be interesting to learn about the algorithm behind how the size is calculated.
- We tried further stamp settings. The offset setting behaved quite understandable. Each of those settings offered a bar to set values in a visual way. Dragging the handle, we got a live preview of the effect the new value would have. Besides for the spacing setting! We found that only for smaller ranges the preview got triggered, but failed to understand why not for larger ones.
- The rotation setting was even better. Suddenly we had dragged the handle for the minimum value farther to the right than the handle for the maximum value, inverting the displayed range! Okay, for rotation this might make sense, but we would not have expected this and the other settings did not allow it either.
- The bar of the size setting offered a fixed minimum and maximum value but also provided a field to enter a specific one, which simply asked for trying values higher than maximum, lower than minimum, zero and negative values. The application dealt with it without producing errors, using only the limits. But the interface still accepted the incorrect value when clicking outside the field. A very tiny red bar appeared to show the value was not valid, but that was easy to get ignored.
- We felt we spent enough time around stamps and set off to something different: the history tab. All our actions were listed and enumerated, the latest one on top. Hovering over the entries, we found the tooltip say "The past". The current entry was "Now". Clicking on a past record, the ones farther on top got faded out and said "The future". What a nice way to communicate this!
- When moving through time, we realized that the redo button in the left sidebar appeared and vanished, seemingly without good reason. We went to the past, the redo appeared. We went a few steps back in the future, not to the beginning, and the redo disappeared. But why? We could have still stepped through all future items by using redo until reaching again our latest status. We went back further into the past, and the redo button got displayed. Clicking on the same entry again, the "Now", the button vanished again!
- Going back in time, we suddenly had a canvas state where some of our custom stamp brush lines got turned into lines of those gray rectangles instead! Moving through history, we could not make them convert again to our actual ones. We tried to export the drawing to an image, and the gray boxes were exported! We decided to reload the page - and suddenly we had our custom stamp brush line back. Really weird.
- Back to the history, we wanted to see what happened when we were at a past state and made a change. We wanted to move an element, and a text element got created. Right, the text tool was still selected. Why didn't we see this? We realized that this tool did not change the cursor to provide a hint which tool you currently had selected - like other tools we had tried. Inconsistent and not helpful.
- We browsed a bit further. Checking the settings dialog we found that an autosave feature was active. Something to look into separately.
- We went further to open a drawing, expecting to open files we had first exported - but no, to our surprise it offered us the drawings we had previously worked on! Even the one we had created before clearing the local storage. Seemed we missed something. So we went back to the DevTools application tab and found that the drawing information had been stored in the IndexedDB. Nice that users don't lose their drawings; not nice that they are not informed that their drawings fill up the browser's storage. We deleted all entries for one of our images and reloaded the page, and the image was indeed not found, instead opening a new canvas.
- We opened one of the other images - and got caught by surprise again. The drawing showed several elements in a strange way, cutting them off. And: our wonderful default gray box stamps had reappeared.
We felt this was a good point to stop our session and take a few minutes to reflect.
Looking Back
Rachel found that pairing was awesome for generating ideas. The four minute timer felt just perfect. At one point towards the end she shared she was getting tired and running out of ideas. However, she knew she had to push herself just a bit further so the other one could take over navigating.
She addressed she found it hard to take notes during the session, as even navigating the other one to note something down would have interrupted the flow. I offered the option to take notes together in the beginning but then did not bring it up again. In my recent pair testing sessions I was trying to not enforce my ways on others too much and instead learn about new approaches.
Rachel said the session was fun, and it was great that the navigator could really take the time and focus to observe, having their eyes out. This helped a lot when facing weird behavior.
I found myself hearing us saying lots of things like "wait, what was that, did you see that?", "that was weird", "hmmm" (the sound of frowning), "strange", "ohhh!", "that was unexpected". We made lots of explicit assumptions and also tried to verify them which I really liked.
In general I found it harder to come up with great test ideas for this kind of application that was so different from what I normally tests. I had used lots of drawing applications already, but not having the focus on testing. It was great to try testing something completely different, and it was even better to do it together to generate more and better ideas.
Overall, this product really surprised me. In my previous sessions, the test target quickly showed issues or quirky behavior we frowned upon. This time, the product looked indeed quite nice in the beginning! A positive surprise. Then we started to find issues. More of them. When it finally came to opening existing drawings, the surprise had turned from pleasant to unpleasant. Still, this stop was yet again a fun learning session and I am already looking forward to the last scheduled ones!
Today on my #TestingTour: @racheljoi! Had a great #PairTesting session exploring an online drawing application together. What a surprising product this was! Thank you Rachel for this fun learning session ๐— Elisabeth Hocke (@lisihocke) October 13, 2018
No comments:
Post a Comment