Had a really fun session with @lisihocke today. If you get the opportunity to join her on her #TestingTour I'd really recommend it! https://t.co/bfs9fXncaE— Claire Reckless (@clairereckless) September 30, 2018
The Goal: ExploreClaire did not aim to focus on a specific topic. She shared that she recently moved into a test lead role where she's not quite so hands on so it would be great to have a testing session where we could focus on doing some actual testing. She was happy to pair with me and see what happens.
Being for longer on this testing tour already with several session done, I now have a whole list of potential test targets ready to be tackled. For our session, I selected a sub-set of 5 applications that might fit our general exploration goal. At the beginning of our session, Claire decided to go for Evite, an online invitation service we both did not know.
We had already agreed in advance to pair in the strong-style way. Claire was familiar with this approach, so I simply shared screen control and explained shortly the mob timer that I prefer to use when pairing with new people. We set the timer to 4 minutes and off we went.
Stumbling and Frowning
Throughout our session, we stumbled on several issues, frowned on even more and documented all note-worthy points in a mind map. Here's what we found.
- We started off on the Evite homepage, wondering whether we needed to register to create an invitation or how far we could come without signing up.
- Right in the beginning, as first navigator, Claire asked me to open the browser's DevTools. When scrolling through the homepage, having a look around what this online service offered, we viewed several tabs highlighting the platform's features. When hovering over one them, showing you can handcraft invitations, we noticed in the DevTools that errors got thrown. An image was not found and a video not loaded.
- As we had the DevTools opened and docked to the right side of the screen, its width got reduced. We noticed that this resulted in having the top navigation menu displayed as menu bottom on the top left. When opening it, we found a completely differently looking menu with differently ordered items. This felt strange and definitely not consistent.
- We decided to search for invitation designs, using not the usual suspects that got offered but something different. We searched for "life role-playing" which gave us 0 search results back. We got suggestions to design our own invitation as well as several design proposals to start from, however, we missed what we knew from other applications to give us suggestions based on the search (like Google or Amazon does, for example). We noted it down as feature idea.
- Going through the suggested invitation designs, we noticed that some of them had been replaced by advertisements instead, looking a lot like the other invitation designs. We did not like that at all, being a dark pattern tricking the user into clicking on these advertisements.
- We browsed the offered categories, choosing "Birthday for Kids". The suggested designs seemed to be indeed filtered by category, and everything looked like a new search, our previous search term not maintained. We decided to use more filters and selected to see only free designs. Doing so, the result page told us that no templates matched our search although we had seen free templates for the category before. We realized the reason was that it indeed had maintained the former "life role-playing" search!
- We removed our custom search term and then tried out further filters, like for color or animations. When filtering for themes, we noticed sometimes strange gaps in the search result grid, leading to an inconsistent layout.
- We viewed the details of an invitation template. It showed the selected design on top, with navigation arrows on both sides. Browsing through designs this way, we found designs that had not been in our previous search result. Later I saw the tool-tip indeed says "Previous Design in Category", still, this felt not intuitive.
- We chose to have a deeper look into the invitation template itself. We found that the design on top displayed placeholders for information we could fill in a form below. We instantly thought of security testing, if that would be our product to test.
- The template offered a preview button, so we decided to use it to see how the invitation would look like. As we still had the DevTools opened, we saw that validation errors had been thrown. Only by scrolling down further we now also saw fields marked as required - not instantly obvious for users.
- The required fields of the form were title, host and date/time. We filled title and host and decided to add a co-host. Interestingly, the co-host showed not only a name field but also an email address. Strange. We decided to leave the email out for now. Date and time were two different fields; we went for only providing a date. Trying to preview now, we got once again validation errors. The co-host email was required which was not obvious before, and the time as well. Well then, we added a time and any value in the email address field, having the validation errors vanish. Preview again failed due to the invalid email address. Okay, provided a valid email format.
- We had selected an invitation template including a photo. It offered a default picture, but required us to upload a custom one before previewing. The validation message said "Please upload a photo first by tapping on the camera" - but which camera? Having scrolled down, viewing the form, it was not clear what was needed to do. Scrolling up, we saw a camera icon next to the photo.
- Back to the start. Choosing a template, filling out required fields, uploading an actual image this time. Clicked somewhere else on the page by mistake - and saw we lost our photo. The upload did not actually upload the photo yet.
- We learned we had to confirm the upload by pressing a "Done" button which saved the image. However, hovering over our uploaded photo, the tool-tip still showed "No file chosen".
- We noticed the upload photo button had changed into an edit button. Clicking it, a dialog was opened, titled "Custom Image". Hm, we couldn't select any standard image, so why custom? Also, the question asked whether we'd like to "modify" our image or upload a new one; the related button was labeled "Edit", however. Inconsistent.
- We chose to edit our photo, noticing it shortly displayed the previous photo before our custom image which felt strange. We could zoom it, we could rotate it - in 45° angles! We would have expected 90° as in other applications, however, it made sense to us that more angles are offered for customizing invitations. We could drag and drop the image to a new position. Editing the photo again, we noticed the edit mode had discarded our selected zoom level, showing the image again in original size. Not nice.
- Now, having fulfilled all prerequisites, we were ready to try the preview. Pressing the button opened a dialog telling us we could now preview the invitation - this came as a surprise to us! There was no loading bar or any other signal that would justify a separate dialog instead of simply loading the preview.
- The preview was opened in a new tab, informing us on the tab that we're viewing a preview. The page was displayed as read-only, we could not interact with it. Strangely, the entered co-host was not displayed anywhere.
- We decided to add more information and see what gets displayed on preview. The message from host was displayed. The created poll did not. Strange, as we would have expected to see a preview of the actual invitation including all information.
- We found that the preview tab could not simply be reloaded, another preview had to be requested, getting a different URL.
- We created a what to bring list; and found the field validation there to simply be incorrect. Filling only items to bring without quantity would validate the name field of the first item to bring, telling us to "Please fill out at least one field" - which we clearly had done. We added the quantity for the first item, left it empty for the other items, and the list got created successfully; only including the first item.
- We found we could still change designs and our details got maintained. Phew. We decided to save the draft by clicking the related button, and got referred to a sign in page. The offered Google login button was framed by a red border, making it look like one of the field validation errors that we had received before.
- Well, we did not want to register, but there was no option to cancel or return to the previous screen. So we used the browser back button - and found ourselves back at the selected template, but with all our previously added information lost!
Focused, Fun, Fruitful
Claire shared that she enjoyed the session. It got her tester brain going again! She's doing lots of management these days so this was refreshing and welcome to her. She felt that the session was really productive, too. We found lots of things on all kind of levels, functionality, usability, even security. And time flew so fast! Using the mob timer set to 4 minutes, she was often surprised how fast these 4 minutes passed. She said this kept us focused, switching our roles frequently helped a lot. Also, we didn't go down too many rabbit holes and stayed on track.
From my point of view, it was very easy to collaborate with Claire. We knew each other only vaguely from online, we haven't even met yet, but I felt we spoke the same language. We did not have to explain what we meant, we had a shared basis and could instantly test together. It was fun! Another interesting point for me was that Claire was the first of my pair testing partners to ask me to open the DevTools from the very start. I really loved that fact as this is exactly what I do when testing on my own. When pairing, I try not to impose myself too much and see how others tackle testing problems. This time it was refreshing for me to see her having the same approach. And these tools let us discover so many things already, not even diving deep, simply by having the console and network tabs open.
Reflecting on what kind of issues we found testing for 75 minutes only, considering that this is a productive application, I really wonder how the people developing it approach testing and quality. I don't mean to assume bad intention, I am not the one to judge their circumstances and challenges. Seeing an application like this just makes me very curious. When we find so many clear issues on the surface, not even deeply knowing the application or testing further regarding security or accessibility or anything else, it really makes me wonder what else is there to find.
All in all, our session was again enlightening, we practiced our testing and collaboration skills, exchanged ideas and approaches. It was great. And the best thing: We will meet each other in real life quite soon at Agile Testing Days 2018 as both Claire and I are speaking there!