Showing posts with label api. Show all posts
Showing posts with label api. Show all posts

Sunday, August 18, 2019

#CodeConfident: Journey - Part 2

My Journey project made some progress! It's still far from where I'd like it to be, yet here's another part of my coding journal - including three more pairing sessions and a brief detour to my former Rest Practice project. Enjoy! :)

July 14

  • pairing session with Parveen Khan
  • when preparing for my pairing session, I removed all built directories and have the project build from scratch - which indeed proved my assumption that this would fix the profile service! :D
  • in the beginning I told her about the background of the app, did a short run through of the app, explained what had been generated and what I added or changed, and what are potential next things to do
  • Parveen wanted to explore the app and then do first changes together
  • when Parveen went through the app, we noticed several things, she raised lots of great questions and she triggered several ideas what to do next
    • TODO: the create / update challenge dialog showed the "influences" label in lowercase although it should be uppercase
    • TODO: the journal entry index page should sort entries by default by date in a descending way with the latest one on top (right now sorted by creation date)
    • TODO: for the index pages, the pagination thresholds should be defined, e.g. only showing 10 items per page
    • IDEA: Parveen asked me whether I wanted to use the app to instantly publish journal entries to my blog - and I thought of that indeed! Or at least to have a quick copy button for the description field so I could easily copy it over to my blog posts and make my own life easier
    • TODO: showing the challenge id for journal entries is not really useful, should rather show the tag instead (had that one on my list, yet it's great to hear confirmation from a different person that this should change)
    • TODO: the challenge delete dialog should not only ask for confirmation of the deletion but also inform what will happen with potential related journal entries
    • TODO: when trying to delete a challenge that still has journal entries, an unhelpful error 500 is thrown and the error message is not useful at all
  • then we went on with making small changes to the app and seeing their impact
  • for the challenge index view, we added a column for the hypothesis
  • for the journal entry index view, we wanted to do the same, yet decided not to dive deeper here for now and solve that puzzle yet
  • instead we changed the challenge detail view to not show the challenge id in the title yet the tag instead, and removed the tag from the rest of the detail page
  • IDEA: we found it would be great to see related journal entries also in the challenge detail view
  • Reflecting:
    • Lisi: having a practice project offers a great and safe learning environment to get your toes wet
    • Parveen: you need to have some idea of what app you want, then you start with small steps, then you get used to it, and you keep trying
    • Parveen: having me as trusted navigator gave her the confidence to do these changes herself; also, I gave her the confidence by telling her "it's fine, don't worry", "we can just delete it, we have it under version control" and letting her do it on her own and try things out; Lisi: it was super useful to practice explaining things I learned (some of them recently), especially in a simple way, really helps my own learning; this is also something such safe learning environments are perfect for
    • Parveen: was super great to both explore an app and make changes within just one hour! Now she understands why developers get into the zone, even forgetting to eat, doing things step by step; all this really helps to get into their shoes and create empathy
    • Lisi: was really cool, got a lot of ideas out of the session :D

July 14 (cont)


July 21


July 24


July 25


July 27


July 28

  • looking at examples from work
  • tried more things
  • really need to read more deeper first
  • nothing committed yet

August 11

  • pairing with Shivani Gaba
  • We planned to set out to pair on my Rest Practice project for which Shivani had feedback to share based on her experience with automated API tests.
  • Shortly before the call I realized that the framework used in this project was relying on Java 8 which I recently had removed from my computer so we could not run the tests. After spending some time back and forth how to install OpenJDK 8 so I would not need the Oracle version, we decided to go through the feedback first. (Later I found this site where I could download a pre-built version of OpenJDK 8 for Windows: https://adoptopenjdk.net/)
  • Instead of always using an explicit status code (e.g. 200), Shivani recommended to use predefined response status types so that the test becomes more readable and less error-prone.
    https://docs.oracle.com/javaee/7/api/javax/ws/rs/core/Response.Status.html
    https://www.programcreek.com/java-api-examples/javax.ws.rs.core.Response.Status
  • In the current tests I asserted for certain body properties. Shivani recommended to implement a JSON helper instead which would provide utility methods like comparing the actual response to an expected JSON. This comes in handy especially if you have a lot more fields to test for. Also, this helper could offer generic convenience methods for creating a JSON object with certain parameters, adding and removing properties, or merging JSONs. Shivani offered to provide a sample - thank you!!
  • Shivani recommended to store the complete response in a variable from which we can then extract headers, body, code whatever we need/want to use for each test only what's needed for the specific test.
  • All endpoints were repeated in each test. Shivani would rather extract them as global enums (e.g. something like "Employee.GET_ID"), or at least declare them as variable, and re-use them, which would also increase readability and maintainability. If they change, you only need to change them in one place.
  • One open question in the practice project was how to get a response without given/when/then when it's only about the setup. Shivani shared how she would wrap the setup in a custom method (e.g. a post method) to hide this implementation details from the rest of the test, which makes it more readable and more maintainable as well.
  • At this point we decided to switch to my Journey project and have a look at how integration tests are done there. I explained the background and purpose of the project, what JHipster already brought with it, and that most of the tests got auto-generated and need improvement.
  • We took the test for creating a journal entry as an example to start with. I shared that what surprised me with the generated tests was that they check the database directly instead of calling the API again to verify if the record really got created. This might be good as it's about testing the integration with the database, yet I was not so sure about depending on the database state here. Shivani did not work with Spring and its test context yet, yet this surprised her as well.
  • Shivani shared that oftentimes integration tests are only used for the happy path, yet they are perfect for negative testing as well. For example, what if a mandatory field is not there? How do we handle that? We decided to implement a test for that to learn how it's done with the Spring framework. We discovered we could simply set a mandatory field to null before creating the DTO out of the entity which led to a bad request we could assert for.
  • We learned more about how the Spring framework builds requests and uses matchers for assertions.
  • We did not yet find a way how to provide custom error messages this way, like we could do for AssertThat().
  • We found a more readable and re-usable way how to write the tests, extracting the expected response and the builder which made the test more lean. We could do this globally to reduce duplication.
    https://www.logicbig.com/how-to/code-snippets/jcode-spring-mvc-junit-test.html
  • Reflection:
    • I was really sorry for the rough start and the delay before we could work hands-on together. I was not as prepared for the session as I wanted to be, had a bad day. In the session I gained a lot new insights and good practices how to improve readability and maintainability of API tests. Thanks so much for sharing your experience Shivani!
    • Shivani shared she was happy to pair with me. It got her really motivated to work on her own project idea now; something she had in mind for longer yet did not start yet :)

August 15

  • Pairing session with Gem Hill on unit tests
  • Gem is starting her own testing tour and I felt super honored to be the first stop on her tour! :D For more details see https://letstalkabouttests.xyz/index.php/testing-tour/
  • We started with me explaining the background of my Journey project, what it's supposed to do and also the code generation part, showing Gem what's there.
  • We wanted to dive deeper into unit tests, and we had quite some generated tests to look into to understand them better and also critically question them.
  • We decided to dive into the frontend unit tests using Jest as Gem's team is now also working with TypeScript and Jest, so this was a perfect match; and also for me it was great as currently I am working on frontend unit tests as well at work in a similar setup.
  • I shortly introduced Jest and it's super useful watch feature where it runs all time in the background, running tests for changed files to quickly provide feedback while developing (my colleagues love that feature).
  • We decided to look at the test for the challenge detail component first as most simple starting point.
  • We went through the test from top til bottom, trying to formulate our understanding, making assumptions clear, voicing questions and parts we don't understand yet.
  • Then we had a look at the actual implementation to map it to what we saw in the test.
  • We noticed that the tested ngOnInit function only subscribed to the challenge, yet the subscription was not tested. This part is basic functionality provided from the framework - which is normally not what we want to test. So it's always the question where do we trust the framework implementation?
  • We had a closer look at the mocking part. Interesting to dive deeper into would be the ComponentFixture class which is provided by Angular as fixture for testing and debugging components. This is something we're not using at work so I'm curious to find out more.
  • The test is importing a test module which comes with a lot of mocks for services.
  • When looking at the test module we came across the same number used for ids again, "123" - it seems the JHipster developers used the same number for all kinds of ids. To make sure our test is independent we changed the id for the used challenge and the tests promptly failed. Here we could see how Jest points out differences. I am also using the Jest plugin for Visual Studio Code which even shows the error inline. That's a nice feature, yet for me it's easier and quicker to see what went wrong in the console output.
  • We decided to move on to another test and chose the assumed next smallest one, the one for the delete component.
  • Here we came across the fakeAsync() and tick() methods which caught our interest. Once again we enjoyed the Visual Studio Code feature to hover over a method to get its description :) We learned that tick() simulates the asynchronous passage of time.
  • We discussed spies which allow to mock return values of functions to stay independent from other parts of the app.
  • Gem suggested to interact with the locally running app and see what's actually happening. This triggered the idea that I should install the Redux DevTools so we could inspect even closer. For now, we deleted a challenge and checked the respective requests sent. This way we saw that opening the delete dialog sends a GET request for the respective challenge - why? We had a look at the implementation and saw the defined selector. Was this the reason? We checked all references yet it was only defined here. We could not yet clearly see how things connect here, something to find out later.
  • We had a look at the implementation for deletion and found a method to close the dialog on cancel. Once again we asked ourselves, is this worth writing a test for? Is it worth the effort of maintaining such a test? This should normally be code that's not touched very often so it's probably not worth it.
  • We discussed mutation testing, an approach my team is currently trying out and Gem used quite often in the past. This is a great way to see if our tests are actually valuable and would detect issues and where we are missing tests. In my team we plan to use this as better guideline for what to write tests for than mere coverage which is not as helpful.
  • Another part that caught our interest was that on deletion, an event gets broadcasted and the dialog closed. In the test, however, not the actual methods but respective spy methods had been used: dismissSpy() and broadcastSpy(). We assumed this makes the unit test even more "unit". Another point to dive into deeper!
  • We moved on to the next test, this time the more complex one for updating the component. Once again, we started from top, including the imports. Gem called out that it's interesting that a HttpResponse is imported, came unexpected.
  • We also once more saw that the test was named "Challenge Management Update Component", as it was generated - we both found it weird that the word "management" was included here.
  • We had a look at the test for saving. It's really great that the generated tests included comments for given / when / then which made them easier to read and understand. This test, interestingly, also showed a comment for the tick() method which the other test lacked!
  • We checked the differences for updating and creating a challenge, both on test and implementation side.
  • What caught our eye was a boolean called isSaving. We wondered what it would do? We decided to first interact with the app again to see what's actually happening. A GET request, a PUT request. Nothing unexpected. Then we had a look at the implementation for isSaving. And we found this boolean is set to true only during the time the actual saving process is taking place. The value is only used in the tests. We both found this very weird. We would not expect this kind of implementation just for the tests, we never saw something like this in the products we worked on. We would only expect to add this during debugging to see what's going on; yet for the test it felt strange and it didn't feel to provide any value. I took a note to remove this implementation later on.
  • Reflection:
    • Time was flying and we soon got to the end of our 90min timebox.
    • We both found it really nice to take time and dissect unit tests, challenging them, asking why, finding issues and questions while learning about the product. It proved again the point that you don't need to know code to the extent a developer does to be able to use your skills and strengths as a tester on code.
    • I absolutely enjoyed our pairing session. When pairing we both bring our skills, expertise and knowledge to the table, and Gem and I had both overlapping and differing knowledge which fit together nicely.
    • When pairing I realized how valuable it is to also look at the imports! I normally skipped them, yet we can already raise questions and improve our understanding here.
    • It was great that Gem reminded me to interact with the real product again. I tend to focus too narrowly on what's in front of me which in this case is the code base, so it's easy to get disconnected. I noticed this several times now already during my code-confident challenge, catching myself on things I should know better. Gem pointed out this needs a context switch which is not easy. I agree, and it also increased my empathy towards my developers a lot more.
    • Gem noticed me taking notes and realized she should also think about how to take notes better for her next stops on her testing tour.
    • Gem found JHipster really intriguing. All the weirdness and bloatiness aside it seems to be really valuable to quickly kick-start a project and be able to learn with it. She plans to generate her own project for further learning purpose.
    • We both found it super easy to pair with each other, even though we only met a few times in real life or virtually and paired only once before. Time flew by and we both were energized afterwards!
    • I'm super looking forward to hearing more about Gem's testing tour and the lessons she will learn on her way :)
  • TODO:
    • install the Redux DevTools
    • remove isSaving boolean

What else?

The next Ministry of Testing Power Hour is coming up! I'm excited to be the one answering all your questions about pairing and mobbing. Just ask them at the club, and on August 22nd I'll answer them from my experience.
Also, the call for collaboration for European Testing Conference is open! Seize the opportunity to participate in this amazing conference next year by proposing your session ideas. The submission alone is worth it as you will get invaluable feedback.
Posted by at No comments:
Labels: , , , , , , , , , , , , , ,

Monday, April 22, 2019

#CodeConfident: Rest Practice

My second #CodeConfident challenge is finally done! It was all about RESTful services. Here's my coding journal, as raw as it is.

March 3


March 4


March 5

  • decided to follow a tutorial to create own REST service, then test it with Rest Assured: https://spring.io/guides/tutorials/bookmarks/ This way I will not only learn how to use the library, but gain a deeper understanding how RESTful services are built
  • also, it's only one step further into the learning zone, as the tech stack is still familiar
  • set up base project using Spring Initializr; added Serenity and REST Assured dependencies; adapted license and readme files
  • created new GitHub repo, cloned it, committed base project, configured repo

March 6


March 9


March 11


March 12

  • pairing session with Amitai Schleier
  • we walked through the project, I explained my reasoning behind
  • Amitai: the build step includes test, that comes unexpected
  • found the included Gradle wrapper was not executable, marked it as such
  • we had to add the Lombok annotation to the classpath; this came unexpected, did not already come with the repo or Gradle; we deferred identifying the root cause here; was a great reminder to run a project locally on a fresh computer from time to time to see what's missing
  • Amitai: shared he would desire fast tests that can run without the app running, just triggered by a key stroke; however, this comes from the perspective of test driving an app, we're in a different context here
  • we learned that the extract method was not instantly comprehensible, not clear what it's doing
  • learned about fluent interfaces (see https://en.wikipedia.org/wiki/Fluent_interface); Amitai: they are nicely readable but might not be instantly comprehensible
  • we both shared concerns like: do we really need to have multiple given/when/thens in a test when we only want to test one action? How to make sure the deletion test really deleted the record, e.g. by calling it again? How can we declare the base path in a nice way at one place?
  • what to test for in which test? Amitai: in general he is thinking of what if this test fails, how hard do I have to think? When it goes red, how obvious is it what I have to do?
  • Amitai: uses the following principle when it comes to tests: make it work, make it right, make it fast
  • Amitai: using auto-format on save a lot; uses a plugin for Intellij: "Save Actions"
  • we committed several changes together throughout the session :)
  • we left TODOs in the code to make it obvious for any one else what's troubling and what is still worked on
  • learned several IntelliJ shortcuts (for MacOS): that were new to me:
    • cmd+9: toggle version control tool window; then cmd+D: on versioned file shows the diff
    • option+cmd+K: commit (& push)
    • option+P: push
    • option+Up/Down: extend / shrink selection
    • after test run, double-click on test lets us jump to the test
  • retrospective:
    • Amitai: already shared his feelings throughout; that was one key learning for him after all the coaching and pairing and mobbing: we don't have to wait to talk about it later, we can talk about it now
    • we were feeling the same pain; how much do we have to say to set up a test; it's a screenful and shouldn't be --> could improve it a bit by reformatting and extracting the test data setup in a separate private helper method (did so); this matters as a test should be so simple that we don't need a test for it; it's supposed to serve us
    • Amitai in general does like the given / when / then structure
    • learned about "TDD as if you meant it" as subcategory; implement class inside test until you come across the first business object (see https://cumulative-hypotheses.org/2011/08/30/tdd-as-if-you-meant-it/)
    • we shared similar concerns, like what does the test actually do, test the test, add assertion that content was actually deleted, multiple given when then steps, test setup, etc.
    • we didn't use a timer, pairing was fluent, frequently switched, quite natural; we both thought of addressing it but decided to give it a go and it worked well
    • I really learned a lot, curious to see how different people tackle different problems, which issues do they see, how they debug and find solutions; e.g. Amitai explored using code and the tests, would have used Postman rather as last resort to find out the behavior
    • love the hands-on approach, already improved things right now, have more pointers what to improve next
  • TODO: think about formatter, start using the plugin
  • TODO: build step includes test
  • TODO: see TODOS in the code ;)

March 12 (cont)

  • realized when summarizing the results from the coding session that pushing from my laptop revealed my work email address; and this time we pushed 3 times, so I could not only amend the last commit like I did before (see https://www.git-tower.com/learn/git/faq/change-author-name-email)
  • this time I had to rebase and edit the commits; above link showed how to do it:
    git rebase -i -p 4f72861fe2fbee02da716f3442cec263e762b4b6
    git commit --amend --author="lisihocke " --no-edit
    git rebase --continue
  • problem: I pushed and merged, and new commits had been created, but the old ones were still there!
  • rebased again and tried to drop these commits, but push did not do anything
  • learned I need to drop them and then force push to replace the remote repo with my local one, this finally did the trick and the concerned commits are gone; feeling relieved!
    git push -f
  • lesson learned, for the second time: be careful what you push..

March 18

  • pairing session with Toyer Mamoojee
  • walked him through the project, explained the reasoning behind, the targeted scope, steps taken so far, open questions
  • we agreed to review what is there for our first session and later have the option to try a different test framework and/or do performance/load testing
  • Toyer: first observation: usually we have test only one endpoint per file
  • naming conventions for tests: have to agree with the team on that, yet should be consistent; Toyer: test name should include what you are testing for, "getEmployee" feels not enough, better "getEmployeeDetails"; also, there's a mixture right now with using "get" for reading or receiving information and "create" for posting
  • Toyer: current test setup class might be sufficient for a practice project for now; would normally have the config in a separate file
  • Toyer: the smoke test looks fine; what they usually do is to additionally include a test for the structure, data types, i.e. including contract testing here as well already
  • Toyer: get first employee: are you sure this id exists? would extract it as constant; for test data setup they are using a separate database that can be queried, I could think about a catalog here; data should be built somewhere else to avoid duplication, in a different class generating the payload; building the payload might be more elegant as well, instead of adding properties I might create a map; creating it upfront to ensure independence makes sense; maybe can include randomizer here as well to detect more issues
  • Toyer: getting information out of the response becomes impossible when you have asynchronous APIs that only return a 200, then you have to query another system
  • Toyer: what else I could test for is that only 1 record is created instead of 2, checking the count of what comes back
  • maybe I could also check if the generated id is not null
  • Toyer: consider splitting tests based on risk; create itself might be crucial and working, only the details might be wrong
  • TODO: revise project based on suggestions

March 27


April 21

  • took up challenge again after a longer period of travels and conferences
  • removed superfluous semicolon (bothered me ever since I detected it and did not find the time to remove it earlier)
  • thought about running the Spring app together with the tests so you don't have to start the app in advance for the tests, like it's done for Restful Booker Platform; decided that this is out of scope for this challenge, would be interesting for a real app though
  • renamed request body to payload
  • potential for future: REST Assured can convert back to Java objects, don't need JSON
  • ran with coverage, however nothing found covered; learned I need to run the tests on the same JVM for it to work, see https://groups.google.com/forum/#!topic/rest-assured/pzBJQqtkvS0 and https://blog.jayway.com/2014/07/04/integration-testing-a-spring-boot-application/ also, need to run as JUnit not as Gradle task it seems
  • doing so I got the error "Error running 'EmployeeTests': Command line is too long. Shorten command line for EmployeeTests or also for JUnit default configuration." --> this fixed it: https://devis.cool/quick-fix/quickfix-intellij-idea-command-line-is-too-long-shorten-command-line-for/
  • still, no coverage found; assume I still need the same JVM; using the Serenity Runner cannot run in same JVM, also issues with SpringRunner
  • interesting post: https://medium.com/@manu.me/bdd-simplified-with-springboot-b56ffdcadb2b
  • did not work so far; also having bootRun as pre-step for test did not trigger tests; however: it's okay to not solve everything in this scope
  • stopped, started Test Automation University course by Bas Dijkstra that I wanted to do for so long already: "Automating your API tests with REST Assured" --> filled several knowledge gaps I had and was not aware of - awesome course!!
  • added checks for content type
  • this way found out how to test for the response content of plain text responses
  • observation: by distracting myself / getting my mind to focus on different but related things (the course) I could solve other issues I had before; was really worth it to "unstuck" myself
  • added test for updating an employee record
  • considering the challenge scope officially done; I'm up for the next challenge! :)

By the way...

First: I learned once again that others appreciate the idea of keeping coding journals as well for their own learning.
Second: I did not have to pause my code-confident challenge yet! I've now indeed continued playing a non-casual computer game at least once a week, and it feels great. Definitely worth following my passion and carving out time for it!

Third: My feeling is I have to speed up. Only two challenges of five completed, let alone my final proof of concept app.

Last but not least: I've been pretty busy the last weeks. The good thing: although I could not work on my personal challenges, I still practiced! I've attended Automation in Testing, TestBash Brighton and Mob Programming Conference. A lot was hands-on learning which I really appreciate. Also, I can't but share one of my absolute highlights with you! Thank you Angie Jones for your encouragement, you're the best!

Code-Confident at TestBash Manchester

Remember when I submitted to Test.bash(); in the last hour? Well, I did not exactly get accepted for this very technical format - yet they selected me for TestBash Manchester! Simply couldn't say no to that. So, I will have yet another challenge this year: craft a presentation out of my ongoing challenge. Already playing with some ideas in my head, curious how they will work out with an audience! :)
Posted by at No comments:
Labels: , , , , , , , ,

Tuesday, October 23, 2018

Testing Tour Stop #24: Pair Exploring Voice Experiences with Gem

At this stop of my testing tour I had the honor to pair test with Gem Hill. I got to know her as active podcaster and enjoyed listening to several of the already 100 episodes of Let's Talk About Tests Baby. This year, I had the pleasure to meet Gem in person for the first time at SwanseaCon. We had attended each other's talks, and right after she heard me speaking about my testing tour she scheduled a session with me!
Gem shared with me that she also experienced a pair testing session with Maaret Pyhäjärvi, just like I did on my very first tour stop. She even did a full podcast episode about it, which I can only highly recommend! Since I've met Gem, I was looking forward to our pair testing session. With good reason, I learned a bunch.

Test Setup

Gem proposed to pair test on voice experiences. As we paired remotely we agreed to use a simulator instead of a real device for testing. My experiences with voice apps are very limited, they narrow down to a simple learning project during one of my company's hackrdays last year. Also, I am not a user of voice apps myself. Therefore, I was eagerly looking forward to our session. Some interesting blog posts I came across during preparation were the following.
Gem kindly agreed to prepare a test setup. As she is currently working on voice apps at the BBC, she suggested to tackle the BBC Alexa skill which is basically a player for all BBC radio stations and podcasts. We used the Alexa Simulator and the latest released skill version, publicly available for everyone.

Challenges of Testing Voice Applications

At first, we checked the happy path to start the app and play a radio channel or podcast. Just by doing so, we discovered that the audio player functionality showed issues. Instead of playing the requested source, it triggered a warning that "AudioPlayer is currently an unsupported namespace". Interestingly, the warning message box was not completely displayed on our screen. Well, we're not testing Amazon's services here. Checking later, I found that this is a known limitation of the simulator and audio playback would work on a real device.

Then we tried to change from one radio station to another one and stumbled again. Asking to "switch" to another station opened another skill, TuneIn, attempting to play the requested station there. Unexpected and not desired! We thought maybe native speakers would not say "switch", but rather "change"; however, this command was unknown. What about "go to"? Again, we switched skills. Interestingly, the feature to switch stations is indeed advertised in the skill description.

Let's try to ask for the news, a use case we deemed quite common when thinking about radio stations. To our surprise, the player started the podcast "Breaking the news". Seems a synonym had been stored for "the news". Hm, what about only "news"? The skill told us that this radio station was unknown. We tried "weather" instead, another common information you might expect to get from a radio station. To our surprise, the skill answered with "I can't do that". Strange, we thought if the request was unknown it would fall back to the "unknown radio station" case. We tried "cheese", which again was interpreted as unknown radio station, just as expected. To test the simulator a bit, we tried a clearly misspelled input, "ra7dio 0n3", and this got recognized as unknown podcast now. But why? We tried several cases, using written words as well as voice input, however, we could not determine why the skill reacted in three different ways to unknown input values.

We looked at the JSON input and output the simulator showed us. Everything looked good here. So we moved on and tried different languages, like asking for the Welsh radio station "nan Gàidheal". The skill understood "nine gale" and asked "Should I play nine gale?" Yet another response we hadn't triggered before! We confirmed, and the skill answered that it couldn't find the station and included a long error message within the response. Interesting finding! However, it was again caused by the simulator and not a real-life example. We used written input, and the API could not deal with the special character included, which would simply not happen when using voice input.

We tried a few more things like seeing whether the station "WM 95.6" got recognized when pronouncing the "." as "point" as well as "dot", or only using "WM". All worked. We tried similarly pronounced words and unclear mumbling to see whether one could be mapped to a station. There were many more options we did not try, like checking any customer reviews on the skill, which in themselves provided lots of useful input for testing.

Throughout our session, Gem shared bits and pieces of wisdom when it comes to testing voice apps.
  • The simulator and real devices behave very differently, so normally they always test on real devices. Furthermore, the simulator is way slower than real devices; another reason to use the latter.
  • They do lots of API testing, checking the JSON output, to ensure the implemented functionality is working as expected.
  • They learned how to test efficiently. Why? Well, when you explore without automation you have to listen to the skill welcome and introduction text again and again and again and again... This does not only take a lot of time, it is also very quickly annoying. Very. Quickly. So, exploratory testing without automation can indeed by very inefficient in this case so they try to automate as much as possible on API level.
  • As the team got more and more mature, they now started to think more and more about how to increase testability. With all automation in place, you still have to test with the real devices as it's the only way to get the real experience.
  • It is really hard to design voice experiences. For example, you have no idea whether you have a very experienced user who might get annoyed by having to listen to too much navigation, or a newbie, who simply needs more instructions.
  • Always ask yourself: Are you still testing the skill, or already the device? Testing the device does not make sense, we can't change it anyways. Going through the list of things users actually say and what Alexa understands, however, does make sense as it allows you to add those as synonyms and make the skill more user-friendly.

Retro Time

We quickly found ourselves at the end of our 90 minutes time box. Gem felt the session went very well, using Zoom to share screen control was amazing, she had lots of fun and the time flew by. For another session she would spend more time in the children's skill BBC offered as it had more branching narratives.

For me the session was awesome, Gem provided a perfect test setup for us to practice. I agreed that a more complex skill would be interesting, however, it was also nice to start with such a "basic" skill as the radio player. This challenged us to come up with good test ideas and we still found issues for such a basic skill. I really appreciated Gem for sharing her knowledge with me, I learned a lot in our session and my respect for people designing and testing voice experiences grew even further. Gem shared it's a real challenge and they are learning something new every day. By the way, a fun fact I learned today: both of us are normally listening to audiobooks when at home! It was a real pleasure testing with Gem and I'm already looking forward to seeing her again in real life.

End of October, the end of my experiment, is getting closer. This was the second last stop on my testing tour 2018. One more to go, so stay tuned!
Posted by at No comments:
Labels: , , , , , , , , ,

Saturday, July 21, 2018

Testing Tour Stop #16: Pair Exploring an API with Thomas

Yet another stop on my testing tour? Yet another stop on my testing tour! Yesterday I learned together with Thomas Rinke. We shortly met each other at Agile Testing Days 2017 where we had a great conversation over lunch. Like many of the other people I paired with, Thomas is really engaged in the community as well, something I really appreciate. He is on the conference board for the German Testing Day, and also actively helping his colleagues grow. He already did what I wanted to do with my own company's testing community for a long time: watching great conference talks and discussing them together. Still on my "to try" list!


The Preparation Phase

About a week before our session, we started brainstorming and aligning ourselves on our topic. Thomas was up for exploration as well as automation. I proposed to go for an exploratory testing session, pairing the strong-style way. Thomas shared he never experienced it himself but was intrigued to give it a try. He had come across Alan Richardson announcing Thingifier, his new demo app for API testing. Awesome! I wanted to have a look at that myself already, so we chose it as our system under test. We directly went for version 1.1, however, as the previous one did not start up successfully on our computers.

In case you haven't explored an API yet, I recommend Maaret's fantastic article Exploratory Testing an API. For our session we agreed to use Postman as our tool of choice and to run the app on my computer, sharing screen control. Have I mentioned already how much I love Zoom for not only offering great video quality and recording functionality, but also free control sharing that's easy to use and actually works?

The Session Introduction

We started out setting everything up and getting any open questions out of the way. Interestingly, Thomas shared with me that he was nervous and excited, thinking "well, what could go wrong, it's going to be great". Especially testing with "strangers" was something extraordinary, and we didn't know each other so well yet. The thing was: I was nervous, too. I always am for my pair testing sessions. Fear is even one of my reasons that I started the whole testing tour. Doing this is still scary for me. However, I already see that it is continuously getting better over time. "If it's scary, do it more often" really applies for me here.

Thomas also shared he prepared for the session upfront. For example, he read Llewellyn Falco's blog post about strong-style pairing and listened to Woody Zuill's podcast episode about mob programming on Agile Uprising. Both really recommended!

I learned Thomas was running Windows as operating system, so I advised him to rather not use shortcuts as they are mapped differently on macOS and this doesn't work easily when sharing control, as I had learned in my session with Pranav. Due to the preparation done beforehand, everything could be quickly clarified so we could activate the mob timer and start testing together.

The Testing Part

Thingifier is described as a "hard coded model of a small TODO application". Our first goal was to get an overview of the capabilities of the application and understand its model. We started out using the given documentation and sent first requests to see how the application worked and behaved. Interestingly, later on I saw that the app's release note for version 1.1 said "When the app is running, visit http://localhost:4567/documentation in a browser to see the docs." This was in contradiction to the actual documentation that could be found directly on "http://localhost:4567/".

Thomas started out as navigator, but when it came to him to take over keyboard control as driver, we found that Postman and Zoom screen control sharing did not work well together. He could do right-click actions with his mouse, but not left-click. We discovered that it worked in Chrome, however, and when going back to Postman it now worked here as well. Would be an interesting topic to explore itself!

When getting to know Thingifier's Rest API, we came across the following findings and starting points for conversations.
  • The documentation spoke of todos and tasks. On first scanning over the documentation, it looked like these terms had been used for the same thing, so it seemed like an inconsistency. However, on second read it became clear that you had standalone todos on the one hand, and projects that have todos as so called "tasks" assigned on the other. Nonetheless, this naming differentiation caused us to stumble. Later I found that when using a category guid as resource instead of a todo guid, it's successfully accepted as task, but then calling the task collection returned a 400 (Bad Request).
  • When adding a task to a project, the request returned a 201 (Created). Repeating the same request did not add the task a second time which was good, but the request still returned 201 (Created). We considered that not critical but also not completely clean, we would have rather expected a message that the relation already existed.
  • We stumbled a bit regarding the naming of resources. For collections the singular form was used (e.g. "/todo"), for sub-collections of an element the plural form (e.g. "/todo/:guid/categories"). We wondered whether there was a related naming convention for Rest APIs. Intuitively we would have used the plural form for all kind of collections, as the request was returning an array of all items. For example "/todo" returned an array of "todos", so we would rather have named it "/todos". Further research showed that it indeed seems to be best practice to use the plural form for collections.
  • For testing purposes we would have loved to see existing relations in the request responses, e.g. when querying a project to see its assigned tasks and categories.
  • Creating a todo created it with a random guid, and it was marked as not completed by default; just as expected.
  • When checking all given relationships between entities, we found them to be inconsistent. Categories and todos were implemented in both directions, so you could see all todos of a category as well as all categories of a todo. However, for example, you could see all projects of a category, but not all categories of a project.
  • We had already tried three of the four offered relationships, and thought everything seemed okay. To complete our understanding we tried the fourth as well - and it failed. "/todo/:guid/categories" resulted in a 400 (Bad Request). Fun fact number one: It's so often the last thing we try that unveils an issue. Fun fact number two: I could not reproduce this issue anymore when writing this blog post; it seems we missed something here.
  • When querying a collection, we would have liked to directly see in the response how many elements were returned, at least to help testing.
  • When requesting a resource that did not exist, we received an error message as expected. Its wording was quite technical but expressive.
  • We deleted all todo items and then queried for the collection of all todos. The request returned only an empty response without value ( i.e. {} ). We would have preferred to receive an empty array of the queried entity instead to keep the response expressive (i.e. {"todos": []} ).
After we had gotten more familiar with the application, we decided that it was time to go deeper on a certain part of the API. We agreed to have a more detailed look at creating todos as we perceived it as core part while still being small and graspable.
  • Sending a request with an empty body returned a 400 (Bad Request), as expected.
  • When providing the framing brackets {} as body we received the error message that the field "title" was mandatory.
  • When providing a body with the title field set to an empty string, we received the error message that the title must not be empty. Providing a blank resulted in the same message.
  • Providing a dot (.) as title worked, just as the Polish special character ł. Previously I learned those additional Polish characters are not included in the basic Latin character set so they can can cause problems e.g. when storing values in a database.
  • Providing a double quote (") returned a MalformedJsonException. We tried to escape the character, using a backslash (\), but it returned "Expected ',' instead of '"'". We wondered how to escape a character in a REST API request body but left that for future investigation.
  • We provided a quite long title of more than 256 characters. The request returned a 201 (Created) as expected. However, when then querying for all todos, the request only returned "Expected ',' instead of '"'". We found that this was not related to the long value provided, but due to our previous tries to escape characters. We indeed had to restart the app to reset the given test data.
  • We switched to the guid field. We found that if the guid is explicitly specified on todo creation, the given one is taken. In case the guid already existed, the todo is not created again but the existing todo is updated. We discovered that there seemed to be no validation regarding the length of the guid. Later I found you could even provide an empty string as guid which made it impossible to request the specific todo.
  • According to documentation, both POST and PUT methods were offered. However, their purpose felt mixed up. Both should create todos in case they did not exist yet, and both should update an already existing todo. However, I found that my gut feeling is mixed up here as well, as I thought in terms of HTTP, however REST only demands a uniform interface.

The Retrospective

As for all test sessions, we left some time for a short retrospective at the end to reflect how it went and what could be improved. Thomas started with sharing his thoughts, and by doing so provided wonderful feedback! Here's what I understood.
  • It was a lot of fun and overall a great thing.
  • He had seen on Twitter that I invited basically the whole world to pair test with me and thought that was a really cool thing. He told himself he could not lose anything there and wanted to join in.
  • The communication beforehand was great. He really liked that I reached out in time upfront, about one week before our session. That we clarified the technical setup, aligned on the topic to pair on, agreed on the style of pairing. That both of us could contribute when choosing our topic and how we wanted to collaborate.
  • He learned something during the session which was great. He had the feeling that it was also worth it for me, not that I thought "oh no, Postman, exploring an API, how boring".
  • He found that I was open, friendly, and welcoming. He really liked that I was looking forward to our session myself.
  • The session offered lots of first-timers for him. It was his first time of using Postman, first time of having a REST API, first time of trying the strong style way of pairing, first time exploring an API. He considered these quite a lot of firsts and therefore felt triggered to prepare for the session which he thought was good. He felt a certain sense of commitment and bindingness to do so. He wanted to explore an API for so long already, and our session triggered him to really do it now.
  • Thomas said the execution of the session and our alignment were good as well. Strong-style pairing was really great, too, he had a good feeling about it. He shared that he felt the trust given and had trust in me himself. He referred to Llewellyn's blog post which emphasized the importance of trust as well.
  • What was not completely perfect from his point of view was that we worked on different operating systems. Thomas felt he was slowed down as he always had to use the GUI for operations like copy and paste. This was not optimal as he felt that I was so much faster when I was driving. He said that the upfront warning about the mismatched shortcuts was great, and in the end this was okay for him, but he still felt I was always faster.
  • Thomas valued the results of our session. We found some behavior working as expected, some surprising things, and some classics.
First of all I had to thank him for his great, structured, constructive feedback. It is so helpful to receive detailed feedback also on the positive things, showing which things are going in the right direction besides those that could be improved. Really appreciated!

Besides that I shared that for me nothing is too basic to pair on. I learned that I always learn something, especially with different partners who bring different perspectives. Sometimes it's great to see a different approach, but sometimes it's also great to see when we share approaches or think in similar directions. I started out on my testing tour because I did not know where I stood regarding my own testing skills and knowledge, if what I was doing made sense at all.

We both shared the approach to first start with the base frame and then dive into the details. As Thomas added, there's also the other approach of going fully destructive at once. However, he normally likes to see the happy path first, how the application was intended to behave. He made better experience to first provide feedback around that to developers instead of instantly coming up with rare strange cases causing the implementation to fail.

The whole communication with Thomas upfront and throughout the session was so easy, I really enjoyed it. Furthermore, I really appreciated that Thomas created a safe environment for both of us from the very start of our session. He shared right at the beginning that he was nervous and excited, which made it really safe for me to share my thoughts and feelings with him and put everything on the table. Neither of us had a problem to call something out during the session, as this trust had been established beforehand already. 

Thomas shared that he loves to experiment and try new things in order to see which of them work. He also wanted to give mob testing a try, especially after having experienced strong-style pair testing now. With the strong-style approach you don't have any chance to lose focus, it's simply awesome to have two people or even the whole team fully concentrating on the task at hand.

The Inspiration

In the very end, Thomas gave me some feedback on my whole tour and what I am doing. He said it's a real inspiration for him to also get out of his comfort zone. For example he had recently decided to invest the time to join the Tuesday Night Testing and really valued the event, having great conversations and experience exchange. Honestly, this is so great to hear as it's exactly what I am trying to do: show people how to safely get our of their comfort zones and how valuable this can be for their personal development as well as for the rest of us learning from them.
Thomas also presented the example of Kim Knup, who just decided to tackle Alan Richardson's note taking experiment and made this decision a public commitment. This created the same sort of bindingness. Thomas shared he really wanted to use some of his time focusing on learning. We both encountered people who do not take any time to learn, not even half an hour. Many seem not to belief they are entitled to use their time for learning, "away from their tasks", a few might also not want to grow (which is a pity). Thomas said that sometimes we should do the things that we think are great fun, and sometimes we should do things that are on the edge of our area of responsibility. I so much agree.

I am already looking forward to meeting Thomas in real life again at the next Agile Testing Days end of this year. We are both going to join the Web Application Security tutorial by Dan Billing. Also, Thomas plans to join Toyer's and my workshop of Finding a Learning Partner in the Testing Community and hopes to find an accountability partner for himself there! So awesome. I really hope he will, alongside many more.
Posted by at No comments:
Labels: , , , , , , , , , , , , , , , , , ,
Subscribe to: Posts (Atom)